UNIVERSITAT POLITÈCNICA DE CATALUNYA

Programa de Doctorat:

AUTOMÀTICA, ROBÒTICA I VISIÓ

Tesi Doctoral

Understanding Human-Centric Images:
From Geometry to Fashion

Edgar Simo Serra

Directors:
Francesc Moreno Noguer
Carme Torras

May 2015

Abstract
Understanding humans from photographs has always been a fundamental
goal of computer vision. Early works focused on simple tasks such as detecting the location of individuals by means of bounding boxes. As the ﬁeld
progressed, harder and more higher level tasks have been undertaken. For
example, from human detection came the 2D and 3D human pose estimation in which the task consisted of identifying the location in the image orspace of all diﬀerent body parts, e.g., head, torso, knees, arms, etc. Human
attributes also became a great source of interest as they allow recognizing
individuals and other properties such as gender or age. Later, the attention
turned to the recognition of the action being performed. This, in general,relies on the previous works on pose estimation and attribute classiﬁcation.
Currently, even higher level tasks are being conducted such as predicting
the motivations of human behaviour or identifying the fashionability of anindividual from a photograph.
In this thesis we have developed a hierarchy of tools that cover all theserange of problems, from low level feature point descriptors to high levelfashion-aware conditional random ﬁelds models, all with the objective ofunderstanding humans from monocular RGB images. In order to build
these high level models it is paramount to have a battery of robust andreliable low and mid level cues. Along these lines, we have proposed two
low-level keypoint descriptors: one based on the theory of the heat diﬀusion on images, and the other that uses a convolutional neural network tolearn discriminative image patch representations. We also introduce distinct
low-level generative models for representing human pose: in particular we
present a discrete model based on a directed acyclic graph and a continuous
model that consists of poses clustered on a Riemannian manifold. As midlevel cues we propose two 3D human pose estimation algorithms: one that
estimates the 3D pose given a noisy 2D estimation, and an approach thatsimultaneously estimates both the 2D and 3D pose. Finally, we formulate
higher level models built upon low and mid level cues for understanding humans from single images. Concretely, we focus on two diﬀerent tasks in thecontext of fashion: semantic segmentation of clothing, and predicting the
fashionability from images with metadata to ultimately provide fashion advice to the user.
In summary, to robustly extract knowledge from images with the presence of humans it is necessary to build high level models that integrate low
and mid level cues. In general, using and understanding strong features iscritical for obtaining reliable performance. The main contribution of thisthesis is in proposing a variety of low, mid and high level algorithms forhuman-centric images that can be integrated into higher level models forcomprehending humans from photographs, as well as tackling novel fashionoriented problems.
Keywords: human-centric imaging, feature descriptors, human pose estimation, generative models, semantic segmentation, conditional randomﬁelds, convolutional neural networks, fashion.

Acknowledgements

I would like to ﬁrstly thank both my advisors Francesc and Carme, to whom I owe thisopportunity. I would also like to thank everyone who supported me during this long
journey, and in particular, Rasputin and Nuryev.
This work has been partly sponsored by the following:- A four-year PhD scholarship (BES-2012-054905).
- A four-month scholarship to visit the Toyota Technological Institute at Chicago(EEBB-I-14-08177).
- A four-month scholarship to visit the Nakamura Laboratory in Japan (EEBB-I15-09283).
- Projects CINNOVA (201150E088), PAU+ (DPI2011-27510), ViSEN (ERA-Net
Chistera PCIN-2013-047), and Intellact (FP7-ICT2009-6-269959).
- My own stubbornness.

iii

In loving memory of Misi

Contents

Contents

vii

List of Figuresix

List of Tablesxii

1 Introduction
1.1 Contributions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Publications . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
1.2 Thesis Overview . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

1
2
3
4

2 Overview
2.1 Low Level Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Image Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Simple Priors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.2 Mid Level Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Image Features . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Prior Models . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Bag-of-Words . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.3 High Level Cues . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
2.4 Machine Learning Models . . . . . . . . . . . . . . . . . . . . . . . . . .
Logistic Regression . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Support Vector Machines . . . . . . . . . . . . . . . . . . . . . . . . . .
Deep Networks . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Conditional Random Fields . . . . . . . . . . . . . . . . . . . . . . . . .

7
7
8
10
10
10
13
14
14
14
15
15
16
18

3 Feature Point Descriptors
3.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.2 Deformation and Illumination Invariant (DaLI) Descriptor . . . . . . . .
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Deformation and Light Invariant Descriptor . . . . . . . . . . . . . . . .
Deformation and Varying Illumination Dataset . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Deep Architectures for Descriptors . . . . . . . . . . . . . . . . . . . . .

21
21
23
25
26
34
39
49

vii

viii

Contents
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Learning Deep Descriptors . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

50
50
53
59

4 Generative 3D Human Pose Models
4.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.2 Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.3 Linear Latent Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Deformation Modes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.4 Directed Acyclic Graphs . . . . . . . . . . . . . . . . . . . . . . . . . . .
Discrete Joint Model . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.5 Geodesic Finite Mixture Models . . . . . . . . . . . . . . . . . . . . . . .
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
4.6 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .

63
63
65
67
67
68
69
70
71
73
74
77
80

3.4

5 3D Human Pose Estimation
83
5.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 83
5.2 Related work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 85
5.3 3D Pose Estimation from Noisy 2D Observations . . . . . . . . . . . . . 86
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 87
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 92
5.4 Joint 2D and 3D Pose Estimation . . . . . . . . . . . . . . . . . . . . . . 94
Method . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 95
Results . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 100
5.5 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 102
6 Probabilistically Modelling Fashion
105
6.1 Introduction . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 105
6.2 Segmentation of Garments in Fashion Images . . . . . . . . . . . . . . . 107
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 108
Clothing a Person . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 109
Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 115
6.3 Modelling Fashionability . . . . . . . . . . . . . . . . . . . . . . . . . . . 120
Related Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 122
Fashion144k Dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 123
Discovering Fashion from Weak Data . . . . . . . . . . . . . . . . . . . . 126
Experimental Evaluation . . . . . . . . . . . . . . . . . . . . . . . . . . . 129
6.4 Summary . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 136
7 Conclusions
137
7.1 Future Work . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 138
7.2 Trends in Computer Vision and Machine Learning . . . . . . . . . . . . 139
Bibliography141

List of Figures

2.1
2.2
2.3
2.4
2.5
2.6

Human detection with HOG descriptors . . . . . . . . . . . . . . . . . . . .
Examples of saliency detection . . . . . . . . . . . . . . . . . . . . . . . . .
Architecture of a convolutional neural network . . . . . . . . . . . . . . . .
2D human pose estimation . . . . . . . . . . . . . . . . . . . . . . . . . . . .
Convolutional Neural Network architecture . . . . . . . . . . . . . . . . . .
Example of a Conditional Random Field model . . . . . . . . . . . . . . . .

3.1
3.2
3.3
3.4

10
11
12
13
18
20

SIFT descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 22
Comparing DaLI against SIFT, DAISY, LIOP, and GIH. . . . . . . . . . . . 23
Visualization of a DaLI descriptor. . . . . . . . . . . . . . . . . . . . . . . . 24
Flowchart of the algorithm used to calculate the DaLI and DaLI-PCA descriptors . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 25
3.5 DaLI descriptor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 27
3.6 Invariance of the DaLI and DI descriptors to non-rigid deformations andillumination changes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 29
3.7 Evaluation of the descriptor robustness on synthetic sequences . . . . . . . 30
3.8 Representation of an image patch . . . . . . . . . . . . . . . . . . . . . . . . 32
3.9 Diﬀerent mesh triangulations . . . . . . . . . . . . . . . . . . . . . . . . . . 32
3.10 Preserving edge information . . . . . . . . . . . . . . . . . . . . . . . . . . . 34
3.11 Deformable and varying illumination dataset . . . . . . . . . . . . . . . . . 36
3.12 Extracting patches from points of interest . . . . . . . . . . . . . . . . . . . 37
3.13 DaLI performance for diﬀerent parameter values So , β, σ and wmax . . . . . 38
3.14 The ﬁrst 7 frequencies of the ﬁrst 20 components of the PCA basis . . . . . 40
3.15 PCA approximation of the DaLI descriptor . . . . . . . . . . . . . . . . . . 41
3.16 Mean detection rates obtained by scaling regions of interest with diﬀerent
factors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 42
3.17 Detection rate when simultaneously varying deformation level and illumination conditions . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 43
3.18 DaLI descriptor results for varying only deformation or illumination . . . . 44
3.19 Sample results from the dataset . . . . . . . . . . . . . . . . . . . . . . . . . 45
3.20 True positive, false positive, false negative and true negative image patch pairs 46
3.21 Mean detection accuracy on two real world videos . . . . . . . . . . . . . . 47
3.22 Overview of the network architectures used . . . . . . . . . . . . . . . . . . 51
ix

x

List of Figures

3.23 Pairs of corresponding samples from the Multi-view Stereo Correspondence
dataset . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 53
3.24 Precision-recall curves for diﬀerent network architectures . . . . . . . . . . . 55
3.25 Precision-recall curves for diﬀerent network hyperparameters . . . . . . . . 55
3.26 Precision-recall curves for diﬀerent levels of mining . . . . . . . . . . . . . . 57
3.27 Precision-recall curves for diﬀerent number of ﬁlters and fully-connected layers 57
3.28 Precision-recall curves for the generalized results over the three dataset splits 58
3.29 Samples of matches retrieved with our descriptor . . . . . . . . . . . . . . . 60
4.1
4.2
4.3
4.4
4.5
4.6
4.7
4.8
4.9

Example of a GPDM with a 3-dimensional latent space. . . . . . . . . . . .
Examples of 3D body shape scans . . . . . . . . . . . . . . . . . . . . . . .
Eﬀect of σthreshold in linear latent models. . . . . . . . . . . . . . . . . . . .
Dimensionality of the Linear Latent Space . . . . . . . . . . . . . . . . . . .
Directed Acyclic Graph models for 3D human pose . . . . . . . . . . . . . .
Directed Acyclic Graph modeltraining and sampling . . . . . . . . . . . . .
Geodesics and mixture models . . . . . . . . . . . . . . . . . . . . . . . . .
Sampling from the Geodesic Finite Mixture Model . . . . . . . . . . . . . .
Human pose regression with the GFMM . . . . . . . . . . . . . . . . . . . .

64
66
68
69
71
72
73
78
80

5.1 Example of a multiview 3D pose estimation approach . . . . . . . . . . . . 84
5.2 Convolutional Neural Network for 3D Pose Estimation . . . . . . . . . . . . 85
5.3 3D human pose estimation from noisy observations. . . . . . . . . . . . . . . 86
5.4 Flowchart of the method for obtaining 3D human pose from single images . 87
5.5 Exploration of the solution space . . . . . . . . . . . . . . . . . . . . . . . . 91
5.6 Exploring the space of articulated shapes. . . . . . . . . . . . . . . . . . . . 91
5.7 Choosing human-like hypothesis via anthropomorphism. . . . . . . . . . . . 92
5.8 Comparison of diﬀerent error functions for 3D human pose estimation. . . . 93
5.9 3D Human Pose Estimation Results . . . . . . . . . . . . . . . . . . . . . . 95
5.10 Results on the TUD Stadmitte sequence. . . . . . . . . . . . . . . . . . . . . 96
5.11 Simultaneous estimation of 2D and 3D pose. . . . . . . . . . . . . . . . . . . 96
5.12 Overview of our method for simultaneous 2D and 3D pose estimation . . . . 97
5.13 2D part detectors. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 98
5.14 Sample frames from the HumanEva dataset. . . . . . . . . . . . . . . . . . . 101
5.15 Two sample frames from the TUD Stadmitte sequence . . . . . . . . . . . . 102
6.1 Garment segmentation example . . . . . . . . . . . . . . . . . . . . . . . . . 106
6.2 Fine-grained annotations of the Fashionista dataset. . . . . . . . . . . . . . 107
6.3 Visualization of object masks and limbs. . . . . . . . . . . . . . . . . . . . . 110
6.4 Visualization of clothelets . . . . . . . . . . . . . . . . . . . . . . . . . . . . 111
6.5 Wordnet-based similarity function . . . . . . . . . . . . . . . . . . . . . . . 114
6.6 Confusion matrix for our approach on the Fashionista v0.3 dataset. . . . . . 116
6.7 Per class results for our model. . . . . . . . . . . . . . . . . . . . . . . . . . 118
6.8 Visualization of diﬀerent features . . . . . . . . . . . . . . . . . . . . . . . . 120
6.9 Qualitative clothing segmentation results . . . . . . . . . . . . . . . . . . . 121
6.10 Example of an outﬁt recommendation . . . . . . . . . . . . . . . . . . . . . 122
6.11 Anatomy of a post from the Fashion144k dataset . . . . . . . . . . . . . . . 122
6.12 Normalization of votes . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 126
6.13 Visualization of the density of posts and fashionability by country . . . . . 126

List of Figuresxi

6.14 An overview of the CRF model and the features used by each of the nodes . 127
6.15 Illustration of the type of deep network architecture used to learn features . 128
6.16 Visualization of mean beauty and dominant ethnicity by country . . . . . . 130
6.17 True and false positive fashionability prediction examples . . . . . . . . . . 132
6.18 Visualization of the dominant latent clusters . . . . . . . . . . . . . . . . . . 133
6.19 Visualizing pairwise potentials between nodes in the CRF . . . . . . . . . . 134
6.20 Examples of recommendations provided for our model . . . . . . . . . . . . 135
6.21 Visualization of the temporal evolution of the diﬀerent trends in Manila and
Los Angeles . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 135

List of Tables

2.1

Diﬀerent types of computer vision cues . . . . . . . . . . . . . . . . . . . . .

8

3.1 DaLI computation time and mesh complexity for diﬀerent triangulations . .
3.2 DaLI parameters . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . .
3.3 Evaluation results on the dataset for all descriptors . . . . . . . . . . . . . .
3.4 Comparison of performance and descriptor size . . . . . . . . . . . . . . . .
3.5 Eﬀect of normalizing image patches for various descriptors . . . . . . . . . .
3.6 Eﬀect of network architectures . . . . . . . . . . . . . . . . . . . . . . . . .
3.7 Comparison of network hyperparameters . . . . . . . . . . . . . . . . . . . .
3.8 Eﬀect of mining samples on performance . . . . . . . . . . . . . . . . . . . .
3.9 Eﬀect of number of ﬁlters and fully-connected layer . . . . . . . . . . . . . .
3.10 Precision-recall area under the curve for the generalized results . . . . . . .

33
35
46
48
49
54
54
56
56
58

4.1
4.2
4.3

Comparison of generative pose models . . . . . . . . . . . . . . . . . . . . .
Inﬂuence of the number of latent states . . . . . . . . . . . . . . . . . . . .
Comparison of the Geodesic Finite Mixture Model with other approaches. .

66
71
79

5.1
5.2
5.3

Comparison of results on the HumanEva dataset. . . . . . . . . . . . . . . . 94
Results on the HumanEva dataset. . . . . . . . . . . . . . . . . . . . . . . . 100
Comparison against the state-of-the-art. . . . . . . . . . . . . . . . . . . . . 102

6.1 Overview of the diﬀerent types of potentials used in the proposed CRF model109
6.2 Comparison against the state-of-the-art. . . . . . . . . . . . . . . . . . . . . 115
6.3 Evaluation on the foreground segmentation task . . . . . . . . . . . . . . . . 117
6.4 Inﬂuence of pose. . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 117
6.5 Oracle performance for diﬀerent superpixel levels . . . . . . . . . . . . . . . 118
6.6 Diﬀerent results using only unary potentials in our model . . . . . . . . . . 118
6.7 Importance of the diﬀerent potentials in the full model . . . . . . . . . . . . 119
6.8 Statistics of the Fashion144k dataset . . . . . . . . . . . . . . . . . . . . . . 124
6.9 Mean fashionability of cities with at least 1000 posts . . . . . . . . . . . . . 125
6.10 Overview of the diﬀerent features used . . . . . . . . . . . . . . . . . . . . . 125
6.11 Eﬀect of various attributes on the fashionability . . . . . . . . . . . . . . . . 130
6.12 Fashionability prediction results . . . . . . . . . . . . . . . . . . . . . . . . . 131
6.13 Evaluation of features for predicting fashionability . . . . . . . . . . . . . . 132
xiiChapter 1

Introduction

I may not have gone where I
intended to go, but I think I haveended up where I intended to be.
Douglas Adams

Computer vision is a relatively new ﬁeld with roughly half a century of tradition.
It is a well known anecdote that it originally started out as a summer project for a ﬁrst
year undergraduate student in 1966, whose task was to “connect a television camera to
a computer and get the machine to describe what it sees.” Had computer vision notbeen grossly underestimated and the project succeeded, this thesis would not have beenpossible and the author would be likely enjoying a long drink at the beach. However,seeing that this thesis is indeed ﬁnally completed, we come to the conclusion the project
was not able to complete its ambitious task. Not only that, currently computer vision
is a thriving ﬁeld that is getting closer and closer to solving the 1966 summer problem.
In the beginning, computer vision was seen as more of a mathematical problem,
which was additionally limited by the computational resources of the time. Duringthat period, many of the tools and basic approaches we still use today were developed.
Only in the last decade has the technology advanced suﬃciently to tackle lofty computer
vision problems and obtain reliable results that are making it into real world applications
everywhere. One may argue that this started with the SIFT descriptor (Lowe, 2004),
the most cited paper in computer vision1 , which eventually allowed the appearance ofother notable works such as the deformable parts model (Felzenszwalb et al., 2008) for
object detection. These models began to obtain signiﬁcant results in identifying objects
in natural photographs.
With the progress of time, datasets have increased in size and machine learning has
played a larger and larger role in computer vision, to the point that it has now become
an indispensable tool. One of the most important recent breakthroughs has been theuse of Convolutional Neural Networks (CNN) for classiﬁcation (Krizhevsky et al., 2012)1
Indeed, as it is common among computer vision researchers, we compulsively track the number ofcitations the SIFT paper has, which at the time of this writing is of 28,407 citations.

1

2

Chapter 1. Introduction

which has outperformed existing approaches by a large margin. This has created an
important resurgence of Artiﬁcial Neural Networks (ANN) in computer vision, as they
are the basis for these techniques. In compliance with the current trends we rely heavily
on machine learning throughout this thesis.
One particular area of computer vision is the understanding of human-centric images. This encompasses many diﬀerent problems such as detection of individuals, 2D/3Dpose estimation, attribute prediction, gaze prediction, clothing segmentation, etc. In
particular, tasks such as face detection are already a reality and omnipresent for almost
all types of digital cameras, while other problems such as assessing image memorability
are still in their incipiency. In this thesis we focus on 3D human pose estimation, clothes
segmentation, and predicting fashionability. For this purpose we have developed generative models for 3D human pose and feature point descriptors, which by themselves
are unable to perform these tasks, but play a fundamental role in the algorithms thatare able to do so.
Although our ultimate objective is higher level comprehension of human-centric
images and problems such as the estimation of fashionability, this is unable to be performed without leveraging robust low level cues. Therefore it is critical to have a strong
grasp of mid and low level algorithms. For this purpose we exploit a swath of existing
features while supplementing them with our own. We show that our features are able
to complement the existing ones while palliating their deﬁciencies.

1.1

Contributions

We organize the contributions into four groups: feature point descriptors, generative 3D
human pose models, 3D human pose estimation, and probabilistically modelling fashion.
1. Feature point descriptors. We propose two diﬀerent feature point descriptors:
one based on heat diﬀusion and another based on Convolutional Neural Networks(CNN). In the ﬁrst approach we model the image patch as a 3D surface. Afterwards we calculate the heat diﬀusion for logarithmically sampled time intervals
and perform a Fast Fourier Transform on the resulting heat maps. We show that
this approach is robust to both deformation and illumination changes. This descriptor, which we call DaLI , was published in (Simo-Serra et al., 2015b). The
second work consists of using CNN to learn discriminative representations of image patches. This is done by using a Siamese CNN architecture, trained withpairs of corresponding patches. We propose learning with a sampling-based approach that in combination with heavy mining, which we denote “fracking”, is able
to signiﬁcantly outperform hand-crafted features. This work is currently undersubmission (Simo-Serra et al., 2014c).
2. Generative 3D human pose models. We present two diﬀerent generativemodels for parameterizing the 3D human pose: a Directed Acyclic Graph (DAG)
and a mixture model for Riemannian manifolds. The DAG model uses discrete 3D
poses obtained from clustering and learns the joint distribution of the resulting 3D
poses and a latent space. By not having any loops, it is possible to eﬃciently map
from the latent pose space to the 3D space and vice versa. This work was publishedas part of (Simo-Serra et al., 2012). For the second approach we model the 3D
human pose by considering that it can be represented as data on a Riemannian
manifold. This model is shown to preserve physical properties, such as distances

1.1. Contributions

3

between joints, much better than competing approaches. Additionally, we showthat this scales well to large datasets and allows real-time sampling from themodel. This work was published in (Simo-Serra et al., 2014b), with an extension
for estimating velocities published in (Simo-Serra et al., 2015c).
3. 3D human pose estimation. We propose two diﬀerent algorithms for 3D human pose estimation from single monocular images. In the ﬁrst work we assume
we have a noisy estimation of the 2D human pose and use a linear formulationthat allows us to project forward these estimations to the 3D space. We then use
the distances between the 3D joints to estimate the anthropomorphism of the hypotheses and obtain a ﬁnal solution. This work was published in (Simo-Serra
et al., 2012). The second approach performs the 2D and 3D human pose estimation jointly in a single probabilistic framework. This is done by extending thepictorial structures Bayesian formulation (Felzenszwalb and Huttenlocher, 2005)
to 3D, which naturally leads to a hybrid generative-discriminative model. To estimate the 3D pose we sample from a generative 3D pose model and then evaluate
the hypotheses using a bank of discriminative 2D part detectors. We perform thisalternate optimization until convergence. This work was published in (Simo-Serra
et al., 2013).
4. Probabilistically modelling fashion. We ﬁnally tackle two diﬀerent problems
in the context of fashion: semantic segmentation of clothing, and predicting the
fashionability of people in images. In the ﬁrst problem we attempt to do ﬁnegrained recognition of garments in an image. We propose a Conditional Random
Fields (CRF) model that labels the diﬀerent superpixels in the image. This is doneby leveraging a set of strong unary potentials in combination with ﬂexible pairwise
potentials that are able to signiﬁcantly outperform the state-of-the-art. This work
was published in (Simo-Serra et al., 2014a). For the second problem of estimatingfashionability, we have created a novel large dataset called Fashion144k generated
by crawling posts from the largest online social fashion website. We aggregate
votes or “likes” as a proxy for the fashionability of the posts and propose a large
assortment of features that we compress using a deep Neural Network. Finally
we interlace the features using a CRF model. The resulting model is then able
to naturally learn diﬀerent types of users, outﬁts, and settings, which are used
to predict the fashionability of a post, and even give fashion advice. This work
was published in (Simo-Serra et al., 2015a). Both these last two works are thefruition of collaborating with Prof. Raquel Urtasun and Prof. Sanja Fidler, fromthe University of Toronto.
Our focus is on high performance, and when necessary, fast models. We publish thecode and the datasets when possible2 in order to ensure our contributions are beneﬁcial
to the community. We hope this will encourage other researchers to use and compare
against our approaches.

Publications
The following is a list of the publications derived from this thesis:2

http://www.iri.upc.edu/people/esimo/

4

Chapter 1. Introduction
Simo-Serra, E., Ramisa, A., Alenyà, G., Torras, C., and Moreno-Noguer, F.
Single Image 3D Human Pose Estimation from Noisy Observations. In IEEE
Conference on Computer Vision and Pattern Recognition, 2012.
Simo-Serra, E., Quattoni, A., Torras, C., and Moreno-Noguer, F. A Joint Modelfor 2D and 3D Pose Estimation from a Single Image. In IEEE Conference on
Computer Vision and Pattern Recognition, 2013.
Simo-Serra, E., Torras, C., and Moreno-Noguer, F. Geodesic Finite Mixture
Models. In British Machine Vision Conference, 2014.
Simo-Serra, E., Fidler, S., Moreno-Noguer, F., and Urtasun, R. A High Performance CRF Model for Clothes Parsing. In Asian Conference on Computer
Vision, 2014.
Simo-Serra, E., Torras, C., and Moreno-Noguer, F. DaLI: Deformation andLight Invariant Descriptor. International Journal of Computer Vision, 1:1–1,
2015.
Simo-Serra, E., Torras, C., and Moreno-Noguer, F. Lie Algebra-Based Kinematic Prior for 3D Human Pose Tracking. In International Conference on Machine Vision and Applications, 2015.
Simo-Serra, E., Fidler, S., Moreno-Noguer, F., and Urtasun, R. Neuroaesthetics
in Fashion: Modeling the Perception of Fashionability. In IEEE Conference on
Computer Vision and Pattern Recognition, 2015.
Additionally a publication is still under submission:
Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., and Moreno-Noguer, F.
Fracking Deep Convolutional Image Descriptors. arXiv preprint arXiv:1412.6537,
2014.

1.2

Thesis Overview

We group the work done conceptually into four major chapters as done for our contributions and then organize them from low to high level. Each of these chapters has an
introductory section, several sections explaining diﬀerent techniques, each corresponding roughly to a single publication, and a summary of the work. We additionally include
a chapter to give an overview of our eﬀorts as well as establishing some groundwork forsome techniques we will use throughout the thesis. Finally the last chapter provides a
general wrap-up and concludes the thesis.
Chapter 2: Overview. This introductory chapter gives a high level overview of thecurrent state of the art of computer vision for human-centric images. We also
introduce basic concepts of tools that we will use throughout the thesis.
Chapter 3: Feature Point Descriptors. This chapter focuses on our work on feature point descriptors. It gives a short introduction and then has two major
sections that cover our 2015 IJCV and a paper under submission respectively.

1.2. Thesis Overview

5

Chapter 4: Generative Models for 3D Human Pose. This chapter focuses on thethree diﬀerent 3D human pose models that we use in the rest of the thesis, especially in Chapter 5. We ﬁrst give an introduction followed by an in-depth review
of models used in the past decade. The following two sections refer to the models
used in our 2012 and 2013 CVPR papers. Afterward, we describe the approachof our 2014 BMVC paper in detail. We conclude with a summary of the diﬀerent
models.
Chapter 5: 3D Human Pose Estimation. This chapter focuses on our two approaches
for estimating 3D human pose from single images. We ﬁrst overview the approaches and elaborate on the related work in the ﬁeld. The following two sections are devoted to our 2012 and 2013 CVPR papers, respectively. Lastly, wesummarize the results.
Chapter 6: Probabilistically Modelling Fashion. In this chapter we look at the
problem of modelling fashion. We start with the motivation of this problem andits impact in society before going into details of our 2014 ACCV and 2015 CVPR
papers, respectively. The results are summarized at the end.
Chapter 7: Conclusions. The last chapter gives a high-level discussion of our contributions, how our work stands in the ﬁeld, and current developments. We alsopose open questions and sketch directions for future research.

Chapter 2

Overview

Scientific progress goes “boink”?
HobbesIn this thesis we attempt to tackle very challenging high level computer vision
problems. In order to do this, it is necessary to rely on robust low and mid level cues.
In this chapter we will explain and give notions of the diﬀerent cues and algorithms thatcan form part of these high level computer vision models. Of course the concepts of highand low are relative. In this chapter, and by extension the rest of this thesis, we shallconsider as high level models those that perform tasks such as the ones we will present
in Chapter 6, i.e., semantic segmentation of clothing and predicting fashionability froman image. Mid level cues will include pose estimation models such as those presented
in Chapter 5 and foreground-background segmentation algorithms. Finally, low levelcues will refer to features such as the prior distributions we describe in Chapter 4 or
the feature point descriptors we will present in the next chapter.
We shall additionally discuss several machine learning models used throughout thisthesis. In particular we will focus on logistic regression classiﬁers, Support Vector
Machines (SVM), deep networks, and Conditional Random Fields (CRF). We shallformulate the diﬀerent models, explain how they can be learnt, and brieﬂy state someusage cases and applications of them.
This overview is not meant to be an exhaustive list of all the diﬀerent cues andmachine learning models commonly used in computer vision. Instead, it is meant tobe a rough overview with a focus on the tasks considered in this thesis. A list of thediﬀerent cues we will discuss and their relationship to this thesis can be seen in Table 2.1.
This chapter is divided into four sections that address low, mid, high level cues, andmachine learning models, respectively.

2.1

Low Level Cues

We understand low level cues as simple features or priors that do not encode very
proliﬁc information. They are meant to be “austere” and fast to compute, and are7

8

Chapter 2. Overview

Level

Feature

Type

C3

Maps
Maps
Maps
Regions
Sparse, Maps
Likelihood PriorX

Low

ColourGabor Filter
Edge Detector
Region Histogram
Descriptors
PDF Prior
Detectors
Foreground Segmentation
Saliency
Order 2 Pooling
GIST
CNN
Pose
Attributes
Bag-of-Words

Maps, Bounding Boxes
Maps
Maps
Regions
Image
Regions, Image
Spatial Prior
Generic
Regions, Image

Semantic Segmentation

Maps

Mid

High

C4

C5

C6
X
X

X
X

X
X
X

X
X
X
X
X
X

X

X
X
X
X
X

Table 2.1: Different types of computer vision cues. Overview of some commonlyused cues in computer vision and their relationship with this thesis. We categorize
them into low, mid and high level cues and based on their types. “Maps” generate
2D or 3D matrices where a value or vector corresponds to each pixel, respectively.
“Regions” correspond to larger areas such as superpixels. “Image” type cues provide
a single vector that corresponds to the entire image. In the right four columns we
indicate to which chapter the cue relates to.
always integrated into some larger model as they, by themselves, are unable to complete
any task. However, this does not mean they are useless. For certain problems, just by
combining and leveraging diﬀerent weak low level cues it is possible to obtain signiﬁcantperformance. As an example we refer to the approach of (Krähenbühl and Koltun,2011) that performs semantic segmentation with good results by only employing simple
color features with some smoothing terms in a Conditional Random Fields framework.

Image Features
There are a variety of potential low level image features. The most simple one is to
directly use the RGB color values. This is most commonly used in Convolutional Neural
Networks (Krizhevsky et al., 2012) after normalizing the values, that is, subtracting themean and dividing by the standard deviation of the pixel values. It is also possible toconvert the RGB values to other color spaces such as HSV, YUV or CIE L*a*b. In
general, it is not clear which color space is the best for a speciﬁc application and thus
it requires experimental validation.

2.1. Low Level Cues

9

Since individual pixels do not contain much information, it is standard to insteadgroup them into diﬀerent regions which conceptually correspond to parts of the same
object in the image (Achanta et al., 2012). This grouping is known as pooling and has
two major eﬀects: it ﬁrst allows a much more eﬃcient representation of the image as it is
now reduced to a set of non-overlapping regions, and secondly, each region contains more
information and thus becomes increasingly discriminative. One of the most popularalgorithms is the gPb superpixel approach (P.Arbelaez et al., 2011). When working
with these larger image regions that no longer consist of a single pixel but rather agroup of pixels, it is common to compute histograms of their color values. This gives acompact representation of the color of the image patch. Furthermore, these histograms
are normalized so that the total sum of all their elements is 1, allowing the diﬀerent
image regions to be directly compared regardless of the number of pixels they have.
Instead of trying to encode color information, it is also standard to try to directlyencode textural information. One of the most common representations is using 2D
Gabor ﬁlters, which are Gaussian kernel functions modulated by a sinusoidal planewaves and can be written as such:

 ′2

f2
x + γ 2 y ′2′g(x, y; f, θ, φ, γ, ) =
exp
i2πf
x
+
φ
exp −
πγη
2σ 2
′
x = x cos θ + y sin θ
y ′ = −x sin θ + y cos θ

(2.1)

,

where f is the frequency of the sinusoidal factor, θ represents the orientation, φ is thephase oﬀset, σ is the standard deviation of the Gaussian envelope, and γ is the spatialaspect ratio. By computing the responses of a bank of Gabor ﬁlters with diﬀerent
orientations and frequencies it is possible to get a local representation of the image
texture. It is also standard in this case to calculate the normalized histogram of the
responses as a texture representation of an image region or superpixel as described
before.
Another approach is to detect edges in the image. This is commonly done by ﬁrst
blurring the image to reduce the image noise, before proceeding to convolve the image
with simple Sobel ﬁlters. This ﬁlters are designed to detect lines of diﬀerent orientation,e.g., horizontal and vertical lines. The output map resulting from convolving the image
with the Sobel ﬁlters can then be used as an indication of where edges are in the image.
This information is generally useful when attempting to distinguish contours of objects.
A slightly more advanced approach consists in using feature point descriptors. These
are generally compact vector representations of small image patches. The most wellknown approach is the SIFT descriptor (Lowe, 2004). This descriptor ﬁrst computes
the gradient of a grayscale image, and then uses a rectangular grid in which all thesections build a histogram of the gradient values. These values are then weighted
by a Gaussian centered on the patch. All the histograms are ﬁnally concatenated intoa single vector which is normalized such that the sum of all the elements is one. As
SIFT was originally designed to be used sparsely, there have been several alternatives
proposed to be calculated densely on the image. The most popular one is the Histogramof Oriented Gradients (HOG) descriptor (Dalal and Triggs, 2005) shown in Fig. 2.1(e),while a more eﬃcient modern variant is the DAISY descriptor (Tola et al., 2010), whichuses overlapping circles instead of a rectangular grid.

10

Chapter 2. Overview

(a)

(b)

(c)

(d)

(e)

(f)

(g)

Figure 2.1: Human detection with HOG descriptors. The HOG detectors cue
mainly on silhouette contours. (a) The average gradient image over the training data.
(b) Each “pixel” shows the maximum positive SVM weight in the block centered on the
pixel. (c) Likewise for the negative SVM weights. (d) A test image. (e) Its computed
HOG descriptors. (f,g) The HOG descriptors weighted by the positive and negative
SVM weights, respectively. Figure reproduced from (Dalal and Triggs, 2005).

Simple Priors
It is also standard to use simple priors in addition to image features in computer vision.
A very common prior is to use the location of a pixel within an image. This is based on
the knowledge that for images taken by humans, in general, the subject is well focusedand centered on the image. By using the position within the image as a feature it ispossible to capture this relationship. For image patches either a histogram is calculatedor the mean position for all the pixels in the patch is used.
When dealing with any sort of variable, there are usually some values that are more
likely than others. An example would be the orientation of humans in images. In
general they will be standing upright; it is less likely that they will be horizontal. In
order to capture the prior distribution it is common to use mixture models to estimate
the Probability Density Function (PDF) of the variable. This will help a model take
into account which states or values are more likely.

2.2

Mid Level Cues

The cues we will discuss in this section are more elaborated than the low level ones,
being generally the result of some more complex algorithm. Yet, they are able to capture
concepts which are more useful when performing higher level tasks. The downside isthat they are slower to compute.

Image Features
Object detectors are often used as image feature. The most simple detectors consist
in using a template of the object to be detected and convolving it with the image. In
general, this is not done directly on the raw pixel values but on extracted descriptors,
and the response correlates with the presence of the object. An example is the caseof human detection (Dalal and Triggs, 2005) which was performed by convolving atemplate of Histogram of Oriented Gradients (HOG) descriptors at diﬀerent scales. A

2.2. Mid Level Cues
Input

Method 1

11
Method 2 Annotations

Input

Method 1

Method 2 Annotations

Figure 2.2: Examples of saliency detection. For input images we show the resultof applying Method 1 (Hou and Zhang, 2007) and Method 2 (Itti and Koch, 2000).
In order to obtain a ground truth of the saliency, all the images were labelled by fourannotators. The black area of the annotations corresponds to non-salient regions, thegrey area was a region identiﬁed as salient by at least one of four annotators, and thewhite area is the region selected by all four annotators. Figure reproduced from (Houand Zhang, 2007).
representation of the approach can be seen in Fig. 2.1. As a feature it can be either used
the output of the algorithm, in this case a bounding box of the object, or the associatedlikelihood map, which gives a spatial prior of where the objects is most likely to be in
the image.
Another widely used approach is to run a segmentation algorithm to ﬁrst attempt
to distinguish between what is likely to be the central object and what is likely tobe the background in the image. One possible algorithm that can be used with thispurpose is a Conditional Random Fields (CRF) model in which each pixel can have astate of being either foreground or background. Then, by using Gaussian prior on thecolor values for foreground and background, in addition with simple potentials between
neighbouring pixels, it is possible to obtain fast segmentation results (Krähenbühl and
Koltun, 2011). It is also possible to use more accurate and robust algorithms, although
at a higher computational cost (Carreira and Sminchisescu, 2012).
An interesting feature proposed recently consists of trying to identify salient regionsof the image. This is inspired by how the human visual cortex is able to quickly discerninteresting objects to focus attention on. Saliency maps are in particular useful to descry
the predominant regions of an image in order to focus on them (Todt and Torras, 2004).
One of the more standardized approaches consists of extracting the spectral residueof an image and then constructing the saliency map in the spatial domain (Hou andZhang, 2007). Several saliency maps are depicted in Fig. 2.2.

12Chapter 2. Overview

Figure 2.3: Architecture of a convolutional neural network. This network takes
224x224 RGB images as input, and outputs a 1000-dimensional vector that corresponds
to the probability of the image belonging to one of 1000 diﬀerent classes. It has 5 convolutional layers and 3 densely connected layers. This network was originally designed forthe ILSVRC2012 challenge for a 1000 class classiﬁcation challenge, where it obtained
the ﬁrst place. Figure reproduced from (Krizhevsky et al., 2012).
Instead of computing simple histograms of basic features, it is also possible to use
higher order pooling. For example, (Carreira et al., 2012) performs second order poolingof SIFT descriptors with impressive results. In this approach the outer product ofall descriptors extracted from an image region is computed. Then, either the max orthe average of the resulting vector is used as a cue. More speciﬁcally, given a set of n
descriptors x1 , . . . , xn of a region, the pooling is simply
Gavg (x1 , . . . , xn ) =1X T
xi · xi ,
n

(2.2)

i

for the average pooling case, andGmax (x1 , . . . , xn ) = max xT
i · xi ,i(2.3)

for the max pooling case. This approach yields stronger statistics of the region, allowing
simple classiﬁers using these features to be much more discriminative than instead
relying on more complex classiﬁers with simple features.
For full images, calculating a global descriptor which represents the entire image instead of a local area is also a possibility. One of the most popular global descriptorsof this type is the GIST descriptor (Oliva and Torralba, 2001). This descriptor uses
spectral and coarsely localized information in order to approximate a set of perceptual dimensions of the image such as naturalness, openness, etc. This results in a
544-dimension descriptor that is what is considered a “hand-crafted” feature, i.e., it
is manually designed instead of beingautomatically learnt by a computer algorithm.
Recently, learning deep Convolutional Neural Networks (CNN) to extract features has
been used to obtain similar features with great results (Girshick et al., 2014). The networks to extract these features are usually trained on very large datasets for the task ofimage classiﬁcation. After being trained, one or more of the ﬁnal layers of the network
are removed and the output of the new last layer is used as a mid level representation of
the image. One of the most popular networks to extract these features is the “AlexNet”

2.2. Mid Level Cues

13

Figure 2.4: 2D human pose estimation. Some examples of 2D human pose estimations on diﬀerent images. For each image pair, the bounding boxes of all individual
parts are shown on the left, and the resulting joint skeleton is shown on the right. Figure reproduced from (Yang and Ramanan, 2011).
from (Krizhevsky et al., 2012), shown in Fig. 2.3. In this particular case the secondto-last 4096-dimensional layer is used as a global representation of the image. Thesefeatures have been found to generalize very well to tasks other than the one originallytrained for (Donahue et al., 2013). See (Rubio et al., 2015) for an example of using
these features in a robotics application.

Prior Models
When dealing with human-centric image tasks, having an estimation of the pose canbe a very useful feature. In general a fast state-of-the-art algorithm such as (Yang andRamanan, 2011) is used to estimate the 2D joints of all the individuals in an image.
This model relies on a mixture of HOG templates for detecting diﬀerent body parts.
Then, it models co-occurrence and spatial relationships with a tree structure that allows
eﬃcient inference with dynamic programming. Once the pose is estimated, the locationof the joints within the image can be used as a spatial prior. Furthermore, the relativepose of the human can also be used as a feature when performing tasks such as action
recognition. Some examples of 2D human pose estimation can be seen in Fig. 2.4.
Another useful source of information consists in computing attributes representing
some mid level information (Farhadi et al., 2009). For example, in the case of facerecognition, this kind of information would consist in whether or not a person wears

14

Chapter 2. Overview

glasses and her/his gender or ethnicity. In general, these attributes are obtained bytraining discriminative models on annotated datasets.

Bag-of-Words
A widely used approach to obtain features for regions or entire images is the bag-ofwords model. This model was originally designed for extracting features from text, inwhich the bag-of-words is was just a dictionary of words. For a given input text, thenumber of occurrences that appear in the text of each word in the dictionary is counted.
This is a simple way of obtaining a sparse representation that can then be used asfeatures when training classiﬁers or other algorithms.
This approach is not limited to discrete features such as words in text, but canalso be used for continuous features such as Gabor ﬁlter histograms or feature point
descriptors. In this case, instead of using a dictionary of words, a codebook made of alist of representative features is constructed. A query feature is then made discrete byassigning it to the most similar entry in the codebook. This allows for a more global
representation to be learned for either regions or whole images. The downside of thisrepresentation is that it does not take into account the spatial layout of the input data.

2.3

High Level Cues

Usually, high level cues represent the ultimate goal to be achieved. However, they canalso be used as features in certain applications. We will brieﬂy mention the possibilityof using semantic segmentation algorithms as features for other models. Semantic segmentation diﬀers from foreground segmentation in the sense there are more than twoclasses (foreground and background) and thus it is no longer possible to use eﬃcientalgorithms such as graphcuts (Boykov et al., 2001) to solve the problem. The output
of a semantic segmentation algorithm is a label for all the pixels in the image and ishighly related to the detection task, that is, regions in the image that belong the sameclass can be considered a detection of that particular object. It is not uncommon forthese algorithms to use detectors as features.

2.4

Machine Learning Models

There is no doubt that machine learning plays a fundamental role in modern computer
vision. From classifying objects in images to reasoning about the scene, it allows exploiting large sets of data to generate models that are capable of making predictions
or taking decisions. There are several approaches to machine learning. In this section
we shall consider the supervised learning problem which consists of designing models
that are able to predict labels from input features. These models have a number of parameters which instead of being set manually, are algorithmically chosen such that theyminimize the divergence between the model predictions and the known labels for annotated data.
In this section we present four diﬀerent commonly used supervised models: logistic
regression, support vector machines, deep networks, and conditional random ﬁelds. For
each of the models we formulate the prediction rules and the learning function that is2.4. Machine Learning Models

15

optimized when learning the parameters of the model. We will also give some notions
on how they can be used and learned in practice, along with several examples.

Logistic Regression
The logistic regression is a linear model that despite its name is used for classiﬁcationand not regression. In particular, it outputs a single [0, 1] value that is interpretable asa probability. The model parameters are a vector of weights wT = [w1 , . . . , wn ] of thesame length as the input xT = [x1 , . . . , xn ], which may or may not include a bias term.
The probability of x belonging to class y = +1 is written as:
1
.
(2.4)
1 + e−wT x
The model parameters are optimized by minimizing the negative log-likelihood of thepredictions with an additional regularization term:


X
1
T
log 1 + e−yw x ,(2.5)
min wT w + C
w 2
flr (x) =(x,y)

where C > 0 is the regularization parameter, y = {−1, +1} is the label of a particulartraining sample, and x are the features of the same sample. This optimization can besolved by using gradient-based methods. In particular we use the trust region Newton
method implementation of LIBLINEAR (Fan et al., 2008).
We note that the standard formulation is for the two-class case. In order to generalize
to the m-class case it is common to use a one-vs-all approach in which for each of thepossible m classes, a logistic regression is trained to predict only that particular class,using all the other classes as negatives. By concatenating all the individual logistic
regression outputs a m-dimensional vector is obtained. The class with the largest value
will be the prediction for the sample.

Support Vector Machines
In contrast to logistic regression, Support Vector Machines (SVM) are non-probabilistic
models. While they can be linear, usually non-linear variants are used. The decisionfunction, or predicted class of x is:
fsvm (x) = sgn wT φ(x) + b ,(2.6)

where φ(x) maps x into a high-dimension space, and b is a bias, which is made explicitin this case.
The model parameters w and b are found by maximizing the margin or the cleanest
possible split between the training examples of diﬀerent classes in the dataset. This isdone by introducing slack variables ξi , which measure the degree of misclassiﬁcation of
the data sample xi . We consider two possible labels for each sample yi = {−1, +1}.
Thus the parameters are optimized by:
N

X
1 T
w w+C
ξi
w,b,ξ
2
i=1

subject to yi wT φ(xi ) + b ≥ 1 − ξi ,min

ξi ≥ 0,i = 1, . . . , N

(2.7)

16

Chapter 2. Overview

where C > 0 is the regularization parameter, and N is the number of training samples.
Due to the high dimensionality of w usually the dual problem is optimized:
1 T
α Qα − eT α
2subject to yT α = 0min
α0 ≤ αi ≤ C,

(2.8)

i = 1, . . . , N

where e = [1, . . . , 1]T is a vector of all ones, Q is an N × N positive semideﬁnite matrix,
Qij = yi yj K(xi , xj ), and K(xi , xj ) = φ(xi )T φ(xj ) is the kernel function.
After the optimization process, the optimal w satisﬁes:w=

NXyi αi φ(xi ) ,

(2.9)

i=1

which when combined with Eq. (2.6) yields an equivalent decision function for an input
x based on the dual:!
NX
fsvm (x) = sgn
yi αi K(xi , x) + b .
(2.10)
i=1

We notice that in this case we have a new hyperparameter which is the choice of
kernel function K(xi , x). While there are many diﬀerent options available, we consider
the widely used Radial Basis Function (RBF) kernel deﬁned by:K(xi , x) = exp −γ|xi − x|2 ,(2.11)

where γ is the smoothness parameter of the kernel.
The optimization of the parameters of a SVM is a convex quadratic programming
problem in both the primal (Eq. (2.7)) and the dual (Eq. (2.8)). We use the heuristicbased approach of LIBSVM (Chang and Lin, 2011) to optimize the dual.

Deep Networks
Artiﬁcial Neural Networks (ANN) have a long tradition in computer science. However,
only recently has their usage exploded in the ﬁeld of computer vision. In particular the
networks used, broadly coined “deep networks”, have two important properties: theyuse convolutional layers to lower the number of parameters, and they have many layers,
hence the name “deep”. This recent widespread usage has been instigated by the very
good performance obtained in speciﬁc computer vision tasks. This is due to many small
improvements in combination with a signiﬁcant increase in the available computational
power. Up until now the training of networks with over 50 million parameters has
been infeasible in a reasonable amount of time. Despite many minor improvements
necessary for the improved performance of these networks, the underlying mathematics
and formulation remains unchanged from several decades back.
An ANN is a directed acyclic graphical model, in which the nodes are called “neurons”. The standard feed-forward network we shall consider is a network consisting ofvarious layers. Each layer is only connected to both the previous layer and the nextlayer. We can then write the output of a layer l as:
xl = σ (wl )T xl−1 + bl ,(2.12)

2.4. Machine Learning Models

17

where wl is the weight vector of the layer, bl is the bias term, xl−1 is the output of theneurons in the previous layer, and σ(·) is a non-linear activation function. Commonly
the hyperbolic tangent or rectiﬁed linear unit (ReLU) activation function is used. For
more than one layer the chain rule is applied, i.e., the output of a layer is used as the
input for the next layer.
In general these networks have a large number of parameters or weights w. In orderto learn these weights a technique called back-propagation is used. We assume we havea loss function ∆(y, y∗ ) where y is the prediction of the network and y∗ is the ground∂∆.
truth or true label we want to predict. Back-propagation consists of computing ∂w
This is done by ﬁrst computing the error of the last layer L as:
δL =


∂∆
⊙ σ ′ (wL )T xL−1 + bL ,∂y

(2.13)

where y = xL is the output of the last layer, ⊙ is the Hadamard product or elementwise product of two vectors, and σ ′ in the derivative of the activation function. The
errors of the other layers can then be written as a function of the next layer as:




T
l
l+1
l+1
δ =
⊙ σ ′ (wl )T xl−1 + blδw
(2.14)
Finally the derivatives for the weight k of the neuron j in the layer l can be computedas:∂∆l= xl−1(2.15)k δj ,l∂wjk
and the bias for the layer l becomes:∂∆= δl .
∂bl(2.16)

When learning the network, the features are propagated through the network for eachsample using Eq. (2.12) in what is called a forward pass. Afterwards the loss ∆(y, y∗ )is computed for that given sample’s true label y∗ and the output of the network y,
and is used to then perform a backwards pass. This consists of propagating the error
backwards through the network by ﬁrst using Eq. (2.13) for the top layer and thenEq. (2.14) for the remaining layers. Finally the partial derivatives of all the weights
with respect to the loss are computed and used to update the weights and bias terms:
wi+1 = wi − λ∂∆∂wi(2.17)

where λ is the learning rate hyperparameter which controls the rate at which the weights
are changed. This is done iteratively until some convergence criterion is met.
The usual approach to optimize the network is to use stochastic gradient descent,which is a variant of gradient descent in which only a subset of samples are used at∂∆each iteration to provide an estimate of ∂w
i . This approach, in general, converges fasterthan standard gradient descent, which is fundamental for networks that can have over
50 million parameters.
In computer vision, instead of the fully connected network previously described, it
is common to use what are known as Convolutional Neural Networks (CNN). The
main diﬀerence here is that the output of a layer is obtained by convolving a ﬁlter,

18

Chapter 2. Overview

INPUT
32x32

C1: feature maps
6@28x28

Convolutions

C3: f. maps 16@10x10
S4: f. maps 16@5x5
S2: f. maps
6@14x14

SubsamplingC5: layer F6: layer
120
84

Convolutions

OUTPUT
10

Gaussian connections
Full connection
Subsampling
Full connection

Figure 2.5: Convolutional Neural Network (CNN) architecture. The convolutional layers consist of several ﬁlters that are convolved with the input. By performing
convolutions, fewer parameters are needed as they are shared when generating the output feature maps. In order to improve eﬃciency, subsampling layers are used in whichanother convolution operator is applied in order to reduce the size of the feature maps.
After the convolutional layers, fully connected layers are used in which spatial information is lost. Figure reproduced from (Lecun et al., 1998).
that is, the weights w are not independent for each output neuron but shared for all
the neurons in the layer. In this way 2D spatial information is conserved. In generalinstead of having a single 2D output feature map, more than one ﬁlter is used. Whilethe number of parameters decreases when using convolutional layers, the number ofcalculations necessary increases. For this purpose it is common to also use subsamplinglayers in which a convolution operator is used to decrease the size of the feature map.
An example of a convolutional neural network can be seen in Fig. 2.5.
In particular the CNN models used in computer vision tend to have many layers,and parameters in the order of tens of millions. In order to learn these models a large
amount of data is necessary. To avoid overﬁtting the model to the training data many
diﬀerent techniques are used. The most common method consists of data augmentation.
In this case a set of synthetic deformations are applied to the image, e.g., cropping,rotating, ﬂipping horizontally, etc., in order to increase the number of training samples.
This has been shown to help the network to generalize better.
As the speciﬁcs of each deep network are highly dependent on the task, we shalldefer the explanation of details to the chapters in which they are used.

Conditional Random Fields
When it comes to probabilistic models, one of the most important ones in computer
vision is the Conditional Random Fields (CRF) model (Laﬀerty et al., 2001). These area class of models used for structured prediction, that is, modelling output data that hasa speciﬁc structure which takes the context into account. An example would be semantic
segmentation of clothing in which although the pixels are being labelled, if there are
pixels that belong to say the “boots” class, then there should be no pixels belonging tofor example “heels”, “sneakers” or “pumps” classes. In particular they are discriminative
undirected probabilistic graphical models which encode known relationships between
observations and construct consistent interpretations.
As indicated by its name, a CRF is modelling the conditional distribution p(y|x) ofa random variable over the corresponding sequence labels y, globally conditioned on a

2.4. Machine Learning Models

19

random variable over sequences to be labeled or observations x. Note that p(x) is not
explicitly modelled as it is observed. Additionally the components of y are considered
to come from a ﬁnite set of discrete states, however, x can come from a continuous
distribution.
Definition 2.4.1 Let G be a factor graph over y. Then p(y|x) is a Conditional Random Field (CRF) if for any fixed x, the distribution p(y|x) factorizes according to G.
If F = {ψA } is the set of factors in G, and each factor takes the exponential family
form, then the conditional distribution can be written as,p(y|x) =Y1TfA (xA , yA ) ,exp wAZ(x)(2.18)

ψA ∈G


P Q
T f (x , y ) is the partition function which ensures
where Z(x) = y ψA ∈G exp wA
A A
A
this is a probability, and wA and fA (xA , yA ) are the weights and feature functions forthe factor ψA , respectively. Note that weights can be shared among diﬀerent factors;
it is not uncommon for templates to be used for many diﬀerent cliques in the graph.
Furthermore, it is also typical to write the factors as potential functions, i.e., φ(y) =wT f (x, y), where x is dropped for notation simplicity.
In order to perform inference we can compute the Maximum A Posterior (MAP)
estimate which consists of:X
TfA (xA , yA ) .
(2.19)wAy∗ = arg max p(y|x) = arg max
y

y

ψA ∈G

Exact inference therefore consists of evaluating all the possible assignments of y. For
particular cases such as tree structures it is possible to eﬃciently compute the exact
marginals. However, the general case is an NP-hard problem making it infeasible toevaluate all the states. Instead, it is usually solved by using approximation algorithms.
We shall use a message passing algorithm called distributed convex belief propagation (Schwing et al., 2011) to perform inference. It belongs to the set of LP-relaxation
approaches and has convergence guarantees, unlike other algorithms such as loopy belief propagation.
In order to perform learning we ﬁrst formulate the conditional log-likelihood:XTwAfA (xA , yA ) − log Z(x) .
(2.20)ψA ∈GThe learning problem can then be posed as minimizing the negative log-likelihood withan additional regularization term


X
X
T
log Z(x) −min CwT w +wAfA (xA , yA ) ,(2.21)
w

(x,y)

ψA ∈G

where C is once again the regularization parameter.
To learn the weights we will consider the primal-dual method of (Hazan and Urtasun,2010), which is a structured prediction framework. It is based on message passing
and has the characteristic that it has guaranteed convergence. It has been shown to be
more eﬃcient than other structured prediction learning algorithms.

20

Chapter 2. Overview

Figure 2.6: Example of a Conditional Random Field (CRF) model. This CRF
attempts to localize the diﬀerent objects in the scene using appearance and geometrycues. Additionally, relative geometry of the diﬀerent objects is considered. Furthermore
it is simultaneously performing scene classiﬁcation using the context provided from thediﬀerent objects in the scene. Figure reproduced from (Lin et al., 2013).
The main advantage of CRF models is the ﬂexibility in deﬁning the graph G anddesign of the diﬀerent feature functions fA (x, y). Both must be heavily tailored to theapplication to capture the context and structure of the problem. As an example we
describe the problem of joint 3D pose estimation of objects and scene detection from
(Lin et al., 2013) shown in Fig. 2.6. In this particular case, there are two types of randomvariables y: the scene variable s, which represents the type of scene; and the object
variables y (i) , which encode the 3D location of the object in the scene. Appearance
factors are deﬁned on all the nodes. The objects additionally have geometrical features,and there are factors capturing the geometric relationship between them. Finally there
are factors between the scene and object nodes that capture the co-occurrences betweenthem.

Chapter 3

Feature Point Descriptors

Representing a small part of an image, usually referred to as an image patch, as acompact vector allows performing many diﬀerent useful tasks, e.g., ﬁnding the relativepose between two images or locating objects in images. Feature point descriptors are a
way of representing these patches. It is well known that images contain a large amountof redundant information, by ﬁnding a smaller description of these patches it is possible
to compare them in a more discriminative and eﬃcient manner.
In this chapter we introduce two diﬀerent descriptors developed as part of this thesis:
the Deformation and Illumination Invariant (DaLI) descriptor and Convolutional Neural
Network (CNN) based descriptors. We will discuss the implementation of both andsummarize their strengths and possible usages.

3.1

Introduction

Feature point descriptors, i.e. the invariant and discriminative representation of local
image patches, is a major research topic in computer vision. The ﬁeld reached maturity
with SIFT (Lowe, 2004), and has since become the cornerstone of a wide range ofapplications in recognition and registration. Descriptors are usually identiﬁed by theirinvariant properties. For example a descriptor that is invariant to rotation will be able
to recognize patches independent of their orientation. On the other hand an illumination
invariant descriptor would be useful facial recognition, but it would not be able todistinguish between night and day.
We show a summary of the aforementioned ubiquitous SIFT (Lowe, 2004) descriptor
which initiated the widespread usage of descriptors in Fig. 3.1. By convolving the image
patch with Gaussians to approximate the image gradient, computing histograms indiﬀerent spatial bins, and then normalizing the resulting vector, Lowe was able to createa strong representation of a local image patch. The resulting descriptor is invariant
to uniform scaling and orientation, while additionally being partially invariant to aﬃnedistortion and illumination changes.
While the most commonly used descriptors rely on convolving with Gaussians, wehave focused on developing alternative descriptors that allow for more expressive rep21

22

Chapter 3. Feature Point Descriptors

Image Gradients

SIFT Descriptor

Figure 3.1: SIFT descriptor. The well known SIFT descriptor consists of 3D histograms of image gradients over spatial coordinates and gradient orientations. Left:
The image gradients are weighted by a Gaussian window, indicated by the red circle. The length of the arrows corresponds to the sum of gradient magnitudes on agiven direction. Right: The gradients in each of the 4 × 4 spatial blocks are collected inhistograms with 8 bins each to form the ﬁnal 128D SIFT descriptor. Released by Indif
under CC-BY-SA-3.0, edited by the author.

resentations of local image patches. In particular we have developed a descriptor basedon embedding an image as a 3D mesh and then simulating the heat diﬀusion along thesurface. We show that this representation is robust to non-rigid deformations. Furthermore, by performing a logarithmic sampling of the diﬀusion of heat at diﬀerent timeintervals and then computing the Fast Fourier Transform, we can make the descriptor
also robust to illumination changes. We have created a deformation and illumination
dataset in order to evaluate this descriptor and show it outperforms all other descriptors for this task.
Our second line of work consists of instead of hand-crafting these features, attempting to learn them using Convolutional Neural Networks (CNN). As it is not possible tolearn these networks directly such as done for classiﬁcation problems, we propose learning the network using a Siamese architecture. This consists of considering two image
patches and whether or not they should correspond to the same point simultaneously.
Both image patches are propagated forward through the network giving two diﬀerent
descriptors. Then we calculate the L2 distance between both descriptors and apply aloss function that is meant to minimize the distance for two patches corresponding to
the same object and maximize the distance for two patches corresponding to diﬀerent objects. Afterwards, the error gradients are propagated backwards through the
network for each patch. We use a dataset of patches extracted from Structure from
Motion (Winder et al., 2009) for training, validation and testing. We will show that
our sampling scheme in conjunction with large amounts of mining of samples is able toobtain a very large increase of performance over SIFT.

3.2. Deformation and Illumination Invariant (DaLI) Descriptor
Reference Image

SIFT

DAISY

LIOP

GIH

23
DaLI

n=1
n=2
n=3
n=4
n=5
n=6
n=7
n=8
n=9
n=10

Figure 3.2: Comparing DaLI against SIFT (Lowe, 2004), DAISY (Tola et al.,2010), LIOP (Wang et al., 2011) and GIH (Ling and Jacobs, 2005). Input
images correspond to diﬀerent appearances of the object shown in the reference images,
under the eﬀect of non-rigid deformations and severe changes of illumination. Coloured
circles indicate if the match has been correctly found among the ﬁrst n top candidates,
where n ≤ 10 is parameterized by the legend on the right. A feature is considered
as mismatched when n > 10 and we indicate this with a cross. Note that the DaLI
descriptor yields a signiﬁcantly larger number of correct matches.

3.2

Deformation and Illumination Invariant (DaLI)
Descriptor

Building invariant feature point descriptors is a central topic in computer vision witha wide range of applications such as object recognition, image retrieval and 3D reconstruction. Over the last decade, great success has been achieved in designing descriptors invariant to certain types of geometric and photometric transformations. For instance, the SIFT descriptor (Lowe, 2004) and many of its variants (Bay et al., 2006; Ke
and Sukthankar, 2004; Mikolajczyk and Schmid, 2005; Morel and Yu, 2009; Tola et al.,2010) have been proven to be robust to aﬃne deformations of both spatial and intensity domains. In addition, aﬃne deformations can eﬀectively approximate, at least on
a local scale, other image transformations including perspective and viewpoint changes.
However, as shown in Fig. 3.2, this approximation is no longer valid for arbitrary deformations occurring when viewing an object that deforms non-rigidly.
In order to match points of interest under non-rigid image transformations, recent
approaches propose optimizing complex objective functions that enforce global consistency in the spatial layout of all matches (Cheng et al., 2008; Cho et al., 2009; Leordeanu
and Hebert, 2005; Sanchez et al., 2010; Serradell et al., 2012; Torresani et al., 2008).
Yet, none of these approaches explicitly builds a descriptor that goes beyond invariance to aﬃne transformations. An interesting exception is (Ling and Jacobs, 2005),
that proposes embedding the image in a 3D surface and using a Geodesic Intensity
Histogram (GIH) as a feature point descriptor. However, while this approach is robust
to non-rigid deformations, its performance drops under light changes. This is because
a GIH considers deformations as one-to-one image mappings where image pixels onlychange their position but not the magnitude of their intensities.
To overcome the inherent limitation of using geodesic distances, we propose a noveldescriptor based on the Heat Kernel Signature (HKS) recently introduced for nonrigid 3D shape recognition (Gębal et al., 2009; Rustamov, 2007; Sun et al., 2009), andwhich besides invariance to deformation, has been demonstrated to be robust to global

24

Chapter 3. Feature Point Descriptors
DaLI Descriptor Slices at Frequencies w = {0, 1, 2, 3, 4, 5}

z
2
x

y

2

2

Figure 3.3: Visualization of a DaLI descriptor. The central idea is to embed
image patches on 3D surfaces and describe them based on heat diﬀusion processes. The
heat diﬀusion is represented as a stack of images in the frequency domain. The images
show various slices of our descriptor for two diﬀerent patches.

isotropic (Bronstein and Kokkinos, 2010) and even aﬃne scalings (Raviv et al., 2011).
In general, the HKS is particularly interesting in our context of images embedded on 3D
surfaces, because illumination changes produce variations on the intensity dimensionthat can be seen as local anisotropic scalings, for which (Bronstein and Kokkinos, 2010)
still shows a good resilience.
Our main contribution is thus using the tools of diﬀusion geometry to build a descriptor for 2D image patches that is invariant to both non-rigid deformations and photometric changes. An example of two descriptors are shown in Fig. 3.3. To construct
our descriptor we consider an image patch P surrounding a point of interest, as a surface
in the (u, v, βI(u)) space, where (u, v) are the spatial coordinates, I(u) is the intensity
value at (u, v), and β is a parameter which is set to a large value to favor anisotropic
diﬀusion and retain the gradient magnitude information. Drawing inspiration from theHKS (Gębal et al., 2009; Sun et al., 2009), we then describe each patch in terms of theheat it dissipates onto its neighborhood over time. To increase robustness against 2D
and intensity noise, we use multiple such descriptors in the neighborhood of a point,
and weigh them by a Gaussian kernel. As shown in Fig. 3.2, the resulting descriptor(which we call DaLI, for Deformation and Light Invariant) outperforms state-of-the-art
descriptors in matching points of interest between images that have undergone non-rigid
deformations and photometric changes.
We propose alternatives to both alleviate the high cost of the heat kernel computation and to reduce the dimensionality of the descriptor. In particular we investigate
topologies with varying vertex densities. This allows reducing the eﬀective size of theunderlying mesh, and hence to speed up the DaLI computation time by a factor of over
4 with respect to a simple square mesh grid. In addition, we have also compacted thesize of the ﬁnal descriptor by a factor of 50× using a Principal Component Analysis(PCA) for dimensionality reduction. As a result, the descriptor we propose here can becomputed and matched much faster when compared to (Moreno-Noguer, 2011), while
preserving the discriminative power.
For evaluation, we acquired a challenging dataset that contains 192 pairs of real
images, manually annotated, of diverse materials under diﬀerent degrees of deformation
and being illuminated by radically diﬀerent illumination conditions. Fig. 3.2-left shows3.2. Deformation and Illumination Invariant (DaLI) Descriptor

Image
Patch

25

Principal
Component
Analysis

DaLI-PCA
Descriptor

Fast Fourier
Transform
0.96%

DaLI
Descriptor

Heat Kernel Signature

Meshing
0.03%

Laplace-Beltrami
0.70%

Eigendecomposition
92.81%

SpectralExpansion
5.50%

Figure 3.4: Flowchart of the algorithm used to calculate the DaLI and DaLIPCA descriptors. The percentages below each of the steps indicate the total amountof the contribution of that step to the computation time. Observe that 99% of thecomputation time corresponds to the Heat Kernel Signature calculation and speciﬁcallyalmost entirely to the eigendecomposition of the Laplace-Beltrami operator.
two samples of our dataset. We believe this is the ﬁrst deformation and illumination
dataset for evaluating image descriptors using real-world objects, and have made thedataset along with the code of the DaLI descriptor publicly available1 .

Related WorkThe SIFT descriptor (Lowe, 2004) has become the main reference among feature point
descriptors, showing great success in capturing local aﬃne deformations including scaling, rotation, viewpoint change and certain lighting changes. Since it is relatively slow
to compute, most of the subsequent works have focused on developing faster descriptors (Bay et al., 2006; Calonder et al., 2012; Ke and Sukthankar, 2004; Mikolajczyk andSchmid, 2005; Tola et al., 2010). Scale and rotation invariance has also been demonstrated in (Kokkinos et al., 2012) using a combination of logarithmic sampling andmulti-scale signal processing, although that requires large image patches which makethe resulting descriptor more sensitive to other deformations. Indeed, as discussedin (Vedaldi and Soatto, 2005), little eﬀort has been devoted to building descriptors
robust to more general deformations.
The limitations of the aﬃne-invariant descriptors when solving correspondences between images of objects that have undergone non-rigid deformations are compensatedby enforcing global consistency, both spatial and photometric, among all features (Belongie et al., 2002; Berg et al., 2005; Cheng et al., 2008; Cho et al., 2009; Leordeanu
and Hebert, 2005; Sanchez et al., 2010; Serradell et al., 2012; Torresani et al., 2008), or
introducing segmentation information within the descriptor itself (Trulls et al., 2013,2014). In any event, none of these methods speciﬁcally handles the non-rigid nature of
the problem, and they rely on solving complex optimization functions for establishingmatches.
An alternative approach is to directly build a deformation invariant descriptor. With
that purpose, recent approaches in two-dimensional shape analysis have proposed using diﬀerent types of intrinsic geometry. For example, (Bronstein et al., 2007; Ling andJacobs, 2007) deﬁne metrics based on the inner-distance, and (Ling et al., 2010) proposes using geodesic distances. However, all these methods require the shapes to besegmented out from the background and represented by binary images, which is diﬃcult
to do in practice. In (Ling and Jacobs, 2005), it was shown that geodesic distances, in
combination with an appropriate 3D embedding of the image, were adequate to achieve1

http://www.iri.upc.edu/people/esimo/research/dali/

26

Chapter 3. Feature Point Descriptors

deformation invariance in intensity images. Nonetheless, this method assumes that pixels only change their image locations and not their intensities and, as shown in Fig. 3.2,
is prone to failure under illumination changes.
There have also been eﬀorts to build illumination invariant descriptors. Such works
consider strategies based on intensity ordering and spatial sub-division (Fan et al.,2012; Gupta and Mittal, 2007, 2008; Gupta et al., 2010; Heikkilä et al., 2009; Tanget al., 2009; Wang et al., 2011). While these approaches are invariant to monotonicallyincreasing intensity changes, their success rapidly falls when dealing with photometricartifacts produced by complex surface reﬂectance or strong shadows.
The DaLI descriptor we propose can simultaneously handle such relatively complex
photometric and spatial warps. Following (Ling and Jacobs, 2005), we represent theimages as 2D surfaces embedded in the 3D space. This is in fact a common practice, although it has been mostly employed for low level vision tasks such as image
denoising (Sochen et al., 1998; Yezzi, 1998) or segmentation (Yanowitz and Bruckstein.,1989). The fundamental diﬀerence between our approach and (Ling and Jacobs, 2005)
is that we then describe each feature point on the embedded surface considering theheat diﬀusion over time (Gębal et al., 2009; Lévy, 2006; Sun et al., 2009) instead of using a Geodesic Intensity Histogram. As we will show in the results section this yields
substantially improved robustness, especially to illumination changes. Heat diﬀusion
theory has been used by several approaches for the analysis of 3D textured (Kovnatsky
et al., 2011) and non-textured shapes (de Goes et al., 2008; Lévy, 2006; Reuter et al.,2006; Rustamov, 2007), but to the best of our knowledge, it has not been used before
to describe patches in intensity images.
One of the main limitations of the methods based on the heat diﬀusion theory is
the high complexity cost they require. The bottleneck of their computation lies on an
eigendecomposition of a nv × nv Laplacian matrix (see Fig. 3.5), where nv is the number of vertices of the underlying mesh. This has been addressed by propagating the
eigenvectors across diﬀerent mesh resolutions (Shi et al., 2006; Wesseling, 2004) or using matrix exponential approximations (Vaxman et al., 2010). In this work, an annularmulti-resolution grid will be used to improve the eﬃciency of the DaLI computation.
Additionally, PCA will be used to reduce the dimensionality of the original DaLI descriptor (Moreno-Noguer, 2011), hence speeding up the matching process as well.

Deformation and Light Invariant Descriptor
Our approach is inspired by current methods (Gębal et al., 2009; Sun et al., 2009) thatsuggest using diﬀusion geometry for 3D shape recognition. In this section we show how
this theory can be adapted to describe 2D local patches of images that undergo nonrigid deformations and photometric changes. A general overview of the diﬀerent steps
needed to compute the DaLI and DaLI-PCA descriptors can be seen in Fig. 3.4 and are
explained more in detail below.
Invariance to Non-Rigid Deformations
Let us assume we want to describe a 2D image patch P , of size SP × SP and centeredon a point of interest p. In order to apply the diﬀusion geometry theory to intensity
patches we regard them as 2D surfaces embedded in 3D space (Fig. 3.5 bottom-left).
More formally, let f : P → M be the mapping of the patch P to a 3D Riemannian3.2. Deformation and Illumination Invariant (DaLI) Descriptor

27

DaLI Descriptor Slices at Frequencies w = {0, 2, 4, 6}

x1

z2
x2

y

1

y2

Image Embedding

Descriptor for Single Pixels
x1

SI−HKS

6

y1
4

x2
y2

2
0
0

z2
5

10

Freq(w)

15

20

Figure 3.5: DaLI descriptor. Our central idea is to embed image patches in 3D
surfaces and describe them based on heat diﬀusion processes. We represent the heatdiﬀusion as a stack of images in the frequency domain. The top images show various
slices of our descriptor for two diﬀerent patches. The bottom-right graph depicts thevalue of the descriptor for the pixels marked by colour circles in the upper images. Notethat corresponding pixels have very similar signatures. However, the signature may
signiﬁcantly change from one pixel to its immediate neighbor. For instance, z2 is at
one pixel distance from x2 , but their signatures are rather diﬀerent. As a consequence,using the signature of a single point as a descriptor is very sensitive to 2D noise in thefeature detection process. We address this by simultaneously considering the signatureof all the pixels within the patch, weighted by a Gaussian function of the distance tothe center of the patch.
manifold M. We explicitly deﬁne this mapping by:
f : u → (u, v, βI(u)) ∀u ∈ P ,(3.1)

where I(u) is the pixel intensity at u = (u, v)⊤ , and β is a parameter that, as we will
discuss later, controls the amount of gradient magnitude preserved in the descriptor.
Several recent methods (Gębal et al., 2009; Lévy, 2006; Reuter et al., 2006; Rustamov, 2007; Sun et al., 2009) have used the heat diﬀusion geometry for capturing the
local properties of 3D surfaces and performing shape recognition. Similarly, we describe
each patch P based on the heat diﬀusion equation over the manifold M:


∂
△M +
h(u, t) = 0 ,∂twhere △M is the Laplace-Beltrami operator, a generalization of the Laplacian to nonEuclidean spaces, and h(u, t) is the amount of heat on the surface point u at time t.

28

Chapter 3. Feature Point Descriptors

The solution k(u, v, t) of the heat equation with an initial heat distribution ho (u, t) =δ(u − v) is called the heat kernel, and represents the amount of heat that is diﬀused
between points u and v at time t, considering a unit heat source at u at time t = 0.
For a compact manifold M, the heat kernel can be expressed by following spectral expansion (Chavel, 1984; Reuter et al., 2006):
k(u, v, t) =∞X

e−λi t φi (u)φi (v) ,

(3.2)

i=0

where {λi } and {φi } are the eigenvalues and eigenfunctions of △M , and φi (u) is thevalue of the eigenfunction φi at the point u. Based on this expansion, (Sun et al., 2009)
proposes describing a point p on M using the Heat Kernel Signature
HKS(p, t) = k(p, p, t) =∞X

e−λi t φ2i (p) ,

(3.3)

i=0

which is shown to be isometrically-invariant, and adequate for capturing both the local
properties of the shape around p (when t → 0) and the global structure of M (whent → ∞).
However, while on smooth surfaces the HKS of neighboring points are expected tobe very similar, when dealing with the wrinkled shapes that may result from embedding
image patches, the heat kernel turns to be highly unstable along the spatial domain(Fig. 3.5 bottom-right). This makes the HKS particularly sensitive to noise in the 2D
location of the keypoints. To handle this situation, we build the descriptor of a point
p by concatenating the HKS of all points u within the patch P , properly weighted by
a Gaussian function of the distance to the center of the patch. We therefore deﬁne thefollowing Deformation Invariant (DI) descriptor:
DI(p, t) = [HKS(u, t) · G(u; p, σ)]∀u∈P ,(3.4)

where G(u; p, σ) is a 2D Gaussian function centered on p having a standard deviation
σ, evaluated at u. Note that for a speciﬁc time instance t, DI(p, t) is a SP × SP array.
The price we pay for achieving robustness to 2D noise is an increase of the descriptor
size. That is, if HKS(p, t) is a function deﬁned on the temporal domain R+ discretized
into nt equidistant intervals, the complete DI descriptor DI(p) = [DI(p, t1 ), . . . , DI(p, tnt )]will be deﬁned on SP × SP × nt , the product of the spatial and temporal domains.
However, note that for our purposes this is still feasible, because we do not need tocompute a descriptor for every pixel of the image, but just for a few hundreds of points
of interest. Furthermore, as we will next discuss, the descriptor may be highly compacted if we represent it in frequency domain instead of time domain, and even further
compacted by using dimensionality reduction techniques such as Principal Component
Analysis (PCA).
Invariance to Illumination Changes
An inherent limitation of the descriptor introduced in Eq. (3.4) is that it is not illumination invariant. This is because light changes scale the manifold M along the intensity
axis, and the HKS is sensitive to scaling. It can be shown that an isotropic scaling of
the manifold M by a factor α, scales the eigenvectors and eigenvalues of Eq. (3.2) by3.2. Deformation and Illumination Invariant (DaLI) Descriptor

29

100
90

Descriptor Change(%)

Deformation Level

Illumination Changes

Def11

DaLI
DI

Def10
Def9

80

Def8

70

Def7

60

Def6
50
Def5
40

Def4

30

Def3

20

Def2

10

Def1

0
1.0

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.01

Def0

Intensity Gain

Figure 3.6: Invariance of the DaLI and DI descriptors to non-rigid deformations and illumination changes. Top row and left column images: Diﬀerent
degrees of deformation and light changes applied on the top left reference patch P0 .
Deformations are applied according to a function Def(·) ∈ {Def0, . . . , Def11}, where
Def11 corresponds to the maximal deformation. Light changes are produced by scaling
the intensity of P0 by a gain g ∈ [0, 1]. Bottom Graph: Given a deformation Def(·)
and a gain factor g, we compute the percentage of change of the DI descriptor bykDI(P0 ) − DI(Def(gP0 ))k/kDI(P0 )k. The percentage of change for DaLI is computed ina similar way. Observe that DaLI is much less sensitive than DI, particularly to illumination changes.
factors 1/α and 1/α2 , respectively (Reuter et al., 2006). The HKS of a point αp ∈ αM
can then be written as∞X
λi φ2 (p)1tHKS(αp, t)=e− α2 t i 2 = 2 HKS(p, 2 ) ,
α
α
α

(3.5)

i=0

which is an amplitude and time scaled version of the original HKS.
Nonetheless, under isotropic scalings, several alternatives have been proposed to remove the dependence of the HKS on the scale parameter α. For instance, (Reuter et al.,2006) suggests normalizing the eigenvalues in Eq. (3.2). In this work we followed (Bronstein and Kokkinos, 2010), that applies three consecutive transformations on the HKS.
First, the time-dimension is logarithmically sampled, which turns the time scaling into
a time-shift, that is, the right-hand side of Eq. (3.5) begets α−2 HKS(p, −2 log α + log t).
Second, the amplitude scaling factor is removed by taking logarithm and derivative∂
w.r.t. log t. The Heat Kernel then becomes ∂ log
t log HKS(p, −2 log α + log t). The
time-shift term −2 log α is ﬁnally removed using the magnitude of the Fourier transform, which yields SI-HKS(p, w), a scale invariant version of the original HKS in thefrequency domain. In addition, since most of the signal information is concentrated in
the low-frequency components, the size of the descriptor can be highly reduced com-

30

Chapter 3. Feature Point Descriptors

Def.+Ill.

Def.

Ill. (Def.)

Ill. (No Def.)

Normalized DaLI (SI−HKS)

6
Distance

Distance

8
6

60

150

Def.+Ill.
Def.
Ill. (Def.)
Ill. (No Def.)

Def.+Ill.
Def.
Ill. (Def.)
Ill. (No Def.)4

4

Def.+Ill.
Def.
Ill. (Def.)
Ill. (No Def.)

50
40

100
Distance

10

Normalized DaLI (HKS)

DaLI (HKS)

8

Def.+Ill.
Def.
Ill. (Def.)
Ill. (No Def.)

Distance

DaLI (SI−HKS)
12

30
20

50

2
2
0

10

0

50

100

150
Frame

200

250

300

0

0

50

100

150
Frame

200

250

300

0

0

100

200

300

0

Frame

0

50

100

150
Frame

200

250

300

Figure 3.7: Evaluation of the descriptor robustness on synthetic sequences.
In the top-left we show two sample images (the reference image and one speciﬁc frame)from all four diﬀerent scenarios we consider. In the top-right we show an 3D view of therendering process, with the light position placed near the mesh and producing patternsof diﬀerent brightness on top of the surface. The bottom row depicts the descriptor
distance between every input frame and the reference image for diﬀerent descriptor
variants.
pared to that of HKS(p, t) by eliminating the high-frequency components past a certain
frequency threshold wmax .
As we will show in the results section, another advantage of the SI-HKS signature is that although it is speciﬁcally designed to remove the dependence of the HKS
on isotropic scalings, it is quite resilient to anisotropic transformations, such as those
produced by photometric changes that only aﬀect the intensity dimension of the manifold M. Thus, we will use this signature to deﬁne our Deformation and Light Invariant(DaLI) descriptor:
DaLI(p, w) = [SI-HKS(u, w) · G(u; p, σ)]∀u∈P .
Again, the full DaLI(p) descriptor is deﬁned as a concatenation of wmax slices in thefrequency domain, each of size SP × SP .
Fig. 3.5-top shows several DaLI slices at diﬀerent frequencies for a patch and a
deformed version of it. As said above, observe that most of the signal is concentrated
in the low frequency components. In Fig. 3.6 we compare the sensitivity of the DI and
DaLI descriptors to deformation and light changes, simulated here by a uniform scalingof the intensity channel. Note that DaLI, in contrast to DI, remains almost invariant tolight changes, and it also shows a better performance under deformations. In the results
section, we will show that this invariance is also accompanied by a high discriminability,
yielding signiﬁcantly better results in keypoint matching than existing approaches.

3.2. Deformation and Illumination Invariant (DaLI) Descriptor

31

In order to get deeper insight about the properties of the DaLI descriptor, we havefurther evaluated the HKS and SI-HKS descriptor variants on a synthetic experiment,
in which we have rendered various sequences of images of a textured 3D wave-likemesh under diﬀerent degrees of deformation and varying illumination conditions. The
surface’s reﬂectance is assumed to be Lambertian and the light source is moved near
the surface, producing lighting patterns that combine both shading and the eﬀects of
the inverse-square falloﬀ law.
We have analyzed four particular situations: Def.+Ill., varying both deformationand the light source position; Def., varying deformation and keeping the light source at
inﬁnity; Ill. (Def.), starting with a largely deformed state which is kept constant along
the sequence and varying the light source position; and Ill. (No Def.), varying the lightsource position while keeping the surface ﬂat. The mesh deformation in the ﬁrst two
sequences, corresponds to a sinusoidal warp, in which the amplitude of the deformation
increases with the frame number. The varying lighting conditions in all experiments
except the second, are produced by smoothly moving the light source on a hemispherevery close to the surface. Two frames from each of these sequences are shown in the
top-left of Fig. 3.7.
For the evaluation, we computed the L2 norm between pairs of descriptors at thecenter of the ﬁrst and n-th frames of the sequence. The results are depicted in Fig. 3.7bottom. When computing the distances, we consider two situations: normalizing theintensity of the input images so that the pixels follow a distribution N (0, 1), and directly using the input image intensities. The most interesting outcome of this experiment is how the non-normalized SI-HKS descriptor has comparable distances for all
the scenarios. On the other hand, the normalized versions (SI-HKS and HKS) seemto distinguish largely between whether there is or is not deformation. It is also worth
noting that this normalization creates some instability at the earlier frames while thenon-normalized SI-HKS descriptor starts at nearly 0 error and increases smoothly for
all scenarios. Note also the low performance of the non-normalized HKS descriptor under illumination changes as seen by the exponential curves for the illumination scenarios
Ill. (Def ) and Ill. (No Def ), and the large ﬂuctuations for both the deformation Def andthe illumination changing scenario Def. + Ill . This indicates the importance of the logarithmic sampling and Fourier transform process we apply to make HKS illumination
invariant.
Handling In-Plane Rotation
Although DaLI tolerates certain amounts of in-plane rotation, it is not designed for thispurpose. This is because with the aim of increasing robustness to 2D noise, we built thedescriptor using all the pixels within the patch, and their spatial relations have beenretained. Thus, if the patch is rotated, the descriptor will also be rotated.
In order to handle this situation, during the matching process we will consider
several rotated copies of the descriptors. Therefore, given DaLI(p1 ) and DaLI(p2 ) we
will compare them based on the following metricd(p1 , p2 ) = arg min kRθi (DaLI(p1 )) − DaLI(p2 )kθiwhere k · k denotes the L2 norm and Rθi (DaLI(p)) rotates DaLI(p) by an angle θi . Thisparameter is chosen among a discrete set of values θ.

32

Chapter 3. Feature Point Descriptors

(a)

(b)

(c)

Figure 3.8: Representation of an image patch. (a) Image patch. (b) Representation of the patch as a triangular mesh. For clarity of presentation we only depict the(u, v) dimension of the mesh. Note that besides the vertices placed on the center of the
pixels (ﬁlled circles) we have introduced additional intra-pixel vertices (empty circles),that provide ﬁner heat diﬀusion results and higher tolerance to in-plane rotations. (c)Deﬁnition of the angles used to compute the discrete Laplace-Beltrami operator.

(a) Dense Square(b) Dense Circular(c) Annular

Figure 3.9: Different mesh triangulations. Upper half of three diﬀerent triangulations of a 11 × 11 image patch. The shading on the left half of the mesh indicatesthe density of the meshing. Dark red shading indicates high density and lighter red
shading corresponds to low density. (a) Dense Square Mesh, with the same topology
as in Fig. 3.8. By using circular meshes (b, c), we reduce the number of vertices andthus, the computation time of the heat kernel. In the case of the annular mesh (c), a
further reduction of the number of nodes is achieved by having a variable resolutionof the mesh that is more dense at the center. The edges of the annular mesh preserve
symmetry around the central point in order to favor uniform heat diﬀusion.
This rotation handling will not be necessary when using Principal Component Analysis to compress the descriptor size as we describe in Section 3.2.
Implementation Details
We next describe a number of important details to be considered for the implementationof the DaLI descriptor.
Geometry of the embedding. For the numerical computation of the heat diﬀusion, it is necessary to discretize the surface. We therefore represent the manifold M
on which the image patch is embedded using a triangulated mesh. Fig. 3.8(b) shows
the underlying structured 8-neighbour representation we use. Although it requires introducing additional virtual vertices between the pixels, its symmetry with respect to3.2. Deformation and Illumination Invariant (DaLI) Descriptor

33

Mesh Type# Pixels# Vertices (nv )# Faces (nf )Time (s)Dense Square

1681

3281

6400

1.988

Dense Circular

1345

2653

5144

1.509

Annular

1345

1661

3204

0.460

Table 3.1: DaLI computation time and mesh complexity for different triangulations. We consider a circular patch with outer radius S = 20, and inner radius
So = 10 (for the Annular mesh).
both axes provides robustness to small amounts of rotation, and more uniform diﬀusions than other conﬁgurations.
As seen in Fig. 3.4, nearly all the computation time of the DaLI descriptor is spentcalculating the Laplace-Beltrami eigenfunctions of the triangulated mesh. In the following subsection we will show that this computation turns to have a cubic cost on thenumber of vertices of the mesh, hence, important speed gains can be achieved by lowering this number. For this purpose we further considered a circular mesh (Fig. 3.9(b)),
and a mesh with a variable density, like the one depicted in Fig. 3.9(c), where a lower
resolution annulus is used for the pixels further away from the center.
By using an annular mesh with an inner radius So = S/2, where S is the size ofthe outer radius, we were able to speed up the computation of the DaLI descriptor bya factor of four compared to the Dense Squared conﬁguration (see Table 3.1). Most
importantly, this increase in speed did not result in poorer recognition rates.
Another important variable of our design is the magnitude of the parameter β in
Eq. (3.1), that controls the importance of the intensity coordinate with respect to the(u, v) coordinates. In particular, as shown in Fig. 3.10, large values of β allow our descriptor to preserve edge information. This is a remarkable feature of the DaLI descriptor, because besides being deformation and illumination invariant, edge information isuseful to discriminate among diﬀerent patches.
Discretization of the Laplace-Beltrami operator In order to approximate theLaplace-Beltrami eigenfunctions on the triangular mesh we use the cotangent scheme
described in (Pinkall and Polthier, 1993). We next detail the main steps.
Let {p1 , . . . , pnv } be the vertices of a triangular mesh, associated to an image patch
embedded on a 3D manifold. We approximate the discrete Laplacian by a nv × nv
matrix L = A−1 M where A is a diagonal matrix in which Aii is proportional to thearea of all triangles sharing the vertex pi . M is a nv × nv sparse matrix computed by: Pk mik if i = j
Mij =
−mij
if pi and pj are adjacent0otherwise+−+−
where mij = cot γij+ cot γij
, and γij
and γij
are the two opposite angles depicted inFig. 3.8(c), and the subscript ‘k’ refers to all neighboring vertices of pi .
The eigenvectors and eigenvalues of the discrete La-place-Beltrami operator can thenbe computed from the solution of the generalized eigenproblem MΦ = ΛAΦ, where Λ34Chapter 3. Feature Point Descriptors
w=2

w=4

w=6

w=8

β = 1000

β = 100

β = 10

w=0

Figure 3.10: Preserving edge information. Larger values of the parameter β in
Eq. (3.1) allow the descriptor to retain edge information. Each row depicts the DaLI
descriptor at frequencies w = {0, 2, 4, 6, 8} for a diﬀerent value of β computed on the
car image from Fig. 3.5. Observe that for low values of β there is blurring on the higher
frequencies of the descriptor.

is a diagonal matrix with the eigenvalues {λi } and the columns of Φ correspond to theeigenvectors {φi } in Eq. (3.2).
Note that the computational cost of the eigendecomposition is cubic in the sizeof M, i.e., O(n3v ). As discussed in the previous subsection, we mitigate this cost bychoosing mesh topologies where the number of vertices is reduced. In addition, since
the eigenvectors φi with smallest eigenvalues have the most importance when calculating
the HKS from Eq. (3.3), we can approximate the actual value by only using a subset
formed by the nλ eigenvectors with smallest eigenvalues. Both these strategies allow
the HKS calculation to be tractable in terms of memory and computation time.
Finally, Table 3.2 summarizes all the parameters that control the shape and size of
the DaLI descriptor. The way we set their default values, shown between the parentheses, will be discussed in Section 3.2.

Deformation and Varying Illumination Dataset
In order to properly evaluate the deformation and illumination invariant properties of
the DaLI descriptor and compare it against other state-of-the-art descriptors, we havecollected and manually annotated a new dataset of deformable objects under varying
illumination conditions. The dataset consists of twelve objects of diﬀerent materials
with four deformation levels and four illumination conditions each, for a total of 192
unique images. All images have a resolution of 640 × 480 pixels and are grayscale.
The types of objects in the dataset are 4 shirts, 4 newspapers, 2 bags, 1 pillowcase
and 1 backpack. They were chosen in order to evaluate all methods against as many
diﬀerent types of deformation as possible. The objects can be seen in the top of Fig. 3.11.

3.2. Deformation and Illumination Invariant (DaLI) Descriptor

Symbol

35

Parameter Description. (Default Value)S

Outer radius of the annulus. (20)So

Inner radius of the annulus. (10)βMagnitude of the embedding. (500)σStandard deviation of Gaussian weighting. ( S2 )nλ# of eigenvectors of the Laplace-Beltrami operator. (100)nt# of intervals in the temporal domain. (100)wmax# of frequency components used. (10)θiRotation angles for descriptor comparison. ({−5, 0, +5})nv

# of mesh vertices.(1661)

nf# of triangular faces in the mesh. (3204)

npca# of PCA components for the DaLI-PCA. (256)

Table 3.2: DaLI Parameters. We show the parameters of the DaLI descriptor withthe default values in parenthesis.
Deformation and Illumination Conditions
The pipeline to acquire the images of each object consisted of, while keeping the deformation constant, changing the illumination before proceeding to the next deformationlevel. All images were taken in laboratory conditions in order to fully control the settings for a suitable evaluation.
The reference image was acquired from an initial conﬁguration where the object wasstraightened out as much as possible. While deformations are fairly subjective, as they
were done incrementally over the previous deformation level, they are representativeof increasing levels of deformation. Diﬀerent deformation levels of an object with thesame illumination conditions are shown in the middle-left of Fig. 3.11.
The illumination changes were produced by using two high power focus lamps. The
ﬁrst one was placed vertically over the object, at a suﬃcient distance to guarantee a
uniform global illumination of the object’s surface. The second lamp was placed at asmall elevation angle and close to the object, in order to produce harsh shadows andlocal illumination artifacts. By alternating the states of these lamps, four diﬀerent
illumination levels are achieved: no illumination, global illumination, global with local
illumination, and local illumination. The diﬀerent illumination conditions for constantdeformation levels can be seen in the bottom-left of Fig. 3.11. Note that even withmoderate deformations, the presence of the local illumination causes severe appearancechanges.
Manual Annotations
To build the ground truth annotations, we initially detected interest points in all images using a multi-scale Diﬀerence of Gaussians ﬁlter (Lowe, 2004). This yielded ap-

36

Chapter 3. Feature Point Descriptors

Shirt #1

Shirt #2

Newspaper #1

Newspaper #2

Reference Images
Shirt #3
Shirt #4

Newspaper #3

Newspaper #4

Deformation Level
Deform. Level #0 Deform. Level #1 Deform. Level #2 Deform. Level #3

Bag #1

Bag #2

Backpack

Pillowcase

Manual Annotationof Correspondences

Illumination Changes
Ill. Conditions #0 Ill. Conditions #1 Ill. Conditions #2 Ill. Conditions #3
Light
Sources
CameraObjectFigure 3.11: Deformable and varying illumination dataset. Top: Reference
images of the twelve objects in the dataset. Each object has four deformation levels
and four illumination levels yielding a total of 16 unique images per object. Middle-left:
Sample series of images with increasing deformation levels, and constant illumination.
Bottom-left: Sample images of the diﬀerent illumination conditions taken for a deformation level of each object. The illumination conditions #0, #1, #2 and #3 correspond tono illumination, global illumination, global+local illumination, and local illumination,
respectively. Middle-right and bottom-right: Examples of feature points matched acrossimage pairs. The ﬁrst column corresponds to the reference image for the object. Thesefeature points are detected using Diﬀerences of Gaussians (DoG) and are matched bymanual annotation. Each feature point consists of image coordinates, scale coordinatesand orientation.

3.2. Deformation and Illumination Invariant (DaLI) Descriptor
Original Patch

RotatedScaled37
CroppedFigure 3.12: Extracting patches from points of interest. Top: Outline of theprocess used to obtain patches for evaluating image descriptors. For each feature point,
we initially extract a square patch centered on the feature, and whose size is proportional
to the scale factor of the interest point. The patch is then rotated according to theorientation of the feature point, and ﬁnally scaled to a constant size and cropped to
be in a circular shape. Bottom: Sample patches from the dataset, already rotated andscaled to a constant size in order to make them rotation and scale invariant.

proximately between 500 − 600 feature points per image, each consisting of a 2D image
coordinate and its associated scale.
These feature points were then manually matched for each deformation level against
the undeformed reference image, resulting in three pairs of matched feature points. All
matches were done with top-light illumination conditions (Ill. Conditions #1, Fig. 3.11)
to facilitate the annotation task and maximize the number of repeated features between
each pair of images. The matching process yielded between 100 and 200 point correspondences for each pair of reference and deformed images. The same feature points areused for all illumination conditions for each deformation level. The middle-right images of Fig. 3.11 show a few samples of our annotation. Note that the matched points
are generally not near the borders of the image to avoid having to clip when extracting
image patches.
As we will discuss in the experimental section, in this work we seek to compare therobustness of the DaLI and other descriptors to only deformation and light changes.
Yet, although the objects in the dataset are not globally rotated, the deformations do
produce local rotations. In order to compensate for this we use the SIFT descriptor
as done in (Mikolajczyk and Schmid, 2005) to compute the orientation of each featurepoint, and align all corresponding features. When a feature point has more than one
dominant orientation, we consider each of them to augment the set of correspondences.

38

Chapter 3. Feature Point Descriptors
D efor m at ion O nly[0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20]

β=500 β=1000 β=2000

So

So

88%

σ=S
σ=S/2

β=250

β=500 β=1000 β=2000

σ=2S

wmax

58%

[5,10,15,20]

β=125

wmax

β=250

So

Illumination [0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20]
Only[5,10,15,20]

β=125

So

So

wmax

σ=S
σ=S/2

I llum inat ion O nly
67%

So

σ=2S

wmax

[5,10,15,20]

wmax[5,10,15,20]

wmax[5,10,15,20]

Deformation
Only

So

So

[5,10,15,20]

So

80%

D efor m at ion + I llum inat ion C hanges
S

S

S

S

σ=S
σ=S/2

71%

σ=2S

wmax

[5,10,15,20]

wmax[5,10,15,20]

wmax[5,10,15,20]ooooDeformation[0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20] [0,5,10,15,20]
Illumination
Combined β=125 β=250 β=500 β=1000 β=2000

64%

Figure 3.13: DaLI performance for different values of the parameters
So , β, σ and wmax . We compute and average the matching rate for the Shirt#1 and the Newspaper #1 objects in the dataset using So ∈ {0, 5, 10, 15, 20} pixels,β ∈ {125, 250, 500, 1000, 2000}, σ ∈ { S2 , S, 2S} and wmax ∈ {5, 10, 15, 20} for three scenarios: both deformation and illumination changes, only deformation changes, and only
illumination changes. The graphs depict the results of this 4D parameter exploration,
where the colour of each square represents the percentage of correctly matched points
for a speciﬁc combination of the parameters. In order to visualize the diﬀerences, we
scale the values separately for each scenario. The best parameters for each scenario aremarked in red and can be seen to vary greatly amongst themselves. We use a compromise, and for all the experiments in this section we set these parameters (highlighted
in green) to β = 500, So = 10, σ = S2 and wmax = 10.
Evaluation Criteria
In order to perform fair comparisons, we have developed a framework to evaluate local
image descriptors on even grounds. This is done by converting each feature point into asmall image patch which is then used to compute descriptors. This allows the evaluationof the exact same set of patches for diﬀerent descriptors.
For each feature point we initially extract a square patch around it, with a size
proportional to the feature point’s scale. In the results section we will discuss the valueof the proportionality constant we use. The patch is then rotated by the feature point’sorientation using bilinear interpolation, and scaled to a constant size, which we have
set to 41 × 41 pixels as done in (Mikolajczyk and Schmid, 2005). Finally, the patch is
cropped to a circular shape. This results in a scale and rotation invariant circular image
patch with a diameter of 41 pixels. The steps for extracting the patches are outlinedin the top of Fig. 3.12, and the bottom of the ﬁgure shows a few examples of patches3.2. Deformation and Illumination Invariant (DaLI) Descriptor

39

from the dataset.
Given these “normalized patches” we then assess the performance of the descriptors
as follows. For each pair of reference/deformed images, we extract the descriptors of allfeature points in both images. We then compute the L2 distance between all descriptors
from the reference and the deformed image. This gives a distance matrix, which isrectangular instead of square due to the creation of additional feature points when
there are multiple dominant orientations. Patches that have diﬀerent orientations but
share the same location are treated as a unique patch. As evaluation metric we use adescriptor-independent detection rate, which is deﬁned for the n top matches as:
Detection Rate(n) =100 · Nc (n)
,
N

(3.6)

where Nc (n) is the number of feature points from the reference image that have thecorrect match among the top n candidates in the deformed image, and N is the total
number of feature points in the reference image.
For the experimental results we will discuss in the following section, we consider
three diﬀerent evaluation scenarios: deformation and illumination, only deformation,and only illumination. In the ﬁrst case we compare all combinations of deformation andillumination with respect to the reference image which has no additional illumination(ill. conditions #0) and no deformation (deform. level #0). This represents a total of
15 comparisons for each object. In the second case we consider only varying levels ofdeformation for each illumination condition, which yields 12 diﬀerent comparisons per
object (three comparisons per illumination level). When only considering illumination,
each deformation level is compared to all illumination conditions. Again, this gives rise
to 12 comparisons per object (three comparisons per deformation level).

Results
We next present the experimental results, in which we discuss the following main issues:
an optimization of the descriptor parameters, a PCA-based strategy for compressing
the descriptor representation, and the actual comparison of DaLI against other stateof-the-art descriptors, for matching points of interest in the proposed dataset. Finally,
we analyze speciﬁc aspects such as the performance of all descriptors in terms of theirsize, the beneﬁts of normalizing the intensity of input images, and a real application inwhich the descriptors are compared when matching points of interest in real sequencesof a deforming cloth and a bending paper.
Choosing Descriptor’s Parameters
We next study the inﬂuence and set the values of the DaLI parameters of Table 3.2. As
the size SP of the patch is ﬁxed, causing the descriptor radius S to be also ﬁxed, we
will look at ﬁnding the appropriate value of other parameters, namely the magnitude
β of the embedding, the degree σ of smoothing within the patch, the inner radius of
the annulus So and the dimensionality wmax of the descriptor in the frequency domain.
In order to ﬁnd their optimal values, we used two objects in the dataset (Shirt #1 andNewspaper #1), and computed matching rates of their feature points for a wide rangeof values for each of these parameters.
It is worth to point out that the number of eigenvectors nλ of the Laplace-Beltrami
operator was set to 100 in all cases. Note that this value represents a very small portion

40
PCA

Chapter 3. Feature Point Descriptors
1

2

3

4

5

6

7

8

9

10

11

12

13

14

15

16

17

18

19

20

w=0
w=1
w=2
w=3
w=4
w=5
w=6

Figure 3.14: The first 7 frequencies of the first 20 components of the PCA
basis. These are computed from images of two objects from the dataset (Shirt #1
and Newspaper #1). Each vector is normalized for visualization purposes. Positive
values are displayed in green while negative values are displayed in blue. Most of thecomponents do not contain much information at frequencies w > 6 and thus they are
not displayed, although they are considered in the DaLI-PCA descriptor.
of all potential eigenvectors, in the order of two thousands (equal to the number ofvertices nv ). Using a lesser number of them would eventually deteriorate the results,
while not providing a signiﬁcant gain in eﬃciency, and using more of them, almost
did not improve the performance. Similarly, the number nt of intervals in which thetemporal domain is split is set to 100. Again, this parameter had almost no inﬂuence,neither in the performance of the descriptor nor in its computation time.
Figure 3.13 depicts the results of the parameter sweeping experiment. We display therates for three scenarios: when considering both deformation and illumination changes,
only deformation changes, and only illumination changes. The most inﬂuential parameters are the weighting factor σ and to a lesser extent the magnitude of the embedding
β. We see that for a wide range of parameters, the results obtained are very similar
when considering both illumination and deformation, however, there is a balance tobe struck between both deformation and illumination invariance. By increasing deformation invariance, illumination invariance is reduced and vice-versa. Finally we use acompromise, and the parameters we choose for all the rest of experiments are β = 500,
So = 10, σ = S2 and wmax = 10, besides the nλ = 100 and nt = 100 we mentioned
earlier.
Compression with PCA
The DaLI descriptor has the downside of having a very high dimensionality, as its size is
proportional to the product of the number of vertices nv used to represent the patch andthe number of frequency components wmax . For instance, using patches with a diameterof 41 pixels and considering the ﬁrst 10 frequency slices, results in a 13,450-dimensionaldescriptor (1345 pixels by 10 frequency slices), requiring thus large amounts of memory
and yielding slow comparisons. However, since the descriptor is largely redundant, it
can be compacted using dimensionality reduction techniques such as (C. Strecha andFua, 2012; Cai et al., 2011; Philbin et al., 2010).
As a simple proof of concept, we have used Principal Component Analysis for3.2. Deformation and Illumination Invariant (DaLI) Descriptor

41

Def.+Ill.

0.85

Precision

Illumination
0.8

Deformation
Mean

0.75

0.7

0.65
100

200

300

400

500

600

700

800

900

1000

Dimension

w=1

w=2 w=3

w=4

w=5

w=6

Diff.

PCA

DaLI

w=0

Figure 3.15: PCA approximation of the DaLI descriptor. Top: DaLI-PCA
performance for diﬀerent compression levels. Note that the overall mean precision does
not vary much for npca > 256 components. Bottom: Comparison of an original DaLI
descriptor with its compressed DaLI-PCA version obtained using 256 PCA components.
For visualization purposes the values are normalized and the diﬀerence shown in thethird row is scaled by 5×.

performing such compression. The PCA covariance matrix is estimated on 10,436
DaLI descriptors extracted from images of the Shirt #1 and Newspaper #1. The
npca ≪ nv · wmax largest eigenvectors are then used for compressing an incoming fullsize DaLI descriptor. The resulting compacted descriptor, which we call DaLI-PCA, canbe eﬃciently compared with other descriptors using the Euclidean distance. Fig. 3.14
shows the ﬁrst 7 frequencies of the ﬁrst 20 vectors of the PCA-basis. It is interesting
to note that most of the information can be seen to be in the lower frequencies. Thiscan be considered an experimental justiﬁcation for the frequency cut oﬀ applied withthe wmax parameter, which we have previously set to 10.
In order to choose the appropriate dimension npca of the PCA-basis, we have used
our dataset to evaluate the matching rate of DaLI-PCA descriptors for diﬀerent compression levels. The results are summarized in Fig. 3.15-top, and show that using
fewer dimensions favors deformation invariance (actually, PCA can be understood asa smoothing that undoes some of the harm of deformations) while using more dimensions favors illumination invariance. The response to joint deformation and illumination
changes does not improve after using between 200 − 300 components, and this has been
the criterion we used to set npca = 256 for the rest of the experiments in this section.
In Fig. 3.15-bottom we compare the frequency slices for an arbitrary DaLI descriptor
and its approximation with 256 PCA-modes. Observe that the diﬀerences are almost
negligible.

42

Chapter 3. Feature Point Descriptors

Mean Detect. Rate (%)

80
DaLI
DAISY
SIFT
LIOP
Pixel Diff.
NCC
GIH

70
60
50
40
30
20

3x

5x

7x

9x

Support Region Scale Factor

Figure 3.16: Mean detection rates obtained by scaling regions of interest
with different factors. While a 3× scale factor does lower the overall performance,
the diﬀerence between a 5×, 7× or 9× scale factor is minimum for descriptors other thanweighted pixel diﬀerences (Pix. Diﬀ.) or normalized cross covariance (NCC), which
do improve as interest regions increase in size. The results of the graph correspondto the average of the mean detection rates with Deformation+Illumination changes,
Illumination-only changes and Deformation-only changes.

Comparison with Other Approaches
We compare the performance of our descriptors (both DaLI and DaLI-PCA) to that of
SIFT (Lowe, 2004), DAISY (Tola et al., 2010), LIOP (Wang et al., 2011), GIH (Lingand Jacobs, 2005), Normalized Cross Correlation (NCC) and Gaussian-weighted PixelDiﬀerence. SIFT and DAISY are both descriptors based on Diﬀerences of Gaussians(DoG) and spatial binning which have been shown to be robust to aﬃne deformations
and to certain amount of illumination changes. LIOP is a recently proposed descriptor
based on intensity ordering making it fully invariant to monotonic illumination changes.
GIH is a descriptor speciﬁcally designed to handle non-rigid image deformations, but
as pointed out previously, it assumes these deformations are the result of changing theposition of the pixels within the image and not their intensity. NCC is a standard regionbased metric known to possess illumination-invariant properties. Finally, we compare
against a Gaussian-weighted pixel diﬀerence using the same convolution scheme as used
for the DaLI descriptor. Standard parameters suggested in the original papers are used
for all descriptors except for the LIOP descriptor in which using a larger number ofneighboring sample points (8 instead of 4 neighbors) results in a higher performanceat the cost of a larger descriptor (241,920 instead of 144 dimensions). The LIOP and
SIFT implementations are provided by VLfeat (Vedaldi and Fulkerson, 2008). We use
the authors’ implementation of DAISY and GIH.
The evaluation is done on the dataset presented in Section 3.2. All the descriptors
are therefore tested on exactly the same image patches in order to exclusively judge thecapacity of local feature representation. Yet, as mentioned in Sec. 3.2, the dataset still3.2. Deformation and Illumination Invariant (DaLI) Descriptor

100

DaLI
DAISY
SIFT
LIOP
Pixel Diff.
NCC
GIH

100

50

0
2

4

6

8

100

50

0

10

2

4

0
6

8

50

0

10

2

4

50

0
4

6

8

0
2

4

8

4

0
6

8

10

8

2

4

6

n

2

4

8

10

8

10

6

8

10

6

8

10

n

0
4

6

8

50

0

10

2

4

n

100

0

6

0

10

n

50

10

100

2

Mean Detect. Rate (%)

50

6

50

10

100

n

2

n
Mean Detect. Rate (%)

Mean Detect. Rate (%)

6

8

50

n

50

10

100

4

0

100

n

2

50

10

Mean Detect. Rate (%)

Mean Detect. Rate (%)

Mean Detect. Rate (%)

Deform. Level #2

8

100

2

Deform. Level #3

6

6

100

n

100

4

n

100

n

2

Mean Detect. Rate (%)

4

0

10

Mean Detect. Rate (%)

2

8

Mean Detect. Rate (%)

100

50

6

50

n
Mean Detect. Rate (%)

Mean Detect. Rate (%)

Mean Detect. Rate (%)

Deform. Level #1

n100

Ill. Conditions #3
Mean Detect. Rate (%)

DaLI−PCA

Ill. Conditions #2
Mean Detect. Rate (%)

Ill. Conditions #1
Mean Detect. Rate (%)

Deform. Level #0

Ill. Conditions #0

43

100

50

0
2

4

6

n

8

10

50

0
2

4

n

Figure 3.17: Detection rate when simultaneously varying deformation level
and illumination conditions. Each graph represents the average of the mean detection rate between the reference image (ill. conditions #0 and deform. level #0) and all
images in the dataset under speciﬁc light and deformation conditions.

requires setting the scale factor to use for the points of interest. This value corresponds
to the relative size of each image patch with respect to the scale value obtained fromthe DoG feature point detector. For this purpose, we evaluated the response of all
descriptors for scale factors of 3×, 5×, 7× and 9×. The results are shown in Fig. 3.16.
Although the SIFT implementation uses a default value of 3×, we have observed thatthe performance of all descriptors improves by increasing the patch size. Note that thisdoes not result in a higher computational cost, as the ﬁnal size of the patch is normalized
to a circular shape with a diameter of 41 pixels. The maximum global response for all44

Chapter 3. Feature Point Descriptors

SIFT
LIOP
Pixel Diff.
NCC
GIH

100

50

0

DAISY
SIFT
LIOP
Pixel Diff.
NCC
GIH

4

6

8

100

50

0

10

2

4

6

8

50

0

10

2

4

6

8

10

n

n

n

Ill. Conditions #1

Ill. Conditions #2

Ill. Conditions #3

Mean Detect. Rate (%)

DaLI

Mean Detect. Rate (%)

Illumination-only

2

DaLI−PCA

Mean Detect. Rate (%)

DAISY

100

Deform. Level #3

100

Mean Detect. Rate (%)

DaLI

Deform. Level#2
Mean Detect. Rate (%)

DaLI−PCA

Mean Detect. Rate (%)

Deformation-only

Deform. Level #1

100

50

0
2

4

6

n

8

10

100

50

0
2

4

6

n

8

10

50

0
2

4

6

8

10

n

Figure 3.18: DaLI descriptor results for varying only deformation or illumination. Top: Results when varying only the deformation while keeping the illumination
conditions constant. It can be seen that both DaLI and DAISY largely outperformthe rest of descriptors. Bottom: Results of varying only the illumination conditions
while keeping the deformation level constant. Note that only DaLI remains robust
to illumination changes. The performance of DAISY falls roughly a 20% compared to
DaLI.
descriptors is achieved when using a 7× scale factor, which is the value we use for all
the experiments reported below.
The results for concurrent deformation and illumination are summarized in Fig. 3.17.
DaLI consistently outperforms all other descriptors, although the more favorable results are obtained under large illumination changes. The performance of DAISY isvery similar to that of DaLI when images are not aﬀected by illumination artifacts. In
this situation, the detection rates of DAISY are approximately between 2 − 5% below
to those obtained by DaLI. However, when illumination artifacts become more severe,
the performance of DAISY rapidly drops, yielding detection rates which are more than
20% below DaLI. SIFT, LIOP, and Pixel Diﬀerence yield similar results, with SIFT
being better at weak illumination changes and LIOP better at handling strong illumination changes. Yet, these three methods are one step behind DaLI and DAISY.
NCC generally performs worse except in situations with large illumination changes,
where it even outperforms DAISY. On the other hand, GIH performs quite poorly evenwhen no light changes are considered. This reveals another limitation of this approach,in that it assumes the eﬀect of deformations is to locally change the position of image
pixels, while in real deformations some of the pixels may disappear due to occlusions.
Although our approach does not explicitly address occlusions, we can partially handle

3.2. Deformation and Illumination Invariant (DaLI) Descriptor
Reference Image

SIFT

DAISY

LIOP

45

GIH

DaLI

n 10
n=10

n=9

n=8

n=7
=8
n=6

n=5

n=4

n=3

n=2

n=1

Figure 3.19: Sample results from the dataset. As in Fig. 3.2, the colour of thecircles indicates the position n of the correct match among the top candidates. If n > 10
we consider the point as unmatched and mark it with a cross.
them by weighing the contribution of the pixels within each patch, by a function decreasing with the distance to the center. Thus, most of the information of our descriptor
is concentrated in a small region surrounding the point of interest, hence making it lesssensitive to occlusions. The results also show that the compressed DaLI-PCA follows
a similar pattern as DaLI, and specially outperforms DAISY under severe illumination
conditions.
In Fig. 3.18 we give stronger support to our arguments by independently evaluating
deformations and illumination changes. These graphs conﬁrm that under deformationonly changes, DaLI outperforms DaLI-PCA and DAISY by a small margin of roughly
3%. Next, SIFT, LIOP, and Pixel Diﬀerence yield similar results, roughly 20% below
DaLI in absolute terms. GIH and NCC yield also similar results, although their performance is generally very poor. When only illumination changes are considered, both
DaLI and DaLI-PCA signiﬁcantly outperform other descriptors, by a margin larger than
20% when dealing with complex illumination artifacts. The only notable diﬀerence inthis scenario is that the NCC descriptor outperforms SIFT and Pixel Diﬀerence. As
GIH is not invariant to illumination changes, it obtains poor results. Similarly, since
LIOP is designed to be invariant to monotonic lighting changes, it does not performthat well in real images that undergo complex illumination artifacts.
In summary, the experiments have shown that DaLI globally obtains the best performance. Its best relative response when compared with other descriptors is obtained
when the deformations are mild and the light changes drastic. Some sample results onparticular images taken from the dataset can be seen in Fig. 3.19. Additionally, numeric results for the best candidate (n = 1 in Eq. 3.6) under diﬀerent conditions for all

46

Chapter 3. Feature Point Descriptors

Descriptor

Deformation

Illumination

Deformation+Illumination

DaLI-PCA

67.42585.12268.368

DaLI

70.57789.89572.912

DAISY

67.37375.40266.197SIFT

55.82260.76053.431

LIOP

58.76360.01452.176

Pixel Diﬀ.

54.71465.61054.382NCC

38.64362.04241.998GIH

37.45928.55631.230

Table 3.3: Evaluation results on the dataset for all descriptors. Results are
obtained by averaging the ﬁrst match percentage values over all images being tested
under all diﬀerent conditions.

True Positive Pairs

False Positive Pairs

False Negative Pairs

True Negative Pairs

Figure 3.20: True positive, false positive, false negative and true negative
image patch pairs. These were obtained using the DaLI descriptor on the dataset.
Note that most of the false negatives are due to large orientation changes across featurepoints.
descriptors are shown in Table 3.3.
Finally, examples of particular patch matches are depicted in Fig. 3.20. The true
positives pairs can be seen to be matched despite large changes. On the other hand,the false negatives seem largely generated by diﬀerences in orientations of the featurepoints: they correspond to the same patch, only rotated. The false positives share somesimilarity, although they are mainly from heavily deformed images.
Descriptor Size Performance
Since larger descriptors may a priori have an unfair advantage, we next provide resultsof an additional experiment in which we compare descriptors having similar sizes. The
LIOP we calculate in this case uses 4 neighbours instead of the 8 neighbours we considered before, which results in a smaller size, although also in a lower performance. GIH

3.2. Deformation and Illumination Invariant (DaLI) Descriptor

47

Deforming Cloth

Mean Detect. Rate (%)

100

DaLI
DAISY
SIFT

80

60

40

20
20

40

60

80

100

120

140

160

Frame

Paper Bending

Mean Detect. Rate (%)

100

80

60
DaLI
DAISY

40

SIFT
20
20

40

60

80

100

120

140

160

Frame

Figure 3.21:
Mean detection accuracy on two real world videos
from (Moreno-Noguer and Fua, 2013). For each sequence we show three example frames from each video in the top row. In the bottom row we plot the accuracy
for each frame for three descriptors: DaLI, DAISY and SIFT. Additionally the meanfor each descriptor is displayed as a dashed line.

is originally 176-dimensional, thus the results are the same as in Table 3.3. NCC andPixel Diﬀ, are not considered for this experiment as their size is 41 × 41 = 1681.
Results are shown in Table 3.4. We can see that the 128-dimensional DaLI-PCA
outperforms all other descriptors except the 256-dimensional DaLI-PCA. It is worth

48

Chapter 3. Feature Point Descriptors

Descriptor

Size

Deformation

Illumination

Deformation+Illumination

DaLI-PCA
SIFT
LIOP
DAISY
GIH
DaLI-PCA

128
128
144
200
176
256

67.45
55.82
54.01
67.37
37.46
67.43

82.34
60.76
44.89
75.40
28.56
85.12

67.71
53.43
44.45
66.20
31.23
68.37

Table 3.4: Comparison of performance and descriptor size.

noting the large performance gain obtained over the standard SIFT descriptor.

Effects of Intensity Normalization
We next extend the analysis we introduced in Sect. 3.2 in which we evaluated SI-HKS
and HKS with and without pre-normalizing the intensity of input images. We will
also consider SIFT and DAISY, which have been the most competitive descriptors inprevious experiments. Since SIFT/DAISY implementations require the pixels to be
in a [0, 1] range, we have normalized each image patch so that the pixels follow the
distribution N (0.5, (2 · 1.956)−1 ). This makes it so that on average 95% of the pixels
will fall in [0, 1]. Pixels outside of this range are set to either 0 or 1.
We compare the DaLI descriptor (both its SI-HKS and HKS variants), DAISY and
SIFT, with and without normalization. Results are shown in Table 3.5. We can seethat for DAISY and SIFT, since they perform a ﬁnal normalization stage, the results do
not have any signiﬁcant change. In the case of the DaLI descriptor, though, we observethat there is a rather signiﬁcant performance increase when using the SI-HKS variant
over the HKS one, even with patch normalization. This demonstrates again that therole of the Fourier Transforms applied in HKS to make it illumination invariant go far
beyond a simple normalization. In addition, SI-HKS compresses the descriptor in thefrequency domain and is one order of magnitude smaller than the HKS variant.

Evaluation on Real World Sequences
This section describes additional experiments on two real world sequences of deforming
objects, taken from (Moreno-Noguer and Fua, 2013). One consists of a T-Shirt being
waved in front of a camera (Deforming Cloth) and the other consists of a piece of paper
being bended in front of a camera (Paper Bending). We use points of interest computedwith the Diﬀerences of Gaussians detector (DoG) and follow the same patch extraction
approach as in the rest of the paper. The points of interest are calculated for the ﬁrst
frame in each sequence and then propagated using the provided 3D ground truth to theother frames. We use the same descriptor parameters as in the rest of the experiments,and seek to independently match the points of interest in the ﬁrst frame to those of all
the other frames.

3.3. Deep Architectures for Descriptors

49

Normalization?

Deformation

Illumination

Deformation+
Illumination

DaLI (SI-HKS)
DaLI (SI-HKS)
DaLI (HKS)
DaLI (HKS)

No
Yes
No
Yes

70.58
70.38
66.27
67.20

89.90
88.60
84.21
84.62

72.91
72.28
67.83
69.42

DAISY
DAISY
SIFT
SIFT

No
Yes
No
Yes

67.37
67.08
55.82
55.05

75.40
75.59
60.76
61.83

66.20
66.27
53.43
53.21

Descriptor

Table 3.5: Effect of normalizing image patches for various descriptors.
As we can observe in Fig. 3.21, DaLI outperforms both DAISY and SIFT2 . We
obtain a 5.5% improvement over DAISY on the Deforming Cloth sequence and a 4.1%
improvement on the Paper Bending sequence. Note that these sequences do not haveas complicated illumination artifacts as our dataset, an unfavorable situation for our
descriptor. Yet, DaLI still consistently outperforms other approaches along the whole
sequence.

3.3

Deep Architectures for Descriptors

While most descriptors use hand-crafted features (Lowe, 2004; Bay et al., 2006; Kokkinos
et al., 2012; Trulls et al., 2013), including the DaLI descriptor (Simo-Serra et al., 2015b)
presented in the previous section, there has recently been interest in using machinelearning algorithms to learn descriptors from large databases.
In this section we draw inspiration on the recent success of Deep Convolutional
Neural Networks on large-scale image classiﬁcation problems (Krizhevsky et al., 2012;
Szegedy et al., 2013) to build discriminative descriptors for local patches. Speciﬁcally,
we propose an architecture based on a Siamese structure of two CNNs that share theparameters. We compute the L2 norm on their output, i.e. the descriptors, and use aloss that enforces the norm to be small for corresponding patches and large otherwise.
We demonstrate that this approach allows us to learn compact and discriminative representations.
To implement this approach we rely on the dataset of (Brown et al., 2011), whichcontains over 1.5M grayscale 64×64 image patches from diﬀerent views of 500K diﬀerent
3D points. With such large datasets it becomes intractable to exhaustively explore
all corresponding and non-corresponding pairs. Random sampling is typically used;however, most correspondences are not useful and hinder the learning of a discriminant
mapping. We address this issue with aggressive mining of “hard" positives and negatives
and which proves fundamental in order to obtain discriminative learned descriptors. In
2Again, we only compare against DAISY and SIFT, as these are the descriptors which have been
more competitive in the experiments with the full dataset.

50

Chapter 3. Feature Point Descriptors

particular, in some of the tests we obtain up to a 169% increase in performance with
SIFT as a baseline.

Related Work
Recent developments in the design of local image descriptors are moving from carefullyengineered features (Lowe, 2004; Bay et al., 2006) to learning features from large volumesof data. This line of works includes unsupervised techniques based on hashing as wellas supervised approaches using Linear Discriminant Analysis (Brown et al., 2011; Gong
et al., 2012; C. Strecha and Fua, 2012), boosting (Trzcinski et al., 2012), and convex
optimization (Simonyan et al., 2014).
We explore solutions based on deep convolutional networks (CNNs). CNNs havebeen used in computer vision for decades, but are currenly experiencing a resurgencekickstarted by the accomplishments of (Krizhevsky et al., 2012) on large-scale image
classiﬁcation. The application of CNNs to the problem of descriptor learning has already been explored by some researchers (Jahrer et al., 2008; Osendorfer et al., 2013).
These works are however preliminary, and many open questions remain regarding thepractical application of CNNs for learning descriptors, such as the most adequate network architectures and application-dependent training schemes. We aim to provide
a rigorous analysis of several of these topics. In particular, we use a Siamese network (Bromley et al., 1994) to train the models, and experiment with diﬀerent network
conﬁgurations inspired by the state-of-the-art in deep learning.
Additionally, we demonstrate that aggressive mining of both “hard” positive and negative matching pairs greatly enhances the learning process. Mining hard negatives isa well-known procedure in sliding-window detectors (Felzenszwalb et al., 2010), where
the number of negative samples is virtually unlimited and yet most negatives are easily
discriminated. Similar techniques have been applied to CNNs for object detection (Girshick et al., 2014; Szegedy et al., 2013).

Learning Deep Descriptors
Given an intensity patch x ∈ RN , the descriptor of x is a non-linear mapping D(x)that is expected to be discriminative, i.e. descriptors for image patches corresponding
to the same point should be similar, and dissimilar otherwise.
In the context of multiple-view geometry, descriptors are typically computed forsalient points where scale and orientation can be reliably estimated, for invariance.
Patches then capture local projections of 3D scenes. Let us consider that each image
patch xi has an index pi that uniquely identiﬁes the 3D point which roughly projects
onto the 2D patch, from a speciﬁc viewpoint. Therefore, taking the L2 norm as asimilarity metric between descriptors, for an ideal descriptor we would wish that(0
if p1 = p2.
(3.7)
dD (x1 , x2 ) = kD(x1 ) − D(x2 )k2 =∞if p1 6= p2
We propose learning descriptors using a Siamese network (Bromley et al., 1994),
i.e. optimizing the model for pairs of corresponding or non-corresponding patches,
as shown in Fig. 3.22a. We propagate the patches through the model to extract thedescriptors and compute their L2 norm, which is a standard similarity measure for3.3. Deep Architectures for Descriptors(a) Siamese network.

51

Name

Layer 1

Layer 2

Layer 3Layer 4

CNN3_NN1

32x7x7x2 pool

64x6x6x3 pool

128x5x5
x4 pool

128
-

CNN3

32x7x7x2 pool

64x6x6x3 pool

128x5x5
x4 pool-

CNN2a_NN1

32x5x5x3 pool

64x5x5x4 pool128--

CNN2b_NN1

32x9x9x4 pool

64x5x5x5 pool

128
-

-

CNN2

64x5x5x4 pool

128x5x5
x11 pool--

CNN1_NN1

32x9x9x14 pool

128
-

-

-

(b) Various convolutional neural network architectures.

Figure 3.22: Overview of the network architectures used. Left: Schematic of aSiamese network, where pairs of input patches are processed by two copies of the same
CNN. Right: Diﬀerent CNN conﬁgurations considered.
image descriptors. We then compute the loss function on this distance. Given a pair ofpatches x1 and x2 we deﬁne a loss function of the forml(x1 , x2 , δ) = δ · lP (dD (x1 , x2 )) + (1 − δ) · lN (dD (x1 , x2 )) ,(3.8)

where δ is the indicator function, which is 1 if p1 = p2 , and 0 otherwise. lP and lN
are the partial loss functions for patches corresponding to the same 3D point and to
diﬀerent points, respectively. When performing back-propagation, the gradients are
independently accumulated for both descriptors, but jointly applied to the weights, as
they are shared.
Although it would be ideal to optimize directly for Eq. (3.7), we relax it, using amargin m for lN (·). In particular, we consider the hinge embedding criterion (Mobahi
et al., 2009)lP (dD (x1 , x2 )) = dD (x1 , x2 )

(3.9)

lN (dD (x1 , x2 )) = max(0, m − dD (x1 , x2 )) .

(3.10)

Convolutional Neural Network Descriptors
When designing the structure of the CNN we are limited by the size of the input data,in our case 64×64 patches from the dataset of (Brown et al., 2011). Note that larger
patches would allow us to consider deeper networks, and possibly more informative descriptors, but at the same time they would be also more susceptible to occlusions. We
consider networks of up to three convolutional layers, followed by up to a single additional fully-connected layer. We target descriptors of size 128, the same as SIFT (Lowe,2004); this value also constrains the architectures we can explore.

52

Chapter 3. Feature Point Descriptors

As usual, each convolutional layer consists four sub-layers: ﬁlter layer, non-linearity
layer, pooling layer and normalization layer. Since sparser connectivity has been shown
to improve performance while lowering parameters and increasing speed (Culurcielloet al., 2013), except for the ﬁrst layer, the ﬁlters are not densely connected to theprevious layers. Instead, they are sparsely connected at random, so that the mean
number of connections each input layer has is constant.
Regarding the non-linear layer, we use hyperbolic tangent (Tanh), as we found it
performs better than Rectiﬁed Linear Units (ReLU). We use L2 pooling for the pooling
sublayers, which were shown to outperfom the more standard max pooling (Sermanet
et al., 2012). Normalization has been shown to be important for deep networks (Jarrett
et al., 2009) and fundamental for descriptors (Mikolajczyk and Schmid, 2005). We use
subtractive normalization for a 5×5 neighbourhood with a Gaussian kernel. We will
later justify these decisions empirically.
An overview of the architectures we consider is given in Fig. 3.22b. We choose a
set of six networks, from 2 up to 4 layers. The architecture hyperparameters (numberof layers and convolutional/pooling ﬁlter size) are chosen so that no padding is needed.
We consider models with a ﬁnal fully-connected layer as well as fully convolutionalmodels, where the last sublayer is a pooling layer. Our implementation is based on
Torch7 (Collobert et al., 2011).
Stochastic Sampling Strategy and Mining
Our goal is to optimize the network parameters from an arbitrarily large set of training
patches. Let us consider a dataset with N patches and M ≤ N unique 3D patch indices,each with ni associated image patches. Then, the number of matching image patchesor positives NP and the number of non-matching images patches or negatives NN inthe dataset is
NP =

M
X
ni (ni − 1)
i=1

2

and

NN =

M
X

ni (N − ni ) .

(3.11)

i=1

In general both NP and NN are intractable to exhaustively iterate over. We approach
the problem with random sampling. For gathering positives samples we can randomly
choose a set of BP 3D point indices {p1 , · · · , pBP }, and choose two patches with corresponding 3D point indices randomly. For negatives it is suﬃcient to choose BN random
pairs with non-matching indices.
However, when the pool of negative samples is very large random sampling willproduce many negatives with a very small loss, which do not contribute to the global
loss, and thus stiﬂe the learning process. Instead, we can iterate over non-corresponding
patch pairs to search for “hard” negatives, i.e. with a high loss. In this manner it
becomes feasible to train discriminative models faster while also increasing performance.
This technique is commonplace in sliding-window classiﬁcation.
Therefore, at each epoch we generate a set of BN randomly chosen patch pairs,
and after forward-propagation through the network and computing their loss we keep
M “hardest” negatives, which are back-propagated through the
only a subset of the BN
network in order to update the weights. Additionally, the same procedure can beused over the positive samples, i.e. we can sample BP corresponding patch pairs andprune them down to the BPM “hardest” positives. We show that the combination of

3.3. Deep Architectures for Descriptors

53

Figure 3.23: Pairs of corresponding samples from the Multi-view Stereo
Correspondence dataset. Top row: ‘Liberty’ (LY). Middle row: ‘Notre Dame’ (ND).
Bottom row: ‘Yosemite’ (YO).
aggressively mining positive and negative patch pairs allows us to greatly improve thediscriminative capability of learned descriptors. Note that extracting descriptors withthe learned models does not further require the Siamese network and does not incur
the computational costs related to mining.
Learning
We normalize the dataset by subtraction of the mean of the training patches and division by their standard deviation. We then learn the weights by performing stochastic
gradient descent. We use a learning rate that decreases by an order of magnitude every ﬁxed number of iterations. Additionally, we use standard momentum in order toaccelerate the learning process. We use a subset of the data for validation, and stop
training when the metric we use to evaluate the learned models converges. Due to the
exponentially large pool of positives and negatives available for training and the small
number of parameters of the architectures, no techniques to cope with overﬁtting areused. The particulars of the learning procedure are detailed in the following section.

Results
For evaluation we use the Multi-view Stereo Correspondence dataset (Brown et al.,2011), which consists of 64×64 grayscale image patches sampled from 3D reconstructionsof the Statue of Liberty (LY), Notre Dame (ND) and Half Dome in Yosemite (YO).
Patches are extracted using the Diﬀerence of Gaussians detector (Lowe, 2004), anddetermined as a valid correspondence if they are within 5 pixels in position, 0.25 octaves
in scale and π/8 radians in angle. Fig. 3.23 shows some samples from each set, whichcontain signiﬁcant changes in position, rotation and illumination conditions, and often
exhibit very noticeable perspective changes.
We join the data from LY and YO to form a training set with over a million patches.
Out of these we reserve a subset of 10,000 unique 3D points for validation (roughly
30,000 patches). The resulting training dataset contains 1,133,525 possible positive
patch combinations and 1.117×1012 possible negative combinations. This skew is common in correspondence problems such as stereo or structure from motion; we address
it with aggressive mining.
A popular metric for classiﬁcation systems is the Receiving Operator Characteristic(ROC), used e.g. in (Brown et al., 2011), which can be summarized by its Area Under
the Curve (AUC). However, ROC curves can be misleading when the number of positive

54

Chapter 3. Feature Point Descriptors

Architecture

Parameters

PR AUCSIFT
CNN1_NN1
CNN2
CNN2a_NN1
CNN2b_NN1
CNN3_NN1
CNN3—68,352
27,776145,088
48,57662,784
46,272

0.361
0.032
0.379
0.370
0.439
0.289
0.558

Table 3.6: Effect of network architectures. We look at the eﬀects of network
depth, and fully convolutional networks vs networks with a fully-connected layer. The
PR AUC is calculated on the validation set for the top-performing iteration.
Architecture

PR AUCSIFT
CNN3
CNN3 ReLU
CNN3 No Norm
CNN3 MaxPool

0.361
0.558
0.442
0.511
0.420

Table 3.7: Comparison of network hyperparameters. We compare the fully
convolutional CNN3 architecture with diﬀerent hyperparameters settings such as Tanh
and ReLU rectiﬁcation layers, without normalization, and with max pooling instead of
L2 pooling. The best results are obtained for Tanh units, normalization, and L2 pooling(i.e. ‘CNN3’). The PR AUC is computed on the validation set for the top-performing
iteration.
and negative samples are very diﬀerent (Davis and Goadrich, 2006), and is alreadynearly saturated for the baseline descriptor SIFT. A richer metric is the Precision-Recall
curve (PR). We benchmark our models with PR curves and their AUC. In particular,
for each of the 10,000 unique points in the validation set we randomly sample twocorresponding patches and 1,000 non-corresponding patches, and use them to computethe PR curve. We rely on the validation set for the LY+YO split to examine diﬀerent
conﬁgurations, network architectures and mining techniques.
Finally, we evaluate the top-performing models over the test set. We follow the sameprocedure as for validation, compiling the results for 10,000 points with 1,000 noncorresponding matches each, now over 10 diﬀerent folds. We run three diﬀerent splits,
for generalization: LY+YO (tested on ND), LY+ND (tested on YO), and YO+ND(tested on LY).
We will consider all hyperparameters to be the same unless otherwise mentioned,
i.e. a learning rate of 0.01 that decreases ever 10, 000 iterations by a factor of 10. We
M = 128, and no positive mining; i.e.
consider negative mining with BN = 256 and BNM
BP = BP = 128.

3.3. Deep Architectures for Descriptors

55

PR curve, validation set

1

SIFT
CNN1_NN1
CNN2
CNN2a_NN1
CNN2b_NN1
CNN3_NN1
CNN3

0.9
0.8

Precision

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 3.24: Precision-recall curves for different network architectures. We
look at the eﬀect of network depth (up to 3 CNN layers), and fully convolutional
networks vs networks with ﬁnal fully-connected layer (NN1). Fully-convolutional models
outperform models with fully-connected neural network at the end.
PR curve, validation set

1

SIFT
CNN3
CNN3_ReLU
CNN3_NoNorm
CNN3_MaxPool

0.9
0.8

Precision

0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 3.25: Precision-recall curves for different network hyperparameters.
We analyze fully convolutional CNN3 models with Tanh and ReLU, without normalization, and with max pooling instead of L2 pooling. The best results are obtained forTanh units, normalization, and L2 pooling (‘CNN3’ model).
Depth and Fully Convolutional Architectures
The network depth is constrained by the size of the patch. We consider only up to 3 convolutional layers (CNN1-3). Additionally, we consider adding a single fully-connected

56

Chapter 3. Feature Point Descriptors

RP

RN

Cost

PR AUC

1
1
2
4
8
16

1
2
2
4
8
16

—
20%
35%
48%
67%
80%

0.366
0.558
0.596
0.703
0.746
0.538

Table 3.8: Effect of mining samples on performance. Mining factors indicate
the samples considered (BP , BN ), i.e. 1: 128, 2: 256, 4: 512, 8: 1024, and 16: 2048, of
which 128 are used for training. Column 3 indicates the fraction of the computational
cost spent mining.
Architecture

OutputParameters

PR AUCSIFT
CNN3
CNN3 Wide
CNN3_NN1
CNN3_NN1

128D
128D
128D
128D
32D

—
46,272
110,496
62,78450,400

0.361
0.596
0.552
0.456
0.389

Table 3.9: Effect of number of filters and fully-connected layer. The best
results are obtained with fully-convolutional networks with a fewer number of ﬁlters.

layer at the end (NN1). Fully-connected layers increase the number of parameters bya large factor, which increases the diﬃculty of learning and can lead to overﬁtting. We
show the results of the various architectures we evaluate in Table 3.6 and Fig. 3.24.
Deeper networks outperform shallower ones, and architectures with a fully-connected
layer at the end do worse than fully convolutional architectures. In the following experiments with consider only models with 3 convolutional layers.

Hidden Units Mapping, Normalization, and Pooling
It is generally accepted that Rectiﬁed Linear Units (ReLU) perform much better inclassiﬁcation tasks (Krizhevsky et al., 2012) than other non-linear functions. They are,however, ill-suited for regression tasks (such as the problem in hand), as they can only
output positive values. We consider both the standard Tanh and ReLU. For the ReLU
case we still use Tanh for the last layer. We also consider not using the normalization
sublayer for each of the convolutional layers. Finally, we consider using max poolingrather than L2 pooling. We show results for the fully-convolutional CNN3 architecture
in Table 3.7 and Fig. 3.25. The best results are obtained with Tanh, normalization andL2 pooling (‘CNN3’ in the table/plot). We will use this conﬁguration in the following
experiments.

3.3. Deep Architectures for Descriptors

57

PR curve, validation set

1
0.9
0.8

Precision

0.7
0.6
0.5
0.4
0.3

SIFT
CNN3, no mining
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8
CNN3, mined 16/16

0.2
0.1
0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 3.26: Precision-recall curves for different levels of mining. The best
results are obtained with 8/8 factors, i.e. pools of 1024 samples ﬁltered down to 128.
PR curve, validation set

1
0.9
0.8

Precision

0.7
0.6
0.5
0.4
0.3

SIFT
CNN3 (mined 2/2)
CNN3 Wide (mined 2/2)
CNN3_NN1 128D (mined 2/2)
CNN3_NN1 32D (mined 2/2)

0.2
0.1
0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 3.27: Precision-recall curves for different number of filters and fullyconnected layers. We show several results for diﬀerent number of ﬁlters and fullyconnected layers for dimensionality reduction. The best results are obtained with fullyconvolutional networks.
MiningWe analyze the eﬀect of both positive and negative mining by training diﬀerent models
in which a large, initial pool of BP positives and BN negatives are pruned to a smaller
number of ‘hard’ positive and negative matches, used to update the parameters of theM = 128 andnetwork. We keep this batch size for parameter learning constant, with BN

58

Chapter 3. Feature Point Descriptors

Train

Test

SIFT

LY+YOS
LY+ND
YOS+ND

ND
YOS
LY

0.349
0.425
0.226

CNN3

CNN3

CNN3

CNN3

Improvementmine-1/2

mine-2/2

mine-4/4

mine-8/8(Best vs SIFT)

0.535
0.383
0.460

0.555
0.390
0.483

0.630
0.502
0.564

0.667
0.545
0.608

91.1%
28.2%
169.0%

Table 3.10: Precision-recall area under the curve for the generalized resultsover the three dataset splits.
PR curve, training LY+YO, test ND

1

PR curve, training LY+ND, test YO

0.9

0.9

0.8

0.8

PR curve, training YO+ND, test LY

1

SIFT
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8

0.9
0.8

0.7

0.7

0.7

0.5
0.4

Precision

0.6

Precision

Precision

0.6
0.5
0.4

0.6
0.5
0.4

0.3
0.3

0.3

0.1
0

0.2

SIFT
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8

0.2

0

0.1

0.2

0.3

0.4

SIFT
CNN3, mined 1/2
CNN3, mined 2/2
CNN3, mined 4/4
CNN3, mined 8/8

0.1

0.5

Recall

0.6

0.7

0.8

0.9

1

0

0

0.1

0.2

0.3

0.4

0.2
0.1

0.5

0.6

0.7

0.8

Recall

0.9

1

0

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 3.28: Precision-recall curves for the generalized results over the threedataset splits.
M and positive
BPM = 128, and increase the ratio of both negative mining RN = BN /BNM
mining RP = BP /BP . We keep all other parameters constant. We use the notation
RP /RN , for brevity.
Large mining factors have a high computational cost, up to 80% of the total computational cost, which includes mining (i.e. forward propagation of all BP and BN
samples) and learning (i.e. backpropagating the “hard” positive and negative samples).
In order to speed up the learning process we initialize the CNN3 models with positivemining, i.e. 2/2, 4/4, 8/8 and 16/16, with an early iteration of a model trained onlywith negative mining (1/2).
Results are shown in Table 3.8 and Fig. 3.26. We see that for this particular problem
aggressive mining is fundamental. This is likely due to the extremely large numberof both negatives and positives in the dataset, in combination with models with a low
number of parameters. We observe a drastic increase in performance up to 8/8 mining
factors.

Number of Filters and Descriptor Dimension
We analyze increasing the number of ﬁlters in the CNN3 model, and adding a fullyconnected layer that can be used to decrease the dimensionality of the descriptor. We
consider increasing the number of ﬁlters in layers 1 and 2 from 32 and 64 to 64 and 96,
respectively. Additionally, we double the number of internal connections between layers.
This more than doubles the number of parameters in this network. To analyze descriptor
dimensions we consider the CNN3_NN1 model and change the number of outputs inthe last fully-connected layer from 128 to 32. In this case we consider positive mining
with BP = 256 (i.e. 2/2). Results can be seen in Table 3.9 and Fig. 3.27. The best

3.4. Summary

59

results are obtained with fewer ﬁlters and fully-convolutional networks. Additionally
we notice that mining is also instrumental for models the NN1 layer (compare resultswith Table 3.6).
Generalization
In this section we consider the three dataset splits. We train the best performing
models, i.e. CNN3 with diﬀerent mining ratios, on a combination of two sets, andtest them on the remaining set. We select the training iteration that performs bestover the validation set. The test datasets are very large (up to 633K patches) and
we use the same procedure as for validation, evaluating 10,000 unique points, eachwith 1,000 random non-corresponding matches. We repeat this process over 10 folds,
thus considering 100,000 sets of one corresponding patch vs 1,000 non-corresponding
patches. We show results in terms of PR AUC in Table 3.10, and the corresponding
PR curves are pictured in Fig. 3.28.
We report consistent improvements over the baseline, i.e. SIFT. The performancevaries signiﬁcantly from split to split; this is due to the nature of the diﬀerent sets.
‘Yosemite’ contains mostly fronto-parallel translations with illumination changes and no
occlusions (Fig. 3.23, row 3); SIFT performs well on this data. Our learned descriptors
outperform SIFT on the high-recall regime (over 20% of the samples; see Fig. 3.28),
and is 28% better overall in terms of PR AUC. The eﬀect is much more dramatic on‘Notredame’ and ‘Liberty’, which contains signiﬁcant patch translation and rotation,
as well as viewpoint changes around outcropping non-convex objects which result in
occlusions (see Fig. 3.23, rows 1-2). Our learned descriptors outperform SIFT by 91%
and 169% testing over ND and LY, respectively.
Qualitative analysis
Fig. 3.29 shows samples of matches retrieved with our CNN3-mined-4/4 network, overthe validation set for the ﬁrst split. In this experiment the corresponding patches wereranked in the ﬁrst position in 76.5% of cases; a remarkable result, considering that every
true match had to be chosen from a pool of 1,000 false correspondences. The righthand image shows cases where the ground truth match was not ranked ﬁrst; notice thatmost of these patches exhibit signiﬁcant changes of appearance. We include a failurecase (highlighted in red), caused by a combination of large viewpoint and illumination
changes; however, these misdetections are very uncommon.

3.4

Summary

In this chapter we have presented two vastly diﬀerent approaches at tackling the sameproblem: discriminative representation of local image patches, i.e., feature point descriptors. Our ﬁrst approach is more traditional in the sense it is a hand-crafted descriptor forthe speciﬁc problem of deformation and illumination invariant feature point descriptors.
For our second approach we propose a data-driven approach in which given enough data
we are able to learn the descriptor. Both approaches are complementary and can beused in a variety of diﬀerent applications.
On one hand, heat diﬀusion theory has been recently shown eﬀective for 3D shaperecognition tasks. We have proposed using these techniques to build DaLI, a feature

60

Chapter 3. Feature Point Descriptors

Figure 3.29: Samples of matches retrieved with our descriptor. Each row
depicts the reference patch (in green) and the top matching candidates, sorted fromleft to right by decreasing similarity. The ground truth match is highlighted in blue.
Left: Examples where the ground truth match is retrieved in the ﬁrst position. Right:
Examples in which the ground truth match is ranked at positions 2-6. The last row
shows a failure case, highlighted in red, where the correct match is ranked 632/1000.

point descriptor for 2D image patches, that is very robust to both non-rigid deformations
and illumination changes. The advantages of our method with respect to the stateof-the-art have been demonstrated by extensively testing on a new deformation andvarying illumination evaluation dataset. We have also shown that simple dimensionality
reduction techniques such as PCA can be eﬀectively used to reduce dimensionality
while maintaining similar performance. This seems to give the intuition that further
improvements can be obtained by using more advanced and powerful techniques suchas LDAHash (C. Strecha and Fua, 2012). Work has also been done in optimizing the
calculation speed by means of more complex meshing to reduce the cost of computing
the eigenvectors of the Laplace-Beltrami operator.
We have also shown that it is possible to learn feature point descriptors instead ofmanually deﬁning them. This is a non-standard learning problem due to using small
image patches and not being able directly optimize the model. Instead we propose
learning the model with a Siamese network that uses two patches simultaneously tolearn a deep convolutional network. Furthermore we propose a novel training scheme
based on aggressive mining of both positive and negative correspondences that we haveshown is critical to obtain high performance. We have performed extensive evaluationconsidering a wide range of architectures and hyperparameters to fully evaluate ourmethod. Our results show that given the right training data we are able to obtain a
2.5x performance increase over the widely used SIFT descriptor (Lowe, 2004). While
the results are preliminary, we believe that by creating a much larger and representative
dataset it should be possible to learn a very general descriptor that could be used as adrop-in replacement for existing descriptors to improve performance.
Both presented approaches have been proven to be extremely competitive with thestate-of-the-art and are complementary with existing and widely used diﬀerence of Gaussian based approaches. We have released the code for the evaluation and computation

3.4. Summary

61of the DaLI descriptor3 to encourage other researchers to compare against and use ourdescriptor in other applications. While we have not at the time of this writing released
the code for our deep descriptor framework, we intend to release it for the same reasons.

3

http://www.iri.upc.edu/people/esimo/research/dali/Chapter 4

Generative 3D Human Pose Models

In this chapter we give an overview of diﬀerent models for 3D Human Pose estimation inorder of increasing complexity and expressiveness. This is not meant to be a complete
list and only mentions models used throughout the course of this thesis. In particular
we shall discuss linear latent models, Directed Acyclic Graph (DAG) models, and wewill give a slightly longer description of the Geodesic Finite Mixture Model (GFMM).

4.1

Introduction

There has been a long tradition of generative models in the task of 3D human pose
estimation. This is due to the space of possible 3D poses being extremely large. Additionally, for 3D pose estimation there is no natural grid such as pixels used in 2D pose
estimation. This makes it unnatural to deﬁne discriminative models such as the pictorial structures model (Felzenszwalb and Huttenlocher, 2005) widely used in the 2D forthe 3D case. Furthermore, when tracking 3D poses, there is a strong spatial pose priorwhich encourages searching the solution space near the solution of the previous frame,
which can be incorporated elegantly into generative models.
In this chapter we shall focus on three diﬀerent approaches used in this thesis: linear
latent models, Directed Acyclic Graph (DAG) models, and Geodesic Finite Mixture
Models (GFMM) We will give a special focus to the latter model which is a novel model
proposed in this thesis. For all these poses we consider the static scenario, i.e., we do
not consider motion nor any other temporal information. Additionally all models makeuse of latent variables to model the pose.
Each model has a diﬀerent set of strengths and weaknesses. We focus on four diﬀerent model attributes: complexity, scalability, consistency, and whether it is modelling
the Probability Density Function (PDF) of the data. Complexity roughly corresponds
to the level of technical and algorithmic complexity of the model. As to be expected,
linear models have very low complexity in both formulation and speed, however, theytend to be much weaker representations. As dataset sizes are getting larger and larger,
scalability or the ability of the model to beneﬁt from large amounts of data is becoming more and more important. Additionally for 3D human pose estimation, we know63

64

Chapter 4. Generative 3D Human Pose Models(a) Probability Density Function(b) Trajectory reconstruction(c) Random trajectories drawn from the model

Figure 4.1: Example of a GPDM with a 3-dimensional latent space. Modelare learnt from a walking sequence of 2.5 gaits on the CMU mocap dataset (mocap.cs.cmu.edu). Figures reproduced from (Wang et al., 2005).

the data lies on a manifold for a given individual. Consistency refers to whether or not
an algorithm is able to generate poses only on this manifold, or if it can erroneously
also generate outside the manifold and thus generate non-anthropomorphic poses. Finally, not all models attempt to represent the PDF of the data. In particular linear
models are unable to, while more expressive models are able to. Although there are
more model attributes that can be taken into account, we ﬁnd that these four are themost representative of 3D pose models.
T
For the entire chapter we shall write a 3D pose as the vector x = [pT
1 , · · · , pnv ],where pi are the 3D positions of the skeleton joints. In general the objective is to modelthe distribution of these poses, that is p(x).

4.2. Related Work

4.2

65

Related WorkOne of the more successful families of models for 3D human pose estimation have been
the Gaussian Process family of models. Of which the most important is the GaussianProcess Latent Variable Model (GPLVM) (Lawrence, 2005), in which a continuous lowdimensional latent space is learnt for a single cycle of a motion. This has been extended
to dynamics with the Gaussian Process Dynamic Model (GPDM) proposed by Wanget al. (Wang et al., 2005; Urtasun et al., 2006; Wang et al., 2008; A. Yao and Urtasun,2011), which was also extended to include topological constraints (Urtasun et al., 2007).
A visualization of a 3-dimensional GPDM is shown in Fig. 4.1. Hierarchical variants(hGPLVM) have also been used in a tracking by detection approach (Andriluka et al.,2010). However, Gaussian Processes do not scale well to large datasets due to theirO(n3 ) complexity for prediction. Sparse approximations do exist (Quiñonero-candela
et al., 2005), but in general do not perform as well.
There have been other approaches such as learning Conditional Restricted Boltzmann Machines (CRBM) (Taylor et al., 2010). However, these methods have a very
complex learning procedure that makes use of several approximations and thus it is noteasy to train good models. Li et al. (Li et al., 2010) proposed the Globally CoordinatedMixture of Factor Analyzers (GCMFA) model which is similar to the GPLVM in thesense it is performing a strong non-linear dimensionality reduction. Yet, as in GPLVM
models it does not scale well to large datasets. A more simple approach would be theGaussian Mixture Model (GMM) that have been used extensively to encode the relationship between joints (Sigal et al., 2004; Daubney and Xie, 2011; Sigal et al., 2012).
This is usually done in the angular space. However, as this space is not a vector space
due to an inherent periodicity, they introduce heuristics to mitigate the eﬀect it hason the GMM. On the other hand our proposed GFMM does not have this issue andadditionally provides a much better approximation of the underlying manifold, without
any notable computational slowdown.
Additionally recently there have been many development on statistics on Riemannian manifolds (Pennec, 2006) that have made their way into computer vision (Fletcher
et al., 2004; Pennec et al., 2006; Davis et al., 2007), and in particular human motion (Sommer et al., 2010; Brubaker et al., 2012). The most popular model is the
Principal Geodesic Analysis (PGA) model (Fletcher et al., 2004), in which the data is
projected onto a linear space which is an approximation of the manifold. As this spaceis linear, another linear model, in particular a linear latent model, can be constructedin this approximated space. While this is a simple scalable and consistent approach, it
is not modelling a probability density function.
Not all work has been done on joint-based models. As it is quite common to use
silhouettes for 3D human pose tracking, shape-based models have been also proposed.
Some of the more simple ones model the body with a set of cylinders which are ﬁtted toa joint model (Sigal et al., 2004, 2012). However, there do exist approaches that directly
work with 3D meshes obtained from laser scans. One of the more well-known approaches
applies a Principal Component Analysis (PCA) to the 3D meshes (Anguelov et al., 2005;
Hasler et al., 2009). An example of 3D scans is shown in Fig. 4.2. The downside of using
silhouettes to match 3D mesh models is that it is usually dependent on backgroundsubtraction in order to get reliable silhouette estimates. On the other hand the joint
based models we consider can be used with other types of image evidence as we willshow in the next chapter.

66

Chapter 4. Generative 3D Human Pose Models

Figure 4.2: Examples of 3D body shape scans. These scans are used to obtaina set of deformation basis to represent body shape and 3D pose independently. Figure
reproduced from (Hasler et al., 2009).

Model

Complexity

Scales?

Consistent?

PDF?

GMM (Sigal et al., 2004)
PGA (Fletcher et al., 2004)
GPLVM (Lawrence, 2005)
GPDM (Wang et al., 2005)
hGPLVM (Andriluka et al., 2010)
CRBM (Taylor et al., 2010)
GCMFA (Li et al., 2010)

Low
Low
Low
MediumMedium
High
High

Yes
Yes
No
No
No
Yes
No

No
Yes
No
No
No
No
No

Yes
No
Yes
Yes
Yes
Yes
Yes

PCA
DAG
GFMM

Low
Medium
Low

Yes
Yes
Yes

No
No
Yes

No
Yes
Yes

Table 4.1: Comparison of different generative pose models. We analyze several
commonly used human pose models. We consider the complexity of the models which
we classify as low, medium or high. Models are considered to scale if they can handle
well large amounts of data (∼100k samples) and to be consistent if they use geodesic
distances instead of other metrics. Additionally we show whether or not a model isactually modelling the Probability Density Function (PDF) of the data. The bottom
three correspond to the models used in this thesis.

We show an overview of the aforementioned models in Table 4.1. We focus on fourmodel attributes: complexity, scalability, consistency, and whether or not the model
represents the Probability Density Function (PDF) of the data. We also include thethree models used in this thesis: latent linear model (PCA), Directed Acyclic Graph(DAG) and Geodesic Finite Mixture Model (GFMM). We can see that our models can
all scale well to large data while having a medium to low complexity. Additionally the
GFMM is consistent with the 3D human pose manifold.

4.3. Linear Latent Model

4.3

67

Linear Latent Model

Latent linear models are well known in the ﬁeld of 3D pose estimation and shape recovery (Moreno-Noguer et al., 2009, 2010; Moreno-Noguer and Porta, 2011; Moreno-Noguer
and Fua, 2013). Their extremely low complexity makes them very attractive for many
diﬀerent applications. They consist in ﬁnding a linear transformation that projects data
points to a, generally smaller, latent space. This is very useful as the true degrees of
freedom of high dimensional data is usually much smaller than the dimensionality ofthe data. In general, there is no linear mapping between the pose manifold and the 3D
pose, however, the coarse approximation can be considered suﬃcient when combinedwith additional constraints such as illumination (Moreno-Noguer and Fua, 2013).
One of the more simple yet widely used approaches consists in ﬁnding a linear latent
space using Principal Component Analysis (Tenenbaum et al., 2000). This consists inﬁnding a new linear basis for the data where the vectors are sorted by covariance. By
then discarding the vectors with lower covariance it is possible to ﬁnd a lower dimension
linear subspace that conserves most of the data variability. In the context of deformableobject, the vectors of the new basis can be interpreted as diﬀerent deformation modesof the object. This approach is unsupervised, i.e., no labels are needed for the data.
An alternative is Linear Discriminant Analysis, in which a projection that attempts tomaximize the diﬀerence between the class of each sample is found. This is a supervised
approach, that is, each sample has a label that we wish to predict.

Deformation Modes
We ﬁrst assume that the 3D pose can be represented as a linear combination of a mean
3D pose x0 and nm deformation models Q = [q1 , · · · , qnm ]x = x0 +nm
X

αi qi = x0 + Qα ,

(4.1)

i=1

where α = [α1 , . . . , αnm ]T are the unknown weights that deﬁne the current 3D pose andcan be interpreted as parameters of the learnt latent space corresponding tho the pose x.
The matrix Q can be found by simply calculating the eigenvalues of the covarianceof the training data. Assuming we have N poses xi we can rewrite them in matrix formas(x1 − x0 )T
 (x2 − x0 )T 

 ,
X=
(4.2)
..

.


(xN − x0 )TP
where x0 = N1 i xi .
The covariance C of the training data can then be calculated by
C=

1
XT X .
N −1(4.3)

The eigenvectors V and eigenvalues λ can then be computed by
V−1 CV = diag(λ) .

(4.4)

68

Chapter 4. Generative 3D Human Pose Models
1
0.999

Relative Covariance

0.998
0.997
0.996
0.995
0.994
0.993
0.992
0.991
0.99
5

10

15

20

25

30

35

40

n

m

Figure 4.3: Effect of σthreshold in linear latent models. We plot how the parameter σthreshold aﬀects the amount of covariance retained or relative covariance of the data
for the HumanEva walking action. In particular with nm = 21 we are able to retain
99.99% of the covariance.
Without loss of generality we can assume the eigenvectors V are sorted in descendingorder by their eigenvalues λ.
Finally the matrix Q is just the ﬁrst nm vectors of the eigenvectors V. As a criteria
for selecting the value of nm , the percent of the total variance explained by those vectors
can be used. Thus nm can be chosen byPnm
i=1 diarg min nm subject to σthreshold < PN
.
(4.5)
nm
d
i
i=1
Thus instead of choosing the value of nm directly, it is chosen based on a more interpretable threshold σthreshold , which is the only hyperparameter of the model. Typicalvalues for σthreshold are around 0.99.

Results
We plot the eﬀect of σthreshold in Fig. 4.3. This is the only hyperparameter of themodel and is critical for good performance as it serves as a lower-bound for 3D pose
estimation error. From our experiments we found that setting nm = 21, correspondingto σthreshold = 0.9999, oﬀers a good balance between dimensionality and low error. We
also show a qualitative example of how the dimensionality aﬀects the reconstruction
quality in Fig. 4.4.
The main advantage of these linear models is they are very fast to compute and very
easy to use. They are completely unsupervised and depend on a single hyperparameter.
The main disadvantage is they only allow for linear relationships and are not truly
modelling p(x), i.e., they only model a mapping, not an actually Probability Density
Function (PDF). Additionally were we to sample from the latent space α there is no
guarantee that the poses will be anthropomorphic or human-like. Thus if sampling
additional criteria will have to be used to select only the anthropomorphic poses.
In Section 5.3 we shall show an application in which we beneﬁt from the linear
formulation. In particular we are able to combine the linear pose model with a linear

4.4. Directed Acyclic Graphs

69

Dimension of the Latent Space nm
2
3
4

1

5

1600

1600

1600

1600

1600

1400

1400

1400

1400

1400

1200

1200

1200

1200

1200

1000

1000

1000

1000

1000

800

800

800

800

800

600

600

600

600

600

400

400

400

400

400

200

200

200

200

0
−200

−1200
−1000
−800
−600

0 200

0
−200

−1200
−1000
−800
−600

0 200

6

0
−200

−1200
−1000
−800
−600

0 200

7

0
−200

200
−1200
−1000
−800
−600

0 200

8

0
−200

9

10

1600

1600

1600

1600

1600

1400

1400

1400

1400

1400

1200

1200

1200

1200

1200

1000

1000

1000

1000

1000

800

800

800

800

800

600

600

600

600

600

400

400

400

400

400

200

200

200

200

0
−200

0 200

−1200
−1000
−800
−600

0
−200

0 200

−1200
−1000
−800
−600

0
−200

0 200

−1200
−1000
−800
−600

0
−200

−1200
−1000
−800
−600

0 200

200

0 200

−1200
−1000
−800
−600

0
−200

0 200

−1200
−1000
−800
−600

Figure 4.4: Dimensionality of the Linear Latent Space. We show the eﬀects ofthe dimensionality of the linear latent space for reconstructing. We show the groundtruth in black and the reconstructed pose for nm = {1, . . . , 10}. The ﬁrst three dimensions serve to appropriately position the joints. Additional dimensions improve smaller
details such as matching the hands and the feet.
camera projection model in order to project noisy 2D joint observations into the latent
space. As the formulation is linear, there is an exact solution, which generates a set of
3D hypothesis that can be used to perform 3D pose estimation.

4.4

Directed Acyclic Graphs

Using probabilistic graphical models is very common in computer vision. In particularstar-shaped models (Felzenszwalb et al., 2008, 2010) and tree-shaped models (Yang andRamanan, 2011) have been widely used as exact inference can be performed. In the caseof general graphical models, inference is no longer exact due to the presence of loops
and must be performed by message passing algorithms (Sigal et al., 2012; Andriluka
et al., 2012). We instead opt to use Directed Acyclic Graphs (DAG), that can capture
more complicated relationships in comparison to tree models while still having exact
inference, as there no loops in the graphical model.
This model attempts to characterize p(x) through a latent space. That is p(x) is notmodelled directly, instead it is modelled as p(x) = p(x|h)p(h) where h is a latent space.
In order to be tractable the poses are discretized using a clustering algorithm such as
k-means into a set of base poses X L . Additionally we deﬁne the lower dimensional

70

Chapter 4. Generative 3D Human Pose Models

latent space as another discrete set H. The problem can then be seen as learning acompression function φ(X L ) : X L → H that maps points in the high dimensional local
3D pose space to a lower dimensional space.

Discrete Joint Model
In order to obtain the discrete pose set X L the x poses in 3D are ﬁrst transformed to a
local reference such that they only represent local deformations. Then they are mappedfrom the RN ×3 continuous pose space x to a discrete domain by doing vector quantization of groups of 3D joint positions. More speciﬁcally, the joints are grouped into ﬁvecoarse parts: right arm (ra), left arm (la), right leg (rl), left leg (ll) and torso+head(th). Thus, every pose can be mapped to a discrete vector T = [ra, la, rl, ll, th] ∈ K5
where ra, la, rl, ll, th are cluster indexes belonging to K = {1, 2, .., k} of the corresponding 3D joint positions.
We will now deﬁne a joint distribution over latent variables H ∈ H = {1, .., n}(where n is the number of latent states), and observed variables T . The model is givenby the following generative process:- Sample a latent state i according to p (h0 )- For all the parts associated with arm and leg positions, sample discrete locations:< ra, la, rl, ll > and states < j, l, m, n > according to the conditional distributions: p (hra = j, ra | i), p (hla = l, la | i), p (hrl = m, rl | i) and p (hll = n, ll | i)
- Sample a pair of latent states: < q, w > (associated with the positions of theupper body and lower body joints) according to p (hu = q | hra = j, hla = l) andp (hl = w | hrl = m, hll = n)
- Sample a discrete location th and a state r from p h(th) = r, th | hu = q, hl = w

Given this generative model we deﬁne the probability of a discrete 3D position
T = [ra, la, rl, ll, th] as:X
p (T ) =p (T, H)H=X

p (h0 ) p (hra , ra | h0 ) p (hla , la | h0 )Hp (hrl , rl | h0 ) p (hll , ll | h0 ) p (hu | hra , hla )p (hll , ll | h0 ) p (hu | hra , hla )p (hl | hrl , hll ) p (hth , th | hu , hl) .
The graphical model corresponding to this joint distribution is illustrated in Fig. 4.5a,
where the graph G speciﬁes the dependencies between the latent states. Since H isunobserved, Expectation Maximization can be used to estimate the model parametersfrom a set of training poses. Given that G is a Directed Acyclic Graph we can computeall required expectations eﬃciently with dynamic programming (Huang, 2008). Once
we have learned the parameters of the model we deﬁne our compression function to be:φ(X L ) = arg max p X L , H ,Hand our decompression function to be:
φ−1 (H) = arg max p X L , H .
XL

4.4. Directed Acyclic Graphs

G

71

T

hth

G

th

ra

hu
hra

la

hrl

hr
hll

rl

th

ra

hl
hla

T

hth

ll

hra

la

hl
hla

hrl

hll

h0

h0

(a) UL grouping(b) RL groupingrlll

Figure 4.5: Directed Acyclic Graph models for 3D human pose. We use
a probabilistic graphical model with latent variables G to represent the possible 3D
human pose T from a large set of discrete poses. Latent variables can either be mapped
to the conjoint motion of various parts or be used as internal states containing internal
structure of the pose. We show two diﬀerent type of body part groupings: (a) Upperand Lower body grouping, and (b) Right and Left side grouping.
States

1

2

3

4

5

6

7

UL
RL

123
123

108
107

87
84

64
68

61
77

51
49

46
49

Table 4.2: Influence of the number of latent states. We evaluate them using the
average reconstruction error (in mm). We compare the upper-lower grouping (UL) toa right-left grouping (RL), which can be seen to perform roughly the same. The values
used are highlighted in bold.
Note that the decompression function is not technically speaking the true inverse ofφ(X L ), clearly no such inverse exists since φ(X L ) is many to one. However, we canregard φ−1 (H) as a “probabilistic inverse” that returns the most probable pre-image ofH. Our compression function maps points in K5 to points in H8 . For example, whenk=300 and n=9 we reduce the search space size from 1011 to 106 .

Results
One of the major concerns when building the DAG is what structure the graph shouldhave. This is an open problem and in general is left as a design decision. We experimentwith using two diﬀerent groupings: grouping right and left sides of the body (RL), andgrouping upper and lower body (UL). An overview of both groupings can be seen in
Fig. 4.5. Another hyperparameter is the number of latent states for each node. In orderto minimize the number of hyperparameters we use the same number of states for all
nodes. We experiment on the HumanEVA dataset (Sigal et al., 2010b). We evaluate

72

Chapter 4. Generative 3D Human Pose Models

Log−likelihood

38
36

Val Error

34

Train Error

32
30
28
26
24
22
20
18

0

5

10

15

20

25

30

35

40

Iteration(a) Training of the model

4550(b) Samples from the model

Figure 4.6: Directed Acylcic Graph (DAG) model training and sampling.
We train the DAG model for the training action with 6 latent states and UL grouping.
(a) Log-likelihood of the model for the training and validation splits. (b) Samples fromthe trained model. As we have trained with all three subjects simultaneously we see
three diﬀerent body sizes among the samples.
using the method by compressing and decompressing the testing poses to obtain thereconstruction error ǫrec deﬁned asǫrec (X L ) = X L − φ−1 (φ(X L )) .

(4.6)

We show results for both groupings and number of latent states in Table 4.2. We can seethat in this case both groupings perform nearly the same. While it is likely the graphicalstructure chosen is not optimal, the architecture does not seem to inﬂuence to heavily
in the results. On the other hand the number of latent states is very important to beable to accurately model human pose. However, adding more latent states increases thesize of the model, causing it to overﬁt. For our experiments we chose 6 latent stateswith a Upper and Lower body grouping (UL) that seems to provide a reasonable mixof good performance without having too many degrees of freedom.
The results of training the model can be seen in Fig. 4.6. We can see with 6 latentstates per node the model overﬁts fairly quickly. We rely on a validation split in order toavoid the overﬁtting. Samples taken from the model can be seen in Fig. 4.6b. As we train
with all 3 subjects simultaneously, three diﬀerent body sizes can be identiﬁed. Most of
the variability is, as expected of the walking action, located at the limb extremities.
The advantage of this model with respect to the linear latent model is clear: weare actually able to model and sample from p(x) through the latent space. Therefore
we do not need to perform any additional selection based on anthropomorphism norother criterion. However, the model still has several disadvantages. Namely we aremodelling the pose directly in 3D coordinates. This means that limb lengths may vary
for diﬀerent individuals, i.e., it is not an individual agnostic representation. Additionallythe discretization is also not exempt of issues. It creates artefacts in the 3D pose whensampling in addition to increase the number of hyperparameters of the model. Despite
this, we believe it is a step forward with respect to using linear latent models for thetask of 3D pose generation.

4.5. Geodesic Finite Mixture Models

73

3
2
1
0
−1
−2
−3

−3 −2 −1

0

1

2

3

(a) SO(2) Geodesics(b) Geodesic Finite Mixture Model

Figure 4.7: Geodesics and Mixture Models. (a) Representation of geodesics on
the S 2 manifold. The tangent space ensures that k logo (x)k is the true geodesic distance→→ However, k log (a) − log (b)k is not the geodesic distance of −of −ox.
ab. (b) Illustrationooof the proposed mixture model approach. Each mixture component has its own tangent
space, ensuring the consistency of the model while minimizing accuracy loss.

4.5

Geodesic Finite Mixture Models

In order to deal with problems that arose when using both the linear latent modeland the Directed Acyclic Graph (DAG) model, we investigated into using Riemannian
manifolds for representing the pose and modelling directly on the manifold. This allows
us to obtain a individual-agnostic model in the continuous domain that can scale wellto large datasets. More importantly, this model is consistent with the human posemanifold, i.e., it will only generate poses that have no limb deformation.
It is well known that human motion can be represented as points on a Riemannian
manifold (Brubaker et al., 2012; Sommer et al., 2010). We consider the task of obtaining
a probabilistic generative model of the poses, that is, approximate the Probability
Density Function (PDF) of points on this manifold for a potentially large dataset.
As we consider the situation in which the data lies on a known Riemannian manifold.
We are able to create an eﬃcient completely data-driven algorithm consistent with the
manifold, i.e., an algorithm that yields a PDF deﬁned exclusively on the manifold.
A standard procedure to operate on a manifold is to use the logarithmic map
to project the data points onto the tangent space of the mean point on the manifold (Fletcher et al., 2004; Sommer et al., 2010; Huckemann et al., 2010). After thislinearization, Euclidean statistics are computed and projected back to the manifold using the exponential map. This process is iteratively repeated until convergence of thecomputed statistics. Unfortunately, while this approximation is eﬀective to model datawith a reduced extent, it is prone to fail when dealing with data that covers wide regions of the manifold.
In the proposed ﬁnite mixture model, we overcome this limitation by simultaneously
considering multiple tangent spaces, distributed along the whole manifold as seen in
Fig. 4.7. We draw inspiration on the unsupervised algorithm from (Figueiredo andJain, 2002), which given data lying in an Euclidean space, automatically computes thenumber of model components that minimize a message length cost. By representing
each component as a distribution on the tangent space at its corresponding mean on
the manifold, we are able to generalize the algorithm to Riemannian manifolds and at
the same time mitigate the accuracy loss produced when using a single tangent space.

74

Chapter 4. Generative 3D Human Pose Models

Furthermore, since our model is semi-parametric, we can handle an arbitrarily large
number of samples. This is in contrast to existing non-parametric approaches (Pelletier,2005) whose complexity grows with the training set size.
In this section we will show that our manifold-based ﬁnite mixture model can beeﬀectively used both for non-linear data modeling and regression. In addition, we willshow that the modeling of articulated structures can greatly beneﬁt from our approach,allowing for instance, hallucinating subsets of missing joints of the human body without
any speciﬁc training.

MethodWe next describe our approach, starting with some basic notions on Riemannian geometry and statistics on manifolds. We then integrate these tools in a mixture modelingalgorithm to build consistent generative models.
Manifolds, Geodesics and Tangent Spaces
Manifolds arise naturally in many real-world problems. One of the more well-known isthe manifold representing spatial rotations. For example, when studying human motion,
it is a common practice to use the spatial rotations of the diﬀerent body parts to obtain asubject-agnostic representation of the whole body pose. This is usually done with angle
representations that have an inherent periodicity and thus are not a vector space. By
considering the Riemannian manifold of spatial rotations it is possible to use tangent
spaces as a local vector space representation, and use powerful statistical tools basedon Euclidean metrics. For an in depth description of Riemannian manifolds we refer
the reader to (Boothby, 2003).
Geodesic distances, which we shall denote as d(·, ·), are the shortest distance along
the manifold between two arbitrary points. This distance is generally not equivalent toan Euclidean one. The tangent space is a local vector space representation where theEuclidean distances between the origin and arbitrary points correspond to the geodesic
distances on the manifold. Yet, as seen in Fig. 4.7a, this correspondence does not holdfor two arbitrary points. A point v ∈ M on the tangent space Tc M at c ∈ M canbe mapped to the manifold M and back to the Tc M by using the exponential andlogarithmic maps respectively:
expp :

Tc M −→ Mv
7−→ expc (v) = x(4.7)

M −→ Tc M
x 7−→ logc (x) = v(4.8)

logp :

In general there is no closed-form of the expc and logc maps for an arbitrary manifold. There are, though, approximations for computing them in Riemannian manifolds (Dedieu and Nowicki, 2005; Sommer et al., 2009). Additionally, eﬃcient closedform solutions exist for certain manifolds (Said et al., 2007). An interesting mapping
for us will be the one between the unit sphere S 2 and its tangent space Tc S 2 . Let
x = (x1 , x2 , x3 )⊤ , y = (y1 , y2 , y3 )⊤ be two unit spoke directions in S 2 and v = (v1 , v2 )⊤4.5. Geodesic Finite Mixture Models

75

a point in Tc S 2 . The expc and logc maps are in this case:sin kvk
sin kvk, v2, cos kvk
v1
kvk
kvk


θ
θ
logp (x) = y1
, y2
sin θ
sin θ

expc (v) = R−1
c



(4.9)
(4.10)
1

where Rc is the rotation of c to the north pole, kvk = (v12 + v22 ) 2 ,θ = arccos(y3 ).

y = Rc x andStatistics on Tangent Spaces
While it is possible to deﬁne distributions on manifolds (Pennec, 2006), we shall focus
on approximating Gaussian PDFs using the tangent space. Since it is a vector space,
we can compute statistics that, by deﬁnition of the tangent space, are consistent withthe manifold. For instance, the mean of N points xi on a manifold can be calculated
as (Karcher, 1977)
NXµ = arg mind (xi , c)2 .
(4.11)ci=1This is optimized iteratively using the expc and logc maps,Nµ(t + 1) = expµ(t)δ X
logµ(t) (xi )
N
i=1

!

(4.12),

until kµ(t + 1) − µ(t)k < ǫ for some threshold ǫ, with δ being the step size parameter.
Given the mean, it is then possible to estimate the covariance on the tangent space:
N

Σ=

1 X
logµ (xi ) logµ (xi )⊤ .
N

(4.13)

i=1

Knowing the mean value and the concentration matrix (Γ = Σ−1 ) we can write the
distribution that maximizes entropy on the tangent space as a normal distribution
centered on the point µ ∈ M, corresponding to the origin (ν = 0) in the tangent space:logµ (x)⊤ Σ−1 logµ (x)Nµ (ν, Σ−1 ) = a exp −2!

(4.14)

where a is normalization term that ensures the normal distribution over the tangent
space integrates to unity. If Tµ M ≡ RD this term simpliﬁes toa−1 =q(2π)D det(Σ) .

(4.15)As the normalization term is dependent on the tangent space, it is not always easy
to obtain. However, it is possible to approximate this normalization factor based on
maximum entropy criterion computed directly on the manifold (Pennec, 2006). Since
this situation is very unusual, it is not further explored in this thesis.

76

Chapter 4. Generative 3D Human Pose Models

Unsupervised Finite Mixture Modeling
Our approach for ﬁtting the mixture model relies on (Figueiredo and Jain, 2002), avariant of the EM algorithm that uses the Minimum Message Length criterion (MML)
to estimate the number of clusters and their parameters in an unsupervised manner.
Given an input dataset, this algorithm starts by randomly initializing a large numberof mixtures. During the Maximization (M) step, a MML criterion is used to annihilatecomponents that are not well supported by the data. In addition, upon EM convergence,
the least probable mixture component is also forcibly annihilated and the algorithm
continues until a minimum number of components is reached. (Figueiredo and Jain,
2002) is designed to work with data in an Euclidean space. To use it in Riemannian
manifolds, we modify the M-step as follows.
Each mixture component is deﬁned by its mean µk and concentration matrix Γk =Σ−1
k as a normal distribution on its own tangent space Tµk M:p(x|θ k ) ≈ Nµk 0, Σ−1(4.16)
kwith θ k = (µk , Σk ). Remember that the mean µk is deﬁned on the manifold M, while
the concentration matrix Γk is deﬁned on the tangent space Tµk M at the mean ν k = 0.
Also note that the dimensionality of the space embedding the manifold is larger than
the actual dimension of the manifold, which in its turn is equal to the dimension of
the tangent space. That is, dim(Embedding(M)) > dim(Tc M) = dim(M) = D.
This dimensionality determines the total number of parameters Dθ specifying eachcomponent, and, as we will explain below, plays an important role during component
annihilation process. For full covariance matrices it can be easily found that Dθ =D + D(D + 1)/2.
We next describe how the EM algorithm is extended from Euclidean to Riemmanianmanifolds. Speciﬁcally, let us assume that K components survived after iteration t − 1.
Then, in the E-step we compute the responsibility that each component k takes forevery sample xi :αk (t − 1)p(xi |θ k (t − 1))(i)w k = PK,
(4.17)
k=1 αk (t − 1)p(xi |θ k (t − 1))for k = 1, . . . , K and i = 1, . . . , N , and where αk (t − 1) are the relative weights of eachcomponent k.
In the M-step we update the weight αk , the mean µk and covariance Σk for each of
the components as follows:N

1 X (i) wk
wk =
N
N
i

2
N
X
N (i) (i)µk (t) = arg mindw x , cwk kcαk (t) =(4.18)(4.19)i=1NΣk (t) =


⊤
1 X
(i)
logµk (t) (x(i) ) logµk (t) (x(i) ) wk
wk

(4.20)

i=1

After each M-step, we follow the same annihilation criterion as in (Figueiredo and Jain,
2002), and eliminate those components whose accumulated responsibility wk is below aDθ /2 threshold. A score for the remaining components based on the Minimum Message

4.5. Geodesic Finite Mixture Models

77

Length is then computed. This EM process is repeated until the convergence of the scoreor until reaching a minimum number of components Kmin . If this number is not reached,
the component with the least responsibility is eliminated (even if it is larger than Dθ /2)
and the EM process is repeated. Finally, the conﬁguration with minimum score isretained (see (Figueiredo and Jain, 2002) for details), yielding a resulting distribution
with the formKX
p(x|θ) =αk p(x|θ k ) .
(4.21)
k=1

To use the generative model in regression tasks, we need "to split the mix# intoΓk,A Γk,AB
two components, x = (xA , xB ) with µk = (µk,A , µk,B ) and Γk =. The
Γk,BA Γk,B
regression function can be written as:PK
αk p(xB |θ k,B )p(xA |xB , θ k )p(xA , xB |θ).
(4.22)= k=1 PK
p(xA |xB , θ) =p(xB |θ B )k=1 αk p(xB |θ k,B )Observe that this is a new mixture model p(xA |xB , θ) =weights:
αk p(xB |θ k,B )πk = PK,
j=1 αj p(xB |θ j,B )PK

k=1 πk p(xA |xB , θ k ), with

(4.23)and p(xA |xB , θ k ) = Nµk,A|B (ν k,A|B , Γk,A|B ), where the mean and concentration matrix
can be found to be:ν k,A|B = Γk,AB Γ−1k,B logµk ,B (xB ),(4.24)

Γk,A|B = Γk,A − Γk,AB Γ−1k,B Γk,BA .

(4.25)

Results
We evaluated our approach on the Human3.6M Dataset (Ionescu et al., 2011, 2014)
which has many subjects performing diﬀerent activities. As is common practice (Ionescu
et al., 2014), we model the poses in an individual-agnostic way by normalizing all thelimbs by their segments and only modeling the rotation. The rotation of each joint isrepresented on the S 2 manifold which then are arranged in a tree structure (Tournier
et al., 2009; Sommer et al., 2010). We note that other manifolds do exist such as one
deﬁned by using forward kinematics (Hauberg et al., 2012), however, this approach
does not have closed form solutions of the expc and logc operators. A simpliﬁed modelof the human with 15 joints is used because the dimensionality of the model greatly
increases the amount of the required data. This results in a 24-dimensional manifold,represented by a block-diagonal covariance matrix with 46 non-zero elements.
We divide the dataset into its ﬁve base scenarios: S1 , upper body movements; S2 ,full body upright variations; S3 , walking variations; S4 , variations while seated on achair; and S5 , sitting down on the ﬂoor. We train on subjects 1, 5, 6, 7, 8, 9 and 11 foreach scenario using only the ﬁrst of the actions in each scenario and test on the second
one. The algorithm uses 2000 initial clusters and ǫs = 0.999 for all scenarios.
We evaluate on the task of hallucinating limbs, i.e., probabilistically recovering
missing parts xA from the rest of the body xB . This is done by obtaining the expectation

78

Chapter 4. Generative 3D Human Pose Models(a) Scenario 1(b) Scenario 2(d) Scenario 4(c) Scenario 3(e) Scenario 5

Figure 4.8: Sampling from the Geodesic Finite Mixture Model. We sample
from the diﬀerent mixtures trained on each of the scenarios. For visualization purposes, we perform 3D reconstruction using the limb lengths of subject 1. Note thatthe mixtures themselves represent poses and are independent of limb lengths. For eachscenario, in the top left 30 samples are presented along with 5 particular examples. We
can see there is a lot of diversity that our model is able to capture.

4.5. Geodesic Finite Mixture Models

GFMM
vMF
1-TM
GMM

79

Scenario 1

Scenario 2

Scenario 3

MGJE MJE MLLE

MGJE MJE MLLE

MGJE MJE MLLE

0.446
0.481
0.522
1.111

0.468
0.568
0.640
1.167

0.349
0.470
0.535
1.152

105.8
114.5
123.0
103.1

GFMM
vMF
1-TM
GMM

0.0
0.0
0.0
19.0

110.1
134.8
148.7
106.6

0.0
0.0
0.0
27.5

81.7
110.2
124.9
77.6

Scenario 4

Scenario 5

MGJE MJE MLLE

MGJE MJE MLLE

0.458
0.496
0.548
1.272

0.597
0.698
0.765
1.401

108.2
118.0
130.2
101.0

0.0
0.0
0.0
14.2

135.7
162.3
175.1
127.3

0.0
0.0
0.0
11.3

0.0
0.0
0.0
24.8

Table 4.3: Comparing the Geodesic Finite Mixture Model (GFMM) withother approaches. We compare against von Mises Fisher distributions (vMF), asingle tangent space (1-TM) and an Euclidean Gaussian Mixture Model (GMM) in thelimb reconstruction task. We show results using three diﬀerent metrics: Mean Geodesic
Joint Error (MGJE), Mean Joint Error (MJE), and Mean Limb Length Error (MLLE).
of the posterior p(xA |xB , θ) using a combination of the conditional means (Eq. (4.24))
at each tangent space with
xA = arg min
c

K
X
k=1

d πk ν k,A|B , c

2(4.26)

We compare our method with von Mises Fisher distributions (vMF), a single tangent
space (1-TM) and a Gaussian Mixture Model in Euclidean space (GMM). To compensate the fact that our approach is individual-agnostic, was trained with the GMM withsame individual that was being tested. We deﬁne three metrics: Mean Geodesic Joint
Error (MGJE), that accounts for the rotation error; Mean Joint Error (MJE), representing the Euclidean distance with the ground truth in mm; and Mean Limb Length
Error (MLLE), which is the amount of limb length deformation in mm. All four models
are trained using exactly the same parameters for all scenarios. The results are summarized in Table 4.3. Observe that while the MJE is similar for both our method and
GMM, our approach has roughly one third of the MGJE error. Additionally, as ourmodel is coherent with the manifold, there is no limb deformation (MLLE). Our model
also consistently outperforms the vMF and 1-TM approaches, which once again showsthat multiple tangent spaces with Gaussians can better capture the underlying distribution. The 1-TM approach has the worst MJE error but we can see it has a MGJE
much closer to our model than the GMM. We also outperform the vMF model. We
show four speciﬁc examples of the estimated distributions for our model in Fig. 4.9.
In summary we have proposed a novel data-driven approach for modeling the probability density function of data located on a Riemannian manifold. By using a mixture

80

Chapter 4. Generative 3D Human Pose Models(a) Scenario 1(b) Scenario 2(d) Scenario 4(c) Scenario 3(e) Scenario 5

Figure 4.9: Human pose regression with the GFMM. We use the trained mixtures to perform regression on the left arm (3 limbs) of subjects on the Human3.6M
dataset. On the left we plot 30 random samples from the full mixture. On the top-rightwe plot 30 random samples from the mixture of the conditional distribution corresponding to a particular frames. On the bottom-right the mean of the mixture is shown in red.
The black lines indicate the input sample. As shown in Scenario 4, the distribution canbe multimodal, however, one of the modes is very close to the true limb position.
of distributions, each with its own tangent space, we are able to ensure the consistencyof the model while avoiding most of the linearization error caused by using one single
tangent space. The approach has been experimentally validated on various synthetic
examples that highlight their ability to both correctly approximate manifold distributions and discover the underlying data structure. Furthermore, the approach has beentested on a large and complex dataset, where it is shown to outperform the traditionallyused Euclidean Gaussian Mixture Model, von Mises distributions and using a single
tangent space in a regression task.

4.6

Summary

In this chapter we have presented three generative models of varying complexity andexpressiveness that are suitable for 3D human pose. The linear latent model ﬁnds a
linear transformation of the data that maximizes the covariance of the training data.

4.6. Summary

81

The Directed Acyclic Graph (DAG) model on the other hand deﬁnes a joint distribution
over pose and a set of latent variables p(X L , H), and calculates expectations withdynamic programming. Finally the Geodesic Finite Mixture Model (GFMM) directly
represents the data p(x) with a Gaussian Mixture Model in which each mixture isdeﬁned on a separate local linear representation of the pose manifold.
Each have diﬀerent strengths and weaknesses making them adequate for diﬀerent
tasks. The linear latent model is very simple and fast to calculate, however, it does
not necessarily produce realistic poses. The DAG model is able to model the poseseﬃciently, but it relies on using discrete states for the poses. The GFMM on the other
hand is a fully continuous generative model that is very eﬃcient to sample on, but it
relies on having a large amount of data to be able to train good models. In general,
the optimal model for a particular framework or problem will depend heavily on many
diﬀerent constraints such as time complexity, or required accuracy. The approaches wehave discussed in this chapter have very diﬀerent properties that allow them to covermost if not all usage cases.
In the next chapter we will show full end-to-end systems that are able to exploit
these diﬀerent generative models to predict 3D human poses from single images.

Chapter 5

3D Human Pose Estimation

Estimating the 3D human pose using a single image is a severely under-constrained
problem, as many diﬀerent body poses may have very similar image projections. In thischapter we shall explain two diﬀerent methods for approaching this problem. The ﬁrst
one consists of estimating the 2D human pose, and then propagating the uncertaintyof the 2D estimation to 3D, where the hypothesis can be disambiguated based onanthropomorphism to obtain a single solution. The second approach is based on jointly
estimating the 2D and 3D pose using a combination of 2D discriminative body partdetectors with a strong 3D generative model. By iteratively generating hypothesis andthen ranking them based on how well they match the detector responses, we are able
to ﬁnd a single 2D and 3D human pose estimation.

5.1

Introduction

Computer vision has always had a focus on problems centered around humans and theirenvironment. One of the more traditional computer vision problems has been thatof detecting humans in static images (Dalal and Triggs, 2005). This problem was thenextended to also estimating the 2D human pose from these images in which the Pictorial
Structures (Felzenszwalb and Huttenlocher, 2005) and its variations (Ramanan, 2006;
Andriluka et al., 2009; Yang and Ramanan, 2011; Andriluka et al., 2012) played afundamental role. However, as humans are not 2D structures, many ambiguities can
not be resolved with pure 2D models, leading to the advent of the 3D human pose
estimation problem. In order to simplify the problem, most approaches either tackle
the 3D human pose tracking problem (Lawrence and Moore, 2007; Urtasun et al., 2006;
Zhao et al., 2011; Andriluka et al., 2010), the multiview 3D pose estimation problem (Boand Sminchisescu, 2010; Burenius et al., 2013; Kazemi et al., 2013) (shown in Fig. 5.1)
or a combination of both (Daubney and Xie, 2011; Yao et al., 2012a). On the other
hand the single image monocular problem has only recently gained popularity.
In order to disambiguate the problem, one common approach is to assume that an83

84

Chapter 5. 3D Human Pose Estimation

Figure 5.1: Example of a multiview 3D pose estimation approach. In this approach a random forest is ﬁrst used to classify each pixel in each image as belonging
to a part or the background. The results are then back-projected to a 3D volume.
Corresponding mirror symmetric parts are found across views by using latent variables. Finally, a part-based model is used to estimate the 3D pose Figure reproduced
from (Kazemi et al., 2013).
underlying deformation model is available. Linear models (Balan et al., 2007) or sophisticated dimensionality reduction methods have been used for this purpose (Lawrence
and Moore, 2007; Sminchisescu and Jepson, 2004; Urtasun et al., 2006). Alternatively,
other techniques have focused on learning the mapping from 2D image observations to
3D poses (Okada and Soatto, 2008; Rogez et al., 2008; Sigal et al., 2009). In any event,most of these generative and discriminative approaches rely on the fact that 2D features, such as edges, silhouettes or joints may be easily obtained from the image.
In this thesis, we get rid of the strong assumption that data association may be easily
achieved, and propose two novel approaches for estimating the 3D pose of a personfrom a single image acquired with a calibrated but potentially moving camera. For thispurpose we use a HOG-based discriminative appearance model trained independently
on the PARSE dataset for 2D human pose estimation (Yang and Ramanan, 2011).
By training the 2D appearance model independently, we are able to avoid overﬁtting
the model to the 3D human pose estimation datasets that have very low appearance
variability. Poor generalization is a general trait of models with appearance trained
exclusively on the 3D dataset (Sminchisescu and Jepson, 2004).
The ﬁrst approach we present consists of initially using a 2D human pose estimation
algorithm (Yang and Ramanan, 2011) to obtain a noisy estimation of the 2D pose.
Then by combining a linear latent pose model with a linear camera projection model,
we are able to project the noisy 2D pose estimation into the latent space with a closedform formulation to obtain a large set of pose hypothesis. As the linear latent model is
not a probabilistic model, we rely on an anthropomorphism classiﬁer to disambiguate
the resulting poses to obtain a ﬁnal solution. By exploiting linear formulations andfast classiﬁers, the algorithm is extremely eﬃcient with the runtime dominated by the2D human pose estimation. The downside of this approach is the fact it is a pipelined
approach, i.e., if the 2D pose estimation is incorrect, the 3D pose estimation will also
be incorrect.
Our second approach relaxes the hypothesis that we have a 2D human pose estimation and instead relies directly on a bank of discriminative independent 2D bodypart detectors. We use a Bayesian formulation very similar to the pictorial structuresmodel which consists of the aforementioned 2D body part detectors in conjunction witha strong 3D pose generative model. In particular we use a Directed Acyclic Graph(DAG) to model the pose. By alternatively sampling from the generative model and5.2. Related work

85

Figure 5.2: Convolutional Neural Network for 3D Pose Estimation. The
network is initially pre-trained for the 2D human detection task using layers f cd1,
f cd2, and f cd3. Afterwards training is continued for the 3D human pose estimationtask, which is treated as a regression problem and uses layers f cr1, f cr2, and f cr3.
Figure reproduced from (Li and Chan, 2014).
then evaluating using the discriminative model we are able to jointly estimate the 2D
and 3D pose.
We evaluate both of our approaches numerically on the HumanEva dataset (Sigal
et al., 2010b) and qualitatively on the TUD Stadtmitte sequence (Andriluka et al.,2010). Results for both approaches are competitive with the state-of-the-art despite
our relaxation of restrictions.

5.2

Related work

Without using prior information, monocular 3D human pose estimation is known to be
an ill-posed problem. In order to be disambiguate between the possible solutions, manymethods to favor the most likely shapes have been proposed.
One of the most straightforward approaches consists of modeling the pose deformations as linear combinations of modes learned from training data (Balan et al., 2007).
Since linear models are prone to fail in the presence of non-linear deformations, more accurate dimensionality reduction approaches based on spectral embedding (Sminchisescu
and Jepson, 2004), Gaussian Mixtures (Howe et al., 1999) or Gaussian Processes (Urtasun et al., 2006; Lawrence and Moore, 2007; Zhao et al., 2011) have been proposed.
However, these approaches rely on good initializations, and therefore, they are typicallyused in a tracking context. Other approaches follow a discriminative strategy and use
learning algorithms such as support vector machines, mixtures of experts or random
forest to directly learn the mappings from image evidence to the 3D pose space (Agarwal and Triggs, 2006; Okada and Soatto, 2008; Rogez et al., 2008; Sigal et al., 2009).
Most of the aforementioned solutions, though, oversimplify the 2D feature extraction
problem, and typically rely on background subtraction approaches or on the fact thatimage evidence, such as edges or silhouettes, may be easily obtained from an image, oreven assume known 2D (Salzmann and R.Urtasun, 2010; Ramakrishna et al., 2012).
With regard to the problem of directly predicting 2D poses on images, we ﬁnd that

86

Chapter 5. 3D Human Pose Estimation
Uncertain 2D Detection

Ambiguous 3D Poses
1500

Disambiguated Pose
1600
1400

1000

1200
1000

500

800
600

0

400
200

−500
−1000
−5000

2000

1500

1000

500

0

−500

0
−600−400
−200

500
600
700
800
900

Figure 5.3: 3D human pose estimation from noisy observations. The left
image shows the bounding box results of a body part detector and green dots indicate
the true position of the joints. Note, in the middle, how the bounding box centers do
not match the joint positions. Using the heat map scores of the classiﬁer we represent
the output of the 2D detector by Gaussian distributions, as shown on the right. Using
the distribution of all the joints we sample the solution space and propose an initial setof ambiguous poses. By simultaneously imposing geometric and kinematic constraints
that ensure the anthropomorphism, we are able to pick an accurate 3D pose (shown inmagenta on the right) very similar to the ground truth (black).
one of the most successful methods is the pictorial structure model (Felzenszwalb and
Huttenlocher, 2005) (later extended to the deformable parts model (Felzenszwalb et al.,2008)), which represents objects as a collection of parts in a deformable conﬁguration
and allows for eﬃcient inference. Modern approaches detect each individual part using
strong detectors (Andriluka et al., 2009; Singh et al., 2010; Tian and Sclaroﬀ, 2010; Yang
and Ramanan, 2011) in order to obtain good 2D pose estimations. The deformableparts model has also been extended to use 3D models for 3D viewpoint estimation of
rigid objects (Pepik et al., 2012).
Recently there has been an explosion in deep network approaches in the computer
vision community that have made their way to 3D human pose estimation from single
images (Li and Chan, 2014) using the architecture shown in Fig. 5.2. However, in generalmost works still focus on the 2D human pose estimation problem (Jain et al., 2014a;
Toshev and Szegedy, 2014; Jain et al., 2014b). Currently 3D human pose estimation
using deep networks suﬀers from an overﬁtting to the unnatural images of the training
dataset. That is, in order to obtain a 3D ground truth, the subjects wear special
clothing in conjunction with markers used in the 3D capture system. Appearance
models learnt on these images do not generalize well to natural images. In contrast we
propose using 2D body part detectors trained independently on natural images, whichallow our appearance model to generalize much better.

5.3

3D Pose Estimation from Noisy 2D Observations

In order to robustly retrieve 3D human poses we proposed a new approach in which noisy
observations are modeled as Gaussian distributions in the image plane and propagatedforward to the shape space as shown in Fig. 5.3. This yields tight bounds on the solution
space, which we explore using a probabilistic sampling strategy that guarantees thesatisfaction of both geometric and anthropomorphic constraints. To favor eﬃciency, the5.3. 3D Pose Estimation from Noisy 2D Observations

Single Monocular
2D Im ageRaw
Im age2D PartDetector

2D Gaussians
U

87

EXPLORATION OF AMBIGUOUS HYPOTHESESHYPOTHESES DISAMBIGUATION

Set ofαsam plesSet ofLinear Projection (X,β) sam ples Sort by OCSVM
run on βv ectors
d →β

CMA
Sam pling
GaussianαLinear Projection
U →X →αLinear Projection
α→ X

Set ofX sam ples
Gm eans
Clustering

Set of(X,d) v ectors

Subset of
X sam ples

Calculatedistance v ectors

Sorted set of
X sam ples
Choose best
sam ple3D PoseEstim ation

3D PoseX

Figure 5.4: Flowchart of the method for obtaining 3D human pose fromsingle images.
exploration is performed hierarchically, starting from relatively lax and computationallyeﬃcient constraints up to highly restrictive and costly ones, until one single shapesample is retained. Our methodology outperforms approaches that optimize using only
geometric constraints.
Drawing inspiration from (Andriluka et al., 2010; Sigal and Black, 2006) we propose
retrieving 3D poses from the 2D body part positions estimated by state-of-the-art detectors (Andriluka et al., 2009; Felzenszwalb et al., 2008; Singh et al., 2010; Tian and
Sclaroﬀ, 2010; Yang and Ramanan, 2011). Although these detectors require a much reduced number of training samples, as they individually train each of the parts, they
have shown impressive results in a wide range of challenging scenarios. However, theirsolutions have an associated uncertainty which, combined with the inherent ambiguityof the single view 3D detection, may lead to large errors in the estimated 3D shape.
This is addressed in (Sigal and Black, 2006) by restricting the method to highly controlled settings, and in (Andriluka et al., 2010) by imposing temporal consistency. Other
approaches (Guan et al., 2009; Salzmann and R.Urtasun, 2010; Taylor, 2000) guarantee the single frame solution, but simplify the 2D detection process by either manuallyclicking the position of the 2D joints or directly using the ground truth values obtained
from motion capture systems.
In contrast, the approach we propose naturally deals with the uncertain observations of oﬀ-the-shelf body part detectors by modeling the position of each body partusing a Gaussian distribution that is propagated to the shape space. This sets bounds
on the solution search space, which we exhaustively explore to seek for the 3D pose
conﬁguration that best satisﬁes geometric (reprojection and length) and kinematic (anthropomorphic) constraints. To the best of our knowledge, (Daubney and Xie, 2011)is the only approach that has previously considered noisy observations, but only thoserelated to the root node and not to all the nodes, as we do. In addition, the mentioned
work imposes temporal constraints, while we are able to estimate the 3D pose using
one single frame.

MethodFigure 5.4 outlines our approach, which can be split into three major parts: 2Dpart detection, stochastic exploration of ambiguous hypotheses and disambiguation.
The 2D body part estimation is based on the state-of-the-art detector (Yang and Ramanan, 2011) which is adapted to our usage by obtaining information from the classiﬁer
heatmaps to provide local 2D Gaussian inputs. Following (Moreno-Noguer et al., 2010),

88

Chapter 5. 3D Human Pose Estimation

this uncertainty is propagated from the image plane to the shape space, thus reducing the size of the search space. We then use stochastic sampling to eﬃciently explore
this region and propose a set of hypotheses that satisfy both reprojection and length
constraints. This set of hypotheses must then be disambiguated by using some additional criteria. We show that only minimizing the reprojection and length errors does
not generally give the best results and propose a new method based on coordinate-freegeometry to help disambiguate while ensuring anthropomorphic-like shapes.
2D Body Part Detection
For body part detection we used (Yang and Ramanan, 2011) which learns a mixtureof-parts tree model encoding both co-occurrence and spatial relations. Each part ismodeled as a mixture of HOG-based ﬁlters that account for the diﬀerent appearances
the part can take due to, for example, viewpoint change or deformation. Since the partsmodel is a tree, inference can be eﬃciently done using dynamic programming, even fora signiﬁcant number of parts. The output of the detector is a bounding box for each
body part, which we convert to a Gaussian distribution by computing the covariancematrix of the classiﬁcation scores within the box. This is done because the method we
propose below to estimate the 3D pose takes as input probability distributions.
Estimating Ambiguous Solutions
The Gaussian distributions of each body part will be propagated to the shape space
and used to propose a set of 3D hypotheses that both reproject correctly onto the image
and retain the inter-joint distances of training shapes. However, due to the errors in theestimation of the 2D part location, there is no guarantee that minimizing these errors
will yield the best pose estimate. We will show that this requires applying additional
anthropomorphic constraints.
The approach we use to propagate the error and propose ambiguous solutions is inspired in (Moreno-Noguer et al., 2010), originally applied to non-rigid surface recovery.
However, note that dealing with 3D human poses has an additional degree of complexity, because most joints can only be linked to two other joints. In contrast, when dealing
with triangulated surfaces, each node is typically linked to six nodes. Therefore, theset of feasible human body conﬁgurations is much larger than the set of surface conﬁgurations. This will require using more sophisticated machinery such as integrating
kinematic constraints within the process.
We start out by assuming the matrix A of internal camera parameters to be known
and that the 3D points of the nv joints are expressed in the camera reference frame.
T
We can then write the projection constraints of 3D points x = [pT
1 , · · · , pnv ] to theirTT
2D correspondences u = [u1 , · · · , unv ]i as:
#
"
" #
A2×3
ui
pi ,
= Api =
wi
aT
1
3

(5.1)

where wi is a projective scalar, A2×3 are the ﬁrst two rows of A, and aT
3 is the last one.
T
Since from the last row we have wi = a3 pi , we can write:(ui aT
3 − A2×3 )pi = 0 .

(5.2)

5.3. 3D Pose Estimation from Noisy 2D Observations

89

For each 3D-to-2D correspondence we have 2 linear constraints on pi . As we have nv
correspondences, one for each joint, the 2nv constraints can be written as a linear system
Mx = 0 ,(5.3)

where M is a 2nv × 3nv matrix obtained from the known values of u, and A.
We also assume we have trained a linear latent model with nm dimensions and thushave:
x = x0 + Qα ,(5.4)
where x0 is the mean shape, Q is the nv × nm set of basis of the linear latent model,
and α are the nm weights in the latent space.
We can combine both the linear projection model from Eq.(5.3) and the linear latentmodel from Eq.(5.4) to obtain
MQα + Mx0 = 0 ,(5.5)

in which any set of weights α that is a solution will project at the right set of 2D
correspondences u.
We must now propagate the 2D Gaussian distributions found on the camera planeto the α latent space weights. Following (Moreno-Noguer et al., 2010), the mean of thissubspace can be computed as the least-squares solution of Eq. (5.5),µα = (BT B)−1 BT b ,(5.6)

where B = MQ is a 2nv × nm matrix and b = −Mx0 is a 2nv vector. The componentsof B and b are linear functions of the known parameters ui , Q and A. The same canbe done for the 2nv × 2nv covariance matrix Σu built using the covariances Σui of each
body part. Its propagation yields a nm × nm matrix Σα on the modal weights space,Σ α = JB Σ u JT
B ,

(5.7)

where JB is the nm × 2nv Jacobian of (BT B)−1 BT b.
The Gaussian distribution N (µα , Σα ) represents a region of the shape space containing 3D poses that will most likely project close to the detected 2D joint positions
ui . We will now sample this region and propose a representative set of hypotheses.
Note however, that the mean µα computed in Eq. (5.6) is unreliable, as it is computed from the ui ’s which are not necessarily the true means of the distributions. We
therefore do not draw all samples at once. Instead, we propose an evolution strategy
in which we draw successive batches by sampling from a multivariate Gaussian whosemean and covariance are iteratively updated using the Covariance Matrix Adaptation(CMA) algorithm (Hansen, 2006) so as to simultaneously minimize reprojection andlength errors.
sMore speciﬁcally, at iteration k we draw ns random samples {α̃ki }ni=1from thekk
2
distribution N (µα , M Σα ), where M is a constant that guarantees a certain conﬁdence
level (we set M = 4 in all experiments). Each sample α̃ki is assigned a weight πik
proportional to εlr = εl · εr , the product of the length and reprojection errors:X
˜lij − ltrain σ −1 ,
(5.8)
εl =
ij
ij
εr =

i,j∈N
nv q
X

(ũi − ui )T Σ−1ui (ũi − ui ),

i

(5.9)

90

Chapter 5. 3D Human Pose Estimation

train is the mean distance in all training samples between the i-th and j-th joints,
where lij
σij is the standard deviation, ˜lij is the length between joints i and j in the sample α̃ki ,and the ũi ’s are their corresponding 2D projections.
Given the weights πik for all samples, we then update the mean and covariance ofthe distribution following the CMA strategy. The mean vector µk+1
is estimated as aαconsists ofweighted average of the samples. The update of the covariance matrix Σk+1
αthree terms: a scaled covariance matrix from the preceding step, a covariance matrix
that estimates the variances of the best sampling points in the current generation, and acovariance that exploits information of the correlation between the current and previous
generations. For further details, we refer the reader to (Hansen, 2006).
After each iteration a subset of the samples with smaller weights is retained andprogressively accumulated for additional analysis. Note that instead of trying to optimize the error function, we use the error function with the CMA optimizer as a way
to explore the solution space. When a speciﬁc number of samples (104 in practice) has
been obtained, the problem then becomes how to disambiguate them to ﬁnd one that
represents an anthropomorphic pose.
As the detector input does not provide information on the orientation of the subject,
we consider the possibility of the pose facing both directions by swapping the detectedparts representing the left and right side of the body. This leads to two diﬀerent
distributions which we can then sample from.
Figure 5.5 shows an example of how the solution space is explored. Note that
although the CMA algorithm converges relatively far from the optimal solution withminimal reconstruction error, some of the samples accumulated through the exploration
process are good approximations. This is the key diﬀerence between using a plain CMA,
which just seeks for one single solution, and our approach, that accumulates all samples and subsequently uses more stringent –although computationally more expensive–
constraints to disambiguate.
After exploring the solution space, we have obtained a large number of samples that
represent possible poses that have both low reprojection and length errors. However,since many of these samples are very similar, we reduce their number using a Gaussianmeans clustering algorithm (Hamerly and Elkan, 2003). As shown in Fig. 5.6, we thenconsider the medoid of each cluster to be the candidate ambiguous shape. With thisprocedure, we can eﬀectively reduce the number of samples from 104 to around 102 .

Hypotheses Disambiguation
The set of ambiguous shapes has been obtained by imposing relatively simple but computationally eﬃcient constraints based on reprojection and length errors. In this section
we will describe more discriminative criteria based on the kinematics of the anthropomorphic pose to further disambiguate them until obtaining a single solution.
For this purpose, we will ﬁrst propose using a coordinate-free kinematic representation of the candidate shapes, based on the Euclidean Distance Matrix. Given the 3D
position of the nv joints, we deﬁne the nv × nv matrix D such that, dij = kpi − pj k.
It can be shown that this representation is unique for a given conﬁguration. In addition, as it is a symmetric matrix with zero entries at the diagonal, it can be compactly
represented by the nv (nv − 1)/2 vector
dKin = [d12 , · · · , d1nv , d23 , d24 , · · · , d(nv −1)nv ]T .

(5.10)

5.3. 3D Pose Estimation from Noisy 2D Observations

91

1000

Optimal PCA

500

Initialization

0

Convergence

Large

Best Candidate

α4

−500
−1000

Error

−1500
−2000
−2500
−3000

−2000

−1500

−1000

−500

Small

0

α1
Figure 5.5: Exploration of the solution space. The ﬁgure plots the distributionof samples on the modal weights space and how the covariance matrix is progressively
updated using the CMA algorithm. The two distributions represent both hypothesesof the directions the pose can be facing. In addition, the graph depicts the initial andthe ﬁnal conﬁgurations obtained with the CMA, and an optimal solution computedby directly projecting the ground-truth pose onto the PCA modes. The Best Candidate corresponds to the solution estimated by our approach. Note that although the
CMA does not converge close to the optimal solution, some of the samples accumulated
through the process lie very close, and thus, are potentially good solutions.
2D Projection

Clusters

Set of Candidate Articulated Poses

1800

1800

1800

1800

1800

1600

1600

1600

1600

1600

1400

1400

1400

1400

1400

1200

1200

1200

1200

1200

1000

1000

1000

1000

1000

800

800

800

800

800

600

600

600

600

600

400

400

400

400

200

200

200

400
200
0
−1000

−800

−600

−400

0
−200
−400
−600

−800

−600

−400

0
−200
−400
−600

200

400

200
0
−1000

400

200

400

400

200
0
−1000

−800

−600

−400

0
−200
−400
−600

400

200
0
−1000

−800

−600

−400

0
−200
−400
−600

200
0
−1000

−800

−600

−400

0
−200
−400
−600

Figure 5.6: Exploring the space of articulated shapes. The ﬁrst two plots
represent the 2D projection and 3D view of the shape samples we generate. The colourof the 3D samples indicates the cluster to which they belong. The four graphs on
the right represent the medoids of the clusters, which are taken to be the ﬁnal set of
ambiguous candidate shapes.
Given this unique representation of the pose kinematics, we then propose the treatment of the anthropomorphism as a regression problem. Speciﬁcally, we want to be able
to calculate how diﬀerent a 3D pose is from a set of training poses. We deal with thisproblem by using a one-class Support Vector Machine (OCSVM). The scores computedwith this classiﬁer can then be used to distinguish between clusters to determine themost anthropomorphic one.
In order to be able to properly determine the degree of anthropomorphism, andgiven that we have a limited amount of training data, we need to reduce the size of our92

Chapter 5. 3D Human Pose Estimation
Sample #1
Pose

Sample #2

AlignedPose

Aligned

1600

1600

1600

1600

1400

1400

1400

1400

1200

1200

1200

1000

1000

800

800

600

600

400

400

200

200

1200

1000

1000

800

800

600

600

400

400

200

200

0
−800
−600
−400
700600500400300

−800
−600
−400 400 200

0

−1000
−5001000

Error Value (εlr )SVM Output
Reconst. Err. (mm)Aligned Err. (mm)

500

0

−1000
−500 800 600 400 200

Sample #1

Sample #2

6.883
2.8e-04
199.9
56.4

6.885
-7.6e-03
214.9
167.7

0 −200Figure 5.7: Choosing human-like hypothesis via anthropomorphism. The anthropomorphism factor obtained from the OCSVM can be used to choose more humanlike models. In the top ﬁgures, the black lines represent the ground truth while thecoloured lines represent the diﬀerent poses. Note that although Sample #1 is far morehuman-like than Sample #2, both the error given by εlr (consisting of reprojection error
and limb length errors) and the reconstruction error are almost the same. In contrast,
the output of the SVM (+1: anthropomorphic; -1:non-anthropomorphic) indicates that
Sample #1 resembles more a human-like pose. A good way to validate anthropomorphism is by aligning the pose to the ground truth and measuring the reconstruction
error after alignment.
pose representation and avoid the curse of dimensionality. For this purpose, we will use
again PCA, and we will not directly train the classiﬁer on the whole Euclidean distancevector, but with a linear projection β of it.
One important thing to note is that the projection of the distance vectors dKin to thesubspace β implies a loss of information that can lead to non-anthropomorphic forms
being projected close to anthropomorphic forms. In order to account for this eﬀect, it
is important to remove the clusters with the worst error value εlr . As shown in Fig. 5.7,
this increases the likelihood that the results returned by the OCSVM correspond to ananthropomorphic form.

Results
We evaluated the algorithm on two diﬀerent datasets: the HumanEva dataset (Sigal
et al., 2010a), which provides ground truth, and the TUD Stadtmitte sequence (Andriluka et al., 2010), which is a challenging urban environment with multiple people,

93

800
800

Aligned Error (mm)Reconstruction Error (mm)5.3. 3D Pose Estimation from Noisy 2D Observations

600
400
200
0

Opt. BRec.

OA

BRep. BErr. BLen.

600
400
200
0

Opt. BRec.

OA

BRep. BErr. BLen.

Figure 5.8: Comparison of different error functions for 3D human pose estimation. Left: Reconstruction errors on the HumanEva dataset for the sequence “S2walk”. Right: Same errors after rigid alignment of the shapes with the ground truth
poses. It is more representative of the anthropomorphism of the pose compared to theplain reconstruction error, which only considers the distance between the joints of theretrieved pose and the ground truth. See text for a detailed description of the labels.
but without ground truth for a quantitative evaluation.
Evaluation on the HumanEva dataset
We quantitatively evaluated the performance of our method, using the walking andjogging actions of the HumanEva dataset. For training the PCA and SVM, we used the
motion captured data, independently for each action, for subjects “S1”, “S2” and “S3”,and used the “validation” sequences for testing.
Fig. 5.8 shows the distribution of the results on the “S2 walk” sequence. In Fig. 5.8Left we plot the average reconstruction error of our approach (OA), and compare itagainst the reconstruction error of Opt: the best approximation we could achieve using
PCA; BRec: sample with minimum reconstruction error among all samples generated
during the exploration process; BRep: the sample with minimum reprojection error;
BLen: the sample with minimum length error; and BErr: the sample that minimizes εlr(consisting of reprojection error and limb length errors). Note that neither minimizing
the reprojection error, the length error nor εlr guarantees retrieving a good solution.
We address this by also maximizing the similarity with anthropomorphic shapes. By
doing this, the mean error per joint of the shapes we retrieve is around 230mm. Yet,most of this error is due to slight depth oﬀsets which are hard to control due to thenoise in the input data. In fact, if we perform a rigid alignment between these shapes
and the ground truth ones, the error is reduced to about 100mm (Fig. 5.8-Right).
Finally, numeric results comparing with the state-of-the-art are given in Table 5.1.
Note that this comparison is for guidance only, as diﬀerent methods train and evaluate
diﬀerently. For instance, although (Bo and Sminchisescu, 2010) yields signiﬁcantly
better results, it relies on strong assumptions, such as background subtraction, which
both our approach and (Andriluka et al., 2010; Daubney and Xie, 2011) do not consider.
Therefore, we believe that to truly position our approach, we should compare ourselves
against (Andriluka et al., 2010; Daubney and Xie, 2011). In fact, the performance of all
three methods is very similar, but we remind the reader that (Andriluka et al., 2010;
Daubney and Xie, 2011) impose temporal consistency along the sequence, while we94

Chapter 5. 3D Human Pose Estimation

S1

Walking
S2

S3

3D Output
2D Input

99.6 (42.6)
14.1 (7.5)

108.3 (42.3)
19.1 (8.1)

127.4 (24.0)
26.8 (8.0)(Andriluka et al., 2010)(Daubney and Xie, 2011)(Bo and Sminchisescu, 2010)

89.3
38.2 (21.4)

107 (15)108.7
32.8 (23.1)

113.5
40.2 (23.2)

S1

Jogging
S2

S3

3D Output
2D Input

109.2 (41.5)
18.3 (6.3)

93.1 (41.1)
18.1 (6.0)

115.8 (40.6)
20.9 (6.1)(Bo and Sminchisescu, 2010)

42.0 (12.9)

34.7 (16.6)

46.4 (28.9)

Table 5.1: Comparison of results on the HumanEva dataset. Comparing theresults on the HumanEva dataset for the walking and jogging actions with all threesubjects. All values are in mm with the standard deviation in parentheses if applicable.
2D values are in pixels. Absolute error is displayed for (Andriluka et al., 2010; Daubneyand Xie, 2011), while our approach (OA) and (Bo and Sminchisescu, 2010) are relative
error values. (Andriluka et al., 2010; Daubney and Xie, 2011) do not provide jogging
data.
estimate the 3D pose using just one single image. A few sample images of the results
we obtain are shown in Fig. 5.9.
Testing on Street Images
We have also used the TUD Stadtmitte sequence (Andriluka et al., 2010) to test therobustness of the algorithm. We consider the scenario with multiple people to detect.
This sequence is especially challenging for 3D reconstruction as the camera has a long
focal distance, which ampliﬁes the propagation of the 2D errors to the 3D space.
Since we are dealing with real street images, walking pedestrian poses frequently
do not match our limited training data: pedestrians may either carry an object orhave their hands in their pockets, as seen in Fig. 5.10. Furthermore, the 2D body partdetector generally fails to ﬁnd the correct position of the hands (and consequently thearms) because of these occlusions. Despite these diﬃculties, our method is usually able
to ﬁnd the correct pose.

5.4

Joint 2D and 3D Pose Estimation

In this work, we get rid of the strong assumption that data association may be easily
achieved, and propose a novel approach to jointly detect the 2D position and estimate
the 3D pose of a person from one single image acquired with a calibrated but potentially moving camera. For this purpose we formulate a Bayesian approach combining
a generative latent variable model that constrains the space of possible 3D body poses5.4. Joint 2D and 3D Pose Estimation
Ground Truth

Part Detection

95

Opt. PCA Best Reconst. Best Error
1500

1500

1500

1000

1000

1000

500

500
−1000

0

S1, Walk, Frame 280 Err: 9.8 ± 5.3 px

1000
500

−2000
−1000

0

0
−500
500 1000

1500

500

−2000

−2000
−1000

0

0
−500
500 1000

1000

Ours

−2000
−1000

0

0
−500
500 1000

1000

0
−500
500 1000

1000

1000

Err: 24.8mm Err: 46.6mm Err: 805.0mm Err: 55.3mm
1600

1600

1600

1600

1400

1400

1400

1400

1200

1200

1200

1200

1000

1000

1000

1000

800

800

800

800

600

600

600

600

400

400

400

200

200

200

0
−100 100
0 200

−1000
−800
−600
−400
−200

0
−100 100
0 200

−1000
−800
−600
−400
−200

400
200

0
−100 100
0 200

−1000
−800
−600
−400
−200

0
−100 100
0 200

−1000
−800
−600
−400
−200

S2, Walk, Frame 20 Err: 13.7 ± 5.2 px Err: 22.1mm Err: 102.4mm Err: 284.5mm Err: 102.4mm
1600

1600

1600

1600

1400

1400

1400

1400

1200

1200

1200

1200

1000

1000

1000

1000

800

800

800

800

600

600

600

600

400

400

400

200

200

200

−1000
−500

−600
−400
−200

−1000
−500

−600
−400
−200

400
200

−1000
−500

−600
−400
−200

−1000
−500

−600
−400
−200

S2, Jog, Frame 155 Err: 13.0 ± 8.8 px Err: 15.4mm Err: 68.8mm Err: 157.3mm Err: 68.8mm
1800

1800

1800

1800

1600

1600

1600

1600

1400

1400

1400

1400

1200

1200

1200

1200

1000

1000

1000

1000

800

800

800

800

600

600

600

600

400

400

400

200

200

200

−1000
−500

600
800
1000

−1000
−500

600
800
1000

400
200

−1000
−500

600
800
1000

−1000
−500

600
800
1000

S3, Jog, Frame 280 Err: 17.5 ± 13.0 px Err: 26.3mm Err: 72.6mm Err: 105.3mm Err: 89.5mm

Figure 5.9: 3D Human Pose Estimation Results. Leftmost two columns: Raw
image with 2D ground truth projection, and the 2D detection results with the associated average pixel distance from ground truth. Rightmost four columns: Opt. PCA:
projection of the ground truth on the PCA; Best Reconstruction: the sample with lowest reconstruction error; Best Error: the sample with the lowest error εlr (consistingof reprojection error and limb length errors); and Ours: the solution obtained. Beloweach solution we indicate the corresponding reconstruction error (in mm). Note thatminimizing εlr does not guarantee retrieving a good solution.
with a HOG-based discriminative model that constrains the 2D location of the bodyparts. The two models are simultaneously updated using an evolutionary strategy. In
this manner 3D constraints are used to update image evidence while 2D observations
are used to update the 3D pose. As shown in Fig. 5.11 these strong ties make it possible to accurately detect and estimate the 3D pose even when image evidence is very
poor.

MethodFigure 5.12 shows an overview of our model for simultaneous 2D people detection and
3D pose estimation. It consists of two main components: a 3D generative kinematic
model, which generates pose hypotheses, and a discriminative part model, which weights

Chapter 5. 3D Human Pose Estimation

Frame 7045

96

1600
1400
1200
1000
800
600
400
200
500

Frame 7141

1000

500

0

−500

0
−500

−1000

−1500

1500

1000

500
3000
2000
2500 2000
1500 1000

1000
500

0 −5000

Figure 5.10: Results on the TUD Stadtmitte sequence. Left three columns: Raw
input image, followed by the detected Gaussians and the reprojection of the estimated
3D pose on the scene. Right column: Estimated 3D pose.
Raw

2D with Detectors 3D Output

Example Result

Figure 5.11: Simultaneous estimation of 2D and 3D pose. First column: Raw
input image. Second column: Ground truth 2D pose (green) and the result of ourapproach (red). Additionally, we plot a few part detectors and their correspondingscore, used to estimate 3D pose. Reddish areas represent regions with highest responses.
Third column: 3D view of the resulting pose. Note that despite the detectors not
being very precise, our generative model allows estimating a pose very close to theactual solution. Last Column: We show an example of a challenging scene with several
pedestrians.
the hypotheses based on image appearance. Drawing inspiration from the approach
proposed in (Andriluka et al., 2009) for 2D articulated shapes, we represent this modelusing a Bayesian formulation.
With this purpose, we represent 3D poses as a set of N connected parts. Let
l = {l1 , . . . , lN } be their 2D conﬁguration with li = (ui , vi , si ). (ui , vi ) is the image
position of the center of the part, and si a scale parameter which will be deﬁned below.
In addition let d = {d1 , . . . , dN } be the set of image evidence maps, i.e., the maps forevery part detector at diﬀerent scales and for the whole image. Assuming conditional

5.4. Joint 2D and 3D Pose Estimation

97

PROBABILISTIC GENERATIVE MODEL SAMPLINGDISCRIMINATION BY 2D PART DETECTORS

hth
hu
hra

...

hl
hrl

hla

Best Estimation Found

Sampling

Projection

hll

+

h0

Use discriminative evidence to generate samples that are most likely

Figure 5.12: Overview of our method for simultaneous 2D and 3D pose estimation. Our approach consists of two main building blocks: a probabilistic generativemodel and a set of discriminative 2D part detectors. Our optimization framework simultaneously solves for both the 2D and 3D pose using an evolutionary strategy. A
set of weighted samples are generated from the probabilistic generative model and aresubsequently reweighted by the score given by the 2D part detectors. This process isrepeated until convergence of the method. The rightmost ﬁgure shows results at convergence where the red shapes are the estimated poses and the green ones correspondto the ground truth.
independence of the evidence maps given l, and that the part map di only depends on
li , the likelihood of the image evidence given a speciﬁc body conﬁguration becomes:p (d | l) =NYp (di | li ) .

(5.11)i=1

In (Andriluka et al., 2009), Eq. (5.11) is further simpliﬁed under the assumptionthat the body conﬁguration may be represented using a tree topology. This yields anadditional eﬃciency gain, as it introduces independence constraints between branches,e.g., the left arm/leg does not depend on the right arm/leg. Yet, this causes the issue ofthe double counting, where the same arm/leg is considered to be both the left and rightone. In (Andriluka et al., 2012) this is addressed by ﬁrst solving an optimal tree andafterwards attempting to correct these artefacts using loopy belief propagation. Insteadof using two stages, we directly represent our 3D model using a Directed Acyclic Graph,
which enforces anthropomorphic constraints, and helps preventing the double counting
problem.
Let x = {p1 , . . . , pN } be the 3D model that projects on the 2D pose l, where pi =(xi , yi , zi ) is the 3D position of i-th part center. We write the posterior of x given theimage evidence d by:p (x | d) ∝NY(p (di | li ) p (li | pi )) p (x) .

i=1

In order to handle the complexity of directly modeling p (x), we propose approximating x through a generative model based on latent variables h. This allows us toﬁnally write the problem as:p (x | d) ∝ p (h) p (x | h)|{z}generativeNY(p (di | li ) p (li | pi ))

i=1

|

{z

discriminative}

98

Chapter 5. 3D Human Pose Estimation
1

4

7

10

13

16

19

22

25

28

31

Left Foot

Left Hand

Head

OctaveFigure 5.13: 2D Part Detectors. We visualize the response at diﬀerent octaves forthree diﬀerent part detectors. We overlay the projection of the 3D ground truth in white
to get an idea of the accuracy. The outputs are normalized for visualization purposes,with the dark blue and bright red areas corresponding to lower and higher responses
respectively. Note that while some detectors, such as the head one in the ﬁrst row,
generally give good results, others do not, such as the left hand detector in the middle
row. Our approach can handle these issues by combining these 2D part detectors witha generative model.
where the discriminative and generative components become clearly separated. We will
next describe each of these components.
Discriminative Detectors
Recent literature proposes two principal alternatives for discriminative detectors: theshape context descriptors built by applying boosting on the limbs (Andriluka et al.,2009), and the HOG template matching approach (Yang and Ramanan, 2011). For
our purposes we have found the HOG-based template matching to be more adequate
because it matches our joint-based 3D model better as we can place a detector at eachjoint part instead of having to infer the limb positions from the joints. In addition,
the HOG template matching yields smoother responses, which is preferable when doing
inference.
As mentioned above, each part li has an associated scale parameter si . This parameter is used to pick a speciﬁc scale among the evidence maps. Intuitively, if a partis far away, it should be evaluated by a detector at a small scale (high resolution). We
therefore approximate the scale si as:−1s−1
i = α βzi

(5.12)

5.4. Joint 2D and 3D Pose Estimation99

where α is the focal length of the camera and is used for normalization purposes. The
parameter β will be learned oﬀ-line and used during inference. Note that despite thisparameter constant, the scale at which each part is evaluated is diﬀerent as it depends
on the zi coordinate.
Let t = {t1 , . . . , tN } be the set of templates in the HOG space associated to each
body part. These templates are provided in (Yang and Ramanan, 2011)1 . Given abody part li , its image evidence di is computed by evaluating the template ti overthe entire image for a range of scales si . Figure 5.13 illustrates the response of threepart detectors at diﬀerent scales. By interpreting each detector as a log-likelihood, theimage evidence of a conﬁguration L can be computed as:log p (l | d) ≈ score(l) =NX

ki di (ui , vi , si ) ,(5.13)

i=1

where ki is a weight associated to each detector, which is learned oﬄine. It is meant
to adapt the 2D detectors and 3D generative model due to the fact they were trained
independently on diﬀerent datasets.
Additionally, when evaluating a part detector at a point, we consider a small window
from which we use the largest detector value in order to give additional noise tolerance
to the detector. We ﬁnd this necessary as small 2D errors can have large consequencesin 3D positioning.
Latent Generative Model
We use the Directed Acyclic Graph model explained in Section 4.4. This model discreticizes the poses to be able to learn the joint distribution of the 3D poses and alatent space by using Expectation Maximization. Given that the model does not contain loops, eﬃcient inference can be done using dynamic programming. This allows themodel to be used in a generative fashion in which the latent space is sampled and thenthe most likely 3D pose for the latent space conﬁguration is eﬃciently estimated.
Inference
The inference problem consists of computing:<X ∗ >= arg max
xNY(p (di | li ) p (li | pi )) p (x | h) p (h) .

(5.14)

i=1

We treat this as a global optimization problem where, given a set of 2D detections
corresponding to the diﬀerent parts from a single image, we optimize over both a rigid
transformation and the latent states. Drawing inspiration in (Moreno-Noguer andFua, 2013; Moreno-Noguer et al., 2010), we do this using a variation of the CovarianceMatrix Adaptation Evolutionary Strategy (CMA-ES) (Hansen, 2006), which is a blackbox global optimizer that uses a Gaussian distribution in the search space to minimize
1

Indeed, each of the part detectors provided in (Yang and Ramanan, 2011) is formed by several
templates and we use their maximum score for each coordinate (u, v, s). For ease of explanation werefer to them as a single template.

100

Chapter 5. 3D Human Pose Estimation(Yang and Ramanan, 2011)

Ideal Detector

Our Approach

Err.

2D

2D

3D

Pose

2D

3D

Pose

All

21.7

11.0

106.6

51.6

19.5

237.3

55.3

C1
C2
C3

19.5
22.9
22.8

11.1
11.1
10.8

113.8
109.7
96.2

52.3
51.2
51.2

18.9
19.6
20.0

239.1
245.8
227.1

55.2
55.4
55.4

S1
S2
S3

21.8
21.8
21.6

10.2
10.8
12.3

96.8
108.0
119.0

63.4
44.8
43.7

19.9
18.6
20.1

277.2
206.6
221.4

69.3
46.8
46.6

A1
A2

20.9
22.7

10.7
11.3

106.0
107.2

56.2
46.6

19.3
19.7

254.4
219.0

60.3
50.0

Table 5.2: Results on the HumanEva dataset. We show results for the walking(A1) and jogging (A2) actions with all subjects (S1,S2,S3) and cameras (C1,C2,C3).
We compare with the 2D error obtained using the 2D model from (Yang and Ramanan,2011), based on the same part detectors we use. 2D, 3D and Pose Errors are deﬁned
in the text. Ideal detector corresponds to our approach using Gaussians with 20px
covariance as 2D input instead of the 2D detections.
a function. In our case we perform:
arg max score proj φ
R,t,H

R,t

−1

(h)



!

+ log p φ−1 (h) , h,where projR,t (·) is the result of applying a rotation R, a translation t and projecting
onto the image plane. We then take x∗ = Rφ−1 (h∗ ) + t, that is, we obtain the mostprobable xL∗ given h∗ and perform a rigid transformation to the world coordinates toobtain x∗ .

Results
We numerically evaluate our algorithm on the HumanEva benchmark (Sigal et al.,2010b), which provides 3D ground truth for various actions. In addition, we provide
qualitative results on the TUD Stadtmitte sequence (Andriluka et al., 2010), a clutteredstreet sequence. For both cases, we compute the 2D observations using the detectors
from (Yang and Ramanan, 2011) trained independently on the PARSE dataset (Ramanan, 2006).
We consider three error metrics: 2D error, 3D error and 3D pose error. The 2D
error measures the mean pixel diﬀerence between the 2D projection of the estimated
3D shape, and the ground truth. 3D error is the mean euclidean distance, in mm, withthe ground truth, and the 3D pose error is the mean euclidean distance with the groundtruth after performing a rigid alignment of the two shapes. This error is indicative ofthe local deformation error. We evaluate three times on every 5 images using all threecameras and all three subjects for both the walking and jogging actions, for a total of1318 unique images.

5.4. Joint 2D and 3D Pose Estimation
S1

S2

101
S3

Depth Errors

Other Failures

600
800

600

600

400

400
600

400

400
200

200
400

200

200
0

0

A1

200

0

0
−200

−200

0

−400

−200

−200

−200
−400

−400

−400

−600

−400

−600

−600
−600

−800

−600

−800

−800

−1000
−800
−600

−400
−200

−800
−1100
−1000
−900
−800
−700

−400
−200
0

−3600 −3800 −4000

−3700
−3600
−3500
−3400
−3300

−4000 −4200 −4400

0
200
400

−3600
−3000 −3200 −3400

0−4800 −5000 −5200

600
600

600
400

400

400
400

400
200

200

200
200

200

0

0

0
0

0

A2

−200

−200

−200

−200
−200

−400

−400

−400

−400
−400

−600

−600

−600

−600
−600

−800

−800

−800

−800

−800
400
600
800

−3900
−3800
−3700
−3600
−3500

−800
−600
−400

−3400 −3600 −3800

400
600

−3500
−3400
−3300
−3200
−3100

−1000
100
200
300

−2800 −3000 −3200

−600
−400
−200

−3400 −3600 −3800

Figure 5.14: Sample frames from the HumanEva dataset. We show examples
for both the walking (A1) and jogging (A2) actions. The ﬁrst three columns correspondto successful 2D and 3D pose estimations for the three subjects (S1,S2,S3). The next
two columns show typical failures cases of our algorithm. In the fourth column we seethat occasionally we suﬀer from depth errors, where the 3D pose is correct but its depth
is not. In the last column we plot other failures, mostly caused by very large errors ofthe 2D detector, due to mis-classiﬁcations or self-occlusions.

Table 5.2 summarizes the results of all experiments. We compare our approach using
both Gaussians (20px Cov.) and the detector outputs as inputs. We see that using ideal
detectors, even with large covariances, the absolute error is reduced to 45% of the full
approach. An interesting result is that we outperform the 2D pose obtained by (Yangand Ramanan, 2011), using their own part detectors. This can likely be attributed toour joint 2D and 3D model. Nonetheless, although (Yang and Ramanan, 2011) is not
an action speciﬁc approach as we are, this is still an interesting result as (Pepik et al.,2012) reports performance loss in 2D localization when using a 3D model. Figure 5.14
shows some speciﬁc examples. As expected, performance is better when there are fewer
self-occlusions.
We also compare our results with (Andriluka et al., 2010; Bo and Sminchisescu, 2010;
Daubney and Xie, 2011), and the approach presented in the previous section “From 2D
Observations”. This is just meant to be an indicative result, as the diﬀerent methods
are trained and evaluated diﬀerently. Table 5.3 summarizes the results using the poseerror, corresponding to the “aligned error” mentioned in the previous section. The twoalgorithms that use temporal information (Andriluka et al., 2010; Daubney and Xie,
2011) are evaluated using absolute error. Moreover, (Daubney and Xie, 2011) uses twocameras, while the rest of the approaches are monocular. Due to our strong kinematic
model we outperform all but (Bo and Sminchisescu, 2010). Yet, in this work the 2D
detection step is relieved through background subtraction processes.
Finally, we present qualitative results on the TUD Stadtmitte sequence (Andriluka
et al., 2010), which represents a challenging real-world scene with the presence of distracting clutter and occlusions. Some results can be seen in Fig. 5.15. Note that while
the global pose seems generally correct, there are still some errors in the 3D pose dueto the occlusions and to the fact that the walking style in the wild is largely diﬀerent
from that of the subjects of the HumanEva dataset used to train the generative model.

102

Chapter 5. 3D Human Pose Estimation

S1

Walking (A1,C1)
S2

S3

Joint 2D and 3D

65.1 (17.4)

48.6 (29.0)

73.5 (21.4)

From 2D Observations(Daubney and Xie, 2011)(Andriluka et al., 2010)(Bo and Sminchisescu, 2010)99.6 (42.6)
89.3
38.2 (21.4)

108.3 (42.3)
108.7
107 (15)
32.8 (23.1)

127.4 (24.0)
113.5
40.2 (23.2)

S1

Jogging (A2,C1)
S2

S3

Joint 2D and 3D

74.2 (22.3)

46.6 (24.7)

32.2 (17.5)

From 2D Observations(Bo and Sminchisescu, 2010)109.2 (41.5)
42.0 (12.9)

93.1 (41.1)
34.7 (16.6)

115.8 (40.6)
46.4 (28.9)

Table 5.3: Comparison against state-of-the-art approaches. We present results
for both the walking and jogging actions for all three subjects and camera C1. From
2D Observations refers to the ﬁrst approach presented in this chapter.
Frame 7056

Frame 7126

Figure 5.15: Two sample frames from the TUD Stadtmitte sequence (Andriluka et al., 2010).

5.5

Summary

We have presented two diﬀerent approaches for 3D human pose estimation from single
monocular images. This is an extremely challenging problem as it is severely underconstrained. Despite the diﬃculty we have been able to obtain very promising resultswith both approaches, each with a diﬀerent set of strengths and weaknesses.
In the ﬁrst approach a 2D pose estimation model is used to initially obtain a noisy
observation that can be used for 3D pose estimation. The method consists of a linear
projection model that constrains the possible 3D positions given the 2D observations in
combination with a linear latent model for the 3D pose. By fusing both linear models,
we are able to obtain a linear formulation of the camera constraints on the latent space.
This allows a noisy 2D pose estimation to be projected to the 3D pose space in which a
hypergaussian is obtained. Afterwards anthropomorphism is used as a criteria to discern
the most realistic pose obtained from sampling from the hypotheses. The advantage ofthis formulation is in its simplicity, which allows it to be very computationally eﬃcient,5.5. Summary

103

limited only by the speed of the 2D pose estimation model. However, by being apipelined approach, the quality of the 3D output is limited by that of the 2D input.
For our second approach we employ 2D discriminative detectors directly as the model
input and then perform 2D and 3D pose estimation jointly. This is done by extending
the Bayesian formulation of pictorial structures to 3D and reformulating it as a hybrid
generative-discriminative model. For the generative model we use a Directed Acyclic
Graph, which allows probabilistic representing 3D human poses through a discrete latent
space. The discriminative model consists of reweighting 2D detectors for the 3D humanpose estimation task. We treat the inference as a global optimization problem andsolve it iteratively using state-of-the-art techniques. By performing 2D and 3D pose
estimation jointly, the 3D is able to correct errors caused by occlusions in the 2D
detections.
Our ﬁndings with both approaches show that it is indeed possible to obtain good 3D
human pose estimation results despite the complexity of the problem. This is done byleveraging the prior knowledge of human motion with generative models. Stronger priors
allow much more accurate pose estimation by eliminating non-realistic pose hypotheses.
We are also able to see that performing 2D and 3D pose estimation jointly is beneﬁcial
to performance. By delaying the pose estimation decision to the end and leveraging all
the available information, the noise in the 2D part detectors is minimized.
In the next chapter we will show that the output of pose estimators can be used tocreate strong features when performing higher order reasoning.

Chapter 6

Probabilistically Modelling Fashion

The finest clothing made is aperson’s skin, but, of course,
society demands something morethan this.
Mark Twain

In this chapter we will discuss higher level computer vision tasks in a fashion context.
These high level task push the boundaries of what is possible with modern computer
vision and machine learning, while being the contributions that are able to inﬂuencepeople in their daily life.
In particular we shall focus on two tasks: semantic segmentation of clothings infashion photographs, and prediction of fashionability while providing fashion advice.
The ﬁrst task is based on a standard problem in computer vision, however, the application domain is vastly diﬀerent. In particular the large number of classes with highintra-class variability in combination with the ﬁne-grained level of annotations make it
an extremely challenging problem. In our second task we tackle the novel problem ofpredicting fashionability of an individual in an image with some associated metadata.
We also take it a step further and attempt to give advice to the user on which outﬁts
would suit her/him better.
All the work in this chapter is a result of collaborating with Prof. Raquel Urtasun
and Prof. Sanja Fidler, from the University of Toronto.

6.1

Introduction

Fashion has a tremendous impact on our society. Clothing typically reﬂects the person’ssocial status and thus puts pressure on how to dress to ﬁt a particular occasion. Its
importance becomes even more pronounced due to online social sites like Facebook
and Instagram where one’s photographs are shared with the world. We also live ina technological era where a signiﬁcant portion of the population looks for their dream
partner on online dating sites. People want to look good; business or casual, elegant
105

106

Chapter 6. Probabilistically Modelling Fashion
Input

Truth

(Yamaguchi et al., 2012)

Ours

Figure 6.1: Garment segmentation example. We show our approach in a scenario
where no a priori knowledge of which garments are worn is provided. We compare
against state-of-the-art. Despite some mistakes, our result looks visually much morenatural than the competing method.

or sporty, sexy but not slutty, and of course trendy, particularly so when putting their
picture online. This is reﬂected in the growing online retail sales, reaching 370 billion
dollars in the US by 2017, and 191 billion euros in Europe (Forbes Magazine, 2013).
Computer vision researchers have started to be interested in the subject due tothe high impact of the application domain (Bossard et al., 2012; Bourdev et al., 2011;
Chen et al., 2012; Gallagher and Chen, 2008; Hasan and Hogg, 2010; Jammalamadaka
et al., 2013; Liu et al., 2012b; Wang and Ai, 2011). The main focus has been to inferclothing from photographs. This can enable a variety of applications such as virtual
garments in online shopping. Being able to automatically parse clothing is also key
in order to conduct large-scale sociological studies related to family income or urbangroups (Murillo et al., 2012; Song et al., 2011).
In the context of fashion, Yamaguchi et al. (Yamaguchi et al., 2012), created Fashionista, a dataset of images and clothing segmentation labels. Great performance wasobtained when the system was given information about which garment classes, but nottheir location, are present for each test image. Unfortunately, the performance of thestate-of-the-art methods (Yamaguchi et al., 2012; Jammalamadaka et al., 2013) is rather
poor when this kind of information is not provided at test time. This has been very
recently partially addressed in (Yamaguchi et al., 2013) by utilizing over 300,000 weaklylabeled images, where the weak annotations are in the form of image-level tags. In
this chapter, we show an approach which outperforms the state-of-the-art signiﬁcantly
without requiring these additional annotations, by exploiting the speciﬁc domain of thetask: clothing a person. An example of our result can be seen in Fig. 6.1.
In this chapter we also present a second task in which our goal is to predict how
fashionable a person looks on a particular photograph. The fashionability is aﬀected by
the garments the subject is wearing, but also by a large number of other factors such as
how appealing the scene behind the person is, how the image was taken, how visuallyappealing the person is, her/his age, etc. The garment itself being fashionable is also
not a perfect indicator of someone’s fashionability as people typically also judge how
well the garments align with someone’s “look”, body characteristics, or even personality.

6.2. Segmentation of Garments in Fashion Images

107

backgroundheels

blazerstockings

blousehat

hairwedgescardigantights

top

purse

skin

shoes

jumpershorts

skirt

belt

Figure 6.2: Fine-grained annotations of the Fashionista dataset (Yamaguchi
et al., 2012). Many of the diﬀerent classes are very diﬃcult to distinguish even forhumans. Observe the subtle diﬀerences between some classes such as footwear (heels,wedges, and shoes), blazer and cardigan, or stockings and tights. We also point out that
this dataset has been annotated via superpixels, and thus the ground truth contains
errors when superpixels do not align with the actual garments. We have not modiﬁed
the ground truth segmentation in any way.

6.2

Segmentation of Garments in Fashion Images

The complexity of the task of human semantic segmentation comes from the inherent
variability of pose and cloth appearances, the presence of self-occlusions as well as thepotentially large number of classes. Consider for example Fig. 6.2: an autonomous
system needs to distinguish between blazers and cardigans, stockings and tights, andheels, wedges and shoes, where the intra-class variability is fundamentally much larger
than the inter-class variability. This ﬁne-grained categorization is diﬃcult to resolve
even for humans who are not familiar with the fashion industry. The problem is further
aggravated by the power law distribution of classes, as certain categories have very few
examples. Thus, extra-care has to be taken into account to not over-predict the classes
that are very likely to appear in each image, e.g., skin, hair.
In this section we address some of these challenges and formulate the problem as
the one of inference in a Conditional Random Field (CRF), which takes into accountthe complex dependencies between clothing and human pose. Speciﬁcally, we develop arich set of potentials which encode the person’s global appearance and shape to perform
ﬁgure/ground segmentation, shape and location likelihoods for each garment, which wecall clothelets, and long-range similarity between segments to encourage, for example,
T-Shirt pixels on the body to agree with the T-shirt pixels on the person’s arm. We
further exploit the fact the people are symmetric and dress as such as well by introducing
symmetry-based potentials between diﬀerent limbs. We also use a variety of diﬀerent
local features encoding cloth appearance as well as local shape of the person’s parts. We
demonstrate the eﬀectiveness of our approach on the Fashionista dataset (Yamaguchi
et al., 2012) and show that our approach signiﬁcantly outperforms the existing stateof-the-art.

108

Chapter 6. Probabilistically Modelling Fashion

Related Work
There has been a growing interest in recognizing outﬁts and clothing from still images.
One of the ﬁrst approaches on the subject was Chen et al. (Chen et al., 2006), which
manually built a composite clothing model, that was then matched to input images.
This has led to more recent applications for learning semantic clothing attributes (Chenet al., 2012), which are in turn used for describing and recognizing the identity of individuals (Bourdev et al., 2011; Gallagher and Chen, 2008), their style (Bossard et al., 2012),
and performing sociological studies such as predicting the occupation (Song et al., 2011)
or urban tribes (Murillo et al., 2012). Other tasks like outﬁt recommendations (Liuet al., 2012a) have also been investigated. However, in general, these approaches do
not perform accurate segmentation of clothing, which is the goal of our approach. Instead, they rely on more coarse features such as bounding boxes and focus on producing
generic outputs based on the presence/absence of a speciﬁc type of outﬁt. It is likelythat the performance of such systems would improve if accurate clothing segmentation
would be possible.
Recent advances in 2D pose estimation (Bourdev and Malik, 2009; Yang and Ramanan, 2011) have enabled a more advanced segmentation of humans (Dong et al.,2013). However, most approaches have focused on ﬁgure/ground labeling (Ladicky
et al., 2013; Wang and Koller, 2011). Additionally, pose information has been used asa feature in clothing related tasks such as ﬁnding similar worn outﬁts in the context of
online shopping (Liu et al., 2012b).
Segmentation and classiﬁcation of garments has been addressed in the restrictive
case in which the labels are known beforehand (Yamaguchi et al., 2012). The original
paper tackled this problem in the context of fashion photographs which depicted one
person typically in an upright pose. This scenario has also been extended to the casewhere more than one individual can be present in the image (Jammalamadaka et al.,2013). In order to perform the segmentation, conditional random ﬁelds are used withpotentials linking clothing and pose. However, the performance of these approaches
drops signiﬁcantly when no information about the outﬁt is known a priori (i.e., notags are provided at test time). The paper doll approach (Yamaguchi et al., 2013) usesover 300,000 weakly labeled training images and a small set of fully labeled examples
in order to enrich the model of (Yamaguchi et al., 2012) with a prior over image labels.
As we will show in the experimental evaluation, our method can handle this scenario
without having to resort to additional training images. Furthermore, it consistently
outperforms (Yamaguchi et al., 2012, 2013).
CRFs have been very successful in semantic segmentation tasks. Most approaches
combine detection and segmentation by using detectors as additional image evidence (Yaoet al., 2012b; Fidler et al., 2013). Co-occurrence potentials have been employed to enforce consistency among region labels (Ladicky et al., 2010). Part-based detectors havealso been aligned to image contours to aid in object segmentation (Brox et al., 2011).
All these strategies have been applied to generic segmentation problems, where one isinterested in segmenting classes such as car, sky or trees. Pixel-wise labeling of clothingis, however, a much more concrete task, where strong domain speciﬁc information, suchas 2D body pose, can be used to reduce ambiguities.

6.2. Segmentation of Garments in Fashion Images

109

Type

Name

Description

unary
unary
unary
unary
unary

Simple features (φsimple(yi ))
i,jObject mask (φobj
(y
))
i
i,j
Clothelets (φcloth
(y
))
i
i,j
o2p
Ranking (φi,j(yi ))(y)
and φbias
Bias (φbiasip,j (lp ))jAssortment of simple features.
Figure/ground segmentation ask.
Pose-conditioned garment likelihood masks.
Rich set of region ranking features.
Class biases.

pairwisepairwise

Similarity (φsimilm,n (ym , yn ))Compatibility (φcompi,p (yi , lp ))Similarity between superpixels.
Edges between limb segments and superpixels.

Table 6.1: Overview of the different types of potentials used in the proposed
CRF model.

Clothing a Person
Our proposed Conditional Random Field (CRF) takes into account complex dependencies that exist between garments and human pose. We obtain pose by employing a 2D
articulated model by Yang et al. (Yang and Ramanan, 2011) which predicts the main
keypoints such as head, shoulders, knees, etc. As (Yamaguchi et al., 2012), we will
exploit these keypoints to bias the clothing labeling in a plausible way (e.g., a hat is
typically on the head and not the feet). To manage the complexity of the segmentation
problem we represent each input image with a small number of superpixels (P.Arbelaez et al., 2011). Our CRF contains a variable encoding the garment class (including
background) for each superpixel. We also add limb variables which encode the garment
associated with a limb in the human body and correspond to edges in the 2D articulated
model. We use the limb variables to propagate information while being computationallyeﬃcient.
Our CRF contains a rich set of potentials which exploit the domain of the task. We
use the person’s global appearance and shape to perform ﬁgure/ground segmentationin order to narrow down the scope of cloth labeling. We further use shape and location
likelihoods for each garment, which we call clothelets. We exploit the fact that people
are symmetric and typically dress as such by forming long-range consistency potentials
between detected symmetric keypoints of the human pose. We ﬁnally also use a variety
of diﬀerent features that encode appearance as well as local shape of superpixels.
We will now discuss the diﬀerent potentials in the model. All our potentials belong
to the exponential function family, however, for notation simplicity we will write thelogarithm of the potentials and not make explicit the dependency with the observed
variables.
Pose-aware Model
Given an input image represented with superpixels, our goal is to assign a clothing label(or background) to each of them.
More formally, let yi ∈ {1, · · · , C} be the class associated with the i-th superpixel,
and let lp be the p-th limb segment deﬁned by the edges in the articulated body model.
Each limb lp is assumed to belong to one class, lp ∈ {1, · · · , C}. To encode bodysymmetries in an eﬃcient manner, we share limb variables between the left and right

110

Chapter 6. Probabilistically Modelling Fashion
Image

Person Mask

Limbs

Image

Person Mask

Limbs

Figure 6.3: Visualization of object masks (Carreira and Sminchisescu, 2012)and limbs (obtained via (Yang and Ramanan, 2011)). Note that CPMC typicallygenerates high quality results, e.g. the one in the left image, but can also completely
miss large parts of the body as shown in the image on the right.
part of the human body, e.g., the left and the right leg share the same limb variables.
We propose several domain inspired potentials, the overview of which is presented inTable 6.1. We emphasize that the weights associated with each potential in our CRF
will be learned using structure prediction. We now explain each potential in more detail.
Simple Features: Following (Yamaguchi et al., 2012), we concatenate a diverse setof simple local features and train a logistic regression classiﬁer for each class. In particular, we use colour features, normalized histograms of RGB and CIE L*a*b* colour;
texture features, Gabor ﬁlter responses; and location features: both normalized 2D image coordinates and pose-relative coordinates. The output of the logistic functions arethen used as unary features in the CRF. This results in a unary potential with as many
dimensions as classes:(
wjsimple σjsimple (fi ),if yi = j,
(6.1)
φsimple
(y
)
=
i
i,j
0,
otherwise
where wjsimple is the weight for class j, σjsimple (fi ) is the score of the classiﬁer for classj, and fi is the concatenation of all the features for superpixel i. Notice we have used
C diﬀerent unary potentials, one for each class. By doing this, we allow the weights ofa variety of potentials and classes be jointly learned within the model.
Figure/Ground Segmentation: To facilitate clothing parsing we additionally compute how likely it is that each superpixel belongs to a person. We do this by computing a set of bottom-up region proposals using the Constrained Parametric Min-Cuts(CPMC) approach (Carreira and Sminchisescu, 2012), which repetitively solves a ﬁgure/ground energy minimization problem with diﬀerent parameters and seeds via parametric min-cuts. We take top K (we set K = 100) regions per image and use Order 2
Pooling (O2P) (Carreira et al., 2012) to score each region into ﬁgure/ground (personvs-background). Since we know that there is a person in each image, we take at least
the top scoring segment per image, no matter its score. For images with multiple highscoring segments, we take the union of all segments with scores higher than a learned
threshold (Carreira et al., 2012). We deﬁne a unary potential to encourage the superpixels that lie inside the foreground mask to take any of the clothing labels (and not6.2. Segmentation of Garments in Fashion Images
Image

Background

Skin

Socks

111
JacketBag

Figure 6.4: Visualization of clothelets. We show several clothelets for two diﬀerent
input images.
background):φobji,j (yi ) =(w1cmpc σ cpmc · |¬Mf g ∩ Si |/|Si |,wjcpmc σ cpmc · |Mf g ∩ Si |/|Si |,if yi = 1otherwise

,

(6.2)

where yi = 1 encodes the background class, wjcpmc is the weight for class j, σ cpmc is thescore of the foreground region, Si , Mf g are binary masks deﬁning the superpixel andforeground, respectively, and ¬Mf g is a mask of all pixels not in foreground. Fig. 6.3
shows examples of masks obtained by (Carreira et al., 2012). Note that while in somecases it produces very accurate results, in others, it performs poorly. These inaccurate
masks are compensated by other potentials.
Clothelets: Our next potential exploits the statistical dependency between the location on the human body and garment type. Its goal is to make use of the fact thate.g. jeans typically cover the legs and not the head. We compute a likelihood of eachgarment appearing in a particular relative location of the human pose. In particular,
for each training example we take a region around the location of each joint (and limb),
the size of which corresponds to the size of the joint part template encoded in (Yangand Ramanan, 2011). We average the GT segmentation masks for each class across thetraining examples. In order to capture garment classes that stray away from the pose,
we use boxes that are larger than the part templates in (Yang and Ramanan, 2011).
At test time, the masks for each class are overlaid relative to the inferred pose and normalized by the number of non-zero elements. Areas with no information are assigned
to the background class. The potential is then deﬁned as(
wjcloth (clotheletji · Si )/|Si |,if yi = j
cloth
,
(6.3)
φi,j (yi ) =0,otherwise

112

Chapter 6. Probabilistically Modelling Fashion

where wjcloth is the weight for class j, clotheletji is the clothelet for the j-th class, and ·is the dot product. Fig. 6.4 depicts clothelets for a few sample classes. Note that theselikelihoods are conditioned on the pose.
Shape Features: This potential uses a set of rich features that exploit both the shapeand local appearance of garments. In particular, we use eSIFT and eMSIFT proposedby (Carreira et al., 2012) for region description. Given a region, both descriptors extract
SIFT inside the region and enrich it with the relative location and scale within theregion. Second-order pooling is used to deﬁne the ﬁnal region descriptor. eSIFT andeMSIFT diﬀer slightly in how the descriptors are pooled, eSIFT pools over both theregion and background of the region, while eMSIFT pools over the region alone.
While (Carreira et al., 2012) deﬁnes the features over full object proposals, here
we compute them over each superpixel. As such, they capture more local shape of thepart/limb and local appearance of the garment. We train a logistic classiﬁer for eachtype of feature and class and use the output as our potential:
φo2pi,j (yi ) =(wjo2p σjo2p (ri ),0,

if yi = jotherwise

,

(6.4)

with wjo2p is the weight for class j, σjo2p (ri ) the classiﬁer score for class j, and ri thefeature vector for superpixel i.
Bias: We use a simple bias for the background to encode the fact that it is the classthat appears more frequently. Learning a weight for this bias is equivalent to learninga threshold for the foreground, however within the full model. Thus:φbias(yi ) =(

wbias ,
0,if yi = 1otherwise

,

(6.5)

where wbias is the single bias weight.
Similarity: In CRFs, neighboring superpixels are typically connected via a (contrastsensitive) Potts model encouraging smoothness of the labels. For clothing parsing, wewant these connections to act on a longer range. That is, a jacket is typically split
in multiple disconnected segments due to a T-shirt, tie, and/or a bag. Our goal is toencourage superpixels that are similar in appearance to agree on the label, even though
they may not be neighbors in the image.
We follow (Uijlings et al., 2013) and use size similarity, ﬁt similarity that measures
how well two superpixels ﬁt each other; and colour and texture similarity, with thetotal of 12 similarity features between each pair of superpixels. We then train a logistic
regression to predict if two superpixels should have the same label or not. In order toavoid setting connections on the background, we only connect superpixels that overlap
with the bounding box of the 2D pose detection. Note that connecting all pairs ofsimilar superpixels would slow down inference considerably. To alleviate this problem,
we compute the minimum spanning tree using the similarity matrix and use the top 10
edges to connect 10 pairs of superpixels in each image. We form a pairwise potential6.2. Segmentation of Garments in Fashion Images

113

between each connected pair:φsimilm,n (ym , yn ) =(simil ,
wisimil σm,n0,

if ym = yn = iotherwise

,

(6.6)

simil is the output of the similarity classiﬁer.
where wisimil is the weight for class i, and σm,nLimb Segment Bias: We use a per-class bias on each limb segment to capture alocation speciﬁc bias, e.g., hat only appears in the head:(bias ,
wp,j
if lp = j
(l
)
=
,
(6.7)
φbias
p
p,j
0,
otherwisebias is a weight for part limb p and class j that allows us to compute which
where wp,jclasses are more frequent in each limb.

Compatibility Segmentation-Limbs: We deﬁne potentials connecting limb segments with nearby superpixels encouraging them to agree in their labels. Towards thisgoal, we ﬁrst deﬁne a Gaussian mask centered between two joints. More formally, for
two consecutive joints with coordinates Ja = (ua , va ) and Jb = (ub , vb ), we deﬁne themask based on the following Normal distribution:!
!
q1 kJa − Jb k 0
Ja + Jb
T
, R
R
,
(6.8)
M (Ja , Jb ) = N
2
0
q2
−ub
where R is a 2D rotation matrix with an angle arctan( uvaa −v), and q1 and q2 are twob
hyperparameters controlling the spread of the mask longitudinally and transversely,
respectively. In order to additionally encode symmetry, a single limb segment is used
for both left and right sides of the body, e.g., left and right shins will share the same
keypoint. The strength of the connection is based on the overlap between the superpixels
and the Gaussian mask:
(
comp
wj,p
M (Ja , Jb ) · Si ,if yi 6= 1 and yi = kp = j
comp
,
(6.9)
φi,p (yi , lp ) =0,otherwisecomp
where wi,pis the weight for part p and class j. For computational eﬃciency, edges
with connection strengths below a threshold are not set in the model. Some examplesof the limb segment masks are shown in Fig. 6.3. We can see the masks ﬁt the bodytightly to avoid overlapping with background superpixels.

Full Model: We deﬁne the energy of the full model to be the sum of three types ofenergies encoding unary and pairwise potentials that depend on the superpixel labeling,
as well as an energy term linking the limb segments and the superpixels:
E(y, l) = Eunary (y) + Esimilarity (y) + Elimbs (y, l) .

(6.10)

This energy is maximized during inference. The unary terms are formed by the concatenation of appearance features, ﬁgure/ground segmentation, clothelets, shape features

114

Chapter 6. Probabilistically Modelling Fashion
1

Clothing classes
jumperjumperjumperjumperjumperjumperjumper

0.9

0.8

0.7

0.6

0.5

0.4

0.3

vs
vs
vs
vs
vs
vs
vs

Similarity (dlch )sweatersweatshirt
top
stockings
heels
sunglasses
watch

1.000
0.697
0.557
0.507
0.465
0.341
0.317

Figure 6.5: Wordnet-based similarity function. We use the inverse of the
Leacock-Chodorow Similarity between the classes in the Fashionista dataset to weighour loss function. We display the similarity matrix between all the classes on the left.
Some individual values of the similarity between the jumper class and several otherclasses can be seen on the right.
and background bias for a total of K = (1 + 5C) features
Eunary (y) =

N X
K
X

φunary(yi ) ,
i,j

(6.11)

i=1 j=1

where N is the number of superpixels. The pairwise features encode the similarity
between diﬀerent pairs of superpixels as we describe aboveXφsimil(6.12)
Esimilarity (y) =m,n (ym , yn ) .
(m,n)∈pairs

The limb-superpixel compatibility term is deﬁned as

!
M
C
N
X
X
X

Elimbs (y, l) =φbias
φcomp (yi , lp )  ,p,j (lp ) +
p=1

j=1

(6.13)

i=1

for a total of (M + C) features, with M the number of limb segments.
Learning and Inference

Our model is a multi-label CRF which contains cycles and thus inference is NP-hard.
We use a message passing algorithm, distributed convex belief propagation (Schwing
et al., 2011) to perform inference. It belongs to the set of LP-relaxation approaches, andhas convergence guarantees. This is not the case in other message passing algorithms
such as loopy-BP.
To learn the weights, we use the primal-dual method of (Hazan and Urtasun, 2010)(we use the implementation of (Schwing et al., 2012)), shown to be more eﬃcient thanother structure prediction learning algorithms.
As loss-function, we use the semantic similarity between the diﬀerent classes in orderto penalize mistakes between unrelated classes more than similar ones. We do this6.2. Segmentation of Garments in Fashion Images

115

29 Classes

56 Classes

Method

Y2012

Ours

Y2012

Y2013

Ours

Jaccard index

12.32

20.52

7.22

9.22

12.28

Table 6.2: Comparison against the state-of-the-art. We compare on two diﬀerent
datasets: Fashionista v0.2 with 56 classes and Fashionista v0.3 with 29 classes, against
Y2012 (Yamaguchi et al., 2012) and Y2013 (Yamaguchi et al., 2013).
via Wordnet (Miller, 1995), which is a large lexical database in which sets of cognitivesynonyms (synsets) are interlinked by means of semantic and lexical relationships. We
can unambiguously identify each of the classes with a single synset, and then proceed to
calculate similarity scores between these synsets that represent the semantic similarity
between the classes, in order to penalize mistakes with dissimilar classes more.
In particular, we choose the corpus-independent Leacock-Chodorow Similarity score.
This score takes into account the shortest path length p between both synsets and the
maximum depth of the taxonomy d at which they occur. It is deﬁned as the relationship− log(p/2d). A visualization of the dissimilarity between all the classes in the dataset
can be seen in Fig. 6.5. We therefore deﬁne the loss-function as:(0,
if yi = yi∗y∗(6.14)∆ (yi , yi ) =dlch (yi , yi∗ ),otherwisewith dlch (·, ·) being the inverse Leacock-Chodorow Similarity score between both classes.
For the limb segments we use a 0-1 loss:(
0,
if ki = ki∗(6.15)∆k (ki , ki∗ ) =1,otherwise

Experimental Evaluation
We evaluate our approach on both a the Fashionista dataset v0.3 (Yamaguchi et al.,2012), and the setting of (Yamaguchi et al., 2013) with the Fashionista dataset v0.2.
Both datasets are taken from chictopia.com in which a single person appears wearing
a diverse set of garments. The dataset provides both annotated superpixels as well
as 2D pose annotations. A set of evaluation metrics and the full source code of approaches (Yamaguchi et al., 2012, 2013) are provided. Version 0.2 has 685 images andv0.3 has 700 images. Note that v0.3 is not a superset of v0.2.
We have modiﬁed the Fashionista v0.3 dataset in two ways. First we have compressed the original 54 classes into 29. This is due to the fact that many classes thatappear have very few occurrences. In fact, in the original dataset, 13 classes have 10
or fewer examples and 6 classes have 3 or fewer instances. This means that when performing a random split of the samples into training and test subsets, there is a highprobability that some classes will only appear in one of the subsets. We therefore compress the classes by considering both semantic similarity and the number of instances.

116

0.0
.00
.01
0.0
0.0
.00
.01
.01
.04
.02
.01
.01
0.0
.00
0.0
.01
.00
0.0
0.0
.01
0.0
.01
.07
.09
.00
.03
.14
.12
0.0

0.0
0.0
0.0
.01
.04
.03
0.0
0.0
0.0
0.0
.01
.00
.00
.01
.00
.03
.01
.02
.00
.01
.01
.01
0.0
.00
.02
.02
.01
0.0
.60

bo

pu

ba

ha

dr

up

sh

sw

p
to

ja

ve

bl

je

le

pa

sh

sk

oc
st

tig

be

ts

gs

gs

in

s

se

et

r
te

y

od

ar

rb

s

e

s

pe

lt

0.0
0.0
.01
0.0
0.0
.00
.01
.03
.05
.02
0.0
.02
0.0
.00
.00
0.0
0.0
0.0
0.0
0.0
0.0
.02
.05
.01
0.0
0.0
.17
.10
0.0

kin

0.0
0.0
.00
0.0
0.0
.00
0.0
0.0
0.0
.01
.02
.08
0.0
.17
.01
.01
.03
.01
0.0
.00
.03
0.0
0.0
.02
.10
.20
.00
.00
.02

s
ht

0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
.01
.01
0.0
.01
.00
.03
.01
.03
.00
.00
.00
.03
.00
.00
.47
.09
0.0
0.0
.00
or

.00
.00
.00
0.0
.01
.04
.01
.01
.00
.01
0.0
.02
0.0
.05
.09
.02
.16
0.0
.02
0.0
0.0
.24
.09
.27
.02
.06
.07
.16
.02

irt

0.0
.00
0.0
0.0
0.0
.03
0.0
.02
0.0
.06
0.0
.00
0.0
.01
.00
0.0
0.0
.03
.00
.01
0.0
.15
.17
.10
.02
.08
.06
.07
.00

s
nt

.00
0.0
.00
0.0
0.0
.01
.00
.01
.00
.08
0.0
.02
0.0
.01
.02
.00
.00
0.0
0.0
0.0
.01
.37
.12
.25
.02
.03
.03
.04
.00

gg

0.0
.01
.00
0.0
.00
.02
0.0
0.0
0.0
0.0
.00
.00
0.0
.02
.04
.05
.05
.06
.01
.00
.09
0.0
0.0
0.0
0.0
.01
0.0
0.0
.02

an

0.0
.02
0.0
0.0
.00
.07
0.0
.00
0.0
0.0
0.0
.03
.01
.04
.05
.01
.01
.03
.07
.08
.00
0.0
0.0
.00
.00
.00
0.0
0.0
0.0

ou

.00
.04
.00
.02
.03
.16
0.0
0.0
0.0
0.0
.05
.02
.00
.05
.11
.11
.04
.07
.31
.23
.09
.00
0.0
0.0
.00
.02
0.0
0.0
.02

st

0.0
.00
.00
0.0
.05
.07
0.0
0.0
0.0
0.0
.01
.00
0.0
.03
.02
.12
.04
.19
.03
.04
.15
.00
0.0
0.0
.03
.01
0.0
0.0
.04

ck

0.0
.00
.00
0.0
.00
.04
0.0
0.0
0.0
0.0
0.0
.02
.00
.04
.05
.04
.07
.07
.03
.02
.05
.00
0.0
0.0
.00
.05
0.0
0.0
.02
ea

0.0
.00
.01
0.0
.02
.02
0.0
0.0
0.0
0.0
.04
.01
0.0
.07
.02
.27
.10
.15
.05
.18
.13
.01
.00
.02
.03
.04
0.0
0.0
.01
irt

.00
.05
.02
.00
.04
.10
0.0
0.0
0.0
0.0
.05
.04
0.0
.05
.37
.08
.20
.06
.29
.11
.11
.01
0.0
.00
.02
.03
0.0
0.0
.02

t

.00
.00
.01
0.0
.12
.07
0.0
0.0
0.0
0.0
.05
.05
0.0
.26
.04
.09
.13
.08
.03
.16
.14
.05
.02
.01
.22
.09
.06
0.0
.11
es

0.0
.08
.00
.04
.02
0.0
0.0
0.0
0.0
0.0
0.0
0.0
.70
0.0
.01
0.0
.00
.00
.00
.00
0.0
0.0
0.0
.04
.00
0.0
0.0
0.0
0.0

g

.00
.00
.00
0.0
.03
.02
.00
.02
0.0
.01
.31
.27
0.0
.05
.01
.00
.00
.01
.01
.02
.00
.02
.03
.02
.02
.05
.01
.01
.01

rs

0.0
.00
.00
0.0
.02
.00
0.0
.02
0.0
0.0
.10
.11
0.0
.04
.03
.00
0.0
.01
.01
.02
.02
.00
.02
.00
0.0
.02
0.0
.00
.02

s
ot

.00
0.0
.00
0.0
.06
.00
.03
.11
.09
.35
.03
.03
0.0
.00
.01
.00
0.0
0.0
.00
.00
.00
.03
.02
.01
0.0
.05
.16
.05
0.0

ck

.00
0.0
.00
0.0
.05
0.0
.04
.07
.66
.09
.00
.01
0.0
0.0
.00
.00
.01
.02
0.0
0.0
.00
.04
.29
.04
0.0
0.0
.18
.21
.00
so

s

e
tw

s

ts

en

ie

rm

or

ga

s

ss

se

oe

sh

ac

isc

gl

.00
.00
.01
0.0
.00
.01
.30
.35
.07
.15
0.0
.01
0.0
.00
.00
.00
.01
0.0
.00
.00
0.0
.01
.01
.00
0.0
.01
.01
.00
0.0

o
fo

sk

.00
.00
.02
0.0
.01
.00
.43
.19
.02
.08
.01
.01
0.0
.00
.00
.00
.01
.00
.00
0.0
.00
.00
0.0
.01
.00
.00
.00
.00
.01

m

ha

0.0
.01
.00
.01
.07
.06
.00
0.0
0.0
0.0
.01
.01
0.0
.03
.02
.04
.03
.07
.03
.04
.05
0.0
0.0
.00
0.0
.01
.00
0.0
.00

ce

0.0
.00
.00
.00
.14
.00
0.0
0.0
0.0
0.0
0.0
.00
0.0
.01
.00
.00
.00
.00
.00
.00
.01
.00
0.0
0.0
0.0
0.0
0.0
0.0
.01

as

0.0
.00
.00
.63
.01
.01
0.0
0.0
0.0
0.0
0.0
0.0
.00
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
0.0
.01
0.0
0.0
0.0
0.0

ir

.00
.05
.82
.14
.15
.04
.04
.05
.04
.03
.11
.07
.03
.03
.05
.04
.05
.08
.02
.04
.02
.01
.03
.08
.02
.05
.09
.13
.03
in

.00
.67
.02
.14
.02
.15
0.0
.00
0.0
.00
.00
.00
.11
.01
.02
.02
.01
.02
.02
.01
.03
0.0
0.0
0.0
0.0
0.0
0.0
0.0
.01

ll

.97
.03
.04
.01
.10
.01
.12
.10
.01
.08
.17
.14
.13
.01
.01
.02
.02
.01
.03
.01
.03
.01
.08
.01
.00
.04
.01
.09
.02
nu

null
hair
skin
glasses
accessories
misc garments
footwear
shoes
socks
boots
purse
bag
hat
dress
upper body
shirt
sweater
top
jacketvest
blousejeans
leggings
pants
shortsskirt
stockings
tightsbelt

Chapter 6. Probabilistically Modelling Fashion

Figure 6.6: Confusion matrix for our approach on the Fashionista v0.3
dataset.

For evaluation on Fashionista v0.3 we consider a random 50-50 train-test split. As
previously stated, we do not have information about which classes are present in thescene. We employ the publicly available code of (Yamaguchi et al., 2012) as the baseline.
We evaluate on Fashionista v0.2 according to the methodology in (Yamaguchi et al.,2013). This consists of a split with 456 images for training and 229 images for testing.
Note that (Yamaguchi et al., 2013) uses 339,797 additional weakly labeled images fromthe Paper doll dataset for training, which we do not use.
Following PASCAL VOC, we report the average class intersection over union (Jaccard index). This metric is the most similar to human perception as it considers alltrue positives, true negatives and false positives. It is nowadays a standard measure
to evaluate segmentation and detection (Everingham et al., 2010; Geiger et al., 2013;
Deng et al., 2009).

Comparison to State-of-the-Art: We compare our approach against (Yamaguchi
et al., 2012, 2013). The approach of (Yamaguchi et al., 2012) uses a CRF with very
simple features. We adapt the code to run in the setting in which the labels that appear in the image are not known a priori. Note also that (Yamaguchi et al., 2013) uses
a look-up approach on a separate dataset to parse the query images. The results of thecomparison can be seen in Table 6.2. Note that our approach consistently outperforms
both competing methods on both datasets, even though (Yamaguchi et al., 2013) uses
339,797 additional images for training. We roughly obtain a 60% relative improvement
on Jaccard index metric with respect to (Yamaguchi et al., 2012) and a 30% improvement over (Yamaguchi et al., 2013). The full confusion matrix of our method can beseen in Fig. 6.6. We can identify several classes that have large appearance variationand similar positions that get easily confused, such as Footwear with Shoes and Jeans
with Pants.

6.2. Segmentation of Garments in Fashion Images

117

Method

CPMC

Y2012

Clothelets

Y2013

Ours

Pixel Accuracy
Person/Bck. Accuracy

85.39

77.98
93.79

77.09
94.77

84.68
95.79

84.88
97.37

Table 6.3: Evaluation on foreground segmentation task. We compare
CPMC (Carreira et al., 2012), Y2012 (Yamaguchi et al., 2012), clothelets, Y2013 (Yamaguchi et al., 2013) and our approach for segmenting the foreground on the Fashionista
v0.2 dataset.
Method29 Classes

56 Classes

Y2012

Estimated
GT PoseNo Pose

12.32
12.39
10.54

7.22
7.41
5.22

Ours

Estimated
GT PoseNo Pose

20.52
21.01
16.56

12.28
12.46
9.64

Table 6.4: Influence of pose. We compare against Y2012 (Yamaguchi et al., 2012)
in three diﬀerent scenarios: estimated 2D pose, ground truth 2D pose and no pose
information at all.
Foreground Segmentation: We also evaluate person-background segmentation results. Note that the binary segmentation in our model is obtained by putting all foreground garment classes to the person class. In Table 6.3, we show results for both pixel
accuracy considering all the diﬀerent classes, and the two class case of foreground/background segmentation accuracy. We see that the best results are obtained by the approaches reasoning jointly about the person and clothing. Even a simple model thatuses only clothelets outperform the object mask by nearly 10% despite simply being aunary. Our approach outperforms the baseline CPMC (Carreira et al., 2012) by 12%,and achieves a 4% over (Yamaguchi et al., 2012) and 2% over (Yamaguchi et al., 2013).
Pose Influence: We next investigate the importance of having an accurate pose
estimate. Towards this goal, we analyze three diﬀerent scenarios. In the ﬁrst one, thepose is estimated by (Yang and Ramanan, 2011). The second case uses the ground-truth
pose, while the last one does not use pose information at all. As shown in Table 6.4,
the poses in this dataset are not very complex as performance does not increase greatly
when using ground truth instead of estimated pose. However, without pose information,
performance drops 20%. This shows that our model is truly pose-aware.
A breakdown of the eﬀect of pose on all the classes is shown in Fig. 6.7. Some classes
like hat, belt or boots beneﬁt greatly from pose information while others like shorts,tights or skin do not really change.

118

Chapter 6. Probabilistically Modelling Fashion

1
Ours (GT Pose)
Ours (Estimated Pose)
Ours (No Pose)

0.8
0.6
0.4
0.2
0

lt
be
s
ht
tig ings
k
oc
st
irt
sk
ts
or
sh
s
nt
pa gs
in
gg
le
s
an
je e
s
ou
bl
st
ve
et
ck
ja

p
to tereasw
irt dy
sh bo
r
pe
up
s
es
dr
t
ha

g
ba
e
rs
pu
s
ot
bo
s
ck
so
s
oe
sh ear ts
w
enot
fo arm
g
c ies
is
m sor
s
ce
ac es
s
as
gl
in
sk
ir
ha
ll
nu

Figure 6.7: Per class results for our model. We evaluate with Jaccard index
metric on Fashionista v0.3.
Threshold
Mean superpixels/image

0.16
50

0.10
120

0.05
290

Jaccard index

69.44

83.07

100

Table 6.5: Oracle performance for different superpixel levels. We show results
for the Fashionista v0.3 dataset.
MethodSimple features

Clothelets

eSIFT

eMSIFT

29 Classes
56 Classes

13.80
7.93

8.91
3.02

16.65
9.29

13.65
7.80

Table 6.6: Different results using only unary potentials in our model. BotheSIFT and eMSIFT are ranking features from (Carreira et al., 2012).
Oracle Performance: Unlike (Yamaguchi et al., 2012), we do not use the ﬁne levelsuperpixels, but instead use coarser superpixels to speed up learning and inference.
Table 6.5 shows that using coarser superpixels lowers the maximum achievable performance. However, by having larger areas, the local features become more discriminative.
We also note that the dataset (Yamaguchi et al., 2012) was annotated by labeling theﬁner superpixels. As some superpixels do not follow boundaries well, the ground truth
contains a large number of errors. We did not correct those, and stuck with the original
annotations.
Importance of the Features: We also evaluate the inﬂuence of every potential in ourmodel in Table 6.6. The eSIFT features obtain the best results under the Jaccard index
metric. The high performance of eSIFT can be explained by the fact that it also takes
into account the super pixel’s background, thus capturing local context of garments.
This feature alone surpasses the simple features from (Yamaguchi et al., 2012) despite6.2. Segmentation of Garments in Fashion Images

119

MethodJaccard index

Full Model

12.28

No similarity (φsimilm,n (ym , yn ))No limb segments (φcompi,p (yi , lp ))No simple features (φsimple(yi ))
i,jcloth
No clothelets (φi,j (yi ))No object mask (φobji,j (yi ))

11.64
12.24
10.07
11.94
10.02

o2p(eSIF T )No eSIFT (φi,j(yi ))o2p(eM SIF T )No eMSIFT (φi,j(yi ))

10.70
12.25

Table 6.7: Importance of the different potentials in the full model. We
evaluate the model after removing individually all the diﬀerent potentials to evaluate
their usefulness. We consider the 56 class setting.

that it does not use pose information. By combining all the features we are able toimprove the results greatly. We show some qualitative examples of the diﬀerent feature
activations in Fig. 6.8. We also evaluate the model in a leave-one-out fashion. That is,for each unary we evaluate the rest of the unaries in the model without it. Results areshown in Table 6.7.

Qualitative Results: We show qualitative results for both our approach and thecurrent state-of-the-art in Fig. 6.9. We can see a visible improvement over (Yamaguchi
et al., 2012), especially on person/background classiﬁcation due to the strength of the
proposed clothelets and person masks which in combination give strong cues on person
segmentation. There are also several failure cases of our algorithm. One of the main
failure cases is a breakdown of the superpixels caused by clothing texture. An excess
of texture leads to an oversegmentation where the individual superpixels are no longer
discriminative enough to individually identify (Fig. 6.9-bottom-left), while too muchsimilarity with the background leads to very large superpixels that mix foregroundand background. Additionally it can be seen that failures in pose detection can lead tomissed limbs (Fig. 6.9-bottom-right).

Computation Time: Our full model takes several hours to train and roughly 20
minutes to evaluate on the full test set (excluding feature computation), on a single
machine. On the same machine (Yamaguchi et al., 2012) takes more than twice thetime for inference. Additionally, (Yamaguchi et al., 2012) uses grid-search for training,which does not scale to a large amount of weights, that as we have shown, are able toprovide an increase in performance. Even with only 2 weights, (Yamaguchi et al., 2012)is several times slower to train than our model. Furthermore, (Yamaguchi et al., 2013)
reports a training time of several days in a distributed environment.

120

Chapter 6. Probabilistically Modelling Fashion
Input

Truth

Person MaskeSIFT

Similarity

Simple Feat.

Full Model

Figure 6.8: Visualization of different features. We show various feature activations for example images. For similarity we display the connections between the
superpixels. While both the simple features from (Yamaguchi et al., 2012) and eSIFT
provide decent segmentation results, they have poorly deﬁned boundaries. These arecorrected via person masks (Carreira et al., 2012) and clothelets. Further corrections
are obtained by pairwise potentials such as symmetry and similarity. These results
highlight the importance of combining complementary features. For class colours werefer to Fig. 6.9.

6.3

Modelling Fashionability

In this section we focus on predicting how fashionable a person looks on a particular
photograph. Our aim here is to give a rich feedback to the user: not only whether thephotograph is appealing or not, but also to make suggestions of what clothing or eventhe scenery the user could change in order to improve her/his look, as illustrated inFig. 6.10. We parametrize the problem with a Conditional Random Field that jointly
reasons about several important fashionability factors: the type of outﬁt and garments,
the type of user, the setting/scenery of the photograph, and fashionability of the user’s
photograph. Our model exploits several domain-inspired features, such as beauty, age
and mood inferred from the image, the scene type of the photograph, and if available,meta-data in the form of where the user is from, how many online followers she/he has,the sentiment of comments by other users, etc.
Since no dataset with such data exists, we created our own from online resources. We
collected 144,169 posts from the fashion website chictopia.com to create our Fashion144k

6.3. Modelling Fashionability
Input

Truth

121

Y2012

Ours

Input

Truth

Y2012

Ours

backgroundglassesfootwear

boots

hat

shirt

jacket

hair

jeans

shortstightsaccessories

shoes

purse

skin

sweatervest

leggings

skirt

belt

dress

misc_garments

socks

bag

upper_body

topblousepants

stockings

Figure 6.9: Qualitative clothing segmentation results. Results on Fashionista
v0.2 with 29 classes, comparing our approach with the state-of-the-art Y2012 (Yamaguchi et al., 2012). In the top four rows we show good results obtained by our model.
In the bottom row we show failure cases that are in general caused by 2D pose estimation failure, superpixel failures or chain failures of too many potentials.

dataset. In a post, a user publishes a photograph of her/himself wearing a new outﬁt,typically with a visually appealing scenery behind the user. Each post also contains
text in the form of descriptions and garment tags, as well as other users’ comments. It
also contains votes or “likes” which we use as a proxy for fashionability. We refer the122

Chapter 6. Probabilistically Modelling Fashion
Prediction
Claustrophobic Setting
User Cluster 20
Brown/Blue Jacket (2)Recommendation
Black Casual (7)Black Boots/Tights (4)Black/Blue Going out (3)Figure 6.10: Example of an outfit recommendation. We show how our model isable to provide outﬁt recommendations for hte post on the left. In this case the user
is wearing what we have identiﬁed as “Brown/Blue Jacket”. This photograph obtains ascore of 2 out of 10 in fashionability. Additionally the user is classiﬁed as belonging tocluster 20 and took a picture in the “Claustrophobic” setting. If the user were to wear
a “Black Casual” outﬁt as seen on the right, our model predicts she would improve her
fashionability to 7 out of 10. This prediction is conditioned on the user, setting andother factors allowing the recommendations to be tailored to each particular user.
LOS ANGELES, CANOVEMBER 10, 2014

466 FANS
288 VOTES
62 FAVOURITESGARMENTSTAGSCHICEVERDAYFALL

COLOURS
WHITE-BOOTS

White Cheap Monday Boots
Chilli Beans Sunglasses
Missguided Romper
Daniel Wellington Watch

COMMENTS
Nice!!
Love the top!
cute

...

Figure 6.11: Anatomy of a post from the Fashion144k dataset. It consists
always of at least a single image with additional metadata that can take the form oftags, list of nouns and adjectives, discrete values or arbitrary text.
reader to Fig. 6.11 for an illustration of a post.
As another contribution, we provide a detailed analysis of the data, in terms offashionability scores across the world and the types of outﬁts people in diﬀerent partsof the world wear. We also analyze outﬁt trends through the last six years of posts
spanned by our dataset. Such analysis is important for the users, as they can adapt tothe trends in “real-time” as well as to the fashion industry which can adapt their new
designs based on the popularity of garments types in diﬀerent social and age groups.

Related Work
Fashion has a high impact on our everyday lives. This also shows in the growinginterest in clothing-related applications in the vision community. Early work focusedon manually building composite clothing models to match to images (Chen et al., 2006).
In (Jammalamadaka et al., 2013; Simo-Serra et al., 2014a; Yamaguchi et al., 2013, 2012;

6.3. Modelling Fashionability

123

Yang et al., 2014), the main focus was on clothing parsing in terms of a diverse setof garment types. Most of these works follow frameworks for generic segmentation
with additional pose-informed potentials. They showed that clothing segmentation isa very challenging problem with the state-of-the-art capping at 12% accuracy (SimoSerra et al., 2014a).
More related to our line of work are recent applications such as learning semantic
clothing attributes (Chen et al., 2012), identifying people based on their outﬁts, predicting occupation (Song et al., 2011) and urban tribes (Murillo et al., 2012), outﬁt
recommendations (Liu et al., 2012a), and predicting outﬁt styles (Kiapour et al., 2014).
Most of these approaches address very speciﬁc problems with fully annotated data. In
contrast, the model we propose is more general, allowing to reason about several properties of one’s photo: the aesthetics of clothing, the scenery, the type of clothing theperson is wearing, and the overall fashionability of the photograph. We do not require
any annotated data, as all necessary information is extracted by automatically mining
a social website.
Our work is also related to the recent approaches that aim at modeling the humanperception of beauty. In (Dhar et al., 2011; Gygli et al., 2013; Isola et al., 2014; Khosla
et al., 2014) the authors addressed the question of what makes an image memorable,interesting or popular. This line of work mines large image datasets in order to correlate visual cues to popularity scores (deﬁned as e.g., the number of times a Flickr image
is viewed), or “interestingness” scores acquired from physiological studies. In our work,
we tackle the problem of predicting fashionability. We also go a step further from previous work by also identifying the high-level semantic properties that cause a particularaesthetics score, which can then be communicated back to the user to improve her/his
look. The closest to our work is (Khosla et al., 2013) which is able to infer whether aface is memorable or not, and modify it such that it becomes. The approach is howeververy diﬀerent from ours, both in the domain and in formulation.

Fashion144k Dataset
We collected a novel dataset that consists of 144,169 user posts from a clothing-orientedsocial website chictopia.com. In a post, a user publishes one to six photographs ofher/himself wearing a new outﬁt. Generally each photograph shows a diﬀerent angle ofthe user or zooms in on diﬀerent garments. Users sometimes also add a description of
the outﬁt, and/or tags of the types and colours of the garments they are wearing. Not
all users make this information available, and even if they do, the tags are usually notcomplete, i.e. not all garments are tagged. Users typically also reveal their geographic
location, which, according to our analysis, is an important factor on how fashionability is
being perceived by the visitors of the post. Other users can then view these posts, leavecomments and suggestions, give a “like” vote, tag the post as a “favorite”, or become a“follower” of the user. There are no “dislike” votes making the data challenging to workwith from the learning perspective. An example of a post can be seen in Fig. 6.11.
We parsed all the useful information of each post to create Fashion144k. The oldest
entry in our dataset dates to March 2nd in 2008, the ﬁrst post to the chictopia website.
The last crawled post is May 22nd 2014. We refer the reader to Table 6.8 for detailed
statistics of the dataset. We can see the large diversity in meta-data. Perhaps expected,
the website is dominated by female users (only 5% are male).

124

Chapter 6. Probabilistically Modelling Fashion

Property

Total

Per UserPer Post

posts
users
locations
males
fans
comments
votes
favourites
photostags
colours
garments

144169
14287
3443
5%
277537
13192
3337
-

10.09 (30.48)
116.80 (1309.29)
14.15 (15.43)
106.08 (108.34)
18.49 (22.04)
1.73 (1.00)
3.43 (0.75)
2.06 (1.82)
3.14 (1.57)

1226.60 (3769.97)
20.09 (27.51)
150.76 (129.78)
27.01 (27.81)
1.93 (1.24)
3.66 (1.12)
2.28 (2.06)
3.22 (1.72)

Table 6.8: Statistics of the Fashion144k dataset. We compute various statistics
of our dataset. We show global statistics, statistics for the diﬀerent users, and averages
for all posts.

Measuring Fashionability of a Post. Whether a person on a photograph is truly
fashionable is probably best decided by fashion experts. It is also to some extent a
matter of personal taste, and probably even depends on the nationality and the genderof the viewer. fashionability. In particular, we base our measure of interest on eachpost’s number of votes, analogous to “likes” on other websites. The main issue with votes
is the strong correlation with the time when the post was published. Since the numberof users ﬂuctuate, so does the number of votes. Furthermore, in the ﬁrst months or ayear since the website was created, the number of users (voters) was signiﬁcantly lower
than in the recent years.
As the number of votes follows a power-law distribution, we use the logarithm fora more robust measure. We additionally try to eliminate the temporal dependency by
calculating histograms of the votes for each month, and ﬁt a Gaussian distribution toit. We then bin the distribution such that the expected number of posts for each bin
is the same. By doing this we are able to eliminate almost all time dependency andobtain a quasi-equal distribution of classes, which we use as our fashionability measure,ranging from 1 (not fashionable) to 10 (very fashionable). We can see an overview of
how we obtain the fashionability metric in Fig. 6.12.
The dataset is very rich in information. In order to gain more insights we perform
some data mining. Figure 6.13 shows the number of posts and fashionability scores
mapped to the globe via the user’s geographic information. Table 6.9 reveals some of
the most trendy cities in the world, according to chictopia users and our measure.

6.3. Modelling Fashionability

125City Name

Posts

Fashionability

ManilaLos Angeles
Melbourne
Montreal
Paris
Amsterdam
Barcelona
TorontoBucharest
New York
LondonSan Francisco
MadridVancouver
Jakarta

4269
8275
1092
1129
2118
1111
1292
1471
1385
4984
3655
2880
1747
1468
1156

6.627
6.265
6.176
6.144
6.070
6.059
5.845
5.765
5.667
5.514
5.444
5.392
5.371
5.266
4.398

Table 6.9: Mean fashionability of cities with at least 1000 posts. We computethe mean fashionability of all the posts from the diﬀerent cities in the dataset. We thenshow the most fashionable cities with at least 1000 posts.

FeatureDim.

Description

Fans∆T
Comments
Location
Personal
Style
Scene
Tags
Colours
Singles
Garments

1
1
5
266
21
20
397
209
604
121
1352

Number of user’s fans.
Time between post creation and download.
Sentiment analysis (Socher et al., 2013) of comments.
Distance from location clusters (Simo-Serra et al., 2014b).
Face recognition attributes.
Style of the photography (Karayev et al., 2014).
Output of scene classiﬁer trained on (Xiao et al., 2010).
Bag-of-words with post tags.
Bag-of-words with colour tags.
Bag-of-words with split colour tags.
Bag-of-words with garment tags.

Table 6.10: Overview of the different features used.

126

Chapter 6. Probabilistically Modelling Fashion

Binned Votes

Votes
35000
20000
30000

25000

15000

20000
10000

15000

10000
5000
5000

0
0

200

400

600

800

1000

0
0

2

4

6

8

10

Non-linear
Binning

Logarithm
Logarithm of Votes

Time Normalized Votes
16000

8000

Time
Normalization

6000

14000
12000
10000

4000

8000

2000

4000

6000

2000
0
0

1

2

3

4

5

6

7

8

0

4

2

0

2

4

Figure 6.12: Normalization of votes. We objectively model fashionability using
the votes. In order to remove the temporal component of the votes we perform a nonlinear binning on the logarithm of the votes for a speciﬁc month. We note that it is not
a ﬂat distribution as we might have hoped. This is because we make an assumptionthat the votes have a Gaussian distribution, which is not entirely true.
1.0

0.9

Post Density

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

1.0

Fashionability

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

Figure 6.13: Visualization of the density of posts and fashionability by country. In the top row we show the logarithm of the number of posts per country as arepresentation of post density. In the bottom row we show the mean fashionability ofthe posts for all the countries.

Discovering Fashion from Weak Data
Our objective is not only to be able to predict fashionability of a given post, but wewant to create a model that can understand fashion at a higher level. For this purpose

6.3. Modelling Fashionability

127

Fans
Personal

Location
Scene

ΔT
Comments
Style
Tags

Colours
Singles
Garments

Figure 6.14: An overview of the CRF model and the features used by eachof the nodes.
we make use of a Conditional Random Field (CRF) to learn the diﬀerent outﬁts, typesof people and settings. Settings can be interpreted as where the post is located, bothat a scenic and geographic level. Our potentials make use of deep networks over a wide
variety of features exploiting Fashion144k images and meta-data to produce accurate
predictions of how fashionable a post is.
More formally, let u ∈ {1, · · · , NU } be a random variable capturing the type of user,o ∈ {1, · · · , NO } the type of outﬁt, and s ∈ {1, · · · , NS } the setting. Further, we denotef ∈ {1, · · · , 10} as the fashionability of a post x. We represent the energy of the CRF asa sum of energies encoding unaries for each variable as well as non-parametric pairwise
potentials which reﬂect the correlations between the diﬀerent random variables. We
thus deﬁneE(u, o, s, f ) = Euser (u) + Eout (o) + Eset (s) + Ef ash (f )
uf
of
sf
+ Enp
(u, f ) + Enp(o, f ) + Enp(s, f )
uo
so
us
+ Enp
(u, o) + Enp(s, o) + Enp(u, s)(6.16)We refer the reader to Fig. 6.14 for an illustration of the graphical model. We now
deﬁne the potentials in more detail.
User: We compute user speciﬁc features encoding the logarithm of the number of fansthat the particular user has as well as the output of a pre-trained neural network-based
face detector enhanced to predict additional face-related attributes. In particular, we
use rekognition.com which computes attributes such as ethnicity, emotions, age, beauty,
etc. We run this detector on all the images of each post and only keep the features forthe image with the highest score. We then compute our unary potentials as the output
of a small neural network with two hidden layers that takes as input the user’s highdimensional features and produces an 8D feature map φu (x). We refer the reader toFig. 6.15 for an illustration. Our user unary potentials are then deﬁned asTEuser (u = i, x) = wu,iφu (x)

with x all the information included in the post. Note that we share the features andlearn a diﬀerent weight for each user latent state.

128

Chapter 6. Probabilistically Modelling Fashion
Location

Fans

Scene

Personal

ΔT
Colours

Comments

Singles

Style

Garments

Tags
Softmax

Figure 6.15: Illustration of the type of deep network architecture used tolearn features. We can see that it consists of four network joined together by asoftmax layer. The output of the diﬀerent networks φf , φo , φu , and φs are then usedas features for the CRF.
Outfit: We use a bag-of-words approach on the “garments” and “colours” meta-dataprovided in each post. Our dictionary is composed of all words that appear at least 50
times in the training set. This results in 1352 and 604 words respectively and thus ourrepresentation is very sparse. Additionally we split the colour from the garment in the“colours” feature, e.g., red-dress becomes red and dress, and also perform bag-of-words
on this new feature. We then compute our unary potentials as the output of a small
neural network with two hidden layers that takes as input the outﬁt high dimensionalfeatures and produces an 8D feature map φo (x). We refer the reader to Fig. 6.15 for
an illustration. Our outﬁt unary potentials are then deﬁned asT
Eout (o = i, x) = wo,iφo (x)

with x all the information included in the post. Note that as with the users we sharethe features and learn a diﬀerent weight for each outﬁt latent state.
Setting: We try to capture the setting of each post by using both a pre-trained scene
classiﬁer and the user-provided location. For the scene classiﬁer we have trained amulti-layer perceptron with a single 1024 unit hidden layer and softmax layer on the
SUN Dataset (Xiao et al., 2010). We randomly use 70% of the 130,519 images as thetraining set, 10% as the validation set and 20% as the test set. We use the Caﬀe pretrained network (Jia, 2013) to obtain features for each image which we then use tolearn to identify each of the 397 classes in the dataset, corresponding to scenes such as“art_studio”, “vineyard” or “ski_slope”. The output of the 397D softmax layer is used
as a feature along with the location. As the location is written in plain text, we ﬁrst
look up the latitude and longitude. We project all these values on the unit sphere andadd some small Gaussian noise to account for the fact that many users will write moregeneric locations such as “Los Angeles" instead of the real address. We then perform
unsupervised clustering using geodesic distances (Simo-Serra et al., 2014b) and use thegeodesic distance from each cluster center as a feature. We ﬁnally compute our unarypotentials as the output of a small neural network with two hidden layers that takes
as input the settings high dimensional features and produce an 8D feature map φs (x).

6.3. Modelling Fashionability

129

Our outﬁt unary potentials are then deﬁned asT
Eset (s = i, x) = ws,iφs (x)

with x all the information included in the post. Note that as with the users and outﬁts
we share the features and learn a diﬀerent weight for each settings latent state.
Fashion: We use the time between the creation of the post and when the post was
crawled as a feature, as well as bag-of-words on the “tags”. To incorporate the reviews,
we parse the comments with the sentiment-analysis model of (Socher et al., 2013).
This model attempts to predict how positive a review is on a 1-5 scale (1 is extremely
negative, 5 is extremely positive). We used a pre-trained model that was trained on therotten tomatoes dataset. We run the model on all the comments and sum the scoresfor each post. We also extract features using the style classiﬁer proposed in (Karayev
et al., 2014) that is pre-trained on the Flickr80k dataset to detect 20 diﬀerent imagestyles such as “Noir”, “Sunny”, “Macro” or “Minimal”. This captures the fact that agood photography style is correlated with the fashionability score. We then computeour unary potentials as the output of a small neural network with two hidden layersthat takes as input the settings high dimensional features and produce an 8D feature
map φf (x). Our outﬁt unary potentials are then deﬁned as
T
Ef ash (f = i, x) = wf,iφf (x)

Once more, we shared the features and learn separate weights for each fashionability
score.
Correlations: We use a non-parametric function for each pairwise and let the CRF
learn the correlations. Thus
uf
uf
Enp
(u = i, f = j) = wi,jSimilarly for the other pairwise potentials.
Learning and Inference
We learn our model using a two step approach: we ﬁrst jointly train the deep networks
that are used for feature extraction to predict fashionability as shown in Fig 6.15, andestimate the initial latent states using clustering. We then learn the CRF model (2430weights) using the primal-dual method of (Hazan and Urtasun, 2010). In particular,
we use the implementation of (Schwing et al., 2012). As task loss we use the L1 norm
for fashionability, and encourage the latent states to match the initial clustering. We
perform inference using message passing (Schwing et al., 2011).

Experimental Evaluation
We perform a detailed quantitative evaluation on the 10-class fashionability predictiontask. Furthermore, we provide a qualitative evaluation on other high level tasks suchas visualizing changes in trends and outﬁt recommendations.

130

Chapter 6. Probabilistically Modelling Fashion

Attribute

Corr.

Economy class
Income classlog(GDP)log(Population)-0.137-0.111
0.258
0.231

Attribute

Corr.

age
beautyeye_closed
gender
smile
asian
calm
happy
sad

-0.0250.066
0.022-0.037-0.0230.024
0.023-0.0240.023

Table 6.11: Effect of various attributes on the fashionability. Economy andIncome class refer to a 1-7 scale in which 1 corresponds to most developed or richcountry while 7 refers to least developed or poor country. For the face recognition
features we only show those with absolute values above 0.02. In all cases we show thePearson Coeﬃcients.

1.0

0.9

Dominant Ethnicity Mean Beauty

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

White

Indian

Black

Asian
AmericanIndianFigure 6.16: Visualization of mean beauty and dominant ethnicity by country. We average the beauty and ethnicity scores of all posts and show results for all
the countries.

6.3. Modelling Fashionability

131

Model

Acc.

Pre.

Rec.

IOU

L1

CRF
Deep Net
Log. Reg.
Baseline
Random

29.27
30.42
23.92
16.28
9.69

30.42
31.11
22.54
9.69

28.69
30.26
22.99
10.00
9.69

17.36
18.41
12.55
1.63
4.99

1.46
1.45
1.91
2.32
3.17

Table 6.12: Fashionability prediction results.We show results for classiﬁcation for
random, a baseline that predicts only the dominant class, a standard logistic regression
on our features, the deep network used to obtain features for the CRF and the ﬁnal
CRF model. We show accuracy, precision, recall, intersection over union (IOU), and L1
norm as diﬀerent metrics for performance.

Correlations
We ﬁrst analyze the correlation between fashionability and various features in our model.
We consider the eﬀect of the country on fashionability: in particular, we look the eﬀectof economy, income class, Gross Domestic Product (GDP) and population. Results
are in Table 6.11-left. A strong relationship is clear: poorer countries score lower infashionability than the richer, sadly a not very surprising result.
We also show face-related correlations in Table 6.11-right. Interestingly, but not
surprising, younger and more beautiful users are considered more fashionable. Additionally, we show the mean estimated beauty and dominant inferred ethnicity on theworld map in Fig. 6.16. Brazil dominates the Americas in beauty, France dominates
Spain, and Turkey dominates in Europe. In Asia, Kazakhstan scores highest, followedby China. There are also some high peaks which may be due to a very low number ofposts in a country. The ethnicity classiﬁer also seems to work pretty well, as generally
the estimation matches the ethnicity of the country.
Predicting Fashionability
We use 60% of the dataset as a train set, 10% as a validation, and 30% as test, andevaluate our model for the fashionability prediction task. Results of various model
instantiations are reported in Table 6.12. While the deep net obtains slightly better
results than our CRF, the model we propose is very useful as it simultaneously identiﬁes
the type of user, setting and outﬁt of each post. Additionally, as we show later, theCRF model allows performing much more ﬂexible tasks such as outﬁt recommendationor visualization of trends. Since the classiﬁcation metrics such as accuracy, precision,
recall, and intersection over union (IOU) do not capture the relationship between thediﬀerent fashionability levels, we also report the L1 norm between the ground truth andthe predicted label. In this case both the CRF and the deep net obtain virtually thesame performance.
Furthermore, we show qualitative examples of true positives, false positives, true
negatives and false positives in Fig. 6.17. Note that while we are only visualizing
images, there is a lot of meta-data associated to each image.

Chapter 6. Probabilistically Modelling Fashion

False Positives True Positives

132

Figure 6.17: True and false positive fashionability prediction examples. We
show some examples of the classiﬁcation task obtained with our CRF model.

FeatureSingle feature

Leave one out

Baseline

16.3

23.9

Comments
Tags
∆T
Style
Location
Scene
Fans
Personal
Colours
Singles
Garments

19.7
17.4
17.2
16.3
16.9
16.1
18.9
16.3
15.9
17.2
16.2

21.6
23.7
23.4
23.4
23.3
23.3
23.2
23.1
23.0
22.8
22.7

Table 6.13: Evaluation of features for predicting fashionability. We evaluate
the diﬀerent features for the fashionability prediction task using logistic regression. We
show two cases: performance of individual features, and performance with all but one
feature, which we call leave one out.

Dominant Outfit Cluster Dominant Settings Cluster

6.3. Modelling Fashionability

133

PlaygroundMarketLockers
Vinyard
Snowy Mountains
CoffeeBaseballSprings
Ski
FarmInterior Wide
Beach
VolcanoOutdoors Red
SportsOrchardInterior Small
Claustrophobic
Suburbia
MosqueBlack/Blue Going out
Formal Blue/Brown
Black Boots/Tights
White/Black Blouse/Heels
Brown/Blue Jacket
Navy and Bags
Bags/Dresses
Pink/Blue Shoes/Dress/Shorts
Black/Gray Tights/Sweater
Pastel DressBlack with Bag/Glasses
Blue with Scarf
Shirts and Jeans
Pink Outfit
Black Casual
Heels
Pink/Black Misc.
Shoes and Blue Dress
Pastel Shirts/Skirts
Black Heavy

Figure 6.18: Visualization of the dominant latent clusters. We show results forthe settings and outﬁt nodes in our CRF by country.

In order to analyze the individual contribution of each of the features, we show
their individual prediction power as well as how much performance is lost when afeature is removed. The individual performances of the various features are shown inthe second column of Table 6.13. We can see that in general the performance is very
low. Several features even perform under the baseline (Personal, Scene, and Colours).
The strongest features are Comments and Fans, which, however, are still not a verystrong indicator of fashionability as one would expect. In the leave one out case shownin the third column, removing any feature causes a drop in performance. This meansthat some features are not strong individually, but carry complementary information toother features and thus still contribute to the whole. In this case we see that the mostimportant feature is once again Comments, likely caused by the fact that most usersthat comment positively on a post also give it a vote.
Identifying Latent States
In order to help interpreting the results we manually attempt to give semantic meaning
to the diﬀerent latent states discovered by our model. While some states are harder toassign a meaning due to the large amount of data variation, other states like, e.g., thesettings states corresponding to “Ski” and “Coﬀee” have a clear semantic meaning. A
visualization of the location of some of the latent states can be seen in Fig. 6.18.
By visualizing the pairwise weights between the fashionability node and the diﬀerent
nodes we can also identify the “trendiness” of diﬀerent states (Fig. 6.19). For example,
the settings state 1 corresponding to “Mosque” is clearly not fashionable while the state
2 and 3 corresponding to “Suburbia” and “Claustrophobic”, respectively, have positive
gradients indicating they are fashionable settings.

Chapter 6. Probabilistically Modelling Fashion

Fashionability

134

10
9
8
7
6
5
4
3
2
1

10
5
0
−5
−10
−15

Fashionability

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Setting Class

10
9
8
7
6
5
4
3
2
1

2
1
0
−1
−2

Fashionability

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
User Class
4

10
9
8
7
6
5
4
3
2
1

2
0
−2
−4
−6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20
Outfit Class

Figure 6.19: Visualizing pairwise potentials between nodes in the CRF. By
looking at the pairwise between fashionability node and diﬀerent states of other variables
we are able to distinguish between fashionable and non-fashionable outﬁts and settings.

6.3. Modelling Fashionability

135

Current Outfit:
Pink Outfit (3)Current Outfit:
Pink/Blue Shoes/Dress Shorts (3)Current Outfit:
Pink/Black Misc. (5)Recommendations:
Heels (8)Pastel Shirts/Skirts (8)Black/Gray Tights/Sweater (5)Recommendations:Black/Gray Tights/Sweater (5)Black Casual (5)Black Boots/Tights (5)Recommendations:
Pastel Dress (8)Black/Blue Going out (8)Black Casual (8)Current Outfit:
Blue with Scarf (3)Current Outfit:
Pink/Blue Shoes/Dress Shorts (3)Current Outfit:
Formal Blue/Brown (5)Recommendations:
Heels (8)Pastel Shirts/Skirts (8)Black Casual (8)Recommendations:Black Casual (7)Black Heavy (3)Navy and Bags (3)Recommendations:
Pastel Shirts/Skirts (9)Black/Blue Going out (8)Black Boots/Tights (8)Figure 6.20: Examples of recommendations provided for our model. In parenthesis we see the fashionability of the user as predicted by our model.

Manila0.6

Los Angeles

0.35Black/Blue Going out
Formal Blue/Brown
Black Boots/Tights
White/Black Blouse/Heels
Brown/Blue Jacket
Navy and Bags
Bags/Dresses
Pink/Blue Shoes/Dress/Shorts
Black/Gray Tights/Sweater
Pastel DressBlack with Bag/Glasses
Blue with Scarf
Shirts and Jeans
Pink Outfit
Black Casual
Heels
Pink/Black Misc.
Shoes and Blue Dress
Pastel Shirts/Skirts
Black Heavy

0.30

0.5

0.25
0.4
0.20
0.3
0.15
0.2
0.10
0.1

0.0

0.05

4

8

12

16

20

Trimester

0.00

4

8

12

16

20

TrimesterFigure 6.21: Visualization of the temporal evolution of the different trends in
Manila and Los Angeles. The less signiﬁcant clusters have been manually removed
to decrease clutter.

Outfit Recommendation
An exciting property of our model is that it can be used for outﬁt recommendation.
In this case, we take a post as an input and estimate the outﬁt that maximizes the
fashionability while keeping the other variables ﬁxed. In other words, we are predicting
what the user should be wearing in order to maximize her/his look instead of theircurrent outﬁt. We show some examples in Fig. 6.20. This is just one example of the
ﬂexibility of our model. Other tasks such what is the least ﬁtting outﬁt, what is thebest place to go to with the current outﬁt, or what types of users this outﬁt ﬁts themost, can also be done with the same model.

136

Chapter 6. Probabilistically Modelling Fashion

Estimation Fashion Trends
By incorporating temporal information we can try to visualize the changes in trends
for a given location. In particular we look at the trendiest cites in the dataset, that is
Manila and Los Angeles, as per Table 6.9. We visualize these results in Fig. 6.21. For
Manila, one can see that while until the 8th trimester, outﬁts like “Pastel Skirts/Shirts”
and “Black with Bag/Glasses” are popular, after the 12th trimester there is a boom of“Heels” and “Pastel Dress”. Los Angeles follows a roughly similar trend. For LA however,before the 8th trimester, “Brown/Blue Jacket” and “Pink/Black Misc” are popular,
while afterwards “Black Casual” is also fairly popular. We’d like to note that in the 8thtrimester there appears to have been an issue with the chictopia website, causing very
few posts to be published, and as a consequence, results in unstable outﬁt predictions.

6.4

Summary

In this chapter we have dealt with understanding images in the challenging context of
fashion. This has only recently become the focus of research and is proving to require
many innovations in order to have competitive results. It is an exciting topic that can
prove to be invaluable to users in an era in which one’s personal image is critical to be
successful in society: from applying to universities to ﬁnding a job, and even to succeed
in one’s love life.
First we have tackled the challenging problem of clothing parsing in fashion images.
We have shown that our approach is able to obtain a signiﬁcant improvement overthe state-of-the-art in the challenging Fashionista dataset by exploiting appearance,
ﬁgure/ground segmentation, shape and location priors for each garment as well as similarity between segments and symmetries between diﬀerent human body parts. Despite
these promising results, we believe much can still be done to improve. For example, oneof the most occurring mistakes are missing glasses or other small garments. We believea multi-resolution approach is needed to handle the diversity of garment classes.
We have also presented a novel task of predicting fashionability of users photographs.
In order to evaluate this task we have collected a large-scale dataset by crawling thelargest fashion-oriented social website. Our proposed Conditional Random Fields model
is able to reason about settings, users and their fashionability. We are able to predictthe visual aesthetics related to fashion, which can also be used to analyze fashion
trends in the world or individual cities, and potentially diﬀerent age groups and outﬁtstyles. Furthermore our model can also be used for outﬁt recommendation. This is an
important ﬁrst step to building more complex and powerful models that will be able
to understand fashion, trends, and users a whole in order to improve the experience ofusers in the modern day society. We will make the dataset and code public in hopesthat this will inspire other researchers to tackle this challenging task.
It is important to note that to obtain our competitive results in these challenging
tasks, many of the techniques and approaches developed in other chapter of this thesis
have played a fundamental role. In order to perform high level tasks such as predicting
fashionability it is essential to have a good grasp on mid and low level cues. It is nolonger suﬃcient to focus on a very niche problem with a narrow understanding of one’sﬁeld.

Chapter 7

Conclusions

In this thesis we have analyzed the problem of understanding human-centric images atvarious levels. Our work is a push towards more ambitious goals such as estimatingfashionability and proposing better outﬁts which we presented in Chapter 6. In order
to be able to tackle these higher level tasks, we have presented a swath of techniquesfrom low level image features to full 3D pose estimation algorithms. We ﬁnally showed
the importance of using strong low and mid level cues in high level tasks.
To recapitulate the contributions:
1. We have proposed and evaluated two diﬀerent approaches for local feature description. On one hand, we have used heat diﬀusion to robustly describe image
patches undergoing deformation and illumination changes. On the other hand,
we have shown that it is not only possible to learn descriptors using convolutional
networks, but also extremely beneﬁcial when done with high levels of mining denoted “fracking”.
2. For 3D human pose estimation and other problems we have investigated several
generative models. In particular, we presented two novel models for representing
3D human pose. One is based on a Directed Acyclic Graph with a discrete representation of the poses, and is used to learn a mapping between a latent space andthe pose space. The other models the pose as data on a Riemannian manifold
using a Gaussian mixture with local approximations.
3. We have also presented two frameworks for 3D human pose estimation from single
monocular images. We show how it is possible to use noisy 2D observations
to obtain more reliable 3D pose estimations. Additionally, we demonstrate thebeneﬁts of integrating the 2D and 3D pose estimation into a single end-to-end
system that performs both tasks simultaneously.
4. Finally, we show more higher level applications in the challenging context of fashion. In particular, we show results with a Conditional Random Fields model using very strong mid level cues for the complex task of semantic segmentation ofclothing. We also propose the novel high level task of predicting fashionability
from images. We show it is possible to design models that are able to interpretthe image and provide recommendations on how to improve one’s fashionableness.
137

138

Chapter 7. Conclusions

All these works share the same context, that is, they are designed for usage in humancentric images. While the lower level approaches such as descriptors and generativemodels do also generalize to many other problems, the 3D human pose estimation
approaches and the fashion understanding developments are speciﬁc to human-centric
images.
Additionally, we would like to point out we have made most of the code and datasetsof this thesis publicly available1 . We believe this is important for the reproducibility of
our results and for the beneﬁt of the community in general. We acknowledge that there
are still parts of this thesis not made publicly available, however, we have the intentionof releasing more in the near future.

7.1

Future Work

We have set the basis for many diﬀerent tools that we have not yet fully exploited inthis thesis. The most direct line of work is to attempt to take more advantage of our
low and mid level cues in other tasks. In particular our descriptors have not been tested
in full applications yet, although they have been thoroughly evaluated on real images.
For more tangible directions of work, we can refer to our deep learning approachfor feature point descriptors. In particular, while we were able to obtain great improvements, there are still many open questions. For example, is it possible to learn robustness to diﬀerent types of properties, such as rotation or scale, independently? What
exactly is the eﬀect of the training data on a pre-learned network? Is it possible toexploit other datasets in order to obtain better generalizations? It is also not clear
whether the best performing network’s hyperparameters are optimal. This requiresa much more high level large-scale analysis and in particular the creation of a more
complete large dataset for feature point matching.
Our generative model based on clustering on Riemannian manifolds also has shown
great promise, but has not been used in a real 3D pose estimation framework. As thestrength of this model holds on the fact it can quickly estimate distributions for missing
limbs, if it were feasible to integrate it with an occlusion estimator, it would be possible
to handle occlusions in an elegant manner. Furthermore, as we show in (Simo-Serra
et al., 2015c), it is possible to use the same framework for tracking. Therefore anotherlogical extension of the work presented in this thesis would be to tackle the problem of
3D human pose tracking. This would require extending the Bayesian formulation of oursimultaneous 2D and 3D human pose estimation to integrate the temporal component.
Finally, we have only started to explore the possibilities in the world of fashion. We
have presented a novel large dataset called Fashion144k for the prediction of fashionability, but have only started approaching this challenging problem. As shown by our
preliminary results, this is a very complex problem which requires development of muchstronger features and cues, such as our clothing segmentation algorithm, in order to beable to perform reliable predictions. Additionally, due to the large scale of the dataset,
it is imperative for these feature extraction algorithms to be fast and scale well to thesize of the dataset. For this purpose, we think it is fundamental to try to develop new
approaches as well as explore the existing literature to see what cues we can exploit tobuild models that can truly understand fashion.
1

http://www.iri.upc.edu/people/esimo/

7.2. Trends in Computer Vision and Machine Learning

7.2

139

Trends in Computer Vision and Machine Learning

In the elaboration of this thesis, we have exploited many existing tools from the ﬁeldsof computer vision and machine learning. As both these ﬁelds are moving extremely
fast, we would like to brieﬂy mention some major changes we have experienced during
the course of this thesis and give insights as to which direction we believe it is heading.
Probably the strongest change we have been able to experience has been the irruption of deep learning in computer vision. At the beginning of the thesis (2011),
these tools were not widely available nor very relevant, however, since the publicationof (Krizhevsky et al., 2012) we have seen a large inﬂux of incredible results that havebeen overturning most well established techniques (Girshick et al., 2014; Simonyan andZisserman, 2014). While the pure quantitative results of these approaches have been
unbeatable in most tasks, a number of concerned voices have appeared questioning
whether or not we fully understand them. This has led to the current trend of not only
obtaining great results with these systems, but also understanding why we are able toobtain these great results (Zeiler and Fergus, 2014; Chatﬁeld et al., 2014).
Competing with alternative approaches against deep learning in many problems
such as classiﬁcation seems like a lost cause. Not only because it seems like no otheralgorithms can compete with them on certain problems, but additionally the resources
some of the next generation of networks require to train are out of the reach of all
except the largest research laboratories. The unrelenting increase of available data andthe infrastructure needed to process them are a concern for the competitiveness of small
research centers. However, despite the enormous results on classiﬁcations, and more
recently detection, problems, we believe there are still holes that can’t be ﬁlled withpure deep learning approaches. For example structured prediction problems in which
the output is subject to a set of constraints are very complicated to model with neuralnetworks.
While results in problems with more tradition have been greatly aﬀected by the deeplearning tsunami, we have also seen an emergence of very interesting new problems incomputer vision. Related to this thesis we have seen the ﬁrst dataset for segmentation in
fashion images in 2012 (Yamaguchi et al., 2012). In tune with this, we proposed our own
dataset for predicting fashionionability and presented the novel task of giving fashion
advice. On a similar line of research it is also worth to mention the work of (Pirsiavash
et al., 2014), which attempts to assess diﬀerent actions performed by a user, such as
pool diving or ﬁgure skating, and give advice on how to improve the performance of theaction. Another interesting and recent line of research seeks to ﬁnd out what makes
photographs of faces memorable and attempts to modify them to become even more so(Khosla et al., 2013). These are all very high level tasks that were unthinkable just a
few years back. We believe we are at a point of inﬂection and things that were science
ﬁction just a decade back will start to become more and more commonplace.
Another heavily questioned aspect is currently the publication model in computer
vision and machine learning. There is already an antecedent of a major change in themachine learning literature when forty members resigned from the Editorial Board of
the Machine Learning Journal (MLJ), and moved to support the newly created Journalof Machine Learning Research (JMLR) (Jordan, 2001). This was a move away fromrestricted access pay-per-view to a free-access model, which we personally believe has
greatly beneﬁted the community. Along the same lines, during the Computer Vision
and Pattern Recognition (CVPR) 2013 conference in Portland, Oregon, moving to an

140

Chapter 7. Conclusions

open access model was also decided by voting (PAMI-TC, 2013). This is the unavoidableconsequence of the distribution abilities of the internet and the widespread practice of
researchers hosting their papers on their website to encourage diﬀusion.
In 2013 we additionally saw the birth of a completely new publication model.
The new deep learning oriented International Conference on Learning Representations(ICLR) had its ﬁrst edition. Instead of using the well established double-blind review
process with a rebuttal period, they opted for single blind reviewing process, which wascommon in machine learning over a decade ago. This may seem like a step backwards,
however, this is the consequence of moving to an open review model 2 . In this conference papers are put online during submission time and the entire reviewing process ismade transparent, i.e., reviews and responses are public and shown in realtime. Additionally, editing of the paper after the initial submission is actively encouraged. Thisdrastically changes the publication model and is once again the inevitable result of a
new widespread practice: publishing preliminary work on the arXiv website 3 whensubmitting to a top-tier conference such as CVPR. While it is not clear if this newmodel is more fair than double-blind reviewing, it provides an interesting alternative
for a time when it is hard to enforce the anonymousness of the submitting authors.
Finally, very recently we have seen the results of a most interesting experiment. The
denoted “NIPS Experiment” (Price, 2014) consisted of forming two independent programcommittees and assigning a random 10% of submitted papers to both committees. The
ambitious objective of this feat was to quantify the amount of randomness in acceptanceof papers. Anyone who has submitted to a research conference is familiar with thisrandomness. A poll of submitters indicated that most agreed that the divergence would
be found to be around 30%. During the conference in December 2014 the results were
published and it was found that an astounding 57% of accepted papers would havebeen rejected by the other reviewing team. The implications of this result are not yet
understood.
As we can see, not only has there been a landslide in the techniques used for computer vision with the irruption of deep learning, there have also been many changes as
the current publishing model is being questioned. We believe that there are still more
important changes to come in the short term, however, they seem to be going in theright direction and will be beneﬁcial to the future of our research.

23

http://www.iclr.cc/doku.php?id=pubmodel
http://arxiv.org/

Bibliography

A. Yao, J. Gall, L. V. G. and Urtasun, R. (2011). Learning probabilistic non-linear latentvariable models for tracking complex activities. In Neural Information Processing
Systems. 65
Achanta, R., Shaji, A., Smith, K., Lucchi, A., Fua, P., and Susstrunk, S. (2012). Slicsuperpixels compared to state-of-the-art superpixel methods. IEEE Transactions
Pattern Analylis and Machine Intelligence, 34(11):2274–2282. 9
Agarwal, A. and Triggs, B. (2006). Recovering 3d human pose from monocular images.
IEEE Transactions Pattern Analylis and Machine Intelligence, 28(1):44–58. 85
Andriluka, M., Roth, S., and Schiele, B. (2009). Pictorial Structures Revisited: People
Detection and Articulated Pose Estimation. In IEEE Conference on Computer Vision
and Pattern Recognition. 83, 86, 87, 96, 97, 98
Andriluka, M., Roth, S., and Schiele, B. (2010). Monocular 3D Pose Estimation andTracking by Detection. In IEEE Conference on Computer Vision and Pattern Recognition. 65, 66, 83, 85, 87, 92, 93, 94, 100, 101, 102
Andriluka, M., Roth, S., and Schiele, B. (2012). Discriminative Appearance Models for
Pictorial Structures. International Journal of Computer Vision, 99(3). 69, 83, 97
Anguelov, D., Srinivasan, P., Koller, D., Thrun, S., Rodgers, J., and Davis, J. (2005).
Scape: shape completion and animation of people. ACM Transactions on Graphics,
24(3):408–416. 65
Balan, A. O., Sigal, L., Black, M. J., Davis, J. E., and Haussecker, H. W. (2007).
Detailed Human Shape and Pose from Images. In IEEE Conference on Computer
Vision and Pattern Recognition. 84, 85
Bay, H., Tuytelaars, T., and Gool, L. V. (2006). SURF: Speeded up robust features. In
European Conference on Computer Vision, pages 404–417. 23, 25, 49, 50
Belongie, S., Malik, J., and Puzicha, J. (2002). Shape matching and object recognition
using shape contexts. IEEE Transactions Pattern Analylis and Machine Intelligence,24(4):509–522. 25
141

142

Bibliography

Berg, A., Berg, T., and Malik, J. (2005). Shape matching and object recognition using
low distortion correspondences. In IEEE Conference on Computer Vision and Pattern
Recognition, volume 1, pages 26–33. 25
Bo, L. and Sminchisescu, C. (2010). Twin Gaussian Processes for Structured Prediction.
International Journal of Computer Vision, 87:28–52. 83, 93, 94, 101, 102
Boothby, W. M. (2003). An Introduction to Differentiable Manifolds and Riemannian
Geometry. Revised 2nd Ed. Academic, New York. 74
Bossard, L., Dantone, M., Leistner, C., Wengert, C., Quack, T., and Gool, L. V. (2012).
Apparel classifcation with style. In Asian Conference on Computer Vision. 106, 108
Bourdev, L., Maji, S., and Malik, J. (2011). Describing people: A poselet-based approach to attribute classiﬁcation. In International Conference on Computer Vision.
106, 108
Bourdev, L. and Malik, J. (2009). Poselets: Body part detectors trained using 3d humanpose annotations. In International Conference on Computer Vision. 108
Boykov, Y., Veksler, O., and Zabih, R. (2001). Fast approximate energy minimization via graph cuts. IEEE Transactions Pattern Analylis and Machine Intelligence,
23(11):1222–1239. 14
Bromley, J., Guyon, I., Lecun, Y., Säckinger, E., and Shah, R. (1994). Signature
veriﬁcation using a "siamese" time delay neural network. In Neural Information
Processing Systems. 50
Bronstein, A., Bronstein, M., Bruckstein, A., and Kimmel, R. (2007). Analysis of twodimensional non-rigid shapes. International Journal of Computer Vision, 78(1):67–88.
25
Bronstein, M. and Kokkinos, I. (2010). Scale-invariant heat kernel signatures for nonrigid shape recognition. In IEEE Conference on Computer Vision and Pattern Recognition, pages 1704–1711. 24, 29
Brown, M., Hua, G., and Winder, S. (2011). Discriminative learning of local image descriptors. IEEE Transactions Pattern Analylis and Machine Intelligence, 33(1):43–57.
49, 50, 51, 53
Brox, T., Bourdev, L., Maji, S., and Malik, J. (2011). Object segmentation by alignmentof poselet activations to image contours. In IEEE Conference on Computer Vision
and Pattern Recognition. 108
Brubaker, M. A., Salzmann, M., and Urtasun, R. (2012). A Family of MCMC Methods
on Implicitly Deﬁned Manifolds. Journal of Machine Learning Research - Proceedings
Track, 22:161–172. 65, 73
Burenius, M., Sullivan, J., and Carlsson, S. (2013). 3d pictorial structures for multiple
view articulated pose estimation. In IEEE Conference on Computer Vision and
Pattern Recognition. 83

Bibliography

143

C. Strecha, A. Bronstein, M. B. and Fua, P. (2012). LDAHash: Improved matching withsmaller descriptors. IEEE Transactions Pattern Analylis and Machine Intelligence,34(1). 40, 50, 60
Cai, H., Mikolajczyk, K., and Matas, J. (2011). Learning linear discriminant projections for dimensionality reduction of image descriptors. IEEE Transactions PatternAnalylis and Machine Intelligence, 33(2):338–352. 40
Calonder, M., Lepetit, V., Ozuysa, M., Trzcinski, T., Strecha, C., and Fua, P. (2012).
BRIEF: Computing a local binary descriptor very fast. IEEE Transactions PatternAnalylis and Machine Intelligence, 34(7):1281–1298. 25
Carreira, J., Caseiroa, R., Batista, J., and Sminchisescu, C. (2012). Semantic segmentation with second-order pooling. In European Conference on Computer Vision. 12,110, 111, 112, 117, 118, 120
Carreira, J. and Sminchisescu, C. (2012). CPMC: Automatic Object Segmentation
Using Constrained Parametric Min-Cuts. IEEE Transactions Pattern Analylis andMachine Intelligence, 34(7):1312–1328. 11, 110
Chang, C.-C. and Lin, C.-J. (2011). LIBSVM: A library for support vector machines.
ACM Transactions on Intelligent Systems and Technology, 2:27:1–27:27. 16
Chatﬁeld, K., Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Return of the devil
in the details: Delving deep into convolutional nets. CoRR, abs/1405.3531. 139
Chavel, I. (1984). Eigenvalues in Riemannian Geometry. London Academic Press. 28
Chen, H., Gallagher, A., and Girod, B. (2012). Describing clothing by semantic attributes. In European Conference on Computer Vision. 106, 108, 123
Chen, H., Xu, Z. J., Liu, Z. Q., and Zhu, S. C. (2006). Composite templates for
cloth modeling and sketching. In IEEE Conference on Computer Vision and Pattern
Recognition. 108, 122
Cheng, H., Liu, Z., Zheng, N., and Yang, J. (2008). A deformable local image descriptor.
In IEEE Conference on Computer Vision and Pattern Recognition. 23, 25
Cho, M., Lee, J., and Lee, K. (2009). Feature correspondence and deformable object
matching via agglomerative correspondence clustering. In International Conference
on Computer Vision, pages 1280–1287. 23, 25
Collobert, R., Kavukcuoglu, K., and Farabet, C. (2011). Torch7: A matlab-like environment for machine learning. In BigLearn, NIPS Workshop. 52
Culurciello, E., Jin, J., Dundar, A., and Bates, J. (2013). An analysis of the connections
between layers of deep neural networks. CoRR, abs/1306.0152. 52
Dalal, N. and Triggs, B. (2005). Histograms of oriented gradients for human detection.
In IEEE Conference on Computer Vision and Pattern Recognition. 9, 10, 83
Daubney, B. and Xie, X. (2011). Tracking 3D Human Pose with Large Root Node
Uncertainty. In IEEE Conference on Computer Vision and Pattern Recognition. 65,
83, 87, 93, 94, 101, 102

144

Bibliography

Davis, B. C., Bullitt, E., Fletcher, P. T., and Joshi, S. (2007). Population ShapeRegression from Random Design Data. In International Conference on Computer
Vision. 65
Davis, J. and Goadrich, M. (2006). The relationship between PR and ROC curves. In
International Conference in Machine Learning. 54
de Goes, F., Goldenstein, S., and Velho, L. (2008). A hierarchical segmentation ofarticulated bodies. In Proceedings of the Symposium on Geometry Processing, pages
1349–1356. 26
Dedieu, J.-P. and Nowicki, D. (2005). Symplectic methods for the approximation of the
exponential map and the newton iteration on riemannian submanifolds. Journal ofComplexity, 21(4):487 – 501. 74
Deng, J., Dong, W., Socher, R., jia Li, L., Li, K., and Fei-fei, L. (2009). Imagenet:
A large-scale hierarchical image database. In IEEE Conference on Computer Vision
and Pattern Recognition. 116
Dhar, S., Ordonez, V., and Berg, T. (2011). High level describable attributes for predicting aesthetics and interestingness. In IEEE Conference on Computer Vision and
Pattern Recognition. 123
Donahue, J., Jia, Y., Vinyals, O., Hoﬀman, J., Zhang, N., Tzeng, E., and Darrell, T.
(2013). Decaf: A deep convolutional activation feature for generic visual recognition.
CoRR, abs/1310.1531. 13
Dong, J., Chen, Q., Xia, W., Huang, Z., and Yan, S. (2013). A deformable mixture
parsing model with parselets. In International Conference on Computer Vision. 108
Everingham, M., Van Gool, L., Williams, C. K. I., Winn, J., and Zisserman, A. (2010).
The pascal visual object classes (voc) challenge. International Journal of Computer
Vision, 88(2):303–338. 116
Fan, B., Wu, F., and Hu, Z. (2012). Rotationally invariant descriptors using intensity order pooling. IEEE Transactions Pattern Analylis and Machine Intelligence,
34(10):2031–2045. 26
Fan, R.-E., Chang, K.-W., Hsieh, C.-J., Wang, X.-R., and Lin, C.-J. (2008). LIBLINEAR: A library for large linear classiﬁcation. Journal of Machine Learning Research,
9:1871–1874. 15
Farhadi, A., Endres, I., Hoiem, D., and Forsyth, D. (2009). Describing objects by their
attributes. In IEEE Conference on Computer Vision and Pattern Recognition. 13
Felzenszwalb, P., McAllester, D., and Ramanan, D. (2008). A Discriminatively Trained,
Multiscale, Deformable Part Model. In IEEE Conference on Computer Vision and
Pattern Recognition. 1, 69, 86, 87
Felzenszwalb, P. F., Girshick, R. B., McAllester, D., and Ramanan, D. (2010). Object
Detection with Discriminatively Trained Part Based Models. IEEE Transactions
Pattern Analylis and Machine Intelligence, 32(9):1627–1645. 50, 69

Bibliography

145

Felzenszwalb, P. F. and Huttenlocher, D. P. (2005). Pictorial Structures for Object
Recognition. International Journal of Computer Vision, 61:55–79. 3, 63, 83, 86
Fidler, S., Sharma, A., and Urtasun, R. (2013). A sentence is worth a thousand pixels.
In IEEE Conference on Computer Vision and Pattern Recognition. 108
Figueiredo, M. and Jain, A. (2002). Unsupervised Learning of Finite Mixture Models.
IEEE Transactions Pattern Analylis and Machine Intelligence, 24(3):381–396. 73, 76,
77
Fletcher, P., Lu, C., Pizer, S., and Joshi, S. (2004). Principal Geodesic Analysis for
the Study of Nonlinear Statistics of Shape. IEEE Transactions on Medical Imaging,23(8):995–1005. 65, 66, 73
Forbes Magazine (2013). US Online Retail Sales To Reach $370B By 2017; e191B in
Europe. http://www.forbes.com. [Online; accessed 14-March-2013]. 106
Gallagher, A. C. and Chen, T. (2008). Clothing cosegmentation for recognizing people.
In IEEE Conference on Computer Vision and Pattern Recognition. 106, 108
Geiger, A., Lenz, P., Stiller, C., and Urtasun, R. (2013). Vision meets robotics: The
kitti dataset. International Journal of Robotics Research. 116
Girshick, R., Donahue, J., Darrell, T., and Malik, J. (2014). Rich feature hierarchies
for accurate object detection and semantic segmentation. In IEEE Conference on
Computer Vision and Pattern Recognition. 12, 50, 139Gębal, K., Bærentzen, J. A., Aanæs, H., and Larsen, R. (2009). Shape analysis using
the auto diﬀusion function. In Proceedings of the Symposium on Geometry Processing,pages 1405–1413. 23, 24, 26, 27
Gong, Y., Lazebnik, S., Gordo, A., and Perronnin, F. (2012). Iterative quantization:
A procrustean approach to learning binary codes for large-scale image retrieval. In
IEEE Transactions Pattern Analylis and Machine Intelligence. 50
Guan, P., Weiss, A., Balan, A., and M.J.Black (2009). Estimating Human Shape andPose from a Single Image. In International Conference on Computer Vision. 87
Gupta, R. and Mittal, A. (2007). Illumination and Aﬃne-Invariant Point Matching
using an Ordinal Approach. In International Conference on Computer Vision. 26
Gupta, R. and Mittal, A. (2008). Smd: A locally stable monotonic change invariant
feature descriptor. In European Conference on Computer Vision, pages 265–277. 26
Gupta, R., Patil, H., and Mittal, A. (2010). Robust order-based methods for featuredescription. In IEEE Conference on Computer Vision and Pattern Recognition. 26
Gygli, M., Grabner, H., Riemenschneider, H., Nater, F., and Gool, L. (2013). Theinterestingness of images. In International Conference on Computer Vision. 123
Hamerly, G. and Elkan, C. (2003). Learning the k in k-means. In Neural Information
Processing Systems. 90

146Bibliography

Hansen, N. (2006). The CMA Evolution Strategy: a Comparing Review. In Towards
a new evolutionary computation. Advances on estimation of distribution algorithms,
pages 75–102. Springer. 89, 90, 99
Hasan, B. and Hogg, D. (2010). Segmentation using deformable spatial priors withapplication to clothing. In British Machine Vision Conference. 106
Hasler, N., Stoll, C., Sunkel, M., Rosenhahn, B., and Seidel, H.-P. (2009). A statisticalmodel of human pose and body shape. Computer Graphics Forum, 28(2):337–346.
65, 66
Hauberg, S., Sommer, S., and Pedersen, K. S. (2012).
Natural metrics andleast-committed priors for articulated tracking. Image and Vision Computing,30(6):453–461. 77
Hazan, T. and Urtasun, R. (2010). A primal-dual message-passing algorithm for approximated large scale structured prediction. In Neural Information Processing Systems.
19, 114, 129
Heikkilä, M., Pietikäinen, M., and Schmid, C. (2009). Description of interest regionswith local binary patterns. Pattern Recognition, 42(3):425–436. 26
Hou, X. and Zhang, L. (2007). Saliency detection: A spectral residual approach. In
IEEE Conference on Computer Vision and Pattern Recognition. 11
Howe, N. R., Leventon, M. E., and Freeman, W. T. (1999). Bayesian Reconstructionof 3D Human Motion from Single-Camera Video. In Neural Information Processing
Systems, pages 820–826. MIT Press. 85
Huang, L. (2008). Advanced dynamic programming in semiring and hypergraph frameworks. In Coling 2008: Advanced Dynamic Programming in Computational Linguistics: Theory, Algorithms and Applications - Tutorial notes, pages 1–18, Manchester,
UK. Coling 2008 Organizing Committee. 70
Huckemann, S., Hotz, T., and Munk, A. (2010). Intrinsic Shape analysis: Geodesic PCA
for Riemannian Manifolds Modulo Isometric Lie Group Actions. Statistica Sinica,
20:1–100. 73
Ionescu, C., Li, F., and Sminchisescu, C. (2011). Latent Structured Models for Human
Pose Estimation. In International Conference on Computer Vision. 77
Ionescu, C., Papava, D., Olaru, V., and Sminchisescu, C. (2014). Human3.6m: Large
scale datasets and predictive methods for 3d human sensing in natural environments.
IEEE Transactions Pattern Analylis and Machine Intelligence. 77
Isola, P., Xiao, J., Parikh, D., Torralba, A., and Oliva, A. (2014). What makes a photograph memorable? IEEE Transactions Pattern Analylis and Machine Intelligence,
36(7):1469–1482. 123
Itti, L. and Koch, C. (2000). A saliency-based search mechanism for overt and covert
shifts of visual attention. Vision Research, 40:1489–1506. 11

Bibliography

147

Jahrer, M., Grabner, M., and Bischof, H. (2008). Learned local descriptors for recognition and matching. In Computer Vision Winter Workshop. 50
Jain, A., Tompson, J., Andriluka, M., Taylor, G. W., and Bregler, C. (2014a). Learning
human pose estimation features with convolutional networks. In International Conference on Learning Representations. 86
Jain, A., Tompson, J., Lecun, Y., and Bregler, C. (2014b). Modeep: A deep learning
framework using motion features for human pose estimation. In Asian Conference
on Computer Vision. 86
Jammalamadaka, N., Minocha, A., Singh, D., and Jawahar, C. (2013). Parsing clothes
in unrestricted images. In British Machine Vision Conference. 106, 108, 122
Jarrett, K., Kavukcuoglu, K., Ranzato, M., and LeCun, Y. (2009). What is the best
multi-stage architecture for object recognition? In International Conference on Computer Vision. 52
Jia, Y. (2013). Caﬀe: An open source convolutional architecture for fast feature embedding. http://caffe.berkeleyvision.org/. 128
Jordan, M. (2001). Leading ML researchers issue statement of support for JMLR.
http://www.jmlr.org/statement.html. [Online; accessed 3-February-2015]. 139
Karayev, S., Hertzmann, A., Winnemoeller, H., Agarwala, A., and Darrell, T. (2014).
Recognizing image style. In British Machine Vision Conference. 125, 129
Karcher, H. (1977). Riemannian center of mass and molliﬁer smoothing. Communications on Pure and Applied Mathematics, 30(5):509–541. 75
Kazemi, V., Burenius, M., Azizpour, H., and Sullivan, J. (2013). Multiview body part
recognition with random forests. In British Machine Vision Conference. 83, 84
Ke, Y. and Sukthankar, R. (2004). PCA-SIFT: a more distinctive representation for
local image descriptors. In IEEE Conference on Computer Vision and Pattern Recognition, volume 2, pages 506–513. 23, 25
Khosla, A., Bainbridge, W. A., Torralba, A., and Oliva, A. (2013). Modifying thememorability of face photographs. In International Conference on Computer Vision.
123, 139
Khosla, A., Sarma, A. D., and Hamid, R. (2014). What makes an image popular? In
International World Wide Web Conference. 123
Kiapour, M. H., Yamaguchi, K., Berg, A. C., and Berg, T. L. (2014). Hipster wars:
Discovering elements of fashion styles. In European Conference on Computer Vision.
123
Kokkinos, I., Bronstein, M., and Yuille, A. (2012). Dense Scale Invariant Descriptors
for Images and Surfaces. Research Report RR-7914, INRIA. 25, 49
Kovnatsky, A., Bronstein, M., Bronstein, A., and Kimmel, R. (2011). Photometric heat
kernel signatures. In Scale Space and Variational Methods in Computer Vision, pages
616–627. 26

148

Bibliography

Krizhevsky, A., Sutskever, I., and Hinton, G. E. (2012). Imagenet classiﬁcation withdeep convolutional neural networks. In Neural Information Processing Systems. 1, 8,
12, 13, 49, 50, 56, 139
Krähenbühl, P. and Koltun, V. (2011). Eﬃcient inference in fully connected crfs withgaussian edge potentials. In Neural Information Processing Systems. 8, 11
Ladicky, L., Russell, C., Kohli, P., and Torr, P. H. S. (2010). Graph cut based inferencewith co-occurrence statistics. In European Conference on Computer Vision. 108
Ladicky, L., Torr, P. H. S., and Zisserman, A. (2013). Human pose estimation using a
joint pixel-wise and part-wise formulation. In IEEE Conference on Computer Vision
and Pattern Recognition. 108
Laﬀerty, J. D., McCallum, A., and Pereira, F. C. N. (2001). Conditional random ﬁelds:
Probabilistic models for segmenting and labeling sequence data. In International
Conference in Machine Learning. 18
Lawrence, N. D. (2005). Probabilistic Non-linear Principal Component Analysis with
Gaussian Process Latent Variable Models. Journal of Machine Learning Research,
6:1783–1816. 65, 66
Lawrence, N. D. and Moore, A. J. (2007). Hierarchical Gaussian Process Latent Variable
Models. In International Conference in Machine Learning, pages 481–488, New York,NY, USA. ACM. 83, 84, 85
Lecun, Y., Bottou, L., Bengio, Y., and Haﬀner, P. (1998). Gradient-based learning
applied to document recognition. Proceedings of the IEEE, 86(11):2278–2324. 18
Leordeanu, M. and Hebert, M. (2005). A spectral technique for correspondence problems
using pairwise constraints. In International Conference on Computer Vision, pages
1482–1489. 23, 25
Lévy, B. (2006). Laplace-Beltrami Eigenfunctions: Towards an Algorithm that Understands Geometry. In IEEE International Conference on Shape Modeling and Applications. 26, 27
Li, R., Tian, T.-P., Sclaroﬀ, S., and Yang, M.-H. (2010). 3d human motion tracking
with a coordinated mixture ofăfactor analyzers. International Journal of Computer
Vision, 87(1-2):170–190. 65, 66
Li, S. and Chan, A. B. (2014). 3d human pose estimation from monocular images withdeep convolutional neural network. In Asian Conference on Computer Vision. 85, 86
Lin, D., Fidler, S., and Urtasun, R. (2013). Holistic scene understanding for 3d object
detection with rgbd cameras. In International Conference on Computer Vision. 20
Ling, H. and Jacobs, D. (2005). Deformation invariant image matching. In International
Conference on Computer Vision, pages 1466–1473. 23, 25, 26, 42
Ling, H. and Jacobs, D. (2007). Shape classiﬁcation using the inner-distance. IEEE
Transactions Pattern Analylis and Machine Intelligence, 29(2):286–299. 25

Bibliography

149

Ling, H., Yang, X., and Latecki, L. (2010). Balancing deformability and discriminability
for shape matching. In European Conference on Computer Vision. 25
Liu, S., Feng, J., Song, Z., Zhang, T., Lu, H., Changsheng, X., and Yan, S. (2012a). Hi,
magic closet, tell me what to wear! In ACM International Conference on Multimedia.
108, 123
Liu, S., Song, Z., Liu, G., Xu, C., Lu, H., and Yan, S. (2012b). Street-toshop: Crossscenario clothing retrieval via parts alignment and auxiliary set. In IEEE Conference
on Computer Vision and Pattern Recognition. 106, 108
Lowe, D. G. (2004). Distinctive image features from scale-invariant keypoints. International Journal of Computer Vision, 60(2):91–110. 1, 9, 21, 23, 25, 35, 42, 49, 50, 51,
53, 60
Mikolajczyk, K. and Schmid, C. (2005). A performance evaluation of local descriptors.
IEEE Transactions Pattern Analylis and Machine Intelligence, 10(27):1615–1630. 23,
25, 37, 38, 52
Miller, G. A. (1995). Wordnet: a lexical database for english. Communications of theACM, 38(11):39–41. 115
Mobahi, H., Collobert, R., and Weston, J. (2009). Deep learning from temporal coherence in video. In International Conference in Machine Learning. 51
Morel, J. and Yu, G. (2009). ASIFT: A new framework for fully aﬃne invariant image
comparison. SIAM Journal on Imaging Sciences, 2(2):438–469. 23
Moreno-Noguer, F. (2011). Deformation and illumination invariant feature point descriptor. In IEEE Conference on Computer Vision and Pattern Recognition, pages
1593–1600. 24, 26
Moreno-Noguer, F. and Fua, P. (2013). Stochastic exploration of ambiguities for nonrigid shape recovery. IEEE Transactions Pattern Analylis and Machine Intelligence,35(2):463–475. 47, 48, 67, 99
Moreno-Noguer, F., M.Salzmann, Lepetit, V., and Fua, P. (2009). Capturing 3d stretchable surfaces from single images in closed form. In IEEE Conference on Computer
Vision and Pattern Recognition, pages 1842–1849. 67
Moreno-Noguer, F. and Porta, J. (2011). Probabilistic simultaneous pose and non-rigid
shape recovery. In IEEE Conference on Computer Vision and Pattern Recognition,
pages 1289 –1296. 67
Moreno-Noguer, F., Porta, J., and Fua, P. (2010). Exploring ambiguities for monocular
non-rigid shape estimation. In European Conference on Computer Vision, pages
370–383. 67, 87, 88, 89, 99
Murillo, A. C., Kwak, I. S., Bourdev, L., Kriegman, D., and Belongie, S. (2012). Urban
tribes: Analyzing group photos from a social perspective. In CVPR Workshops. 106,108, 123

150

Bibliography

Okada, R. and Soatto, S. (2008). Relevant Feature Selection for Human Pose Estimationand Localization in Cluttered Images. In European Conference on Computer Vision,
pages 434–445, Berlin, Heidelberg. Springer-Verlag. 84, 85
Oliva, A. and Torralba, A. (2001). Modeling the shape of the scene: A holistic
representation of the spatial envelope. International Journal of Computer Vision,
42(3):145–175. 12
Osendorfer, C., Bayer, J., Urban, S., and van der Smagt, P. (2013). Convolutional
neural networks learn compact local image descriptors. In International Conference
on Neural Information Processing. 50
PAMI-TC (2013). The July 2013 edition of the PAMI-TC newsletter. http://www.
computer.org/web/tcpami/july-2013. [Online; accessed 3-February-2015]. 140
P.Arbelaez, M.Maire, C.Fowlkes, and J.Malik (2011). Contour detection and hierarchical image segmentation. In IEEE Transactions Pattern Analylis and MachineIntelligence. 9, 109
Pelletier, B. (2005). Kernel Density Estimation on Riemannian Manifolds. Statistics &Probability Letters, 73(3):297 – 304. 74
Pennec, X. (2006). Intrinsic Statistics on Riemannian Manifolds: Basic Tools for Geometric Measurements. Journal of Mathematical Imaging and Vision, 25(1):127–154.
65, 75Pennec, X., Fillard, P., and Ayache, N. (2006). A Riemannian framework for tensor
computing. International Journal of Computer Vision, 66(1):41–66. 65
Pepik, B., Stark, M., Gehler, P., and Schiele, B. (2012). Teaching 3D Geometry to
Deformable Part Models. In IEEE Conference on Computer Vision and Pattern
Recognition, Providence, RI, USA. accepted as oral. 86, 101
Philbin, J., Isard, M., Sivic, J., and Zisserman, A. (2010). Descriptor learning foreﬃcient retrieval. In European Conference on Computer Vision, pages 677–691. 40
Pinkall, U. and Polthier, K. (1993). Computing discrete minimal surfaces and theirconjugates. Experimental Mathematics, 2(1):15–36. 33
Pirsiavash, H., Vondrick, C., and Torralba, A. (2014). Assessing the quality of actions.
In European Conference on Computer Vision. 139
Price, E. (2014). The NIPS Experiment. http://blog.mrtz.org/2014/12/15/
the-nips-experiment.html. [Online; accessed 3-February-2015]. 140
Quiñonero-candela, J., Rasmussen, C. E., and Herbrich, R. (2005). A Unifying View
of Sparse Approximate Gaussian Process Regression. Journal of Machine Learning
Research, 6:1939–1959. 65
Ramakrishna, V., Kanade, T., and Sheikh, Y. A. (2012). Reconstructing 3D Human
Pose from 2D Image Landmarks. In European Conference on Computer Vision. 85

Bibliography

151

Ramanan, D. (2006). Learning to parse images of articulated bodies. In NIPS, volume 19, page 1129. 83, 100
Raviv, D., Bronstein, M. M., Sochen, N., Bronstein, A. M., and Kimmel, R. (2011).
Aﬃne-invariant diﬀusion geometry for the analysis of deformable 3d shapes. In IEEE
Conference on Computer Vision and Pattern Recognition. 24
Reuter, M., Wolter, F., and Peinecke, N. (2006). Laplace-beltrami spectra as ’shapedna’ of surfaces and solids. Computer Aided Design, 38(4):342–366. 26, 27, 28, 29
Rogez, G., Rihan, J., Ramalingam, S., Orrite, C., and Torr, P. (2008). Randomized
Trees for Human Pose Detection. In IEEE Conference on Computer Vision and
Pattern Recognition. 84, 85
Rubio, A., Villamizar, M., Ferraz, L., nate Sńchez, A. P., Ramisa, A., Simo-Serra, E.,
Moreno-Noguer, F., and Sanfeliu, A. (2015). Eﬃcient Pose Estimation for Complex
3D Models. In International Conference on Robotics and Automation. 13
Rustamov, R. (2007). Laplace-beltrami eigenfunctions for deformation invariant shape
representation. In Eurographics Symposium on Geometry Processing, pages 225–233.
23, 26, 27
Said, S., Courtry, N., Bihan, N. L., and Sangwine, S. (2007). Exact Principal Geodesic
Analysis for data on SO(3). In European Signal Processing Conference. 74
Salzmann, M. and R.Urtasun (2010). Combining Discriminative and Generative Methods for 3D Deformable Surface and Articulated Pose Reconstruction. In IEEE Conference on Computer Vision and Pattern Recognition, pages 647–654. 85, 87
Sanchez, J., Ostlund, J., Fua, P., and Moreno-Noguer, F. (2010). Simultaneous pose,correspondence and non-rigid shape. In IEEE Conference on Computer Vision and
Pattern Recognition, pages 1189–1196. 23, 25
Schwing, A., Hazan, T., Pollefeys, M., and Urtasun, R. (2011). Distributed message
passing for large scale graphical models. In IEEE Conference on Computer Vision
and Pattern Recognition. 19, 114, 129
Schwing, A. G., Hazan, T., Pollefeys, M., and Urtasun, R. (2012). Eﬃcient structured
prediction with latent variables for general graphical models. In International Conference in Machine Learning. 114, 129
Sermanet, P., Chintala, S., and LeCun, Y. (2012). Convolutional neural networks
applied to house numbers digit classiﬁcation. In International Conference on Pattern
Recognition. 52
Serradell, E., Glowacki, P., Kybic, J., Moreno-Noguer, F., and Fua, P. (2012). Robust
non-rigid registration of 2d and 3d graphs. In IEEE Conference on Computer Vision
and Pattern Recognition, pages 996–1003. 23, 25
Shi, L., Yu, Y., and Feng, N. B. W.-W. (2006). A fast multigrid algorithm for meshdeformation. ACM SIGGRAPH, 25(3):1108–1117. 26

152

Bibliography

Sigal, L., Balan, A., and Black, M. (2010a). Humaneva: Synchronized video and motion
capture dataset and baseline algorithm for evaluation of articulated humanămotion.
International Journal of Computer Vision, 87(1-2):4–27. 92
Sigal, L., Balan, A. O., and Black, M. J. (2010b). HumanEva: Synchronized Videoand Motion Capture Dataset and Baseline Algorithm for Evaluation of Articulated
Human Motion. International Journal of Computer Vision, 87(1-2):4–27. 71, 85, 100
Sigal, L., Bhatia, S., Roth, S., Black, M., and Isard, M. (2004). Tracking loose-limbed
people. In IEEE Conference on Computer Vision and Pattern Recognition. 65, 66
Sigal, L. and Black, M. J. (2006). Predicting 3D People from 2D Pictures. In Conferenceon Articulated Motion and Deformable Objects, pages 185–195. 87
Sigal, L., Isard, M., Haussecker, H. W., and Black, M. J. (2012). Loose-limbed People:
Estimating 3D Human Pose and Motion Using Non-parametric Belief Propagation.
International Journal of Computer Vision, 98(1):15–48. 65, 69
Sigal, L., Memisevic, R., and Fleet, D. (2009). Shared kernel information embedding
for discriminative inference. In IEEE Conference on Computer Vision and Pattern
Recognition, pages 2852–2859. IEEE. 84, 85
Simo-Serra, E., Fidler, S., Moreno-Noguer, F., and Urtasun, R. (2014a). A High Performance CRF Model for Clothes Parsing. In Asian Conference on Computer Vision.
3, 122, 123
Simo-Serra, E., Fidler, S., Moreno-Noguer, F., and Urtasun, R. (2015a). Neuroaesthetics in Fashion: Modeling the Perception of Fashionability. In IEEE Conference on
Computer Vision and Pattern Recognition. 3
Simo-Serra, E., Quattoni, A., Torras, C., and Moreno-Noguer, F. (2013). A Joint
Model for 2D and 3D Pose Estimation from a Single Image. In IEEE Conference on
Computer Vision and Pattern Recognition. 3
Simo-Serra, E., Ramisa, A., Alenyà, G., Torras, C., and Moreno-Noguer, F. (2012). Single Image 3D Human Pose Estimation from Noisy Observations. In IEEE Conference
on Computer Vision and Pattern Recognition. 2, 3
Simo-Serra, E., Torras, C., and Moreno-Noguer, F. (2014b). Geodesic Finite Mixture
Models. In British Machine Vision Conference. 3, 125, 128
Simo-Serra, E., Torras, C., and Moreno-Noguer, F. (2015b). DaLI: Deformation andLight Invariant Descriptor. International Journal of Computer Vision, 1:1–1. 2, 49
Simo-Serra, E., Torras, C., and Moreno-Noguer, F. (2015c). Lie Algebra-Based Kinematic Prior for 3D Human Pose Tracking. In International Conference on Machine
Vision and Applications. 3, 138
Simo-Serra, E., Trulls, E., Ferraz, L., Kokkinos, I., and Noguer, F. M. (2014c). Fracking
Deep Convolutional Image Descriptors. CoRR, abs/1412.6537. 2

Bibliography

153

Simonyan, K., Vedaldi, A., and Zisserman, A. (2014). Learning local feature descriptors using convex optimisation. IEEE Transactions Pattern Analylis and MachineIntelligence, 36(8):1573–1585. 50
Simonyan, K. and Zisserman, A. (2014). Very deep convolutional networks for largescale image recognition. CoRR, abs/1409.1556. 139
Singh, V. K., Nevatia, R., and Huang, C. (2010). Eﬃcient Inference with Multiple
Heterogeneous Part Detectors for Human Pose Estimation. In European Conference
on Computer Vision, pages 314–327. 86, 87
Sminchisescu, C. and Jepson, A. (2004). Generative Modeling for Continuous NonLinearly Embedded Visual Inference. In International Conference in Machine Learning, pages 96–. 84, 85
Sochen, N., Kimmel, R., and Malladi, R. (1998). A general framework for low levelvision. IEEE Transactions on Image Processing, 7(3):310–318. 26
Socher, R., Perelygin, A., Wu, J., Chuang, J., Manning, C. D., Ng, A. Y., and Potts,
C. (2013). Recursive deep models for semantic compositionality over a sentimenttreebank. In Conference on Empirical Methods on Natural Language Processing. 125,
129
Sommer, S., Lauze, F., Hauberg, S., and Nielsen, M. (2010). Manifold Valued Statistics, Exact Principal Geodesic Analysis and the Eﬀect of Linear Approximations. In
European Conference on Computer Vision. 65, 73, 77
Sommer, S., Tatu, A., Chen, C., Jurgensen, D., De Bruijne, M., Loog, M., Nielsen,
M., and Lauze, F. (2009). Bicycle Chain Shape Models. In IEEE Conference on
Computer Vision and Pattern Recognition. 74
Song, Z., Wang, M., s. Hua, X., and Yan, S. (2011). Predicting occupation via humanclothing and contexts. In International Conference on Computer Vision. 106, 108,
123
Sun, J., Ovsjanikov, M., and Guibas, L. (2009). A concise and provably informative
multi-scale signature based on heat diﬀusion. In Eurographics Symposium on Geometry Processing, pages 1383–1392. 23, 24, 26, 27, 28
Szegedy, C., Toshev, A., and Erhan, D. (2013). Deep neural networks for object detection. In Neural Information Processing Systems. 49, 50
Tang, F., Lim, S. H., Chang, N., and Tao, H. (2009). A novel feature descriptor invariant
to complex brightness changes. In IEEE Conference on Computer Vision and Pattern
Recognition, pages 2631–2638. 26
Taylor, C. (2000). Reconstruction of articulated objects from point correspondences ina single uncalibrated image. CVIU, 80:349–363. 87
Taylor, G., Sigal, L., Fleet, D., and Hinton, G. (2010). Dynamical binary latent variable
models for 3d human pose tracking. In IEEE Conference on Computer Vision and
Pattern Recognition. 65, 66

154

Bibliography

Tenenbaum, J. B., Silva, V., and Langford, J. C. (2000). A Global Geometric Framework
for Nonlinear Dimensionality Reduction. Science, 290(5500):2319–2323. 67
Tian, T.-P. and Sclaroﬀ, S. (2010). Fast Globally Optimal 2D Human Detection withLoopy Graph Models. In IEEE Conference on Computer Vision and Pattern Recognition. 86, 87
Todt, E. and Torras, C. (2004). Detecting salient cues through illumination-invariant
color ratios. Robotics and Autonomous Systems, 48(2):111–130. 11
Tola, E., Lepetit, V., and Fua, P. (2010). Daisy: An eﬃcient dense descriptor applied towide-baseline stereo. IEEE Transactions Pattern Analylis and Machine Intelligence,32(5):815–830. 9, 23, 25, 42
Torresani, L., Kolmogorov, V., and Rother, C. (2008). Feature correspondence via graphmatching: Models and global optimization. In European Conference on Computer
Vision, volume 2, pages 596–609. 23, 25
Toshev, A. and Szegedy, C. (2014). Deeppose: Human pose estimation via deep neuralnetworks. In IEEE Conference on Computer Vision and Pattern Recognition. 86
Tournier, M., Wu, X., Courty, N., Arnaud, E., and Revéret, L. (2009). Motion compression using principal geodesics analysis. In Proceedings of Eurographics. 77
Trulls, E., Kokkinos, I., Sanfeliu, A., and Moreno-Noguer, F. (2013). Dense
segmentation-aware descriptors. In IEEE Conference on Computer Vision and Pattern Recognition. 25, 49
Trulls, E., Tsogkas, S., Kokkinos, I., Sanfeliu, A., and Moreno-Noguer, F. (2014).
Segmentation-aware deformable part models. In IEEE Conference on Computer Vision and Pattern Recognition. 25
Trzcinski, T., Christoudias, .and Lepetit, V., and Fua, P. (2012). Learning image
descriptors with the boosting-trick. In Neural Information Processing Systems. 50
Uijlings, J. R. R., van de Sande, K. E. A., Gevers, T., and Smeulders, A. W. M. (2013).
Selective search for object recognition. International Journal of Computer Vision,
104(2):154–171. 112
Urtasun, R., Fleet, D. J., and Fua, P. (2006). 3D People Tracking with GaussianProcess Dynamical Models. In IEEE Conference on Computer Vision and Pattern
Recognition. 65, 83, 84, 85
Urtasun, R., Fleet, D. J., and Lawrence, N. D. (2007). Modeling human locomotion with
topologically constrained latent variable models. In Proceedings of the 2nd Conferenceon Human Motion: Understanding, Modeling, Capture and Animation. 65
Vaxman, A., Ben-Chen, M., and Gotsman, C. (2010). A multi-resolution approach toheat kernels on discrete surfaces. ACM SIGGRAPH, 29(4):121. 26
Vedaldi, A. and Fulkerson, B. (2008). VLFeat: An open and portable library of computer vision algorithms. http://www.vlfeat.org/. 42

Bibliography

155

Vedaldi, A. and Soatto, S. (2005). Features for recognition: Viewpoint invariancefor non-planar scenes. In International Conference on Computer Vision, pages
1474–1481. 25
Wang, H. and Koller, D. (2011). Multi-level inference by relaxed dual decomposition
for human pose segmentation. In IEEE Conference on Computer Vision and Pattern
Recognition. 108
Wang, J., Fleet, D., and Hertzmann, A. (2005). Gaussian process dynamical models.
In Neural Information Processing Systems. 64, 65, 66
Wang, J., Fleet, D., and Hertzmann, A. (2008). Gaussian process dynamical models
for human motion. IEEE Transactions Pattern Analylis and Machine Intelligence,30(2):283–298. 65
Wang, N. and Ai, H. (2011). Who blocks who: Simultaneous clothing segmentation forgrouping images. In International Conference on Computer Vision. 106
Wang, Z., Fan, B., and Wu, F. (2011). Local intensity order pattern for feature description. In International Conference on Computer Vision, pages 603 –610. 23, 26,
42
Wesseling, P. (2004). An Introduction to Multigrid Methods. John Wiley & Sons. 26
Winder, S., Hua, G., and Brown, M. (2009). Picking the best daisy. In IEEE Conference
on Computer Vision and Pattern Recognition. 22
Xiao, J., Hays, J., Ehinger, K. A., Oliva, A., and Torralba, A. (2010). Sun database:Large-scale scene recognition from abbey to zoo. In IEEE Conference on Computer
Vision and Pattern Recognition. 125, 128
Yamaguchi, K., Kiapour, M. H., and Berg, T. L. (2013). Paper doll parsing: Retrieving
similar styles to parse clothing items. In International Conference on Computer
Vision. 106, 108, 115, 116, 117, 119, 122
Yamaguchi, K., Kiapour, M. H., Ortiz, L. E., and Berg, T. L. (2012). Parsing clothing in fashion photographs. In IEEE Conference on Computer Vision and Pattern
Recognition. 106, 107, 108, 109, 110, 115, 116, 117, 118, 119, 120, 121, 122, 139
Yang, W., Luo, P., and Lin, L. (2014). Clothing co-parsing by joint image segmentationand labeling. In IEEE Conference on Computer Vision and Pattern Recognition. 123
Yang, Y. and Ramanan, D. (2011). Articulated Pose Estimation with Flexible Mixturesof-Parts. In IEEE Conference on Computer Vision and Pattern Recognition. 13, 69,
83, 84, 86, 87, 88, 98, 99, 100, 101, 108, 109, 110, 111, 117
Yanowitz, S. and Bruckstein., A. (1989). A new method for image segmentation. Computer Vision, Graphics, and Image Processing, 46(1):82–95. 26
Yao, A., Gall, J., and Van Gool, L. (2012a). Coupled action recognition and pose estimation from multiple views. International Journal of Computer Vision, 100(1):16–37.
83

156

Bibliography

Yao, J., Fidler, S., and Urtasun, R. (2012b). Describing the scene as a whole: Joint
object detection, scene classiﬁcation and semantic segmentation. In IEEE Conference
on Computer Vision and Pattern Recognition. 108
Yezzi, A. (1998). Modiﬁed curvature motion for image smoothing and enhancement.
IEEE Transactions on Image Processing, 7(3):345–352. 26
Zeiler, M. and Fergus, R. (2014). Visualizing and understanding convolutional networks.
In European Conference on Computer Vision. 139
Zhao, X., Fu, Y., and Liu, Y. (2011). Human Motion Tracking by TemporalSpatial Local Gaussian Process Experts. IEEE Transactions on Image Processing,20(4):1141–1151. 83, 85