Fashion Style in 128 Floats:
Joint Ranking and Classification using Weak Data for Feature Extraction
Edgar Simo-Serra and Hiroshi Ishikawa
Department of Computer Science and Engineering
Waseda University, Tokyo, Japanesimo@aoni.waseda.jp

hfs@waseda.jp

Abstract
Classiﬁer

We propose a novel approach for learning features fromweakly-supervised data by joint ranking and classification.
In order to exploit data with weak labels, we jointly train afeature extraction network with a ranking loss and a classification network with a cross-entropy loss. We obtain highquality compact discriminative features with few parameters, learned on relatively small datasets without additionalannotations. This enables us to tackle tasks with specialized
images not very similar to the more generic ones in existing fully-supervised datasets. We show that the resulting
features in combination with a linear classifier surpass thestate-of-the-art on the Hipster Wars dataset despite using
features only 0.3% of the size. Our proposed features significantly outperform those obtained from networks trainedon ImageNet, despite being 32 times smaller (128 singleprecision floats), trained on noisy and weakly-labeled data,
and using only 1.5% of the number of parameters.1 .

Sharedparameters

FeatureCNN

FeatureCNN

Feature
CNN

I−

I

I＋

Dissimilar

SimilarFigure 1: Overview of the proposed feature learning approach. We train a feature extraction on weakly annotateddata by jointly training a feature extractor network with aclassification network. For training, an anchor image (center) is provided in conjunction with a similar image (right)
and a dissimilar image (left) according to a metric provided
on the weak noisy annotations. The classification loss lC
serves to learn useful feature maps while the ranking loss
lR on the triplet of Feature CNN encourages them to learna discriminative feature representation.

1. Introduction
With the emergence of large-scale datasets and the appearance of deep networks with millions of parameters, researchers have started to replace hand-crafted global image
features such as GIST [21] with those obtained from intermediate representations of deep networks trained for classification on large datasets [44]. Although this has led to agreat improvement over the previous generation of features,
these networks are learned in a fully-supervised manner onlarge amounts of data with very costly and time-consuming
annotation. Features learned on one dataset can be used onanother, but naturally not all datasets are equal [32], andthus features taken from networks trained on ImageNet [7]will not work as well on datasets with very different visual characteristics, such as the scene classification dataset
Places [49], and vice versa. While unsupervised featurelearning exists as an alternative to supervised learning, thelack of guidance to what to learn given by explicit labels
makes it a much more complex task [19].
However, images obtained from the Internet usually haveassociated metadata which, although often inaccurate, canbe used as weak labels. In this paper, we study how to exploit data with weak labels in order to obtain high-quality
compact discriminative features without additional annotations. With such features, we tackle tasks in which the images are more specific and not very similar to those of existing fully-supervised datasets such as ImageNet or Places.
In this work, we focus on the domain of fashion images,
which have only recently become the focus of research [40].
These images have several characteristics that make them

1 Models available at http://hi.cs.waseda.ac.jp/ esimo/research/stylenet/1

Hipster Wars (1,893 images)

Mean

Places (2,469,371 images)

Mean

Fashion144k (277,527 images)

Mean

ImageNet (1,331,167 images)

Mean

Figure 2: We show example images and the mean image from the Hipster Wars [16], Fashion144k [27], Places [49], andImageNet [7] datasets. In both fashion-related datasets we can make out a human silhouette, although it is significantly
more diffuse in the Fashion144k dataset due to the much larger pose variation. In the Places dataset mean image we cansee a gradient where the top of the image is clearer, likely corresponding to the sky in many images. While in the ImageNetmean we see a much more uniform image with a slightly clearer area in the center of the image. Unlike the other datasets,
Fashion144k only has weak labels. With our approach we are able to exploit the Fashion144k for training to evaluate on the
Hipster Wars dataset, outperforming fully supervised approaches that use larger datasets such as ImageNet or Places.
very challenging to tackle with computer vision. On one
hand, they have small local details such as accessories that
only depend on a very small part of the image, making segmentation very challenging [26, 40, 43]. On the other hand,
we still have to consider more global properties such as the
fashion style [16, 27, 34], which depend jointly on the various items in the image. These difficulties, along with thefact that the images generally have a 3:4 aspect ratio andtend to have much brighter colors, cause the features takenfrom networks trained on ImageNet or Places to generalize poorly to fashion-oriented tasks. While no large fullyannotated fashion dataset exists, there are many datasets,such as the Paperdoll [41] and Fashion144k [27] datasets,that have a large amount of weak labels exploitable forlearning. In this work, we take advantage of these noisylabels to learn compact discriminative features with deep
networks that can then be used in other challenging fashionrelated tasks, such as style classification [16], in which theygreatly outperform the state-of-the-art and other pretrained
CNN baselines, while having 1.5% the number of parameters and being the size of a SIFT descriptor [20]: 128 floats
and 32× smaller than the best competing approach.
Instead of training networks for classification and using an intermediate-layer representation as a feature vector,
we propose performing joint classification and ranking as
shown in Fig. 1. The ranking acts as a soft constraint on theintermediate layer and encourages the model to learn more
representative features guided by the classifier. We performranking by considering three images simultaneously: wefirst pick an anchor image and then pick an image that issimilar to the anchor and one that is different. We establish
a similarity metric by exploiting the weak user-provided labels, which are also used as a classification target. By simultaneously considering both ranking and classification, weare able to outperform approaches that use either ranking orclassification alone. Our approach allows us to efficiently

make use of weak labels to learn features on datasets closer
to the target application, as shown in Fig. 2.
In summary, our novel approach for feature learning:• Exploits large amounts of weakly-labeled data commonly
found on the Internet.
• Learns a compact discriminative representation with few
parameters on relatively small datasets.
• Allows for efficient comparisons by Euclidean distances.
We complement our in-depth quantitative analysis with visualizations for qualitative analysis. In addition to ourfeature extraction network learning approach we present a
novel visualization approach for comparing image similarity between two images by exploiting the change in the extracted features when partially occluded.

2. Related Work
Fashion: Interest in fashion has been growing in thecomputer vision community. Some of the more traditional problems have been semantic segmentation of garments [26, 41, 43], image retrieval [12, 15], and classification of garments and styles [2, 5, 16, 35, 42]. Attempting todirectly predict more esoteric measurements, such as popularity [38] or fashionability [27], have also been recently
studied. As defining absolute metrics is rather complicatedin such a subjective domain as fashion, exploiting relative attributes [17] and learning image similarity [15, 34]
have also been proposed. Many of these approaches rely
on datasets created by crawling the Internet and have large
amounts of exploitable weak labels [27, 39]. Yamaguchi
et al. [39] used these tags to provide priors for semantic
segmentation. Our method is complementary to these approaches and can provide features for greater performance.
Deep Learning: There has been a flurry of new research
focusing on exploiting deep learning in computer vision.
Much of it focuses on improving classification results onlarge datasets such as ImageNet [7] or Places [49]. Thishas led from initial models such as Alexnet [18] to more sophisticated ones such as the VGG models [29] with up to 19
layers; or the Googlenet models [31], that jointly use convolutions of different sizes in each layer. A more structured
analysis of different networks was presented by Chatfield
et al. [4], where they also analyzed using bottleneck layers
to provide features of different sizes. All these approaches
rely on fully supervised datasets. Xiao et al. [37] extended
learning to partially noisy labels. However, they still requireroughly 50% of the labels to be correct and need to learn another network to correct the noisy labels, while only noisylabels suffice for our approach.
Deep Similarity: Instead of learning classification networks, it is possible to directly learn similarity using deepneural networks. A popular approach consists of Siamese
networks [3], in which a pair of inputs is used simultaneously to train a neural network model. The loss encourages
similar inputs to have similar network outputs and dissimilar inputs to have dissimilar network outputs. This method
has been recently applied with great success to local feature descriptors [10, 28, 45] and also for obtaining better
representations of product images [1, 34]. It has also been
extended to triplet images for ranking [11, 36] and has beenvery successfully applied to face recognition [25] in particular. We build upon this concept of image triplets for ourranking loss and show that by combining the ranking withclassification results can be significantly improved.
Weak Data: We can identify two major sources of weaklabels when using deep networks: text [9, 14] and image
tags [22, 23]. Text can generally be found accompanying images and thus can be directly exploited as a formof weak label. Frome et al. [9] use text accompanying
images to train more semantically meaningful classifiers,
while Karpathy and Fei-Fei [14] use them as a form of weakannotation of the objects that lie in the image to perform
localization. More recently, it has been seen that detectors
seem to emerge when training deep networks for classification [48]. This has been utilized to learn models for semantic segmentation [22, 23]. However, as far as we know,
we are the first to propose leveraging user-provided tags tolearn discriminative features for a specific domain.

3. Method
We present a method for learning discriminative featuresfrom weakly-labeled data by jointly training both a feature
extraction network and a classification network. A ranking
loss on triplets of images is applied on the feature extraction network whose output is then fed into the classification network, where a classification loss is employed. After
training, the feature extraction network can be used to provide discriminative features for other algorithms without aneed for the classification network.

3.1. Joint Ranking and Classification
We formulate the problem as a joint ranking and classification problem. A ranking loss on triplets of images is applied on a feature extraction network, while a classification
loss is employed on a classification network that uses theoutput of the feature extraction network. For training, we
assume that we have a set of images with associated weaklabels with large amounts of noise.
For ranking, we take three images as input simultaneously, as shown in Fig. 1. One image (center) is the anchor or reference image, to which the second image (right)
is similar and the last image (left) is dissimilar. We assume
that we have a similarity metric r(·, ·) between the weak labels of a pair of images. We consider two thresholds τs andτd that, given this metric, determine when two images are
similar and dissimilar, respectively. Thus, two images I1
and I2 , with labels y1 and y2 respectively, will be similar
when r(y1 , y2 ) > τs and dissimilar when r(y1 , y2 ) < τd .
We will define each image triplet as T = (I− , I, I+ ) wherer(y− , y) < τd and r(y, y+ ) > τs .
Suppose we have a set T of possible noisy tags, or attributes of an image like red-sweater or even just red. A label l = (lt )t∈T for an image assigns lt ∈ {0, 1} to each tag
t. If a tag applies to an image (e.g., if the tag is red-sweaterand a red sweater appears in the image), the label for theimage assigns 1 to the tag. Note that we assume these tags
to be noisy and not exact. Let |l| be the number of tags thatlabel l assigns 1. We propose using the similarity function
between labels a and b defined as “intersection over union”:|a ∧ b|r(a, b) =,(1)|a ∨ b|where ∧ and ∨ operate on the labels as tag-wise minimum
and maximum, respectively.
Given an image triplet T , we use the feature extraction
network to obtain a triplet of features Tf = (f− , f , f+ ),where f is a feature vector of each image. We then computea distance between two feature vectors and apply a ranking
loss that encourages the distance d+ between the anchor
and the similar image to be smaller than the distance d−between the anchor and the dissimilar image.
For comparing features, we consider the Euclidean distance k · k2 . In contrast with [25], we normalize the distances instead of normalizing the feature pairs to have unitary norm. This changes the hard constraint into a soft constraint, which is essential for being able to learn using aclassification loss and ranking loss simultaneously. We normalize the pair of distances (d− , d+ ) obtained from the feature triplet using the softmax operator:
exp(kf− − f k2 )exp(kf− − f k2 ) + exp(kf+ − f k2 )exp(kf+ − f k2 )d+ =.
exp(kf− − f k2 ) + exp(kf+ − f k2 )

d− =

(2)
(3)

With distances now normalized to the [0, 1] range, we definea ranking loss lR that maximizes the dissimilar distance d−and minimizes the similar distance d+ :lR (d+ , d− ) = 0.5 (d+ )2 + (1 − d− )2 = (d+ )2 , (4)which is 0 only when kf+ − f k2 = 0 and kf− − f k2 > 0.
In contrast to [36], the loss is normalized; because of that,
we do not need to use large amounts of weight decay inorder to ensure that the output of the network does not tend
to infinity. In fact, we find that the implicit regularization
provided by dropout and batch normalization are sufficientand do not rely on weight decay at all.
While the ranking loss lR by itself should be sufficient tolearn discriminative features, we found in practice that it iscritical to complement it with a classification loss. We do
this by employing a separate classification network that uses
the features of the dissimilar image f− and outputs a pretttt
diction value X− = (X−)t∈T , X−= (X−,0, X−,1) ∈ R2for each binary value on each tag. We do not use the anchor
image features f nor the similar image features f+ , as they
form a subset of the training images, unlike the dissimilar
images, which are chosen randomly. With y− as the noisy
target label for the input image, we use multi-label crossentropy loss for classification:lC (X− , y− ) =1 Xttl× (X−
, y−
),
|T |

(5)

t∈T

l× (x, y) = −xy + log (exp(x0 ) + exp(x1 )) .

(6)

Finally, we combine both losses to obtain the model loss:l(d+ , d− , X− , y− ) = lR (d+ , d− ) + αlC (X− , y− ) , (7)where α is a weight to balance the different loss functions.
The classification loss lC affects both the feature extraction network and the classification network, while the ranking loss lR only affects the feature extraction network.

3.2. Feature Extraction Network
We follow the approach of [29] of using 3x3 kernels
for the convolutional filters to keep the number of weights
down for the network and allow increasing the number oflayers. One pixel padding is used to keep the input size andoutput size of the convolutional layers constant. In order toallow efficient learning the entire network from scratch, werely on Batch Normalization layers [13]. Dropout [30] isused to prevent overfitting throughout the architecture.
A full overview of the architecture can be seen in Table 1. We note two important differences with commonlyused networks: firstly, it uses a 3:4 aspect ratio for the input
images as they are dominant in the fashion community; andsecondly, it has a very small number of parameters compared to widely-used models. This is due to using only a

Table 1: Feature extraction network architecture. All convolutional layers have 1 × 1 padding and all layers besides
the max pooling layer have a 1 × 1 stride, while the max
pooling layers have a 4 × 4 stride.
type

kernel size

output size

paramsconvolution
convolution
dropout (25%)

3×3
3×3

384x256x64
384x256x64
384x256x64

1,792
36,928max poolingbatch normalization
convolution
convolution
dropout (25%)

4×4

96x64x64
96x64x64
96x64x128
96x64x128
96x64x128

128
73,856
147,584

max poolingbatch normalization
convolution
convolution
dropout (25%)

4×4

24x16x128
24x16x128
24x16x256
24x16x256
24x16x256

256
295,168
590,080

max poolingbatch normalization
convolution
fully-connected

4×4

3×3
3×3

3×3
3×3

3×3

TOTAL

6x4x256
6x4x256
6x4x128128

512
32,896393,344128

1,572,544

single fully-connected layer with 128 hidden neurons anddecreasing the number of filters before the fully-connected
layer. This allows the model to have high performance
while having only 1,572,544 parameters. In comparison,
the VGG 16 layer network [29] has 134,260,544 parameters when considering only feature extraction.

3.3. Classification Network
Due to the noisy nature of the weak labels, the objectiveof the classification network is to aid the learning of the feature extraction network and not high classification performance. It consists of a batch normalization layer, followed
by a rectified linear unit layer, a linear layer with 128 hidden units, and finally another linear layer which outputs theset of predictions X for classification. This network is kept
small to encourage the propagation of gradients into the feature extraction network and hasten the learning. The initial batch normalization and rectified linear unit layers help
partially isolate the classification network from the feature
extraction network. When learning with 123 tags, the classification network has a total of only 48,502 parameters.

3.4. Joint Learning
Both networks are trained jointly using backpropagation [24]. Instead of using stochastic gradient descent which
is dependent on setting a learning-rate hyperparameter, we
utilize the ADADELTA algorithm [46], which adaptively
sets the learning rate each iteration. No image cropping,momentum, nor weight decay is used. The only image preprocessing consists of subtracting the mean from each color
channel and dividing by the standard deviation.
Initialization is critical for learning both networks: evenwith batch normalization, we were unable to train both networks jointly from scratch. We overcome this issue by
first training the feature extraction network with an additional fully-connected layer for classification (Eq. (5)).
Once the optimization has converged, the additional classification layer is removed from the feature extraction network and the classification network is added with random
weights. Finally, both networks are trained jointly.
Since it is impossible to precompute all the possible values of the similarity metric r(·, ·) for large datasets, we use
a simple sampling approach for the triplet of images whenusing the ranking loss. We initially choose a random anchor
image I. We then randomly sample an image Ir and checkto see if r(I, Ir ) > τs or r(I, Ir ) < τd . In the first case, Ir
is added as the similar image I+ and in the latter case it isadded as the dissimilar image I− to the image triplet. Thisis done until the image triplet is completely formed. If it is
not formed in a set number of iterations, a new anchor image is chosen and the procedure is restarted.

4. Experimental Results
We implement our approach using the Torch7 framework [6]. We train our model on the Fashion144k
dataset [27], and evaluate on the Hipster Wars dataset [16].
We compare our results against publicly available pretrained CNNs, and the state-of-the-art style descriptor [41]baselines. Our approach outperforms all baselines while being more efficient to compute and compact. We also perform additional experiments for the prediction of fashionability and see that we outperform all other approaches in
the accuracy metric. In all cases, our joint classification andranking approach outperforms using either classification or
ranking losses alone, and using a Siamese architecture (described in the supplemental material).

4.1. Cleaning the Dataset
We train on the Fashion144k dataset [27]. Since the images have been obtained from chictopia.com without any
sort of filtering, a large amount of images are not representative of what we wish to learn: a descriptor for fashionstyle. Thus, we would like to clean the data, i.e., take only
the images we consider suitable for training and not others.
An example of images which we wish to classify are shownin Fig. 3. As it is unreasonable to do the cleaning manually,
we train a classifier after a minor amount (6,000 images) ofannotation that can be done in a couple of hours. We willshow that this gives a significant increase in performance.
We annotate the images based on whether or not they
contain a fully-visible person centered in the image (theFigure 3: Examples of clean and dirty images from theFashion144k dataset [27] are shown in the top and bottomrows, respectively. While there is much diversity, the clean
images show figures more or less centered with the wholebody visible, whereas dirty images have strong filters, show
close-ups of objects, and/or are severly cropped.
supplemental material contains more details). We use a 1:1
train-to-test split to finetune the VGG 16 Layers Model [29]pretrained on ImageNet for the binary classification task ofwhether or not an image is suitable for training. We are able
to obtain 94.23% accuracy on the 3,000 test images.
As weak annotations, we use the “color” tags provided
by the Fashion144k dataset which consist of color-garment
tags such as red-sweater or blue-boots, the set of which wedenote by T1 and has 3,180 unique tags. We split the tags
into colors and garments, resulting in a total of 123 uniqueweakly-annotated tags T2 . These tags are the only ones used
when performing classification. However, for the weak
metric (Eq. (1)), we consider the set formed by the unionof the “color” tag set and the split tag set: T = T1 ∪ T2 .
We build a clean version of the Fashion144k dataset by
first filtering out entries for which less than three tags in T2
are assigned 1, to reduce the noise. We additionally filter images using our trained classifier on whether or notthey are suitable. This results in 80,554 training images and8,948 validation images with a 9:1 train to validation split.

4.2. Training the Model
We train with a batchsize of 32 for classification and 16
when jointly training for classification and ranking, due tothe higher memory usage. We use a similarity threshold ofτs = 0.75 and dissimilarity threshold of τd = 0.01. Examples of triplets of images T used for learning can be seen in
Fig. 4. When jointly training for classification and ranking,we set the classification loss weight to α = 0.01 such thatthe losses are of comparable magnitude. We initially train
the feature extraction network with a classification loss. We
then use the best performing model evaluated on the validation set to initialize the weights for the feature extraction
network when using other losses. In particular, we consider
joint classification and ranking loss, only ranking loss, andSiamese loss. We also compare to using the non-cleaned

I−

I

I+

I−

I

I+

Table 2: Comparison with the state-of-the-art on the Hipster
Wars dataset. We evaluate as in [16] by computing the meanof 100 random splits with a 9:1 train to test ratio. For all themodels we additionally display the number of parameters,
and the dimension of the features. Dirty refers to training
on a non-cleaned version of the Fashion 144k dataset. Our
compact features significantly outperform the others.
feature params

Figure 4: Example of triplets of images used for training themodel when using a ranking loss on the cleaned version of
the Fashion144k dataset. For each triplet the anchor image
I is displayed in the center with the dissimilar image I− on
the left and the similar image I+ on the right.
dataset which we denote as “dirty”.

4.3. Hipster Wars Dataset
We evaluate on the Hipster Wars dataset [16], which consists of similar images to the Fashion144k dataset [27] used
to train, but from different sources. The dataset is made upof pictures of upright people in the center of the image; eachcorresponds to one of five styles: hipster, bohemian, goth,preppy, and pinup. The task is to perform 5-way clothingstyle classification. We compare against the state-of-theart which is based on a 39,168-dimensional style descriptor [41] that is built by first estimating the 2D pose, which
is trained in a supervised manner, and then extracting features for 32 × 32 pixel patches around all the pose keypoints. We also consider publicly available standard pretrained networks on ImageNet and Places. All approaches
except ours use fully-supervised noise-free data for training.
We evaluate the features by combining them with a linear SVM [8] with L2 regularization and L2 loss to predict the style. We perform 5-fold cross validation to set
the regularization parameter and evaluate 100 times using
random splits with a 9:1 train to test ratio as done in [16].
We consider the top δ = 0.5 images from each style forclassification. Each of the dimensions of the features arenormalized independently using the training set such thatthe mean is 0 and the standard deviation is 1, except for
approaches that directly learn embeddings, e.g., Ranking,
Joint, and Siamese models. We report accuracy, precision,
recall, and intersection over union (iou) in Table 2. We cansee that all our models outperform all the other approaches.
By cleaning the data and improving the loss objective fromclassification to Siamese, Siamese to Ranking, and finally
Ranking to Joint Classification and Ranking, we are able toimprove performance.
We also consider two other scenarios: no training andfine-tuning. The results for not training and directly usingdim. acc. pre. rec.

iou

Ours Joint
Ours Ranking
Ours Siamese
Ours Classification
Ours Joint Dirty

1.6M
1.6M
1.6M
1.6M
1.6M

128 75.9 75.4 76.5 61.5
128 74.5 74.2 74.5 59.6
128 73.3 72.9 74.0 58.2
128 73.5 71.7 74.1 57.3
128 72.9 72.1 73.1 57.0

Kiapour et al. [16]†‡39,168 70.6 70.6 70.4 54.6

VGG_CNN_M [4]VGG 16 Layers [29]

99M
134M
86M
82M
134M

4096 71.9 72.9 70.9 56.2
4096 70.1 70.5 69.7 54.8
1024 70.4 71.1 69.5 54.2
128 63.5 62.8 63.5 46.3
4096 57.4 57.6 59.4 41.5

VGG_CNN_M_1024 [4]VGG_CNN_M_128 [4]VGG 16 Places [49]† We were unable to reproduce the results of [16] and instead comparewith the results from the confusion matrix they provide in their paper.
‡ Not directly comparable but in the order of hundreds of thousands.

Table 3: Comparison with deep networks using feature distances on the Hipsters Wars dataset. For each image in thedataset, we sort all the remaining images by distance andconsider a top-n match if one of the n nearest images is of
the same class. No training is done at all.
feature

dim.

top-1

top-2

top-3

Ours Joint

128

63.5

79.9

86.3

VGG_CNN_M [4]VGG 16 Layers [29]VGG_CNN_M_128 [4]VGG 16 Places [49]

4096
4096
128
4096

53.2
53.2
44.6
40.1

71.7
71.5
64.0
61.0

81.3
80.4
76.2
72.0

feature distances is shown in Table 3. Our approach clearly
outperforms other approaches, with a 20% increase in performance with respect to 4096-dimensional features and
50% increase with respect to similar size 128-dimensionalfeatures. If we use a single split instead of using 100 random splits and fine-tune the deep networks, we get the results shown in Table 4. Fine-tuning the deep network significantly improves the performance; however, the best performing network is still within 1% of our approach, despiteusing a 32× larger internal feature representation.

4.4. Predicting Fashionability
We also evaluate on the much more complicated task offashionability prediction on the Fashion144k dataset [27].
This consists of rating how fashionable a person in an image is on a scale of 1 to 10. As this is the dataset used fortraining, although with a different objective, we use only

Table 4: Comparison against fine-tuned deep networks on
the Hipster Wars dataset. For the fine-tuned networks, thenumerator is the fine-tuned result and the denominator is theresult of using a logistic regression without fine-tuning.
feature dim.
Ours Joint 128

acc.

pre.

68.4

66.1

rec.

68.4/64.6 67.3/64.2
63.8/63.3 62.6/62.6

Ours
Input

VGG_CNN_M_128

Norm PCA 1 PCA 2 PCA 3 Norm PCA 1 PCA 2 PCA 3

iou

67.9

51.0

68.8/63.0

51.8/46.8

VGG_CNN_M 4096
63.5/61.9 46.5/45.4
VGG 16 Layers 4096
VGG_CNN_M_128 128 62.6/57.2 60.4/55.1 62.1/56.9 44.5/39.0

Table 5: Results on the Fashion144k dataset for the task ofpredicting how fashionable a person is on a scale of 1 to
10. We compare against strong CNN baselines and all variations of our model. We also consider our model architecture with random weights to observe the effect of learning,
and average the results for 10 random initializations. The
model with random weights performs almost the same as
the VGG_CNN_M_128 model that has been pretrained on
ImageNet. Our model outperforms all models in accuracy.
feature

dim.

acc.

pre.

rec.

iou

Ours Joint
Ours Classification
Ours Siamese
Ours Random

128
128
128
128

17.0
14.6
13.9
13.0

14.7
12.7
11.9
10.8

15.2
14.5
24.2
11.5

7.1
6.3
5.8
4.9

VGG 16 Layers [29]VGG 16 Places [49]VGG_CNN_M [4]VGG_CNN_M_128 [4]

4096
4096
4096
128

16.6
15.8
13.2
13.2

15.1
14.0
11.8
10.8

15.7
14.7
11.5
11.7

8.0
7.3
6.0
4.8

the images not used in the training set for evaluation. We
use 8,000 images for training and 948 images for testing.
As with the Hipster Wars dataset, we evaluate the featuresusing a linear SVM with L2 regularization and L2 loss andset the regularization parameter with 5-fold cross validation.
We compare against deep network baselines in Table 5.
We can see our approach is able to significantly outperformthe 128-dimensional feature network. However, it is outperformed by some of the 4096-dimensional features fromdeep networks. This is likely due to the fact that the fashionability score, while correlated with the style of the outfit,
is greatly affected by non-visible factors such as social connections [38], and thus larger features are beneficial. Despite this, our approach outperforms similar networks.

4.5. Visualizing the Style Descriptor
We follow a similar approach to [47] to visualize how the
input image is related to our descriptor. Instead of focusing
on a single neuron, however, we consider the entire style descriptor output by displaying both the norm and projecting
it onto PCA basis. We do this by sliding a 48 × 48 boundingbox around the input image with the mean color of the input

Figure 5: We analyze the relationship between the image
and the style descriptor by moving an occluding box aroundthe image and display the change in the norm of the descriptor and the change of the first three components on PCA
basis computed on all the vectors extracted on the image.
The positive and negative values are encoded in blue andred respectively. The norm is normalized so that the minimum value is white and the maximum value is blue, while
the PCA representations are normalized by dividing by the
maximum absolute value. Large descriptor changes correspond to the location of the individual and the PCA modes
refer to the location of different garments. We compare witha fine-tuned VGG_CNN_M_128 network and see that ourapproach focuses on the figure and not the background.
Image 1

similar

Ours

VGG

Anchor

Ours

VGG

Image 2dissimilar

Figure 6: We analyze how the style of two image matches
by using our descriptor. Blue indicates that the area is dissimilar between the images using the middle anchor image
as a reference, while red represents similar areas in the image. The distance between the descriptors are shown above
each visualization map. Our approach is capable of finding similarities between the fashion styles in the images.
We compare with a fine-tuned VGG_CNN_M_128 network
which can’t identify the clothing changes in the image.
image and calculating the style descriptor. We compare theresulting descriptors with the original image descriptor andvisualize the change. In this way, we can localize the partsof the image that have the greatest effect on the descriptor,
i.e., the parts that our feature extraction network focuses on.
We show examples on the Fashionista dataset [40] in Fig. 5and see that our style descriptor reacts strongly to different
parts of the body for different individuals.

White Spotted
Navy DressBlue Dress

Ampliﬁed Patterns

Independent of
ethnicity andbackgroundBlack Dress

Light Pink andWhite DressTransition fromno pattern topatterns

Figure 7: Visualization of the fashion style space of the Pinup class from the Hipster Wars [16] dataset using t-SNE [33].

4.6. Matching Styles
Instead of considering only a single image, we can consider pairs of images and match parts of the image using thestyle descriptor. This is done in a similar way as visualizing a single style descriptor, that is, by sliding a 48 × 48
pixel bounding box around the image. The difference isthat we consider two feature descriptors f1 = f (I1 ) and
f2 = f (I2 ) corresponding to two different images I1 andI2 simultaneously. We employ the difference between bothfeature vectors to evaluate how well this vector matches thechange of the style descriptor f (·) given an image partiallyoccluded at pixel location (u, v) with a bounding box mask
B(u, v) by using the dot product:
IM (u, v) = (f (I1 ◦ B(u, v)) − f1 ) · (f2 − f1 ) ,(8)

where IM (u, v) is the output map at pixel (u, v), and ◦ is the
Hadamard product or element-wise matrix multiplication.
We show results in Fig. 6 where we can see that our descriptor is effectively capturing the notion of outfit similarity in a reasonable way. Note that this concept of local
outfit similarity was learned automatically from noisy userprovided tags without any pixel level annotations. On theother hand, the fine-tuned VGG_CNN_M_128 model gives
similar maps regardless of the image compared to: it is overfitting to the Hipster Wars dataset.

4.7. Exploring the Fashion Style Space
Finally, we perform a qualitative analysis of the resulting
style descriptors obtained by visualizing the fashion stylespace using t-SNE [33]. The style descriptors can be compared efficiently by using Euclidean distances. We visualize the Hipster Wars “Pinup” class in Fig. 7. Our features
display a remarkable robustness to background changes andfocus on the outfit. They are also able to capture subtleties
such as the transition from pink dresses without patterns tofloral patterns, and group navy dresses with white spots regardless of the background and the wearer’s ethnicity.

5. Conclusions
We have presented a novel approach to weaklysupervised learning of features consisting of joint ranking and classification. This allows learning compact 128dimensional features for more specific types of images that
may be very costly and complicated to annotate. Our
method allows us to learn discriminative features that areable to outperform both the previous state of the art andthe best-performing model trained on ImageNet while being the size of a SIFT descriptor. The proposed joint ranking and classification approach consistently improves results over using either classification or ranking loss alone.
We complement our model with a simple approach to automatically clean the data. In addition to a quantitative analysis, we present a new approach both to visualize the individual descriptor activations and to find similarities between
two style images. Our analysis of the resulting descriptor
shows it is robust to backgrounds and is able to capture finegrained details such as flower patterns on pink dresses.
Acknowledgements: This work was partially supported by
JSPS KAKENHI #26108003 as well as JST CREST.

References[1] S. Bell and K. Bala. Learning visual similarity for productdesign with convolutional neural networks. ACM Trans. on
Graphics (SIGGRAPH), 34(4), 2015. 3[2] L. Bossard, M. Dantone, C. Leistner, C. Wengert, T. Quack,and L. V. Gool. Apparel classifcation with style. In ACCV,2012. 2[3] J. Bromley, I. Guyon, Y. Lecun, E. Säckinger, and R. Shah.
Signature verification using a "siamese" time delay neuralnetwork. In NIPS, 1994. 3[4] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman.
Return of the devil in the details: Delving deep into convolutional nets. In BMVC, 2014. 3, 6, 7
[5] H. Chen, A. Gallagher, and B. Girod. Describing clothing by
semantic attributes. In ECCV, 2012. 2[6] R. Collobert, K. Kavukcuoglu, and C. Farabet. Torch7: A
Matlab-like environment for machine learning. In BigLearn,
NIPS Workshop, 2011. 5[7] J. Deng, W. Dong, R. Socher, L. jia Li, K. Li, and L. Feifei. Imagenet: A large-scale hierarchical image database. In
CVPR, 2009. 1, 2
[8] R.-E. Fan, K.-W. Chang, C.-J. Hsieh, X.-R. Wang, and C.-J.
Lin. LIBLINEAR: A library for large linear classification.
JMLR, 9:1871–1874, 2008. 6[9] A. Frome, G. S. Corrado, J. Shlens, S. Bengio, J. Dean, M. A.
Ranzato, and T. Mikolov. Devise: A deep visual-semantic
embedding model. In NIPS, 2013. 3[10] X. Han, T. Leung, Y. Jia, R. Sukthankar, and A. C. Berg.
MatchNet: Unifying feature and metric learning for patchbased matching. In CVPR, 2015. 3[11] E. Hoffer and N. Ailon. Deep metric learning using triplet
network. In ICLR, 2015. 3[12] J. Huang, R. Feris, Q. Chen, and S. Yan. Cross-domain image retrieval with a dual attribute-aware ranking network. In
ICCV, 2015. 2[13] S. Ioffe and C. Szegedy. Batch normalization: Accelerating
deep network training by reducing internal covariate shift. In
ICML, 2015. 4[14] A. Karpathy and L. Fei-Fei. Deep visual-semantic alignments for generating image descriptions. In CVPR, 2015.
3[15] M. Kiapour, X. Han, S. Lazebnik, A. Berg, and T. Berg.
Where to buy it: Matching street clothing photos in online
shops. In ICCV, 2015. 2[16] M. H. Kiapour, K. Yamaguchi, A. C. Berg, and T. L. Berg.
Hipster wars: Discovering elements of fashion styles. In
ECCV, 2014. 2, 5, 6, 8
[17] A. Kovashka, D. Parikh, and K. Grauman. Whittlesearch: Interactive image search with relative attribute feedback. IJCV,
115(2):185–210, 2015. 2[18] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
NIPS, 2012. 3[19] Y. LeCun, Y. Bengio, and G. Hinton. Deep learning. Nature,
521:436–444, 2015. 1[20] D. G. Lowe. Distinctive image features from scale-invariant
keypoints. IJCV, 60:91–110, 2004. 2[21] A. Oliva and A. Torralba. Modeling the shape of the scene:
A holistic representation of the spatial envelope. IJCV,
42(3):145–175, 2001. 1[22] M. Oquab, L. Bottou, I. Laptev, and J. Sivic. Is object localization for free? – Weakly-supervised learning with convolutional neural networks. In CVPR, 2015. 3[23] G. Papandreou, L.-C. Chen, K. Murphy, and A. L. Yuille.
Weakly- and semi-supervised learning of a deep convolutional network for semantic image segmentation. In ICCV,2015. 3[24] D. Rumelhart, G. Hinton, and R. Williams. Learning representations by back-propagating errors. In Nature, 1986. 4[25] F. Schroff, D. Kalenichenko, and J. Phibin. Facenet: A unified embedding for face recognition and clustering. In CVPR,
2015. 3[26] E. Simo-Serra, S. Fidler, F. Moreno-Noguer, and R. Urtasun.
A High Performance CRF Model for Clothes Parsing. In
ACCV, 2014. 2[27] E. Simo-Serra, S. Fidler, F. Moreno-Noguer, and R. Urtasun. Neuroaesthetics in Fashion: Modeling the Perceptionof Fashionability. In CVPR, 2015. 2, 5, 6
[28] E. Simo-Serra, E. Trulls, L. Ferraz, I. Kokkinos, P. Fua, and
F. Moreno-Noguer. Discriminative Learning of Deep Convolutional Feature Point Descriptors. In ICCV, 2015. 3[29] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In ICLR, 2015.
3, 4, 5, 6, 7[30] N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and
R. Salakhutdinov. Dropout: A simple way to prevent neuralnetworks from overfitting. JMLR, 15:1929–1958, 2014. 4[31] C. Szegedy, W. Liu, Y. Jia, P. Sermanet, S. Reed,
D. Anguelov, D. Erhan, V. Vanhoucke, and A. Rabinovich.
Going deeper with convolutions. In CVPR, 2015. 3[32] A. Torralba and A. Efros. Unbiased look at dataset bias. In
CVPR, 2011. 1[33] L. van der Maaten and G. Hinton. Visualizing data using tsne. JMLR, 9:2579–2605, Nov 2008. 8[34] A. Veit, B. Kovacs, S. Bell, J. McAuley, K. Bala, and S. Belongie. Learning visual clothing style with heterogeneous
dyadic co-occurrences. In ICCV, 2015. 2, 3[35] S. Vittayakorn, K. Yamaguchi, A. C. Berg, and T. L. Berg.
Runway to realway: Visual analysis of fashion. In WACV,2015. 2[36] J. Wang, Y. Song, T. Leung, C. Rosenberg, J. Wang,
J. Philbin, B. Chen, and Y. Wu. Learning fine-grained image similarity with deep ranking. In CVPR, 2014. 3, 4[37] T. Xiao, T. Xia, Y. Yang, C. Huang, and X. Wang. Learningfrom massive noisy labeled data for image classification. In
CVPR, 2015. 3[38] K. Yamaguchi, T. L. Berg, and L. E. Ortiz. Chic or social:
Visual popularity analysis in online fashion networks. In
ACMMM, pages 773–776, 2014. 2, 7
[39] K. Yamaguchi, M. H. Kiapour, and T. L. Berg. Paper dollparsing: Retrieving similar styles to parse clothing items. In
ICCV, 2013. 2[40] K. Yamaguchi, M. H. Kiapour, L. E. Ortiz, and T. L. Berg.
Parsing clothing in fashion photographs. In CVPR, 2012. 1,
2, 7

[41] K. Yamaguchi, M. H. Kiapour, L. E. Ortiz, and T. L. Berg.
Retrieving similar styles to parse clothing. PAMI, 2014. 2,5, 6[42] K. Yamaguchi, T. Okatani, K. Sudo, K. Murasaki, and
Y. Taniguchi. Mix and match: Joint model for clothing andattribute recognition. In BMVC, 2015. 2[43] W. Yang, P. Luo, and L. Lin. Clothing co-parsing by joint
image segmentation and labeling. In CVPR, 2014. 2[44] J. Yosinski, J. Clune, Y. Bengio, and H. Lipson. How transferable are features in deep neural networks? In NIPS, 2014.
1[45] S. Zagoruyko and N. Komodakis. Learning to compare image patches via convolutional neural networks. In CVPR,
2015. 3[46] M. D. Zeiler. ADADELTA: an adaptive learning rate method.
CoRR, abs/1212.5701, 2012. 4[47] M. D. Zeiler and R. Fergus. Visualizing and understanding
convolutional networks. In ECCV, 2014. 7[48] B. Zhou, A. Khosla, À. Lapedriza, A. Oliva, and A. Torralba.
Object detectors emerge in deep scene cnns. In ICLR, 2015.
3[49] B. Zhou, A. Lapedriza, J. Xiao, A. Torralba, and A. Oliva.
Learning deep features for scene recognition using places
database. In NIPS, 2014. 1, 2, 6, 7