Fashion Style in 128 Floats:
Joint Ranking and Classification using Weak Data for Feature Extraction
Edgar Simo-Serra and Hiroshi Ishikawa
Waseda University, Tokyo, Japan

Objective

Model & Training• Learn compact, discriminative representations of images withConvolutional Neural Networks.
• Exploit weak data in the form of incomplete and noisy userprovided tags.
• Optimize for comparisons with L2 distance.

Main features• Able to exploit data with missing and incomplete tags.
• Obtains compact 128-float representations of whole images.
• Adaptable to new datasets without needing annotating.
• Outperforms pre-trained features for fashion style prediction.
http://hi.cs.waseda.ac.jp/˜esimo/research/stylenet/

Key observation
Hipster Wars (1,893 images)

Fashion144k (277,527 images)

Mean

Mean

Places (2,469,371 images)

ImageNet (1,331,167 images)

Mean

Mean

1. Pre-trained imagenet networks limits both architecture and target applications, i.e., images should be similar to imagenet.
2. Lots of data with user-provided tags on the internet. However,these tags are often incomplete and noisy. We want to exploit
this data to train new networks from scratch.
3. Problem? Standard training with classification losses is not robust
to noisy data.
4. Solution: Jointly use a ranking loss with a classification loss.
The ranking loss allows comparing vectors and is robust to noise,
while the classification loss is critical for training.

Experimental results
type kernel size output size

Classiﬁer

Sharedparameters

FeatureCNN

FeatureCNN

I−I

Feature
CNN

I＋

Dissimilar

Similar

Top: Overview of the proposed model.
Right: Architecture of the feature CNN.

convolution
convolution
dropout (25%)

3×3
3×3

384x256x64
384x256x64
384x256x64

max poolingbatch normalization
convolution
convolution
dropout (25%)

4×4

96x64x64
96x64x64
96x64x128
96x64x128
96x64x128

max poolingbatch normalization
convolution
convolution
dropout (25%)

4×4max poolingbatch normalization
convolution
fully-connected

4×4

3×3
3×3

3×3
3×3

3×3

TOTAL

24x16x128
24x16x128
24x16x256
24x16x256
24x16x256

params
1,792
36,928128
73,856
147,584256
295,168
590,080

6x4x256
6x4x256
6x4x128128

512
32,896393,344128

1,572,544

Model is trained from scratch using a classification and ranking loss.
Ranking loss: Defined on triplets of images where one is an anchor
I, one is similar to the anchor I+ with r(I, I+) > τs, and one is dissimilar to the anchor I− with r(I, I−) < τd.
Loss encourages the distance between output of the anchor and similar image d+ = d(I, I+) to be smaller than the distance between theoutput of the anchor and the dissimilar image d− = d(I−, I) [5]:2exp(d−)lR(d−, d+) =exp(d−) + exp(d+)
I−II+I−ITable 1: Linear classifier evaluated on100 random 9:1 train-test splits.

Table 2: Similarity search (no train).

feature params dim. acc. pre. rec. iou
Ours Joint
Ours Ranking
Ours Siamese
Ours Class.
Ours Joint Dirty

1.6M
1.6M
1.6M
1.6M
1.6M

Kiapour et al. [2]†128 75.9 75.4 76.5 61.5
128 74.5 74.2 74.5 59.6
128 73.3 72.9 74.0 58.2
128 73.5 71.7 74.1 57.3
128 72.9 72.1 73.1 57.0

‡ 39,168 70.6 70.6 70.4 54.6

VGG M [1]VGG 16 [4]VGG M 1024 [1]
VGG M 128 [1]
VGG 16 Places [8]feature dim. top-1 top-2 top-3
Ours Joint 128 63.5 79.9 86.3
VGG M [1] 4096 53.2
VGG 16 [4] 4096 53.2
VGG M 128 [1] 128 44.6
VGG 16 Places [8] 4096 40.1

71.7
71.5
64.0
61.0

81.3
80.4
76.2
72.0

Table 3: Fine-tuning on 1:1 split.
feature dim. acc.

pre.

rec.

iou

99M 4096 71.9 72.9 70.9 56.2
66.1
67.9
51.0
134M 4096 70.1 70.5 69.7 54.8 Ours Joint 128 68.4
86M 1024 70.4 71.1 69.5 54.2 VGG M 4096 68.4/64.6 67.3/64.2 68.8/63.0 51.8/46.8
82M 128 63.5 62.8 63.5 46.3 VGG 16 4096 63.8/63.3 62.6/62.6 63.5/61.9 46.5/45.4
134M 4096 57.4 57.6 59.4 41.5 VGG M 128 128 62.6/57.2 60.4/55.1 62.1/56.9 44.5/39.0

Visualizing descriptors
Occlude parts of the image to visualize change [7]. For single descriptors show descriptor norm and PCA basis.
Ours
Input

Norm

PCA 1

PCA 2

Fine-tuned VGG M 128
PCA 3Norm

PCA 1

PCA 2

PCA 3

I+

Problem formulation
1. Consider set of possible tags Tt
2. Assume dataset of images with noisy labels l = (l )t∈T withlt ∈ 0, 1|a∧b|
3. Define similarity between two images as r(a, b) = |a∨b|

Trained on Fashion144k dataset [3] using 80,554 training and 8,948
testing images with |T | = 3, 303 tags. Evaluation on Hipsters Wars
dataset [2] with 1,893 images and 5 class labels.

Classification loss: Auxiliary network used to predict image labelsof the dissimilar image X− with multi-label cross-entropy loss:
1 XtlC (X−, y−) =l×(X−)|T |Visually compare the similarity according to the feature CNN f (·) oftwo images I1 and I2 by occluding with a bounding box B(u, v) :IM (u, v) = (f (I1 ◦ B(u, v)) − f (I1)) · (f (I2) − f (I1))
Image 1Ours

VGG

Anchor

Ours

VGG

Image 2t∈Twith l×(x, y) = −xy + log (exp(x0) + exp(x1))
Joint loss: L(d−, d+, X−, y−) = lR(d−, d+) + αlC (X−, y−)

Implementation
Black-Pants
Blue-Shirt
Gray-Scarf
White-SweaterBlack-Bag
Yellow-Shoes

Aquamarine-Bag
Black-Jeans
Camel-Heels
Brick-Red-Sweater
Ivory-Shorts
Brown-Bag
Ivory-Sunglasses
Dark-Brown-Vest Light-Yellow-Hat
Red-Wedges
Turquoise-BlueVest

Black-Bag
Black-Boots
Black-Sweater
White-Hat
White-Shirt

Black-Boots
Black-CoatGray-Bag
Black-Leggings
Light-Blue-JeansBronze-Scarf
Light-Brown-Coat Dark-Gray-Blouse
White-Sweater
Neutral-Shoes
Gray-Sweater
Tawny-Bag

1. Pre-training with classification loss only.
2. Batches formed by selecting anchor images and randomly sampling until similar/dissimilar criterion is met.
3. Optimization with ADADELTA [6].
4. Fine-tuned VGG to remove poor quality images from trainingcan improve performance.

References[1] K. Chatfield, K. Simonyan, A. Vedaldi, and A. Zisserman. Return of the devil in the details: Delving deep intoconvolutional nets. In BMVC, 2014.
[2] M. Hadi Kiapour, Kota Yamaguchi, Alexander C. Berg, and Tamara L. Berg. Hipster wars: Discovering elements offashion styles. In ECCV, 2014.
[3] Edgar Simo-Serra, Sanja Fidler, Francesc Moreno-Noguer, and Raquel Urtasun. Neuroaesthetics in Fashion: Modeling the Perception of Fashionability. In CVPR, 2015.
[4] K. Simonyan and A. Zisserman. Very deep convolutional networks for large-scale image recognition. In ICLR, 2015.
[5] Jiang Wang, Yang Song, Thomas Leung, Chuck Rosenberg, Jingbin Wang, James Philbin, Bo Chen, and Ying Wu.
Learning fine-grained image similarity with deep ranking. In CVPR, 2014.
[6] Matthew D. Zeiler. ADADELTA: an adaptive learning rate method. CoRR, abs/1212.5701, 2012.
[7] Matthew D. Zeiler and Rob Fergus. Visualizing and understanding convolutional networks. In ECCV, 2014.
[8] Bolei Zhou, Agata Lapedriza, Jianxiong Xiao, Antonio Torralba, and Aude Oliva. Learning deep features for scene
recognition using places database. In NIPS, 2014.