Neuroaesthetics in Fashion: Modeling the Perception of Fashionability
121

Edgar Simo-Serra Sanja Fidler Francesc Moreno-Noguer Raquel Urtasun
1
2
Institut de Informàtica i Robòtica Industrial University of Toronto

2O BJECTIVE

M ODELR ESULTS

We wish to:• Model and understand fashionability and its components• Provide recommendations and advice to users• Extract complementary mid-level features with a deep net• Explicitly model different aspects of fashion using a CRF(u: user, o: outfit, s: setting, f : fashionability)• Latent states manually annotated
CRF Pairwise Terms• Data from crawling the largest fashion social site: chictopia.com• Normalized votes used as a proxy for fashionability

Fans1∆T1Comments
5Location 266
Personal 21
Style 20
Scene 397
Tags 209
Colours 604
Singles 121
Garments 1352

Number of user’s fans
Time between post creation and download
Sentiment analysis [4] of comments
Distance from location clusters [3]Face recognition attributes
Style of the photography [1]Output of scene classifier trained on [5]Bag-of-words with post tags
Bag-of-words with colour tags
Bag-of-words with split colour tagsBag-of-words with garment tags

Deep Network for Feature Extraction• Train four feature extractors jointly maximizing fashionability• Afterwards, the four networks are used independently

10910857065−54−10 3−15 2
110
98
7654
321s

Comments

Singles

Style

Garments

Tags
Softmax

Dataset Statistics
Property Total Per Post
posts 144.169
users 14,287
locations 3443
males 5%fans1226
comments
20.09
votes
150.76favourites
27.01
photos 277,537 1.93
tags 13,192 3.66
colours 3337
2.28
garments
3.22

Trendy Cities
City Name Score

Influence of Compatriots−2−4−6
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

uo

Current Outfit:
Pink Outfit (3)Current Outfit:
Pink/Blue Shoes/Dress Shorts (3)Current Outfit:
Pink/Black Misc. (5)Recommendations:
Heels (8)Pastel Shirts/Skirts (8)Black/Gray Tights/Sweater (5)Recommendations:Black/Gray Tights/Sweater (5)Black Casual (5)Black Boots/Tights (5)Recommendations:
Pastel Dress (8)Black/Blue Going out (8)Black Casual (8)Current Outfit:
Blue with Scarf (3)Current Outfit:
Pink/Blue Shoes/Dress Shorts (3)Current Outfit:
Formal Blue/Brown (5)Recommendations:
Heels (8)Pastel Shirts/Skirts (8)Black Casual (8)Recommendations:Black Casual (7)Black Heavy (3)Navy and Bags (3)Recommendations:
Pastel Shirts/Skirts (9)Black/Blue Going out (8)Black Boots/Tights (8)Visualizing Outfit Trends
Manila0.6

Modelling Fashion with a Conditional Random Field

0Outfit Recomendations• Estimate setting and user state and find outfit state that maximizes fashionability

ΔTColours

2

Sample Predictions

ScenePersonal41098176054−1 32−2 12

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20

1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20Location

Fansf -o Pairwise

False Positives

FASHION 144 K D ATASET

Feature Dim. Description

f -u Pairwise

True Positives

Novel dataset and Conditional Random Field model that canreason about different aspects of fashionability.
http://www.iri.upc.edu/people/esimo/research/fashionability/

f -s Pairwise

Los Angeles

0.35Black/Blue Going out
Formal Blue/Brown
Black Boots/Tights
White/Black Blouse/HeelsBrown/Blue Jacket
Navy and Bags
Bags/Dresses
Pink/Blue Shoes/Dress/Shorts
Black/Gray Tights/Sweater
Pastel DressBlack with Bag/Glasses
Blue with Scarf
Shirts and Jeans
Pink Outfit
Black CasualHeels
Pink/Black Misc.
Shoes and Blue Dress
Pastel Shirts/Skirts
Black Heavy

0.30

0.5

Country Posts SCC MSCS MCS

0.25

Manila 6.627
USA 28.0% 14.86% 3.78 3.76
Los Angeles 6.265
Unknown 21.8%
Melbourne 6.176
UK 5.1% 2.67% 3.80 3.75
Montreal 6.144 Philippines 5.1% 14.54% 3.61 3.72
Paris 6.070
Canada 4.5% 2.95% 3.68 3.76
Amsterdam 6.059
Spain 3.9% 1.52% 3.06 3.75
Barcelona 5.845
Poland 2.5% 1.07% 3.63 3.80
Toronto 5.765
Australia 2.4% 1.76% 3.62 3.75
Bucharest 5.667
France 2.3% 0.46% 3.23 3.75
New York 5.514
Romania 2.0% 6.83% 3.73 3.77
London 5.444
SCC Same Country Comments
San Francisco 5.392
Madrid 5.371 MSCS Mean Same Country Score
Vancouver 5.266 MCS Mean Country Score
Jakarta 4.398
Scores from sentiment analysis [4]Location • Only f node observed during
Scene
training

FansPersonal• Output of deep network usedas unaries•Pairwise
terms
learntColours
Singles • Clustering used to initialize
Garments latent nodes

ΔT
Comments
StyleTags

R ESULTS - P REDICTING FASHIONABILITY• Map of Fashionability
1.0ModelAcc.

Pre.

Rec.

IOU

L1

CRF
Deep NetNo Metadata
Log. Reg.
Baseline
Random

29.27
30.42
19.63
23.92
16.28
9.69

30.42
31.11
17.06
22.54
9.69

28.69
30.26
17.47
22.99
10.00
9.69

17.36
18.41
8.31
12.55
1.63
4.99

1.46
1.45
2.31
1.91
2.32
3.17

0.9

0.8

0.7

0.6

0.5

0.4

0.3

0.2

0.1

0.0

0.40.20
0.3
0.15
0.2
0.10
0.1

0.00.05

4812Trimester

16200.00

48121620

Trimester

Future Work• Incorporate segmentation [2, 6]:
http://www.iri.upc.edu/people/esimo/research/fashion/
References[1] S. Karayev, A. Hertzmann, H. Winnemoeller, A. Agarwala, T. Darrell. Recognizing imagestyle. In BMVC, 2014.
[2] E. Simo-Serra, S. Fidler, F. Moreno-Noguer, R. Urtasun. A High Performance CRF Model forClothes Parsing. In ACCV, 2014.
[3] E. Simo-Serra, C. Torras, F. Moreno-Noguer. Geodesic Finite Mixture Models. In BMVC, 2014.
[4] R. Socher, A. Perelygin, J. Wu, J. Chuang, C. D. Manning, A. Y. Ng, C. Potts. Recursive deepmodels for semantic compositionality over a sentiment treebank. In EMNLP, 2013.
[5] J. Xiao, J. Hays, K. A. Ehinger, A. Oliva, A. Torralba. Sun database: Large-scale scene recognition from abbey to zoo. In CVPR, 2010.
[6] K. Yamaguchi, M. H. Kiapour, T. L. Berg. Paper doll parsing: Retrieving similar styles toparse clothing items. In ICCV, 2013.