CNN-based Style Vector for Style Image Retrieval
Shin Matsuo

Keiji Yanai

Department of Informatics,
The University of Electro-Communications
Chofu, Tokyo 182-8585 JAPAN{matsuo-s,yanai}@mm.inf.uec.ac.jp

ABSTRACTIn this paper, we have examined the eﬀectiveness of “stylematrix” which is used in the works on style transfer andtexture synthesis by Gatys et al. [2, 3] in the context of image retrieval as image features. A style matrix is presented
by Gram matrix of the feature maps in a deep convolutional neural network. We proposed a style vector which are
generated from a style matrix with PCA dimension reduction. In the experiments, we evaluate image retrieval performance using artistic images downloaded from Wikiarts.org
regarding both artistic styles ans artists. We have obtained
40.64% and 70.40% average precision for style search andartist search, respectively, both of which outperformed theresults by common CNN features. In addition, we found
PCA-compression boosted the performance.

1. INTRODUCTION
Figure 1 shows a lion image, a tiger image and the images
the styles of which are modified using the style of Gogh’sStarry Night by the style transfer method proposed by Gatys
et al. [2]. In this figure, the images in the right and theleft have common contents, while the image in the top andthe bottom have common styles. When we classify them interms of object categories, the images in the top are classified
as “lion”, and the images in the bottom are classified as“tiger”. However, we sometimes classify images in terms of
their styles. In such case, the images in the left are classified
as “photo”, while the images in the right are classified as“drawing”.
Classifying images in terms of their styles is expected tohelp various kinds of applications such as style analysis ofimages/videos, and style-based image search. However, inthe computer vision community, recognition of image contents is paid much more attention to than styles. After
Deep Convolutional Neural Network (DCNN) [6] won ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
in 2012, image recognition employing CNN becomes very
common to achieve high performance. Although the perforPermission to make digital or hard copies of all or part of this work for personal orclassroom use is granted without fee provided that copies are not made or distributedfor profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others thanACM must be honored. Abstracting with credit is permitted. To copy otherwise, or republish, to post on servers or to redistribute to lists, requires prior specific permission
and/or a fee. Request permissions from permissions@acm.org.

ICMR’16, June 06-09, 2016, New York, NY, USA
c 2016 ACM. ISBN 978-1-4503-4359-6/16/06. . . $15.00⃝
DOI: http://dx.doi.org/10.1145/2911996.2912057

Figure 1: (Left) Original (content) images, (center) Style image, (right) Images synthesized by styletransfer method.
mance of CNN for 1000-class object category classification
have already outperformed human [4], style classification has
not been explored extensively except [5]. In object classification where classifying object categories regardless of imagestyles is important, CNN is regarded as the best architecture
at present, while it is not confirmed if CNN is the best in
style recognition of images where classifying styles regardlessof contents is needed.
In 2014, Karayev et al. [5] proposed the method on imagestyle recognition employing CNN-based image features andshowed CNN features outperformed the conventional features such as color histogram and GIST with a large margin. However, the classification rates on style classification
are not as high as the rates on object categorization. Therefore, style recognition is regarded as more challenging task
than object classification.
Recently, Gatys et al. proposed an algorithm on artisticstyle transfer [2] which synthesis an image which has thestyle of a given style image and the contents of a given content image. This method replaces the information which aredegraded while the signal of the content image goes forward
through CNN layers with style information extracted thestyle image, and reconstructs a new image which has thesame content as a given content images and the same styleas a given style image as shown in Figure 1. In this method,they introduced “style matrix” which was presented by Grammatrix of the feature maps, that is, correlation matrix between feature maps in a deep convolutional neural network(DCNN). Originally, a style matrix was introduced in texture synthesis with DCNN by the same authors [3].
Then, in this paper, we introduce style vectors which istransformed from style matrix, and examine its eﬀectiveness when using it as image features for image retrieval. By
the experiments, we confirmed style matrix is more eﬀective

than common CNN activation features in image retrieval
tasks. In addition, we found PCA-compression boosted theperformance.

2. RELATED WORKS
Although the number of works on image style analysis is
not so large, several works have been published so far. Mostof them are related to aesthetic analysis of images.
Datta et al. estimates aesthetic rating of images employing various visual features such as color saturation and hue,exposure of light and colorfulness, and wavelet-based texture
co-eﬃcients. Marchesotti et al. [7] also proposed a method
to evaluate aesthetic rating using Aesthetic Visual Analysis(AVA) dataset [8].
Keren [1] proposed a method to classify artistic styles andachieved classification of drawings of Pollock and Dali using
local features. As a recent work, Karayev et al. [5] classified style images using CNN activation features, and showed
CNN feature outperformed conventional features greatly.

Figure 2: Images in “Style Dataset”.

3. PROPOSED METHOD: STYLE VECTOR
Style matrix is used as an representation of image style in
the work on style transfer by Gatys et al. [2], which is computed as Gram matrix of feature maps in the specific layerof CNN. In their method, the style of a given style image
is transferred into a synthesized image by minimizing the
Euclidean loss between the style matrix of both images withback propagation. Because style matrix contains style information of an image, we propose to use style vectors which
is transformed from style matrix as image representation ofimage retrieval.
Style matrix G is a set of correlation values between feature maps in the specific layer l. Style matrix Gl ∈ RMl ×Ml
is represented as
Gl = F l (F l )T(1)

where F l ∈ RMl ×Nl is a feature map in the layer l.
In this paper, we transform a style matrix in each layer
into a feature vector. Because a style matrix is a symmetrical matrix, the number of independent elements are(Ml )(Ml + 1)/2 where Ml is the number of the feature maps
in the layer l. We define a style vector V l as follows:V l = [Gl 1,1 , Gl 2,1 , ..., Gl Ml ,1 , Gl Ml ,2 , ..., Gl Ml ,Ml ](2)

l

We L2-normalize V without/with signed square root before using a raw style vector as a feature vector. We represent L2-normalized of V l without/with signed square root
as S lL2 and S lsgnsqrt , respectively. They are calculated asfollows:
S lL2

=

Vl
∥V l ∥(3)√
sgn(V ) |V l |√∥sgn(V l ) |V l |∥l

S lsgnsqrt=(4)

4. EXPERIMENTS
For experiments, we prepared an artistic pictorial image dataset which contains metadata on image style andartists by gathering them from Wikiart.org, and classified
them with style vectors extracted from five intermediate layers (conv1 1, conv2 1, conv3 1, conv4 1, and conv5 1) of

Figure 3: Images in “Artist Dataset”.

the VGG-16 network [9]. In addition, we also examined the
combined style vectors of all the five layers as well. Finally,
we compared performance with Karayev et al. [5] using thesame dataset.

4.1

DatasetWe created two kinds of datasets by gathering images fromWikiart.org which was used as the data source in the workby Karayev et al. [5]. The first one is “Style dataset”, andthe other is “Artist dataset”.
“Style dataset” contains 100 images of each of 25 style categories which are oﬃcially defined in Wikiart.org. That is,
it contains 2500 images in total. We selected 25 style categories each of which contains more than 100 images among
all the style categories in Wikiart.org. Figure 2 shows someexamples in “Style dataset”.
“Artist dataset” contains 1000 images of the top 10 artists
regarding the number of images in Wikiart.org. We preparedthis dataset for examining if the style vector is eﬀective forclassifying artist drawing styles. Figure 3 shows some examples in “Artist dataset”.

4.2

Evaluation of Style Vector

To search for an image which has similar style to a given
query images, we use nearest neighbor search with respect
to the style vector. In the experiments, to evaluate the eﬀectiveness of a style vector for image style search, we examine
if the nearest image to a query image belongs to the samestyle as the given query image. This is equivalent to nearest
neighbor classification. In section 4.5, we used SVM as aclassifier as well for comparison to the existing work.

4.3

Style Estimation for Two DatasetsWe made experiments with two datasets by 5-fold classification with the nearest neighbor classifier. As a metric, weused standard Euclidean distance.
The convolutional neural network (CNN) we used in theexperiments was 16-layered very deep convolutional neuralnetwork, VGG-16 [9], pre-trained with the ILSVRC-2012
1000-class dataset. We extracted style vectors from the layer
conv1 1, conv2 1, conv3 1, conv4 1 and conv5 1. We usedthem as single features and concatenation of them as combined features. In addition, we extracted fc6 and fc7 from
VGG-16, and fc6 from the standard Alexnet [6]. We usethem as visual features with raw, L2-normalization and L2normalization signed square root. Table 1 shows the resultsfor Style and Artist dataset.
The style vector which was the most eﬀective for both
Style dataset and Artist dataset was the one extracted fromconv5 1 with signed square root and L2 normalization. Therefore, this experimental result revealed that style vectors outperformed conventional CNN features regarding style-based
image search. Especially, signed square root and L2 normalization of style vectors is significant to obtain the best
performance. Note that in the rest part we use signed squareroot L2 normalization for all the style vectors.
Table 2 shows average precision on each category in Style
dataset. The categories in which style vectors outperform
CNN features were “Art Nouveau”, “Modern”, “High Renaissance”, “Post-Impressionism”, and “Symbolism”. Amongthem, the accuracy of “Nouveau Modern” was the highest,
which outperformed CNN features by 0.13. This is because
the images of “Art Nouveau Modern” tends to expose very
unique style but to have common scenes and objects as motifs. For example, as shown in Figure 4 which depicts two
person with unique patterns, CNN features capture persons,while style vectors are expected to represent patterns.
Conversely, the categories where CNN features is better
than style vectors were “Color Field Painting”, “Mannerism
Late Renaissance”, and “Minimalism”. Especially, the accuracy of “Color Field Painting” were better than style vectors
by 0.2. Figure 5 shows a sample of “Color Field Painting”
the nearest neighbor of which was an image of “Abstract
Art” in case of style vectors. This is because style vectors
reflect local patterns rather than whole image structures.
Style all which is an aggregation of the style vectors extracted from five layers tends to be inferior to Style conv5 1.
Exceptionally, in “Ukiyo-e” it achieved the best performance
among all the features. This is because “Ukiyo-e” includes
various scales of unique patterns.
Regarding the results for “Artist dataset’, in the categories of “Chagall”, “Konchalovsky”, and “Gogh”, style vectors clearly outperformed CNN features as shown in Figure 6. In their drawings, common motifs were drawn withunique pattern and touch. This is the similar tendency to
Style dataset. Although “Picasso” also have unique style, theperformance was almost the same in case of style vector and
CNN features. Since “Pissaro” and “Monet” are similar toeach other in terms of their motifs and styles, they tended
to be confused with each other using any of style vectors
or CNN features. The images in the category of “Piranesi”
was correctly classified using any features we used in theexperiments.

4.4 Dimensionality reduction with PCA
Although the most eﬀective style vector was S conv5 1 inFigure 4: Similar images to a sample of “Art Nouveau Modern” by a style vector and a CNN feature.

Figure 5: Similar images to a sample of “Color Field
Painting” by a style vector and a CNN feature.

Figure 6: All per-category APs on “Artist dataset”.
the previous experiment, its dimension was very high, 131, 328 (=512 × (512 + 1)/2), which is 32 times as many as the dimension of CNN features, 4096. To resolve high dimensionality,
we applied PCA to style vectors for dimension reduction.
With PCA compression, we obtained 128-d, 256-d, 512-d,
1024-d, 2048-d, and 4096-d style vectors, and made the same
experiments as the previous ones. Table 3 shows the results
for both Style Dataset and Artist Database.
For Style dataset, 2048-d style vectors achieved the best
performance, while 4096-d vectors was the best for Artist
dataset. Surprisingly, both the results exceeded the resultswith raw style vectors with large margin by 3.44 and 5.1,
respectively. From these results, PCA is helpful not only for
dimension reduction but also boosting of the classification
performance.

4.5

Comparison with the previous work

We made experiments with the same image data as Karayev
et al. [5] for comparison. We used PCA-compressed Style conv5 5(1024d, 2048d and 4096d) as style vectors and fc6 and fc7
as CNN features. For style features, we used SVM as a classifier as well as a nearest neighbor classifier. Table 4 shows
the results including the results in [5]. As a result, style vectors outperformed [5] greatly, which proved the eﬀectivenessof the style vector.

Table 1: Mean APs on two datasets
Style

Artistraw
L2normsgnsqrtraw
L2normsgnsqrtAlex fc6
36.28
39.28
38.00
61.70
61.00
61.90

VGG fc6
33.68
39.44
39.36
59.3
62.9
62.3

VGG fc7
32.04
36.04
36.28
56.7
61.1
60.8

Style conv1 1
15.40
17.36
20.20
34.00
32.60
36.90

Style conv2 1
19.20
21.36
21.16
40.90
42.80
42.80

Style conv3 1
25.48
28.56
29.92
52.80
56.00
56.90

Style conv4 1
31.20
34.48
35.04
60.70
63.80
64.70

Style conv5 1
34.08
39.60
40.64
64.70
68.80
70.40

Style all
25.72
27.36
33.08
51.90
52.20
61.10

Table 2: All per-category APs on the Style dataset
imagenet fc6
vgg fc6
vgg fc7style conv1 1style conv2 1style conv3 1
style conv4 1
style conv5 1
style allMannerism
Late
Renaissance
0.42
0.55
0.53
0.20
0.19
0.32
0.44
0.45
0.39

Table 3:
dataset.

Style

ArtistAbstract
Art
0.26
0.33
0.36
0.05
0.08
0.17
0.21
0.31
0.21

Abstract
Expressionism
0.47
0.54
0.50
0.24
0.22
0.32
0.38
0.46
0.36
Naive
Art
Minimalism Primitivism
0.50
0.38
0.55
0.34
0.56
0.30
0.27
0.15
0.33
0.18
0.38
0.33
0.36
0.31
0.47
0.42
0.38
0.37

Art
Informel
0.34
0.33
0.30
0.20
0.17
0.27
0.31
0.30
0.30

Art
NouveauModern
0.45
0.45
0.42
0.42
0.38
0.46
0.52
0.58
0.50

Neoclassicism
0.60
0.52
0.41
0.28
0.30
0.43
0.54
0.59
0.51

Baroque
0.38
0.43
0.40
0.26
0.24
0.32
0.45
0.49
0.39

Northern
Renaissance
0.44
0.31
0.30
0.23
0.28
0.31
0.34
0.41
0.32

Color
Field
Painting
0.73
0.73
0.77
0.33
0.40
0.52
0.55
0.61
0.55

Pop
Art
0.23
0.29
0.24
0.12
0.16
0.20
0.32
0.33
0.26

Post
Impressionism
0.26
0.28
0.23
0.17
0.18
0.18
0.31
0.35
0.26

APs and features’ dimension on eachconv1 1
conv2 1
conv3 1
conv4 1
conv5 1
conv1 1
conv2 1
conv3 1
conv4 1
conv5 1

128
23.52
27.80
33.60
38.52
41.92
44.1
50.8
62.8
71.6
72.4

256
24.24
28.48
34.28
39.28
43.20
44.7
52.6
65.1
71.8
74.2

512
24.68
29.16
35.28
40.36
43.08
44.8
53.6
65.7
73.3
75.0

1024
24.80
29.60
35.80
41.32
43.24
45.0
54.2
66.1
73.1
75.0

2048
24.92
29.88
36.32
41.32
44.08
45.2
54.1
66.0
73.9
74.5

4096
19.20
30.12
36.80
40.88
43.36
36.8
54.0
65.2
73.5
75.5

original
20.20
21.16
29.92
35.04
40.64
36.9
42.8
56.9
64.7
70.4

Table 4: Comparison with previous work.
[5] style 1024
47.30
53.98
SVM ⇒ 54.94

style 2048
54.27
56.26

style 4096
54.27
57.00

fc6
45.72
—

Cubism
0.28
0.28
0.24
0.12
0.20
0.23
0.24
0.35
0.28

5. CONCLUSIONSIn this paper, we have proposed style vectors which isbased on style matrices used in neural style transfer by Gatys
et al. [2], and have proved that style vectors is more eﬀective
than DCNN features for style classification using the samedataset as Karayev et al. [5] as well as two datasets gatheredfrom Wikiart.org. Especially, they were so eﬀective for theimages which depict common object and scenes with uniquestyles or touches. In addition, it was shown that the performance was boosted by using PCA dimension reduction toseveral thousand dimension. In comparison with Karayev
et al. [5], we achieved about 9.7 point improvements using a4096-d PCA-compressed style vector.
Since image style is diﬀerent from image content, stylevectors which are eﬀective as style representation of images
can be applied for many tasks such as style image searchand image style analysis. We believe the results of our work
is significant and helpful.

Realism
0.19
0.27
0.27
0.13
0.09
0.18
0.14
0.21
0.15

Expressionism
0.26
0.24
0.17
0.13
0.17
0.22
0.22
0.27
0.21

Rococo
0.42
0.45
0.37
0.16
0.15
0.29
0.35
0.42
0.28

High
Renaissance
0.29
0.22
0.21
0.21
0.15
0.19
0.23
0.36
0.19

Romanticism
0.27
0.24
0.23
0.09
0.12
0.14
0.24
0.25
0.15

Impressionism
0.26
0.27
0.25
0.13
0.13
0.29
0.33
0.26
0.29

Surrealism
0.20
0.21
0.16
0.06
0.04
0.10
0.11
0.16
0.11

Symbolism
0.19
0.21
0.22
0.10
0.11
0.12
0.18
0.29
0.15

Magic
Realism
0.37
0.41
0.43
0.27
0.24
0.34
0.41
0.47
0.37

Ukiyo-e
0.70
0.73
0.55
0.41
0.44
0.69
0.74
0.70
0.76

We have several future works: (1) we will apply style vectors for other domains than artistic drawings, and (2) weneed to improve processing time to extract style vectors.
Regarding extraction speed, we expect it can be reduced
much with eﬃcient GPU implementation. Once extracting
style vectors and applying PCA dimension reduction, stylevectors can be used in the same way as common visual features.
Acknowledgement: This work was supported by JSPS
KAKENHI Grant Number 15H05915 (Grant-in-Aid for Scientific Research on Innovative Areas “Innovative SHITSUKSAN Science and Technology”).

6.
fc7
41.35
—

Early
Renaissance
0.61
0.66
0.65
0.32
0.34
0.48
0.53
0.65
0.53

REFERENCES[1] K. D. Painter identification using local features and naive bayes.
In Proc. of IAPR International Conference on Pattern
Recognition, 2002.
[2] L. A. Gatys, A. S. Ecker, and M. Bethge. A neural algorithm ofartistic style. In arXiv:1508.06576, 2015.
[3] L. A. Gatys, A. S. Ecker, and M. Bethge. Texture synthesis andthe controlled generation of natural stimuli using convolutional
neural networks. In Advances in Neural Information Processing
Systems, 2015.
[4] K. He, X. Zhang, S. Ren, and J. Sun. Delving deep into
rectifiers: Surpassing human-level performance on imagenet
classification. In Proc. of IEEE International Conference on
Computer Vision, 2015.
[5] S. Karayev, M. Trentacoste, H. Han, A. Agarwala, A. Darrell,
T. Hertzmann, and H. Winnemoeller. Recognizing image style.
In Proc. of British Machine Vision Conference, 2014.
[6] A. Krizhevsky, I. Sutskever, and G. E. Hinton. Imagenet
classification with deep convolutional neural networks. In
Advances in Neural Information Processing Systems, 2012.
[7] L. Marchesotti and F. Perronnin. Learning beautiful (and ugly)
attributes. In Proc. of British Machine Vision Conference,
2013.
[8] N. Murray, D. Barcelona, L. Marchesotti, and F. Perronnin. Ava:
A large-scale database for aesthetic visual analysis. In Proc. ofIEEE Computer Vision and Pattern Recognition, 2012.
[9] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In Proc. ofInternational Conference on Learning Representation, 2015.