OS2-03

CONFIDENTIAL EXTENDED ABSTRACT.
DO NOT DISTRIBUTE ANYWHERE.

The 19th Meeting on Image Recognition and Understanding

Style Image Retrieval
Using CNN-based Style Vector
Shin Matsuo1,a)1.

Introduction

Figure 1 shows a lion image, a tiger image and the images
the styles of which are modiﬁed using the style of Gogh’sStarry Night by the style transfer method proposed by Gatys
et al. [2]. In this ﬁgure, the images in the right and theleft have common contents, while the image in the top andthe bottom have common styles. When we classify them interms of object categories, the images in the top are classiﬁed as “lion”, and the images in the bottom are classiﬁed as“tiger”. However, we sometimes classify images in terms of
their styles. In such case, the images in the left are classiﬁed as “photo”, while the images in the right are classiﬁed
as “drawing”.
Classifying images in terms of their styles is expected tohelp various kinds of applications such as style analysis ofimages/videos, and style-based image search. However, inthe computer vision community, recognition of image contents is paid much more attention to than styles. After
Deep Convolutional Neural Network (DCNN) [6] won ImageNet Large Scale Visual Recognition Challenge (ILSVRC)
in 2012, image recognition employing CNN becomes very
common to achieve high performance. Although the performance of CNN for 1000-class object category classiﬁcation
have already outperformed human [4], style classiﬁcation has
not been explored extensively except [5]. In object classiﬁcation where classifying object categories regardless of imagestyles is important, CNN is regarded as the best architecture
at present, while it is not conﬁrmed if CNN is the best instyle recognition of images where classifying styles regardlessof contents is needed.
In 2014, Karayev et al. [5] proposed the method on imagestyle recognition employing CNN-based image features andshowed CNN features outperformed the conventional features such as color histogram and GIST with a large margin. However, the classiﬁcation rates on style classiﬁcation
are not as high as the rates on object categorization. Therefore, style recognition is regarded as more challenging task
than object classiﬁcation.
Recently, Gatys et al. proposed an algorithm on artisticstyle transfer [2] which synthesis an image which has thestyle of a given style image and the contents of a given con1a)b)

Keiji Yanai1,b)Department of Informatics,Communications, Tokyo
matsuo-s@mm.inf.uec.ac.jp
yanai@cs.uec.ac.jp

The University of Electro-Fig. 1(Left) Original (content) images, (center) Style image,(right) Images synthesized by style transfer method.

tent image. This method replaces the information which aredegraded while the signal of the content image goes forward
through CNN layers with style information extracted fromthe style image, and reconstructs a new image which has
the same content as a given content images and the samestyle as a given style image as shown in Figure 1. In thismethod, they introduced “style matrix” which was presented
by Gram matrix of the feature maps, that is, correlation
matrix between feature maps in DCNN. Originally, a stylematrix was introduced in texture synthesis with DCNN by
the same authors [3].
Then, in this paper, we introduce style vectors which istransformed from style matrix, and examine its eﬀectiveness when using it as image features for image retrieval. By
the experiments, we conﬁrmed style matrix is more eﬀective
than common CNN activation features in image retrieval
tasks. In addition, we found PCA-compression boosted theperformance.

2.

Related Works

Although the number of works on image style analysis is
not so large, several works have been published so far. Mostof them are related to aesthetic analysis of images.
Datta et al. estimates aesthetic rating of images employing various visual features such as color saturation and hue,exposure of light and colorfulness, and wavelet-based texture

1

The 19th Meeting on Image Recognition and Understanding

co-eﬃcients. Marchesotti et al. [7] also proposed a method
to evaluate aesthetic rating using Aesthetic Visual Analysis(AVA) dataset [8].
Keren [1] proposed a method to classify artistic styles andachieved classiﬁcation of drawings of Pollock and Dali using
local features. As a recent work, Karayev et al. [5] classiﬁed style images using CNN activation features, and showed
CNN feature outperformed conventional features greatly.

3.

Proposed Method: Style Vector

Style matrix is used as an representation of image style in
the work on style transfer by Gatys et al. [2], which is computed as Gram matrix of feature maps in the speciﬁc layerof CNN. In their method, the style of a given style image
is transferred into a synthesized image by minimizing theEuclidean loss between the style matrix of both images withback propagation. Because style matrix contains style information of an image, we propose to use style vectors which
is transformed from style matrix as image representation ofimage retrieval.
Style matrix G is a set of correlation values between feature maps in the speciﬁc layer l. Style matrix Gl ∈ RMl ×Ml
is represented as
Gl = F l (F l )T(1)

where F l ∈ RMl ×Nl is a feature map in the layer l.
In this paper, we transform a style matrix in each layer
into a feature vector. Because a style matrix is a symmetrical matrix, the number of independent elements are(Ml )(Ml + 1)/2 where Ml is the number of the feature maps
in the layer l. We deﬁne a style vector V l as follows:V l = [Gl 1,1 , Gl 2,1 , ..., Gl Ml ,1 , Gl Ml ,2 , ..., Gl Ml ,Ml ] (2)We L2-normalize V l without/with signed square root before using a raw style vector as a feature vector. We represent L2-normalized of V l without/with signed square root
as S lL2 and S lsgnsqrt , respectively. They are calculated asfollows:
S lL2 =S lsgnsqrt =4.

VlkV l kpsgn(V l ) |V l |pksgn(V l ) |V l |kFig. 2

Images in “Style Dataset”.

Fig. 3

Images in “Artist Dataset”.

Wikiart.org which was used as the data source in the workby Karayev et al. [5]. The ﬁrst one is “Style dataset”, andthe other is “Artist dataset”.
“Style dataset” contains 100 images of each of 25 stylecategories which are oﬃcially deﬁned in Wikiart.org. Thatis, it contains 2500 images in total. We selected 25 style categories each of which contains more than 100 images among
all the style categories in Wikiart.org. Figure 2 shows someexamples in “Style dataset”.
“Artist dataset” contains 1000 images of the top 10 artists
regarding the number of images in Wikiart.org. We preparedthis dataset for examining if the style vector is eﬀective forclassifying artist drawing styles. Figure 3 shows some examples in “Artist dataset”.

(3)
(4)

Experiments

For experiments, we prepared an artistic pictorial image
dataset which contains metadata on image style and artists
by gathering them from Wikiart.org, and classiﬁed themwith style vectors extracted from ﬁve intermediate layers(conv1 1, conv2 1, conv3 1, conv4 1, and conv5 1) of theVGG-16 network [9]. In addition, we also examined thecombined style vectors of all the ﬁve layers as well. Finally,
we compared performance with Karayev et al. [5] using thesame dataset.
4.1 Dataset
We created two kinds of datasets by gathering images from

4.2 Evaluation of Style Vector
To search for an image which has similar style to a given
query images, we use nearest neighbor search with respect
to the style vector. In the experiments, to evaluate the eﬀectiveness of a style vector for image style search, we examine
if the nearest image to a query image belongs to the samestyle as the given query image. This is equivalent to nearest
neighbor classiﬁcation. In section 4.5, we used SVM as aclassiﬁer as well for comparison to the existing work.
4.3 Style Estimation for Two Datasets
We made experiments with two datasets by 5-fold classiﬁcation with the nearest neighbor classiﬁer. As a metric, weused standard Euclidean distance.
The convolutional neural network (CNN) we used in theexperiments was 16-layered very deep convolutional neuralnetwork, VGG-16 [9], pre-trained with the ILSVRC-2012

The 19th Meeting on Image Recognition and Understanding

1000-class dataset. We extracted style vectors from the layer
conv1 1, conv2 1, conv3 1, conv4 1 and conv5 1. We usedthem as single features and concatenation of them as combined features. In addition, we extracted fc6 and fc7 from
VGG-16, and fc6 from the standard Alexnet [6]. We usethem as visual features with raw, L2-normalization and L2normalization signed square root. Table 1 shows the resultsfor Style and Artist dataset.
The style vector which was the most eﬀective for both
Style dataset and Artist dataset was the one extracted
from conv5 1 with signed square root and L2 normalization. Therefore, this experimental result revealed that stylevectors outperformed conventional CNN features regarding
style-based image search. Especially, signed square root andL2 normalization of style vectors is signiﬁcant to obtain thebest performance. Note that in the rest part we use signed
square root L2 normalization for all the style vectors.
Table 2 shows average precision on each category in
Style dataset. The categories in which style vectors outperform CNN features were “Art Nouveau”, “Modern”,“High Renaissance”, “Post-Impressionism”, and “Symbolism”. Among them, the accuracy of “Nouveau Modern”
was the highest, which outperformed CNN features by 0.13.
This is because the images of “Art Nouveau Modern” tends
to expose very unique style but to have common scenes andobjects as motifs. For example, as shown in Figure 4 which
depicts two person with unique patterns, CNN features capture persons, while style vectors are expected to represent
patterns.
Conversely, the categories where CNN features is better
than style vectors were “Color Field Painting”, “MannerismLate Renaissance”, and “Minimalism”. Especially, the accuracy of “Color Field Painting” were better than style vectors
by 0.2. Figure 5 shows a sample of “Color Field Painting”
the nearest neighbor of which was an image of “AbstractArt” in case of style vectors. This is because style vectors
reﬂect local patterns rather than whole image structures.
Style all which is an aggregation of the style vectors extracted from ﬁve layers tends to be inferior to Style conv5 1.
Exceptionally, in “Ukiyo-e” it achieved the best performance
among all the features. This is because “Ukiyo-e” includes
various scales of unique patterns.
Regarding the results for “Artist dataset’, in the categories of “Chagall”, “Konchalovsky”, and “Gogh”, stylevectors clearly outperformed CNN features as shown in Figure 6. In their drawings, common motifs were drawn withunique pattern and touch. This is the similar tendency toStyle dataset. Although “Picasso” also have unique style,the performance was almost the same in case of style vector
and CNN features. Since “Pissaro” and “Monet” are similar
to each other in terms of their motifs and styles, they tended
to be confused with each other using any of style vectors orCNN features. The images in the category of “Piranesi”
was correctly classiﬁed using any features we used in theexperiments.

Fig. 4

Similar images to a sample of “Art Nouveau Modern” bya style vector and a CNN feature.

Fig. 5

Similar images to a sample of “Color Field Painting” bya style vector and a CNN feature.

Fig. 6
Table 3

Style

ArtistAll per-category APs on “Artist dataset”.

APs and the dimension of features on each dataset.

conv1 1
conv2 1conv3 1
conv4 1
conv5 1
conv1 1
conv2 1
conv3 1
conv4 1
conv5 1

128
23.52
27.80
33.60
38.52
41.92
44.1
50.8
62.8
71.6
72.4

256
24.24
28.48
34.28
39.28
43.20
44.7
52.6
65.1
71.8
74.2

512
24.68
29.16
35.28
40.36
43.08
44.8
53.6
65.7
73.3
75.0

1024
24.80
29.60
35.80
41.32
43.24
45.0
54.2
66.1
73.1
75.0

2048
24.92
29.88
36.32
41.32
44.08
45.2
54.1
66.0
73.9
74.5

4096
19.20
30.12
36.80
40.88
43.36
36.8
54.0
65.2
73.5
75.5

original
20.20
21.16
29.9235.04
40.64
36.9
42.8
56.9
64.7
70.4

4.4 Dimensionality reduction with PCA
Although the most eﬀective style vector was S conv5 1in the previous experiment, its dimension was very high,131, 328 (= 512 × (512 + 1)/2), which is 32 times as manyas the dimension of CNN features, 4096. To resolve high dimensionality, we applied PCA to style vectors for dimension
reduction. With PCA compression, we obtained 128-d, 256d, 512-d, 1024-d, 2048-d, and 4096-d style vectors, and madethe same experiments as the previous ones. Table 3 shows
the results for both Style Dataset and Artist Database.
For Style dataset, 2048-d style vectors achieved the best
performance, while 4096-d vectors was the best for Artistdataset. Surprisingly, both the results exceeded the resultswith raw style vectors with large margin by 3.44 and 5.1,respectively. From these results, PCA is helpful not only for
dimension reduction but also boosting of the classiﬁcation
performance.

The 19th Meeting on Image Recognition and Understanding

Table 1StyleArtistraw
L2normsgnsqrtraw
L2normsgnsqrtimagenet fc6vgg fc6
vgg fc7style conv1 1style conv2 1style conv3 1
style conv4 1
style conv5 1
style allMannerism
Late
Renaissance
0.42
0.55
0.53
0.20
0.19
0.32
0.44
0.45
0.39

Alex fc6
36.28
39.28
38.00
61.70
61.00
61.90

VGG fc6
33.68
39.44
39.36
59.3
62.9
62.3

Abstract
Art
0.26
0.33
0.36
0.05
0.08
0.17
0.21
0.31
0.21

Abstract
Expressionism
0.47
0.54
0.50
0.24
0.22
0.32
0.38
0.46
0.36
NaiveArt
Minimalism
Primitivism
0.50
0.38
0.55
0.34
0.56
0.30
0.27
0.15
0.33
0.18
0.38
0.33
0.36
0.31
0.47
0.42
0.38
0.37

Table 4
[5]style 1024
47.30
53.98
SVM ⇒ 54.94

Mean APs on two datasets

VGG fc7
32.04
36.04
36.28
56.7
61.1
60.8

Style conv1 1
15.40
17.36
20.20
34.00
32.60
36.90

Table 2

All per-category APs on the Style dataset

Art
Informel
0.34
0.33
0.30
0.20
0.17
0.27
0.31
0.30
0.30

Art
NouveauModern
0.45
0.45
0.42
0.42
0.38
0.46
0.52
0.58
0.50

Neoclassicism
0.60
0.52
0.41
0.28
0.30
0.43
0.54
0.59
0.51

Baroque
0.38
0.43
0.400.26
0.24
0.32
0.45
0.49
0.39

Northern
Renaissance
0.44
0.31
0.30
0.23
0.28
0.31
0.34
0.41
0.32

ColorField
Painting
0.73
0.73
0.77
0.33
0.40
0.52
0.55
0.61
0.55
PopArt
0.23
0.29
0.24
0.12
0.16
0.20
0.32
0.33
0.26

Comparison with previous work.
style 2048
54.27
56.26

style 4096
54.27
57.00

fc6
45.72
48.11

fc7
41.35
45.35

4.5 Comparison with the previous work
We made experiments with the same image data as
Karayev et al. [5] for comparison. We used PCA-compressed
Style conv5 5 (1024d, 2048d and 4096d) as style vectors andfc6 and fc7 as CNN features. For style features, we used
SVM as a classiﬁer as well as a nearest neighbor classiﬁer.
Table 4 shows the results including the results in [5]. As aresult, style vectors outperformed [5] greatly, which proved
the eﬀectiveness of the style vector.

5.

Style conv2 1
19.20
21.36
21.16
40.90
42.80
42.80

Conclusions

In this paper, we have proposed style vectors which isbased on style matrices used in neural style transfer by Gatys
et al. [2], and have proved that style vectors is more eﬀective
than DCNN features for style classiﬁcation using the samedataset as Karayev et al. [5] as well as two datasets gatheredfrom Wikiart.org. Especially, they were so eﬀective for theimages which depict common object and scenes with uniquestyles or touches. In addition, it was shown that the performance was boosted by using PCA dimension reduction toseveral thousand dimension. In comparison with Karayev
et al. [5], we achieved about 9.7 point improvements using
a 4096-d PCA-compressed style vector.
Since image style is diﬀerent from image content, stylevectors which are eﬀective as style representation of images
can be applied for many tasks such as style image searchand image style analysis. We believe the results of our work
is signiﬁcant and helpful.
We have several future works: (1) we will apply style vectors for other domains than artistic drawings, and (2) weneed to improve processing time to extract style vectors.

Cubism
0.28
0.28
0.24
0.12
0.20
0.23
0.24
0.35
0.28

Post
Impressionism
0.26
0.28
0.23
0.17
0.18
0.18
0.31
0.35
0.26

Style conv3 1
25.48
28.56
29.92
52.80
56.00
56.90

Early
Renaissance
0.61
0.66
0.65
0.32
0.34
0.48
0.53
0.65
0.53

Realism
0.19
0.27
0.27
0.13
0.09
0.18
0.14
0.21
0.15

Style conv4 1
31.20
34.48
35.04
60.70
63.80
64.70

Expressionism
0.26
0.24
0.17
0.13
0.17
0.22
0.22
0.27
0.21

Rococo
0.42
0.45
0.37
0.16
0.15
0.29
0.35
0.42
0.28

Style conv5 1
34.08
39.60
40.64
64.70
68.80
70.40

High
Renaissance
0.29
0.22
0.21
0.21
0.15
0.19
0.23
0.36
0.19

Romanticism
0.27
0.24
0.23
0.09
0.12
0.14
0.24
0.25
0.15

Impressionism
0.26
0.27
0.25
0.13
0.13
0.29
0.33
0.26
0.29

Surrealism
0.20
0.21
0.16
0.06
0.04
0.10
0.11
0.16
0.11

Symbolism
0.19
0.21
0.22
0.10
0.11
0.12
0.18
0.29
0.15

Style all
25.72
27.36
33.08
51.90
52.20
61.10

Magic
Realism
0.37
0.41
0.43
0.27
0.24
0.34
0.41
0.47
0.37

Ukiyo-e
0.70
0.73
0.55
0.41
0.44
0.69
0.74
0.70
0.76

Regarding extraction speed, we expect it can be reduced
much with eﬃcient GPU implementation. Once extracting
style vectors and applying PCA dimension reduction, stylevectors can be used in the same way as common visual features.
Acknowledgement: This work was supported by JSPS
KAKENHI Grant Number 15H05915 (Grant-in-Aid for Scientiﬁc Research on Innovative Areas “Innovative SHITSUKSAN Science and Technology”).
References
[1]
[2]
[3]

[4]

[5]
[6]
[7]
[8]

[9]

D, K.: Painter Identiﬁcation Using Local Features and Naive
Bayes, Proc. of IAPR International Conference on Pattern
Recognition (2002).
Gatys, L. A., Ecker, A. S. and Bethge, M.: A Neural Algorithm of Artistic Style, arXiv:1508.06576 (2015).
Gatys, L. A., Ecker, A. S. and Bethge, M.: Texture synthesis
and the controlled generation of natural stimuli using convolutional neural networks, Advances in Neural Information
Processing Systems (2015).
He, K., Zhang, X., Ren, S. and Sun, J.: Delving deep into rectiﬁers: Surpassing human-level performance on imagenet classiﬁcation, Proc. of IEEE International Conference on Computer Vision (2015).
Karayev, S., Trentacoste, M., Han, H., Agarwala, A., Darrell,T. Hertzmann, A. and Winnemoeller, H.: Recognizing ImageStyle, Proc. of British Machine Vision Conference (2014).
Krizhevsky, A., Sutskever, I. and Hinton, G. E.: ImageNet
Classiﬁcation with Deep Convolutional Neural Networks., Advances in Neural Information Processing Systems (2012).
Marchesotti, L. and Perronnin, F.: Learning beautiful (andugly) attributes., Proc. of British Machine Vision Conference(2013).
Murray, N., Barcelona, D., Marchesotti, L. and Perronnin,F.: AVA: A Large-Scale Database for Aesthetic Visual Analysis., Proc. of IEEE Computer Vision and Pattern Recognition(2012).
Simonyan, K. and Zisserman, A.: Very Deep Convolutional
Networks for Large-Scale Image Recognition, Proc. of International Conference on Learning Representation (2015).