DEIM Forum 2021 D21-2

Generating Color Patterns of Impression Words
for Retrieving Abstract Landscape Paintings
Kodai TAKIGAWA†

Takehito UTSURO††† College of Engineering Systems, School of Science and Engineering, University of Tsukuba, Japan
1-1-1 Tennodai, Tsukuba, Ibaraki 305-8573, Japan†† Faculty of Engineering, Information and Systems, University of Tsukuba, Japan
1-1-1 Tennodai, Tsukuba, Ibaraki 305-8573, Japan
Abstract This paper proposes a method to generate color patterns of impression words for retrieving abstract
landscape paintings. As impression words, this paper studied warm, cold, natural, and desolate. First, to eachpainting, we annotate one or more of those four impression words that express the impression of the painting. After
ﬁve dominant colors are extracted from each painting, candidates for colors included in the color pattern of each
impression word are manually selected from the ﬁve dominant colors. Those candidates collected throughout thewhole dataset of abstract landscape paintings are clustered through k-means clustering algorithm. The centers ofeach cluster are selected as the colors included in the color pattern of the impression word. In order to evaluate
the generated color patterns of the four impression words, this paper proposes four variants of distances between
the color pattern of an impression word and the dominant color of an abstract landscape painting. This paper
also evaluates the proposed distances in terms of ranking paintings according to their distances from the query
impression word. The color patterns of the four impression words generated by the proposed method outperformed
the color patterns designed for general purpose use and published by NIPPON COLOR & DESIGN RESEARCHINSTITUTE INC. (NCD).
Key words impression word, abstract landscape painting, k-means clusteringcolor pattern of an impression word. As impression words,

1 Introduction
There exist several image retrieval systems that use impression words as the retrieval query keyword. Those include K-DIME [2], IRIS [7], ART MUSEUM [6], and INMULKansei [4]. However, none of them does not focus on abstract
landscape paintings. The proposed method in this paper focuses on retrieval of abstract landscape paintings throughan impression word as the retrieval query keyword. As forthe color pattern, INMUL-Kansei [4] uses the color patterns
published by NIPPON COLOR & DESIGN RESEARCH INSTITUTE INC. (NCD) [3], [5], and focuses on ﬂower images
and brush paint images. However, the color patterns published by NCD are not appropriate for retrieving abstract
landscape paintings.
This paper proposes a method to generate color patternsof impression words for retrieving abstract landscape paintings. Since the L∗ a∗ b∗ color space ﬁts our impressions [4],the color space is used throughout the processes of extracting dominant colors from a painting, calculating the distance
between a painting and an impression word, generating thethis paper studied warm, cold, natural, and desolate. First,
to each painting, we annotate one or more of those four impression words that express the impression of the painting.
After ﬁve dominant colors are extracted from each painting, candidates for colors included in the color pattern ofeach impression word are manually selected from the ﬁve
dominant colors. Those candidates collected throughout thewhole dataset of abstract landscape paintings are clusteredthrough k-means clustering algorithm with k = 5. The centers of each cluster are selected as the colors included in thecolor pattern of the impression word. In order to evaluate
the generated color patterns of the four impression words,
this paper proposes four variants of distances between thecolor pattern of an impression word and the dominant colorsof an abstract landscape painting.
This paper also evaluates the proposed distance measuresin terms of ranking paintings according to their distancesfrom the query impression word. The color patterns of thefour impression words generated from the dataset of abstract landscape paintings outperformed the color patterns
designed for general purpose use and published by NCD.

Table 1 Color Patterns of Impression Words
Impression Word Generated by the Proposed Method

Published by NCD [3], [5]

warm

cold

naturaldesolate

Abstract Landscape Painting
Extracting
Pixelsk-means
Clustering
Dominant Colors

Extracting
theCenters

Figure 1 Extracting Dominant Colors of a Painting

tween scenery images and impression words. It is stated by

2 Related Works
K-DIME [2] uses neural networks to deﬁne relationships
between images and impression words. Images are interpreted as subsets of perceptual states, each of which is statedas “the signature, or set of low-level descriptive variables, extracted by a set of image processing algorithms that describes
the image in terms of color, texture, and shape features” by
Bianchi-Berthouze [2]. The neural network uses the subsetsof the perceptual states as inputs.
IRIS [7] uses neural networks to deﬁne relationships be-

Kuroda and Hagiwara [7]: “An image is divided into someregions. Next each region is roughly classiﬁed into the sky,earth and water categories by the image recognition method
using a neural network. Then the image characteristics are
extracted from each category.” The neural network uses thecharacteristics as inputs.
ART MUSEUM [6] employs the canonical correlation analysis to deﬁne relationships between paintings and impression
words. A painting has 33 characteristics, and impression
words are expressed as a 30-dimensional vector.
INMUL-Kansei [4] uses the color patterns published by

Abstract Landscape Paintings
with the Impression Word
natural

Dominant
ColorsManually
Selecting
Candidates for
Color Pattern Colors

Generated Color Pattern of
the Impression Word
naturalExtracting
theCenters

k-means
Clustering

Candidates for
Color Pattern Colors
Figure 2 Generating the Color Pattern of the Impression Word, natural

NCD to deﬁne relationships between images and impression

algorithm with k = 5 to the pixels, ﬁve clusters are obtained.

words. The images studied are those of ﬂowers and brush

The centers of those ﬁve clusters are extracted as the colors

paintings. Table 1 shows the color patterns published byincluded in the dominant colors of the painting. Together

NCD [3], [5] for each of the four impression words.

with extracting those ﬁve centers, the number of pixels in

As for the relationships between impression words and im-each cluster is counted and the rate of those numbers amongages, some of those related works employed existing man-the ﬁve clusters is used to deﬁne the proportion of each dom-ually designed color patterns [4], while others proposed toinant color within the overall ﬁve dominant colors as shown

apply neural networks [2], [7] and the canonical correlationin the left of Figure 1.

analysis [6]. Compared to those existing approaches, this paper proposes another approach to deﬁne the relationships
between an impression word and its color pattern, where

3. 2 Generating the Color Pattern of an Impression
Word

we focus on abstract landscape paintings. According to the

Figure 2 shows the procedure of generating the color pat-evaluation results we show in this paper, the color patterns

tern of the impression word, natural. As the ﬁrst step, tocreated through the proposed approach are more suitableeach painting, we manually annotate one or more impression

for retrieving abstract landscape paintings compared to thewords that express the impression of the painting. Next,

manually created color patterns published by NCD.

from each painting that is annotated with the impression

Bhandi and Devi K. A. [1] propose a content based image

word natural, ﬁve dominant colors are extracted. Then, from

retrieval method that utilizes the pre-trained VGG16 deep

each painting, candidates for colors to be included in the

CNN model [8].

color pattern of the impression word natural are manually
selected from the ﬁve dominant colors. Those candidates

3 Overview of the Framework

collected throughout the whole dataset of abstract landscapepaintings are clustered through k-means clustering algorithm

3. 1 Extracting the Dominant Colors of a Painting

with k = 5. Finally, the centers of the ﬁve clusters are ex-

Figure 1 shows the procedure of extracting the dominant

tracted as the colors to be included in the color pattern ofcolors from a painting. First, a painting is converted to pixels

the impression word natural. For each of the four impression∗ ∗ ∗in the L a b color space. By applying k-means clusteringwords warm, cold, natural, and desolate, Table 1 comparesAbstract
LandscapePainting

Abstract
LandscapePainting

Dominant
ColorsDominant
ColorsColor Pattern of
the Impression Word
naturalDistance = 48

Distance = 47

Rankingby DistanceRankingby Distancetothe Impression Word,
naturalDistance = 240

Distance = 48

Distance = 47

Distance = 240

Figure 3 Retrieving Paintings with an Impression Word by Matching the Dominant Colors of a Painting and the Color Pattern of the Impression Wordthe color patterns generated by the proposed method andone-to-one mappings of the ﬁve color pattern colors ciw

those published by NCD.

(i = 1, . . . , 5) of w and the ﬁve dominant colors djp (j =3. 3 Retrieving Paintings with an Impression Word
Figure 3 shows the overview of retrieving abstract landscape paintings with an impression word by matching thedominant colors of a painting and the color pattern of theimpression word. First, a distance measure between a painting and an impression word is selected from those four variants of distance measures proposed in section 4. Then, all

1, . . . , 5) of p. Next, The L∗ a∗ b∗ color space representations of ciw and djp are denoted as (L∗ (ciw ), a∗ (ciw ), b∗ (ciw )) and(L∗ (djp ), a∗ (djp ), b∗ (djp )). The number of all the pixels within
the cluster whose center is the dominant color djp is denoted
as njp (j = 1, . . . , 5). The number of all the pixels within
p is denoted as Np . The distance d(ciw , djp ) (i, j = 1, . . . , 5)between ciw and djp is deﬁned as below:

the paintings are ranked according to their distances in ascending order. Then, we examine whether each of the rankedd(ciw , djp ) =paintings is suitable for the query impression word.

Np
njp



X

x(ciw ) − x(djp )2 1/2x∈{L∗ ,a∗ ,b∗ }Now, we denote the distance between an abstract land-

4 Distance Measures between the Color
Pattern of an Impression Word and Dominant Colors of a Painting

scape painting p and an impression word w as d(p, w), where
its four variants are introduced in the following sections.
4. 1 Minimum Distance over 120 One-to-One Map-

This section deﬁnes the four distance measures between
the color pattern of an impression word and dominant colorsof a painting. In those deﬁnitions, w and ciw (i = 1, . . . , 5)denote an impression word and one of the ﬁve color patterncolors of w, respectively. Also, p and djp (j = 1, . . . , 5) denotean abstract landscape painting and one of the ﬁve dominantcolors of p, respectively. Let the following r(w, p) be oneof the overall 5!=120 one-to-one mappings of the ﬁve color
pattern colors ciw (i = 1, . . . , 5) of w and the ﬁve dominantcolors djp (j = 1, . . . , 5) of p:nr(w, p) =pings of the Color Pattern and Dominant Colors
In this variant, out of the overall 5!=120 one-to-one mappings of the ﬁve color pattern colors ciw (i = 1, . . . , 5) of w andthe ﬁve dominant colors djp (j = 1, . . . , 5) of p, the one whichj(i)gives the minimum of the sum of the distance d(ciw , dp ) overthe ﬁve pairs i = 1, . . . , 5 is selected to deﬁne the distanced(w, p) as below:o⟨ciw , dj(i)p ⟩ i = 1, . . . , 5d(w, p) =min

Xr (∈R(w,p))j(i)Then, let R(w, p) denote the set of the overall 5!=120⟨ciw ,dp⟩ (∈r)d(ciw , dpj(i) )4. 2 Single Minimum Distance Pair of a Color Pattern Color and a Dominant Color

5. 2 Results
With the color patterns generated by the proposed

In this variant, out of the over all 5×5=25 pairs of the ﬁve

method, Figure 4 shows the comparison of the four distancecolor pattern colors ciw (i = 1, . . . , 5) of w and the ﬁve dom-measures between the color pattern of an impression word

inant colors djp (j = 1, . . . , 5) of p, the one which gives theand dominant colors of a painting, where the average of theminimum of the distance d(ciw , djp ) (i, j = 1, . . . , 5) is selected

precision-recall curves of the four impression words is plot-

to deﬁne the distance d(p, w) as below:

ted for each of the four variants. As can be seen from thisd(w, p) =mini,j=1,...,5

comparison, “Min. Pairs over 120 One-to-One Mappings”

d(ciw , djp )slightly outperforms other three distance measures.

4. 3 Minimum Distance Pairs over the Five Color
Pattern Colors / Five Dominant Colors

Next, for each of the four variants of the distance measures, Figure 5 shows the comparison of the color patterns

For the variant of “minimum distance pairs over the ﬁve

generated by the proposed method and those published bycolor pattern colors”, for each of the ﬁve color pattern colors

NCD, where the average of the precision-recall curves of theciw (i = 1, . . . , 5) of w, one dominant color djp (j = 1, . . . , 5)four impression words is plotted for each of the four variants.

of p which gives the minimum of the distance d(ciw , djp ) is

Also, for each of the four impression words, Figure 6 shows

selected. Then, their sum over the ﬁve color pattern colors

the comparison of the color patterns generated by the pro-below is employed as this variant of the distance d(w, p):posed method and those published by NCD. Here, for eachX5d(w, p) =of the four impression words, out of the four variant distancemin d(ciw , djp )measures, the one performing the best is selected for the

j=1,...,5i=1

For the variant of “minimum distance pairs over the ﬁve
dominant colors”, on the other hand, for each of the ﬁve dominant colors djp (j = 1, . . . , 5) of p, one color pattern color ciw(i = 1, . . . , 5) of w which gives the minimum of the distanced(ciw , djp ) is selected. Then, their sum over the ﬁve dominant colors below is employed as this variant of the distancefor each of the two types of the color patterns (the proposed
and those by NCD), Table 3 lists the distance measures selected as optimal. As shown in Figure 5 and Figure 6, forall of the four distance measures as well as for all of the four
impression words, the color patterns generated by the proposed method outperform those published by NCD. Overall,d(w, p):d(w, p) =evaluation. For each of the four impression words as well as

5X

in terms of retrieving abstract landscape paintings, the results show that the color patterns generated by the proposed

min d(ciw , djp )i=1,...,5

method outperform those published by NCD. These results

j=1

support our claim that the proposed approach is suitable for

5 Evaluation and Results

generating the color patterns that are speciﬁcally designed
for the retrieval of abstract landscape paintings.

5. 1 Evaluation ProcedureSpeciﬁc abstract landscape paintings used for generating
the color patterns and the evaluation are those collected fromthe Internet. Their source Web sites

1234, the artists andtheir statistics are shown in Table 2.

6 Conclusion
This paper proposed a method to generate color patterns ofimpression words for retrieving abstract landscape paintings.

In the procedure of retrieving the abstract landscape paint-

We also introduced four distance measures to deﬁne similar-

ings with each of the four words warm, cold, natural, andity between an impression word and an abstract landscapedesolate, we sort all the paintings for evaluation according topainting. In terms of retrieving abstract landscape paint-

their distances in ascending order. Next, we regard the top

ings, the evaluation results show that the color patterns gen-ranked k paintings as predicted as positive. Then, for eacherated by the proposed method outperform those publishedof the four variants of the distance measures and for each ofby NCD. Future work includes introducing the state of thethe four impression words, a precision-recall curve is plottedart deep learning based techniques to represent the relation

by varying the lower bound k from top to the bottom of the

between impression words and abstract landscape paintings.

ranking.

Another future work includes introducing a mechanism of

1：https://www.artfinder.com/beata-belanszky/#/

accepting feedback from users of the system for retrieving

2：https://islesfordartists.com/artists#/sue-charles

abstract landscape paintings.

3：http://poramoralarte-exposito.blogspot.com/search/label/Herv%C3%A9%20Lenouvel
4：https://www.saatchiart.com/account/artworks/425728

References[1] V. Bhandi and S. Devi K.A. Image retrieval using features

Table 2 Statistics of the Dataset of Abstract Landscape Paintings
Impression Word
warm cold natural desolate Total

Web site

ArtistArtﬁnderBeata Belanszky Demko

9

12

0

0

23

Sue Charles

4

3

18

0

34

Hervé Lenouvel

23

30

19

37

78

36

45

37

37

135

44

56

20

27

116

For Generating Color Patterns Islesford Artists Gallery
Por amor al arte
Total

Precision

For Evaluation

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Saatchi Art

Ivan Grozdanovski

Min. over 120 One-to-One Mappings
Min. Pairs over the Five Dominant Colors

Single Min. Pair
Min. Pairs over the Five Color Pattern Colors

0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

RecallFigure 4 Comparison of Four Distance Measures between the Color Pattern of an Impression Word and Dominant Colors of a Painting (by the proposed method)
Table 3 Distance Measures selected as Optimal in Figure 6
Impression Word / Color Pattern
warm

cold

Proposed Method
NCD

naturalNCD

desolate—

Min. over 120 One-to-One Mappings

—

—

[2]
[3]

[4]

[5]

[6]

Single Min. Pair
Proposed Method—

NCD
Proposed Method

Distance Measure—from pre-trained deep CNN. International Journal of Scientiﬁc & Technology Research, 9(6):687–693, 2020.
N. Bianchi-Berthouze. K-DIME: An aﬀective image ﬁlteringsystem. IEEE Multimedia, 10(3):103–106, 2003.
NIPPON COLOR & DESIGN RESEARCH INSTITUTE
INC. 心 を 伝 え る 配 色 イ メ ー ジ. Kodansha, 2008. (inJapanese).
H. Kimoto. An image retrieval system using impressional
words and the evaluation of the system. Journal of Information Processing, 40(3):886–898, 1999. (in Japanese).
S. Kobayashi and (NIPPON COLOR & DESIGN RESEARCH INSTITUTE INC. editor). 配色イメージワーク. Kodansha, 1995. (in Japanese).
T. Kurita, T. Kato, I. Fukuda, and A. Sakakura. Sense
retrieval on a image database of full color paintings. Jour-

Proposed Method Min. Pairs over the Five Color Pattern Colors
NCD

Min. Pairs over the Five Dominant Colors

nal of Information Processing, 33(11):1373–1383, 1992. (inJapanese).
[7] K. Kuroda and M. Hagiwara. An image retrieval system byimpression words and speciﬁc object names-IRIS. Neurocomputing, 43(1-4):259–276, 2002.
[8] K. Simonyan and A. Zisserman. Very deep convolutional
networks for large-scale image recognition. In Proc. 3rd
ICLR, 2015.

Color Pattern by the Proposed Method
Precision

Precision

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Color Pattern by NCD
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

1

Color Pattern by the Proposed Method

Color Pattern by NCD
0

0.1

0.2

0.3

0.4

Color Pattern by the Proposed Method

Color Pattern by NCD
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(b) Single Minimum Distance Pair

Precision

Precision(a) Minimum Distance over 120 One-to-One Mappings

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

0.5

RecallRecall

0.6

0.7

0.8

0.9

1

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Color Pattern by the Proposed Method

Color Pattern by NCD
00.1

Recall

0.3

0.4

0.5

0.6

0.7

RecallMinimum Distance Pairs
(c)

0.2

Minimum Distance Pairs(d)over the Five Color Pattern Colors

over the Five Dominant Colors

Figure 5 Comparison of the Color Patterns of the Proposed Method and NCD (1) (perdistance measure)

0.8

0.9

1

Precision

Precision

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Color Pattern by NCD
Color Patternby the Proposed Method
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Color Pattern by NCD
Color Pattern by the Proposed Method

0

1

0.1

0.2

0.3

0.4

Color Pattern by the Proposed Method

Color Pattern by NCD
0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

0.8

0.9

1

(b) cold

Precision

Precision(a) warm

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

0.5

RecallRecall

0.6

0.7

0.8

0.9

1

1
0.9
0.8
0.7
0.6
0.5
0.4
0.3
0.2
0.1
0

Color Pattern by the Proposed Method
Color Pattern by NCD

0

0.1

0.2

0.3

0.4

0.5

RecallRecall(c) natural(d) desolate

0.6

Figure 6 Comparison of the Color Patterns of the Proposed Method and NCD (2) (perimpression word)

0.7

0.8

0.9

1