Volume 35 (2016), Number 7

Pacific Graphics 2016
E. Grinspun, B. Bickel, and Y. Dobashi(Guest Editors)

Aesthetic Rating and Color Suggestion for Color Palettes
N. Kita and K. Miyata
Japan Advanced Institute of Science and Technology, Japan

Figure 1: The proposed method can rate a given color palette with any number of colors relative to human aesthetic preferences. The
proposed method suggests a compatible color for the given palette, which allows us to expand the palette while retaining color harmony tosupport user exploration of color design.
Abstract
A model to rate color combinations that considers human aesthetic preferences is proposed. The proposed method does notassume that a color palette has a specific number of colors, i.e., input is not restricted to a two-, three-, or five-color palettes.
We extract features from a color palette whose size does not depend on the number of colors in the palette. The proposed rating
prediction model is trained using a human color preference dataset. The model allows a user to extend a color palette, e.g.,
from three colors to five or seven colors, while retaining color harmony. In addition, we present a color search scheme for agiven palette and a customized version of the proposed model for a specific color tone. We demonstrate that the proposed model
can also be applied to various palette-based applications.
Categories and Subject Descriptors (according to ACM CCS): I.3.0 [Computer Graphics]: General—

1. Introduction
Color palettes and patterns are used in various fields, such as graphics, web, packaging, fashion, and interior design. Such design elements are also used in business, e.g., illustrations in documents
and presentations, and play an important role in scientific visualizations. In addition, individuals make personal decisions about aesthetics and harmonious color combinations (e.g., the color of curtains and wallpaper, room theme colors, or even cushions and otherdecorative objects in a room). Therefore, evaluating the aesthetics
and color harmony of color combinations is not only important for
design professionals, it is also important for many people. However, creating and evaluating a set of colors (i.e., a color palette) isdifficult for most people.
Many books, such as [Kob91], that help a user select a color
palette based on a color image (e.g., casual, elegant, or romantic)
have been published. Such books illustrate the relationship between
a color image and a three-color palette. On the web sites, such
as Adobe Color (formerly known as Adobe Kuler) [Col16a] andc 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John
Wiley & Sons Ltd. Published by John Wiley & Sons Ltd.

COLOURlovers [COL16b], users can share the color palettes andpatterns with other community members. These services are useful for novices and designers because they can use others’ palette
designs as references or customize palettes to suit personal requirements and taste. However, customizing a palette composed of N
colors that can be expanded to N + α while retaining the originalcolor harmony remains difficult because choosing a color that iscompatible with the original palette from many possible colors ischallenging and tedious.
We propose a color palette rating model that considers humanaesthetic preferences. In addition, we propose a compatible color
suggestion method, which is based on the color palette rating
model, to expand a given palette while retaining color harmony(Fig. 1). First, we construct a model that predicts the ratings of agiven palette composed of any number of colors by learning froma large dataset of human-rated color palettes. Here, we propose
a feature extraction method that does not depend on the size ofcolor palette, which enables us to rate a color palette composed ofany number of colors. We also propose a color suggestion method,

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

wherein we employ a sampling method to search color candidates
efficiently from a large color space.
In summary, we present the following:• a model that can evaluate a given color palette aesthetics regardless of size, i.e., the number of colors in the palette;• a feature extraction method to learn the weights of the model;• a method to suggest a compatible color for palette expansion,i.e., a color that retains color harmony;• a sampling method to search the compatible color from a color
space.
2. Related Work
Color Harmony Theory. Color harmony theory as a research
topic did not begin until after Newton reported his experimentswith the light spectrum in 1672. Various color harmony theories,
such as Goethe’s thought-provoking color theory [Goe71], were
proposed near the end of the 18th century and throughout the 19th
century. In the early 20th century, the first practical color-ordersystem were derived from the color theories proposed by Munsell [BC69] and Ostwald [OB69]. The color theory proposed byMoon and Spencer [MS44], which was based on the mathematicalanalysis of a user study, represents the beginning of modern color
harmony theory. Following Moon and Spencer, Itten proposed acolor theory which is defined on a hue wheel [Itt74].
Color Harmony Model. Many color harmony models based on
the mathematical analysis of user study results have been proposed [OL06,SBS10,OCLM11,ORLS∗ 12]. However, such models
only evaluate two or three color combinations with a small number of participants (fewer than one hundred), which restricts thegeneralization of the model. On the other hand, O’Donovan et al.
collected 327, 381 human ratings of 22, 376 color themes (palettes)
using Amazon Mechanical Turk (MTurk) and proposed a modeltrained using a machine learning algorithm [OAH11]. Lin et al.
also collected 1, 600 data items relative to how people extract color
themes from images from 160 MTurk participants [LH13].
These models can only rate two-, three-, or five-color palettes,
which is too limited because color palettes are typically composed
as many as seven colors. In contrast to previous approaches, theproposed method is not limited to a specific number of colors ina palette, i.e., our feature extraction method and machine learning method can handle a color palette composed of any numberof colors, which allows us to rate any color palette. Therefore, theproposed method can be applied to a wide range of applications,such as compatible color suggestion, 2D pattern coloring, and otherfields of color design.
Applications. Cohen-Or et al. proposed a method that employed aharmonic template on a hue wheel to harmonize the target image
colors [COSG∗ 06], and Li et al. proposed a fast image recolorization algorithm by employing geodesic distance-based color harmonization [LZNH15]. Sawant et al. applied color harmonization forvideos [SM08], and Zhang et al. employed a color scheme replacement method in their video stream abstraction method for stylization [ZLHM11]. Wang et al. proposed a data-driven approach toconvert a target image based on a given color theme [WYW∗ 10],
while Chang et al. proposed a method for recoloring a photograph based on a color palette [CFL∗ 15]. Lin et al. proposed aprobabilistic factor graph model to color 2D patterns automatically [LRFH13], and Kim et al. proposed a method based oncolor perception theories to assign color to 2D patterns automatically [KYKL14]. Employing the color compatibility rating model
proposed in [OAH11] as the one of the terms to be optimized in
the optimization problems they formulated, Yu et al. proposed outfit synthesis (fashion design) [YYTC12], and, as mentioned previously, Lin et al. employed the model ratings as a global aesthetic
term for 2D pattern coloring [LRFH13]. Chen et al. proposed automatic material suggestion for indoor digital scenes [CXY∗ 15].
Since [YYTC12, LRFH13,CXY∗ 15] employed the rating model
proposed in [OAH11], their method is limited to only five-color
palettes in their methods, which limits the scope of application. In
contrast, we propose a feature extraction method that does not depend on the number of colors in a given palette. Therefore, we cansuggest a compatible color for a given palette with any number ofcolors. For example, given a three-color palette as input, the palettecan be expanded to four-, five-, or even seven-color palettes using
our color suggestion method while maintaining color harmony.
3. Model Training for Color Palette Rating
We employ a machine learning approach to learn the weights ofa model for rating a color palette. First, we extract features from
palettes in a dataset. Then, multivariable regression analysis is applied to learn the weights for each feature vector.
3.1. Dataset
We use a MTurk dataset from O’Donovan et al. [OAH11]. In thedataset, 10, 743 color palette ratings by MTurk users are available.
The palettes were collected randomly from the Adobe Kuler website. Each palette was rated on a scale of 1-5 by 40 participants.
3.2. Feature Extraction
O’Donovan et al. extracted 334 features from a palette [OAH11].
We extract 118 equivalent features from a palette. These include
palette colors, mean, standard deviation, median, max, min, andmax minus min across a single channel in each color space, i.e.,
RGB, CIELAB, HSV, and CHSV† . We also extract plane-fitting
features, i.e., a 2D plane is fit to 3D color coordinates using PCA
in RGB, CIELAB, and CHSV color spaces, while in the HSV color
space, we extract hue entropy and hue joint/adjacent probabilities. Please refer to the original paper for the details. On the other
hand, we exclude features that depend on the number of colors ina palette, i.e., we do not extract colors sorted by lightness, by differences between adjacent colors, and by color differences. Usingsuch features, the total length of the feature vector varies based on
the number of colors in a given palette. This is why O’Donovan’s† A space where hue θ and saturation s are remapped to Cartesian coordinates: d1 = s cos(θ) and d2 = sin(θ).
c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

model can only be applied to a five-color palette. However, our proposed model does not suffer such restrictions.
In addition, we include a color harmony term [OL06]. In addition, a “gradation of lightness” and a “gradation of hue” of colors
in a palette are included in our feature extraction method. These
are based on the fact that as the order of colors in a palette becomesincreasingly linear, human ratings tend to increase. The details offeature extraction are described in the supplemental material. In total, 121 features were extracted and used as for learning and rating.
3.3. Learning Model Weights

where, r(t) is a color palette rating by the learned model,∆E00 (c1 , c2 ) is CIE Delta E of c1 and c2 , and τ is the user-defined
threshold to suggest colors that are substantially different from thecolors in the given palette.
4.2. Color Space Exploration
Since the color space is vast, it is necessary to search color candidates efficiently in order to suggest compatible colors to the user.
It took 12 hours to brute-force search a candidate in the RGB color
space with a Core i7 2.3 GHz, 16 GB RAM machine. Therefore, we
perform the following procedure to sample candidates in the HSVcolor space.

We use LASSO regression [Tib96] in the same way as [OAH11,
LH13] for learning the weights of each feature. The equation for
rating color palette t is given by r(t) = w> f(t) + b, where w is aweight vector, f is a feature vector, and b is a bias term. Then, theweight vector and the bias are leaned with L1 regularization:[0, 360) is the hue of c and Ph1 ,h2 is the probability of co-occurrencemin ∑(w> fi + b − ri )2 + λkwk1 ,(1)of colors with its hues h1 and h2 in the same palette, i.e., Ph1 ,h2 =where ri is an actual user rating (1-5) from the dataset describedin Section 3.1 and λ is a regularization parameter (λ = 2.2 × 10−6by 10-fold cross validation). The dataset is split into learning andtest datasets at a ratio of 6 : 4. Due to the L1 regularization, the extracted features are expected to have adequate weight even if there
are similar features in the extracted feature vector.

bility of co-occurrence of the adjacent hues h1 and h2 in the same
palette, is calculated in the same manner. With this configuration,
we perform rejection sampling using a hue probability distributionfunction to select a candidate hue ccand = (h, s, v) depending on theindex position at which the candidate is to be inserted.

w,b iFirst, we calculate an adjacent hue probability distribution of a
joint
joint joint joint
jointcolor c. Here, pc = [Phc ,0 , Phc ,1 , Phc ,2 , . . . , Phc ,359 ], where hc ∈
joint

joint

4. Suggesting Compatible Colors
Using the proposed model, we can suggest a compatible color for agiven color palette, while in [OAH11], they can only suggest a fifthcolor for a given four-color palette.

# of occurrence of hues h1 and h2 in the same palette. The adjacent hue prob# of palettes in a training datasetadjability distribution of color c denoted by pc , which is the proba-

A case whereby the specified index in a palette is the head or
tail of the palette, we compute the following as the hue probability
distribution function:adj

α · pci + (1 − α) ·1
joint
∑ pck ,
n − 1 c ∈C\{c}Given a color palette with any number of colors, an index position the user would like to insert a color in the palette (e.g., thei-th index from the left of the palette), and optionally the number ofcolors M to be suggested, the proposed method can suggest compatible colors. First, we search a compatible color and add the color
to a given palette. Then, the palette is rated by the rating prediction
model. The top M high-rated colors are suggested to the user. To
avoid suggesting overly similar colors for the given palette to thehuman eye, we calculate CIEDE2000 (∆E00 ) [SWD05] for eachcolor in the palette and the candidate color, and we collect only
candidates with ∆E00 values greater than a threshold τ. Similar tothe τ thresholding, we calculate the ∆E00 for each two candidate
color combinations out of M. Note that we collect only candidates
with ∆E00 values greater than a threshold κ. We explore the color
space with those constraints and select candidate colors by rejection
sampling, which allows us to suggest various colors to the user.
4.1. Compatible Color Suggestion
Given a color palette t = [c1 , c2 , . . . , cn ], we calculate the following equation for a compatible color candidate ccand with the given
palette:maxr(t)subject to∆E00 (ccand , ci ) ≥ τ, ∀i ∈ {1, 2, . . . , n} ,

ccand

(2)

c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

(3)

i

k

where ci is a color adjacent to ccand , n is the number of colors in thegiven palette, ck represents non-adjacent colors to ccand , and α is abalancing parameter of the two terms (α = 0.5 in this study). We
compute Padj with the adjacent color and Pjoint with the remaining
non-adjacent colors. When a candidate is to be inserted between
two colors in a palette, we compute the following hue probability
distribution function:
adj

α·

adj

pci + pc j1
joint
+ (1 − α) ·∑ pck ,
2n − 2 c ∈C\{c
,c }
k

i

(4)

j

where ci and c j are adjacent colors of ccand , and ck represents nonadjacent colors of ccand in the palette. Figure 2 shows the hue probability distribution function calculated for various color palettes
where a candidate is to be inserted at H.
We also sample the candidate’s saturation s with s ∼ N (µs , σs )and the value v with v ∼ N (µv , σv ) from the given color palette,
where N (µ, σ) is a normal probability distribution with mean µ andstandard deviation σ.
In addition, we constrain the color differences of any two color
combinations from M candidates by computing the following constraint, which ensures the diversity of the suggested colors:∆E00 (ccand, ccandjk ) ≥ κ, j 6= k, ∀ j, k ∈ {1, 2, . . . , M} ,(5)

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

color palette and a five-color palette). To do so, we use the following distance function:d(t, t0 ) = EMD∆E00 (t, t0 ),(7)

where EMD∆E00 (t, t0 ) is the Earth Mover’s Distance(EMD) [RTG98], and we employ ∆E00 as the distance metric. The EMD with ∆E00 is defined as follows:
EMD∆E00 (t, t0 ) =n0∑m
i=1 ∑ j=1 f i, j ∆E00 (ti , t j ),n∑mi=1 ∑ j=1 f i, j

(8)where fi, j is a flow from ti to t 0j , and ti and t 0j are the i-th and j-thcolor in each color palette with m and n colors, respectively. The
flow F = [ fi, j ] is computed by solving the following optimization
problem:mminimizen∑ ∑ fi, j ∆E00 (ti , t0j )i=1 j=1

subject tofi, j ≥ 0, 1 ≤ i ≤ m, 1 ≤ j ≤ n,n

∑ fi, j ≤ wt , 1 ≤ i ≤ m,

Figure 2: Hue probability distribution functions for various color
palettes when a color is to be inserted between the index specified
by H.

i

j=1
m

∑ fi, j ≤ wt , 1 ≤ j ≤ m,
0
j

i=1
m

where κ is a user-defined threshold. The proposed sampling method
results in few seconds to several tens of seconds depending on thenumber of samples (a few thousand to 100, 000), and we also findthat the rejection sampling algorithm is approximately 10 times
faster than naive random hue sampling. The pseudo code of therejection sampling algorithm is provided in the supplemental material.

where t is a color palette in the dataset described in Section 3.1 andt0 ∈ T is a color palette in the new dataset composed of a specificcolor to be used for customization. δ is a balancing parameter forthe two terms (δ = 0.6 in this study).
In [CXY∗ 15], they employed a weighted sum of the squared Euclidean distance of all five colors in a HSV space palette to calculate d(t, t0 ). Note that their method can only deal with five-color
palettes; however, since our method is not restricted to five-color
palettes, we must be able to compute the distance of two palettes
with different numbers of colors (e.g., the distance between a three-

j=1

0
j

,

After rerating all palettes in the original dataset, we then combine the additional and original datasets by giving full-rate to all
new color palettes to build a customized dataset for pastel colors
and so on.

6.1. Model Analysis

(6)

i=1

where wt = wt0 = [ 13 , 13 , 13 ] is the weight vector.

6. Results0− min d(t, t )
t0 ∈T
,

!

n
i

We use the general-purpose dataset described in Section 3.1 rather
than a dataset with a specific concept. Therefore, it sometimes does
not give the user a satisfactory rating or color suggestion with themodel if the given palette has specific context, such as pastel colors.
The suggested colors tend to be relatively dark with low saturation.

r0 (t) = δ · r(t) + (1 − δ) · em∑ ∑ fi, j = min ∑ wt , ∑ wt

i=1 j=1

5. Customizing the Model to a Specific Color Tone

To address this issue, we customize the model based on the idea
proposed by Chen et al. [CXY∗ 15], i.e., we rerate the all color
palettes in the dataset using an additional dataset:nHere, we compare our learned model with a previously proposedmodel [OAH11]. The mean absolute errors (MAE) and meansquared errors (MSE) are shown in Table 1. The baseline was computed by the difference between the test and training data calculated by the proposed method [OAH11]. For [OAH11], the MAE
decreased by 33% and MSE decreased by 55% compared to thebaseline. For the proposed method, MAE and MSE decreased by
30% and 51%, respectively. We also performed correlation analysis of ratings for both models (Fig. 3). As can be seen, R2 = 0.56
in [OAH11] and R2 = 0.52 for the proposed method. Although theproposed method is less accurate than [OAH11] in both cases, theMAE and MSE values are reduced compared to the baseline andR2 is greater than 0.5, which indicates that the proposed model canstill rate color palettes with respect to human aesthetic preferenceswith sufficient accuracy. In addition, the proposed model can rate
color palettes with any number of colors, while the model proposedin [OAH11] can only rate a five-color palette. As the learning result with L1 regularization shows, 19 features out of 121 received
zero weights, and the other 102 features received nonzero weights.
The Color Harmony (CH) and Gradation (Grads) terms added toc 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

Table 1: Comparison of MAE and MSE for [OAH11] and the proposed model.
MAE
MSE

Fixed (baseline)
0.267
0.116

[OAH11]
0.180
0.052

Proposed model
0.187
0.057

(a) Gradation (dark → light)(b) Gradation (light → dark)(c) Color palette composed of various color hues(d) Palette expansion from 3-color to 7-color(b) Ours.

(a) [OAH11]

Figure 3: Comparison of correlation coefficient R of human ratingsfor [OAH11] (a) and the proposed model (b).

Table 2: Comparison of correlation coefficient of predicted ratingby the proposed model and human ratings. The R value was calculated with and without the CH and Grads terms.
R value[OAH11]
0.76

w/o
0.71

w/ CH
0.71

w/ Grads
0.72

w/ both0.72

the feature extraction method receive nonzero weights, which contributes to increased correlation. We further analyzed these featuresand found that, although the CH term received nonzero weights, itdoes not sufficiently contribute to correlation. On the other hand,the correlation coefficient R increases by 0.01 due to the Grads term
(Table 2). A more detailed analysis is provided in the supplemental
material.
6.2. Analysis of Compatible Color Suggestion
Figure 4 shows the results of the color suggestions for the given
palettes. The proposed method can suggest appropriate colors forthe gradation color palettes (Figs. 4a and 4b) and can suggest wellchosen colors for a palette that includes various hues (Figs. 4c and4d). Using the proposed color suggestion method, the input threecolor palettes are expanded to four-, five-, or seven-color palettes
while retaining color harmony.
We conducted an experiment to evaluate the performance of thepalette expansion algorithm. For 10 three-color palettes and 10 fivecolor palettes, we applied our compatible color suggestion method(Ours(Best)) to expand these palettes to four-color and six-color
palettes, respectively. For comparison, we generated palettes using
the incompatible color suggestion method (Ours(Worst)), in which
we sampled the lowest rated hue by our model in HSV color space,
and the saturation and value were sampled in the manner describedin Section 4.2. We also generated palettes with additional colors
sampled randomly in RGB color space (Random). For 20 cases,
we asked 17 participants (four females) with normal color vision
to rank the three palettes generated by the different methods relative to Naturalness (“expansion with additional color is natural”)
c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

Figure 4: Palette expansion results of the proposed method. Rows
1-3: a suggested color is added at the index specified by H. Row
4: two types of palettes are expanded from three-color to sevencolor palettes by adding suggested colors side-by-side. τ = 5 for
all results.
and Compatibility (“the expanded palette has compatibility to theoriginal palette”). In the experiment, each participant was shown
an original three-color palette at the top of the display, and three
expanded palettes with four colors were displayed at the bottom,
side-by-side in a random order.
We applied Friedman’s test and if a significant difference (p <0.05) was observed, we proceeded to perform Wilcoxon tests with
Holm corrections for multiple comparison. The results showed that,for 12 of the 20 cases, the palettes expanded by our method were
statistically significant for both the Ours(Worst) and Random methods. In addition, even in cases where our method had no statisticalsignificance, Fig. 5 shows that the palettes generated by our method
were most frequently chosen by participants as the best palette,
which demonstrates the effectiveness of the proposed method.
6.3. Palette Index for Variations of Color Suggestions
As shown in Fig. 6, the suggested colors can vary depending on
where the color is to be inserted because the rating model considers
the order of colors in a palette. One may consider a color palette as
simply a set of colors that is independent of the order of the colors
in the palette, however, we must consider color order because suchsets of colors should be arranged to form a single group. Therefore, we consider that the order is important for us to evaluate theaesthetics of a color palette.
We also apply a palette re-ordering to a palette t, where weselect the highest rated palette from all possible permutations;t = arg maxr∈P(t) r(t), where P(t) denotes the 7! = 5, 040 possible
permutations of a palette in the results shown in Table 3.
In addition, we conduct a user study to analyze user preferencesof a color order of a palette. In the study, we show a participanta pair of color palettes (an original and the re-ordered) listed inTable 3 except for #6 since it is not re-ordered. We shuffle the leftor right side of the two palettes randomly and ask “Which color
palette is more harmonius?” to 48 participants (36 males and 12
females), 27 of them major in design. The age of the participants
lies in 20 – 25. The result is shown in Fig. 7. According to theN. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

count

20
15

17

15

16

10

15

12

6

5

1

0

1

0

20
15

15

10

5

1

0

1

1

0

0

10

7
2

0

0

2

17
11

5

1

0

17

15

15

0

5

5

11
4

0

2

10
2

13
2

0

1

0

4

13

10
0

0

3

method

14

7

4

10
3

3

4
0

Ours(Best)
Ours(Worst)

3

2

2

RandomFigure 5: Number of times that palettes generated by three different methods were chosen by participants as the best compatible palette.
(Top) Color suggestions for three-color palettes to be expanded to four-color palettes. (Bottom) Color suggestions for five-color palettes tobe expanded to six-color palettes. We expanded the palettes listed at the 8th and 10th columns with pastel and retro customized models,
respectively.

(a) τ = 5

Figure 6: The suggested colors (H) are varied depending on theindex of the palette (τ = 5).

(b) τ = 20
Table 3: Comparison of ratings for original and re-orderedpalettes.
original palette
#1
#2
#3
#4
#5
#6

re-ordered palette
3.41
3.49
3.45
3.37
3.21
3.52

3.40
3.43
3.41
3.37
3.09
3.52

#1

#2

#3

#4

79%

frequency

75

60%

50

58%
42%

40%

31%
25

#5

75%

69%

21%

25%

ordering
Original
Re−ordered0

Figure 7: Preference analysis from a user study. For each palette,the participants selected either an original or the re-orderedpalette. The palettes used in the study were taken from #1 – #5
shown in Table 3.

histogram, the re-ordering approach is effective for improving color
palette evaluation of human aesthetics preferences.

Figure 8: τ affects the color suggestion results. With a small τvalue, the results can be similar to the colors already in the given
palette. With a greater τ value, perceptually different but compatible colors can be suggested. For a three-color palette with different
τ values ((a) τ = 5 and (b) τ = 20), the suggested colors are marked
as H.

in Fig. 8a. When the user desires a variety of colors that differ sufficiently from the colors in the original palette, the proposed color
suggestion method can suggest various colors that are sufficient toperceive the differences for the human visual system by setting τ
to a larger value. As can be seen in Fig. 8a, τ = 5 does not give
sufficient differences because the value is close to a just-noticeable
difference (the value around 2.3 [MVEO94]). With a larger value,e.g., τ = 20, in Fig. 8b, the proposed method can suggest a wide
variety of colors.
6.5. Choice of κ.
As shown in Fig. 9a, the color candidates tend to be similar to thecolors in the given palette if κ in Eq. 5 is small. To avoid this,greater κ will suggest colors with more variety (Fig. 9b) while thepalette maintains color harmony.
6.6. Analysis of Model Customization

6.4. Choice of τ.
Figure 8 shows the results of choosing different τ values in Eq. 2.
Although Fig. 6 does give color suggestion results with τ = 5 andsufficient hue variations while maintaining overall color compatibility, it does not always suggest sufficient hue variation, as shown

For the model customization, we collected 1, 403 pastel color
palettes from Pastel_Lªvers [COL16d] and 3, 176 retro color
palettes from the Retro group [COL16e], and we customized themodel with these additional datasets by applying the method described in Section 5.
c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

Table 5: Comparison of ratings for various color palettes with andwithout model customization.
color palette(a) τ = 5, κ = 5(b) τ = 20, κ = 20

Figure 9: κ affects the color suggestion results. Given a four-color
palette with different τ and κ, the suggested colors are inserted atboth ends (the H columns). Top to bottom is higher to lower ratings.
N = 4, M = 5.

Table 4: Comparison of color suggestions with and without themodel customized with pastel colors. The suggested colors arelisted in the columns marked H, and the rating is shown next toit. N = 3, M = 5, τ = 5, and κ = 20.
w/o custom model

model customization
w/o pastel
retro

(a)

2.90

4.24

3.50

(b)

3.18

3.73

2.75

(c)

2.95

3.20

3.12

(d)

3.01

3.16

2.85

(e)

3.21

3.02

3.23

(f)

2.76

2.45

2.94

(g)

2.95

2.24

3.07

(h)

2.84

2.63

4.02

(i)

2.65

1.57

2.66

w/ pastel model

#1

3.16

4.47

#2

3.14

4.35

#3

3.10

4.35

#4

3.07

4.23

#5

3.07

4.18

Table 4 shows a comparison of the color suggestion results withand without the pastel-customized models. As can be seen, the customized model can suggest more compatible colors than the model
without customization.
Table 5 shows a comparison of the ratings results with and without the pastel and retro customizations. We rated various threecolor palettes using these models. As can be seen for the pastelcustomized model, the ratings are higher than those obtained without the customized model for the palette shown by (a) in Table 5.
In addition, the pastel model gives higher ratings to gradation
palettes, as shown in Table 5b and c, while the palette shown by (e)– (i) in Table 5 obtained lower ratings. On the other hand, the Retromodel rated the palette shown in Table 5h higher, and the pastel
color palette shown in Table 5a was also rated higher, which suggests that the retro model gives higher ratings to palettes comprising colors where saturation values are very similar. We also foundthat the pastel model rates were lower for Table 5i, while the retro
model rates it with nealy the same rating as the non-customized
model.
Furthermore, we compared our pastel and retro customized
model to our original and O’Donovan’s model [OAH11]. For thecomparison, we collected 664 pastel color palettes and 1, 222 retro
color palettes from P a s t e l C u t i e s [COL16c] and RETROpolis [COL16f], respectively. Then, we rated the pastel dataset by
our pastel-customized and original model, and O’Donovan’s model(Fig.10a), and the rated retro dataset by our retro-customized andoriginal model, and O’Donovan’s model (Fig.10b). These meansc 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

(a) Pastel dataset(b) Retro dataset

Figure 10: Histograms of ratings of pastel dataset (a) and retro
dataset (b) of our pastel-customized and retro-customized and original model, and O’Donovan’s model [OAH11].

Table 6: Mean and StdDev for Fig. 10.
Dataset
PastelRetro

Mean
StdDev
Mean
StdDev[OAH11]
3.11
0.197
3.06
0.207

Ours (Original)
3.13
0.166
2.95
0.198

Ours (Custom)
3.24
0.719
3.15
0.452

and standard deviations (StdDev) are listed in Table 6. They showed
that, the customized model obtained higher mean ratings in bothcases, and larger StdDevs are considered to these abilities to rate
the specific colors in detail, while O’Donovan’s and our originalmodel can not capture the differences of the specific colors wellsince these models have smaller StdDevs. Although the originalmodel has a less correlation to human ratings in Table 1 and Fig. 3,
the customized version of the proposed models compare favorably
to O’Donovan’s model [OAH11].
6.7. Applications.
Coloring 2D Patterns. We applied the proposed color suggestion
method to a 2D pattern template from COLOURlovers (Fig. 13).
First, we selected a three-color palette with color images (e.g.,
mysterious, noble, dreamy, and progressive) [Kob91]. Then, thethree-color palette was expanded to a five-color palette using the

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

Figure 11: Robot color design. Left to right: segmented model,
model colored by a three-color palette, model colored by a fivecolor palette, and model colored by a seven-color palette while retaining the color image.
proposed color suggestion method. We performed pattern coloringwith three-color (Fig. 13, left) and five-color palette (Fig. 13, right).
As can be seen in the five-color palette and its pattern coloring results, the color image does not change from that of the original
three-color palette. Since the pattern coloring with the five-color
palette has 5! = 120 possible pattern coloring combinations, weemployed the color assignment method proposed in [KYKL14] forcoloring and selected the top-ranked results in Fig. 13.
Coloring segmented 3D model. We also applied the method to
3D model coloring. Given a segmented model, we group these segments into three to seven group, and color them with three-color,
five-color, or seven-color palettes (Fig. 11). It can be noted that suchcolor variations are useful, for example, a user already has a basic
color concept (e.g., three basic colors) and would like to refine thecolor design while retaining the color image.
Photo recoloring. A user can produce enhanced photo-editing
results by incorporating a palette-based photo recoloring
method [CFL∗ 15] with our palette expansion method. First,
we prepare a three-color palette with a target color image. Sucha palette with a certain color image can be obtained from theInternet (e.g., by searching for Pastel or Retro palettes) or from abook [Kob91]. Second, we apply the palette expansion method tothe palette to obtain additional compatible colors. We also preparea target photo to be recolored and apply the palette-based photo
recoloring method to extract a source palette from the photo. Then,each color in the source palette is replaced with our palette colors
using the color transfer function to recolor the photo. Figure 12
and Fig. 17 show the results. Using our palette expansion method,a user can obtain more colors in the palette while retaining its color
image when the colors of the collected palette are limited, which
provide more flexibility.

Figure 12: Recoloring results with Mysterious palettes. The original photo (left) is recolored using the palette-based photo recoloring method with three- (middle) and seven-color palettes (right).
Photo courtesy of Tommie Hansen (Flickr).

(a) Color image: Mysterious(b) Color image: Noble(c) Color image: Dreamy (pastel model)(d) Color image: Progressive (retro model)
Figure 13: Coloring pattern templates. The original three-color
palette color image (left) was expanded to a five-color palette using
the colors suggested by the proposed method while retaining its
color image.
Adding items. A user can add
items with different colors to a
set according to the color suggestions. For example, for a setof pens with different colors(Fig. 14 (left)), the additional
Figure 14: Pen set
pens with different yet compatible colors are added to the set(Fig. 14 (right)). This is useful when providing color variations.
Another example is provided in Fig. 15, in which additional objects (e.g., a blind and a sofa) with colors that are compatible withthe room’s color theme are added to the room. Here, we assign thecolor to a diffuse color of the object.
7. Conclusions and Limitations
Conclusions. In this paper, we have proposed a rating prediction
model for a given color palette that can comprise any number
c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes
A.: Palette-based photo recoloring. ACM Trans. Graph. 34, 4 (July
2015), 139:1–139:11. URL: http://doi.acm.org/10.1145/
2766978, doi:10.1145/2766978. 2, 8
[Col16a] C OLOR A.:, Accessed in Jun 2016. URL: https://color.
adobe.com/. 1[COL16b] COLOUR LOVERS:, Accessed in Jun 2016. URL: http://
www.colourlovers.com/. 1

Figure 15: For a bedroom with an associated color theme (left),a window blind and a sofa with colors assigned according to theextended palette are added to the room (right). Model courtesy of 3D
Bar.

[COL16c] COLOUR LOVERS: P a s t e l c u t i e s, Accessed in Jun
2016. URL: http://www.colourlovers.com/group/P_a_s_t_e_l_C_u_t_i_e_s. 7[COL16d] COLOUR LOVERS: Pastel_lªvers, Accessed in Jun 2016.
URL: http://www.colourlovers.com/group/Pastel_L%
E2%99%A5vers. 6[COL16e] COLOUR LOVERS: Retro, Accessed in Jun 2016.
http://www.colourlovers.com/group/Retro. 6

URL:[COL16f] COLOUR LOVERS: Retropolis, Accessed in Jun 2016. URL:
http://www.colourlovers.com/group/RETROpolis. 7(a) Template(b) Merry(c) Noble

Figure 16: Coloring pattern templates using two color palettes
with different color images.

of colors. We have also proposed a compatible color suggestion
method for palette expansion while retaining color harmony, as
well as a method to explore color space to select color candidates.
With the proposed model, we can evaluate simple palettes, such as
gradations, and we can rate palettes comprising various color tones
and hues.
Limitations. Since the proposed method employs a machinelearning approach, the rating prediction of the model depends
strongly on the quality of the dataset. We specialized our modelby collecting pastel and retro palettes from COLOURlovers withrelatively little effort. However, if only a few datasets are available
or it is difficult to collect datasets from the Internet, it is difficult
to customize the model. It is also difficult to determine how many
palettes need to be customized for sufficient rating accuracy. In addition, we have demonstrated coloring pattern templates as an application; however, the color harmony of a palette is not directly
related to the target pattern mood, which can result in mismatchedcoloring. Figure 16 shows an example, in which we prefer Fig. 16cover Fig. 16b relative to the mood of the target content in the pattern
template (Fig. 16a).
References[BC69] B IRREN F., C LELAND T.: A grammar of color: a basic treatiseon the color system of Albert H. Munsell. Van Nostrand Reinhold Co.,
1969. 2[BPCD11] B YCHKOVSKY V., PARIS S., C HAN E., D URAND F.: Learning photographic global tonal adjustment with a database of input/output image pairs. In Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition (Washington, DC,
USA, 2011), CVPR ’11, IEEE Computer Society, pp. 97–104. URL:
http://dx.doi.org/10.1109/CVPR.2011.5995413, doi:
10.1109/CVPR.2011.5995413. 10[CFL∗ 15]C HANG H., F RIED O., L IU Y., D I V ERDI S., F INKELSTEIN

c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.

[COSG∗ 06] C OHEN -O R D., S ORKINE O., G AL R., L EYVAND T., X U
Y.-Q.: Color harmonization. ACM Trans. Graph. 25, 3 (July 2006),
624–630. URL: http://doi.acm.org/10.1145/1141911.
1141933, doi:10.1145/1141911.1141933. 2[CXY∗ 15] C HEN K., X U K., Y U Y., WANG T.-Y., H U S.-M.: Magic
decorator: Automatic material suggestion for indoor digital scenes. ACM
Trans. Graph. 34, 6 (Oct. 2015), 232:1–232:11. URL: http://
doi.acm.org/10.1145/2816795.2818096, doi:10.1145/
2816795.2818096. 2, 4[Goe71] G OETHE J. V. W.: Goethe’s Color Theory. Van Nostrand Reinhold Co., 1971. 2[Itt74] I TTEN J.: The Art of Color: The Subjective Experience and Objective Rationale of Color. A VNR book. Wiley, 1974. 2[Kob91] KOBAYASHI S.: Color Image Scale. Kodansha International,
1991. 1, 7, 8
[KYKL14] K IM H.-R., YOO M.-J., K ANG H., L EE I.-K.: Perceptuallybased color assignment. Computer Graphics Forum 33, 7 (2014), 309–
318. URL: http://dx.doi.org/10.1111/cgf.12499, doi:
10.1111/cgf.12499. 2, 8
[LH13] L IN S., H ANRAHAN P.: Modeling how people extract color
themes from images. In Proceedings of the SIGCHI Conferenceon Human Factors in Computing Systems (New York, NY, USA,2013), CHI ’13, ACM, pp. 3101–3110. URL: http://doi.acm.
org/10.1145/2470654.2466424, doi:10.1145/2470654.
2466424. 2, 3
[LRFH13] L IN S., R ITCHIE D., F ISHER M., H ANRAHAN P.: Probabilistic color-by-numbers: Suggesting pattern colorizations using
factor graphs.
ACM Trans. Graph. 32, 4 (July 2013), 37:1–
37:12.
URL: http://doi.acm.org/10.1145/2461912.
2461988, doi:10.1145/2461912.2461988. 2[LZNH15] L I X., Z HAO H., N IE G., H UANG H.: Image recoloring using
geodesic distance based color harmonization. Computational Visual Media 1, 2 (2015), 143–155. URL: http://dx.doi.org/10.1007/
s41095-015-0013-5, doi:10.1007/s41095-015-0013-5.
2[MS44] M OON P., S PENCER D. E.:
Geometric formulation ofclassical color harmony∗.
J. Opt. Soc. Am. 34, 1 (Jan 1944),
46–59. URL: http://www.osapublishing.org/abstract.
cfm?URI=josa-34-1-46, doi:10.1364/JOSA.34.000046.
2[MVEO94] M AHY M., VAN E YCKEN L., O OSTERLINCK A.: Evaluation of uniform color spaces developed after the adoption of cielab
and cieluv. Color Research & Application 19, 2 (1994), 105–121.
URL: http://dx.doi.org/10.1111/j.1520-6378.1994.
tb00070.x, doi:10.1111/j.1520-6378.1994.tb00070.
x. 6

N. Kita & K. Miyata / Aesthetic Rating and Color Suggestion for Color Palettes

Figure 17: (Top) Recoloring results with Dreamy palettes. (Middle) Recoloring results with Noble palettes. (Bottom) Recoloring resultswith Retro palettes. The first column shows the original images. Photos courtesy of Celine Nadeau (top) and Ana Kuhnen (bottom) (Flickr), and the
MIT-Adobe FiveK Dataset [BPCD11] (middle).

[OAH11] O’D ONOVAN P., AGARWALA A., H ERTZMANN A.: Color
compatibility from large datasets. ACM Trans. Graph. 30, 4 (July 2011),
63:1–63:12. URL: http://doi.acm.org/10.1145/2010324.
1964958, doi:10.1145/2010324.1964958. 2, 3, 4, 5, 7
[OB69] O STWALD W., B IRREN F.: The color primer: a basic treatise on
the color system of Wilhelm Ostwald. Van Nostrand Reinhold Co., 1969.
2[OCLM11] O U L.-C., C HONG P., L UO M. R., M INCHEW C.: Additivity of colour harmony. Color Research & Application 36, 5 (2011),
355–372. URL: http://dx.doi.org/10.1002/col.20624,
doi:10.1002/col.20624. 2[OL06] O U L.-C., L UO M. R.: A colour harmony model for twocolour combinations. Color Research & Application 31, 3 (2006), 191–
204. URL: http://dx.doi.org/10.1002/col.20208, doi:
10.1002/col.20208. 2, 3
[ORLS∗ 12] O U L.-C., RONNIER L UO M., S UN P.-L., H U N.-C.,C HEN H.-S., G UAN S.-S., W OODCOCK A., C AIVANO J. L., H UER TAS R., T REM Ã L’AU A., B ILLGER M., I ZADAN H., R ICHTER K.:
A cross-cultural comparison of colour emotion for two-colour combinations. Color Research & Application 37, 1 (2012), 23–43. URL:
http://dx.doi.org/10.1002/col.20648, doi:10.1002/
col.20648. 2[RTG98] RUBNER Y., T OMASI C., G UIBAS L. J.: A metric for distributions with applications to image databases. In Computer Vision,
1998. Sixth International Conference on (Jan 1998), pp. 59–66. doi:
10.1109/ICCV.1998.710701. 4[SBS10] S ZABÓ F., B ODROGI P., S CHANDA J.: Experimental modelingof colour harmony. Color Research & Application 35, 1 (2010), 34–
49. URL: http://dx.doi.org/10.1002/col.20558, doi:
10.1002/col.20558. 2[SM08] S AWANT N., M ITRA N. J.: Color harmonization for videos.
In Proceedings of the 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing (Washington, DC, USA, 2008),
ICVGIP ’08, IEEE Computer Society, pp. 576–582. URL: http://dx.doi.org/10.1109/ICVGIP.2008.17, doi:10.1109/
ICVGIP.2008.17. 2[SWD05] S HARMA G., W U W., DALAL E. N.: The ciede2000 colordifference formula: Implementation notes, supplementary test data,
and mathematical observations. Color Research & Application 30,
1 (2005), 21–30. URL: http://dx.doi.org/10.1002/col.
20070, doi:10.1002/col.20070. 3[Tib96] T IBSHIRANI R.: Regression shrinkage and selection via thelasso. Journal of the Royal Statistical Society (Series B) 58 (1996), 267–
288. 3[WYW∗ 10] WANG B., Y U Y., W ONG T.-T., C HEN C., X U Y.Q.: Data-driven image color theme enhancement. ACM Trans.
Graph. 29, 6 (Dec. 2010), 146:1–146:10. URL: http://doi.acm.
org/10.1145/1882261.1866172, doi:10.1145/1882261.
1866172. 2[YYTC12] Y U L.-F., Y EUNG S.-K., T ERZOPOULOS D., C HAN T. F.:
Dressup!: Outfit synthesis through automatic optimization. ACM Trans.
Graph. 31, 6 (Nov. 2012), 134:1–134:14. URL: http://doi.acm.
org/10.1145/2366145.2366153, doi:10.1145/2366145.
2366153. 2[ZLHM11] Z HANG S. H., L I X. Y., H U S. M., M ARTIN R. R.: Online
video stream abstraction and stylization. IEEE Transactions on Multimedia 13, 6 (Dec 2011), 1286–1294. doi:10.1109/TMM.2011.
2165052. 2

c 2016 The Author(s)Computer Graphics Forum c 2016 The Eurographics Association and John Wiley & Sons Ltd.