Generative Colorization of Structured Mobile Web Pages
Kotaro Kikuchi1

Naoto Inoue1
1

Mayu Otani1

CyberAgent

Abstract
Color is a critical design factor for web pages, affecting
important factors such as viewer emotions and the overall
trust and satisfaction of a website. Effective coloring requires design knowledge and expertise, but if this process
could be automated through data-driven modeling, efficient
exploration and alternative workflows would be possible.
However, this direction remains underexplored due to thelack of a formalization of the web page colorization problem, datasets, and evaluation protocols. In this work, we
propose a new dataset consisting of e-commerce mobile web
pages in a tractable format, which are created by simplifying the pages and extracting canonical color styles with acommon web browser. The web page colorization problem
is then formalized as a task of estimating plausible color
styles for a given web page content with a given hierarchicalstructure of the elements. We present several Transformerbased methods that are adapted to this task by prepending
structural message passing to capture hierarchical relationships between elements. Experimental results, including aquantitative evaluation designed for this task, demonstrate
the advantages of our methods over statistical and image
colorization methods. The code is available at https://github.com/CyberAgentAILab/webcolor.

1. Introduction
Color plays an important role in the visual communication of web pages. It is known that colors are associatedwith certain emotions [15] and have a significant impact on
the trust and satisfaction of websites [7]. Effectively coloring web pages to achieve the design goals is a difficult
task, as it requires understanding the theories and heuristics
about colors and their combinations [27]. Another reason
for the difficulty lies in the implicit and practical requirements within a page, such as overall balance, contrast, anddifferentiation of color connotations. Our aim is to overcome these difficulties with data-driven modeling to facilitate new applications such as efficient design exploration
for designers, automated design for non-designers, and automated creation of landing pages for advertised products.

Edgar Simo-Serra2
2

Kota Yamaguchi1

Waseda University

Despite its great industrial potential, the lack of established benchmarks and the need for extensive domain
knowledge for data collection make it difficult to apply datadriven methodologies to web page coloring, which may explain why there are fewer related studies in the literature.
Existing methods for coloring web pages either generatecolor styles for some specific elements [30, 25] or require
already designed web pages [29, 11], and no method has
been investigated that can add plausible color styles to allelements from scratch. Also, the datasets used in thesestudies are not publicly available, making follow-up studies difficult. The image colorization techniques [1, 19] canbe applied to web page, however, they require an input image and the extraction of structural color information fromthe output image, which narrows the applicable scenarios
and would cause extraction errors that degrade quality.
In this work, we construct a new dataset to take the studyof web page coloring one step further. The challenges indata collection are that raw web pages have many redundantelements that do not affect their appearance, and the color
styles defined in their styling sheets are ambiguous as thesame color can be represented differently. To address thesechallenges, we use a common web browser to simplify web
pages and retrieve the browser’s resolved styles as canonicalcolor styles.
The coloring problem is then formalized as a task ofgenerating plausible color styles for web pages provided
with content information and hierarchical structure of theelements. We present several Transformer-based methods
adapted to this task. The hierarchical relationship between
elements affects the color style, since child elements are basically rendered on top of their parent elements. Our hierarchical modeling to capture these characteristics is achievedby contextualizing the features with message passing [8] before feeding them into the Transformer network.
To evaluate the generated color styles in terms of qualityand diversity, we adapt the metric used in image generation
tasks [12] to ours and test whether the resulting pages satisfy the standard contrast criterion. Quantitative and qualitative results demonstrate the advantages of our methods
over statistical and image colorization methods [19].
Our contributions are summarized as follows:• A dataset of e-commerce mobile web pages tailored
for automatic coloring of web pages.
• A problem formulation of the web page coloring andTransformer [28]-based methods that capture hierarchical relationships between elements through structural message passing.
• Extensive evaluation including comparison with astate-of-the-art image colorization method [19].

2. Related Work2.1. Image Recoloring and Colorization
The problem of automatically determining color has
been extensively addressed, especially for images. There
are two common tasks: recoloring, which applies different colors to a color image, and colorization, which applies
colors to a grayscale image. For both tasks, the referencebased approach has been investigated, which is achieved
by finding a mapping between the color or luminance ofthe source and the color of a given reference image or
palette [5, 24, 4, 17]. Volkova et al. [29] adapted this approach for recoloring a web page with a reference image.
The reference-based approach requires additional user
input, whereas the approach that uses color knowledge
learned from data does not. Earlier work in this direction used relatively simple models to learn cost functions,such as color compatibility, and then colorize images or
2D patterns through optimization [22, 21]. Recently, deeplearning-based methods have been actively studied, especially for the image colorization task [1], and ColTran [19]that employs the Transformer network [28] as its basemodel has demonstrated state-of-the-art performance.
There are two concerns in applying image colorization
techniques to web pages. The first one is that the input is
an image, which narrows the applicable scenarios, and thesecond is that the quality could be degraded by errors in extracting the colors from the output image to be used for the
web page. In our problem formulation, these concerns do
not arise because we treat web page coloring as an estimation problem on structural data.

2.2. Data-driven Applications for Web Pages
Research on data-driven applications for web pages has
been actively studied in recent years in different communities, including question answering [6], semantic labeling ofelements [14], information retrieval [23], code generationfrom a screenshot [3], layout generation [16], and GUI photomontage [31]. An earlier work on data-driven web pagestyling is Webzeitgeist [20], a platform that provides several
data and demographics for over 100,000 web pages, but is
no longer publicly available.
Relatively few studies have been done for the automaticcoloring. Gu et al. [11] propose an optimization-based col-Figure 1: Screenshots of randomly selected web pages.

Style<body>

Text color:

Background color:rgba(255,255,255,1.0)
rgba(0,123,255,1.0)<header><div>

Content
HTML tag: button

<img>

<h1>

<button><p>

Image: null

Order: 3

Renderedelement

Text: “Submit”

Figure 2: Data format of a web page in this study.

oring framework by combining several models, including acolor contrast model learned using 500 web pages, and confirm its effectiveness in two tasks: coloring with a prepared
palette and fine-tuning a color design. Zhao et al. [30] address the task of estimating font attributes of an element,
including font color, from the surrounding context. Most
recently, Qiu et al. [25] propose a Transformer-based color
recommendation model for landing pages that is trainedwith the masked prediction and the advertisement performance prediction. Their model uses multiple color palettes
extracted from specific regions of annotated screenshots andcan recommend other colors for the target region.
In summary, there is no established method for adding
plausible color styles to all elements from scratch. There isalso no dataset available to study such a method, therefore,
we begin with the construction of a dataset on the structural
coloring of web pages.

3. Dataset Construction for Web Page Coloring
We first describe a generic preprocessing technique thatconverts web pages into a tractable data format for machinelearning applications. We then describe the details of thedataset we have constructed. Screenshots of randomly selected web pages from our dataset are shown in Fig. 1.

3.1. Data Format and Preprocessing
We represent a web page as an ordered tree, where eachvertex is an element to be rendered on the page and has con-tent and color style information (Fig. 2). We use HTML tagsof elements, their order in siblings in the tree, and low-levelfeatures of images (e.g., average color) and text (e.g., word
counts) as content information, please refer to the supplemental material for details. For color styles, it is not trivial to define what to extract. Also, raw web pages containmany redundant elements for training the coloring model.
We discuss below our color style definition and simplification technique.

change the CSS to a different one that uses absolute selectors such as “#element-1” and is carefully tailored to beequivalent to the original style specification. As a result,the average number of elements, excluding those placed in
the head element, was significantly reduced from 1656.1 to61.4. Note that color styles applied to a simplified web pagecan be applied to the original one if the correspondence ofthe elements is recorded before and after the simplification.

3.1.1

The dataset should have a large enough and diverse set of
web pages. We select the Klarna Product Page dataset [14]as the source of web pages, which provides mobile web
pages for e-commerce in various countries and companies.
Specifically, we use snapshots of the web pages provided in
MHTML format, which can be replayed in a browser almost
identically to the original pages. Although the procedures
described in this section apply to the snapshots, they arealso applicable to web pages crawled in a general way.
We implement the process of the retrieval of the computed values and the simplification of web pages using Selenium WebDriver [26] and Google Chrome [9]. Complex
web pages that, even after simplification, still have morethan 200 elements or a tree depth of more than 30 are excluded. We also exclude web pages encoded in anythingother than utf-8 to simplify text processing. As a result, a
total of 44,048 web pages are left with an average number of
elements of 60.6 and an average tree depth of 9.3. The web
pages of the official training split is divided 9:1 for training
and validation, and those of the official test split for testing, resulting in 27,630 pages for training, 3,190 pages forvalidation, and 13,228 pages for testing.

Color Style Definition

Style information for a web page is defined in Cascading
Style Sheets (CSS), which basically consists of the specifications of target elements (i.e., selector) and styles to be
applied (i.e., properties and their values). Here we consider
two representative properties color and background-color
as the color style, corresponding to the text and backgroundcolors of the element, respectively. There are two possible
choices for where to obtain the values of these properties:
specified or computed values.
The specified values are the values defined in the style
sheets. For elements where values are not specified,browser’s default CSS is applied. Available formats forthe values of color properties include RGBA values, color
keywords, and special keywords, such as “inherit”, whichcauses the use of the same value as the parent element.
Thus, the same color in appearance can be represented indifferent formats, and canonicalizing them requires extra
effort. The computed values, on the other hand, are the values resolved by browser. For our target properties, the values are disambiguated to RGBA format, which can be usedas the specified values without any change in appearance.
We use the computed values as the canonical values for thetarget properties, and we additionally use the ground-truth
specified values for the other properties for visualization.
3.1.2

Web Page Simplification

Raw web pages contain many elements that do not contribute to their appearance on the present screen, including
alternative elements such as elements that only appear on
laptop-sized screens and functional elements such as hidden input elements in submission forms. Many redundantor less relevant elements should be eliminated, as they will
negatively affect the learning of the coloring model.
Here we consider keeping only those elements that contribute to the first view on the mobile screen. A naive simplification method would be to try removing an element andif the appearance of the first view does not change, then really remove that element. Removing elements, however,often breaks the original style when other elements are involved in the CSS selector such as “div > a”. To avoid
such undesirable style corruption, before simplification, we

3.2. Our Dataset

4. Approach
We provide high-level explanations of our approach forthe sake of clarity, and details on message passing, modelarchitecture, and implementation are presented in the supplemental material.

4.1. Overview
We define web page colorization as the task of generatinga color style for each element, given the content informationand the hierarchical structure of the elements. We denotethe index of an element as n, the number of elements on
the page as N , the set of color style as Y = {Yn }N
n=1 , the,andthehierset of content information as C = {Cn }N
n=1
archical structure as T , respectively. Thus, the main objective of this task can be formulated as creating a function fthat generates color styles based on given conditions, i.e.,f : (C, T ) 7→ Ŷ, where a variable with the hat symbol, i.e.
Ŷ, represents the estimate of that variable, i.e., Y.
Following the recent image colorization method [19],we take a two-stage approach using a core model toContent and Style Processing (§4.2)

Content

Core Generation Models (§4.3)

Discrete

Generated

style

style

Transformer Encoder

Hierarchical Message Passing

Transformer Decoder<BOS>

Transformer Encoder

Non-autoregressive

Autoregressive model

model

+

Bottom-up

Transformer Encoder

Top-down

Tree-independent content embedding

Style embedding

Content embedding (tree-dependent)

Hidden vector

Intermediate vector representing styleSpecial embedding

Transformer Decoder

Latent vector

Conditional Variational Autoencoder

Figure 3: Key components of our method for generating discrete color styles. Our method has three variants with different
core generation models, all using the same architecture of the content and style encoders and the style estimation head.
Element-wise content features are contextualized by features from other elements through hierarchical message passing.

generate low-resolution discrete styles X = {Xn }N
n=1 ,i.e., g : (C, T ) 7→ X̂ , and a color upsampler to increase the resolution of the colors to the desired size,i.e., h : (X , C, T ) 7→ Ŷ. For quantization, we divide the RGBA values into eight bins per channel, andthe RGB channels are grouped together. The discrete
text color of an element is then represented
 by a pairof xrgb ∈ {1, . . . , 83 }, xalpha ∈ {1, . . . , 8} . The same
quantization is applied to the background color. We consider three core generation models: autoregressive model,
non-autoregressive model, and conditional variational autoencoder, which are all based on Transformer networks
and have the same architecture for the content and style encoders and the style estimation head. The core model andthe color upsampler are trained independently. Some key
components of our method are shown in Fig. 3.

4.2. Content and Style Processing
The content encoder takes content information for eachelement {Cn }N
n=1 (the index n is omitted hereafter) as input, and referring to the tree structure T , converts theminto d-dimensional content embeddings {hC } that reflect
the hierarchical relationship of the elements. We implement
the content encoder using bottom-up and top-down message
passing with residual connections, which is expressed as:h̄C = MaxPool ({Embc (c) |c ∈ C}) ,(1){hup } = MPup ({h̄C }, hleaf ; T ),(2)

{hdown } = MPdown ({hup }, hroot ; T ),(3)

hC = h̄C + hdown ,(4)where MaxPool(·) is the max-pooling operator, Embc (·)
is the embedding function corresponding to the content c,hleaf and hroot are learnable parameters, respectively. In

the bottom-up message passing MPup , the value hup of
an element is computed using the value h̄C of the element
and the values hup of the elements children or hleaf if theelement is a leaf node. The top-down message passing
MPdown is defined similarly in the reverse direction.
The style encoder maps a discrete color style X to astyle embedding hX ∈ Rd independently for each element.
Specifically, we represent an RGBA color by a vector obtained by merging two embeddings corresponding to discrete RGB and alpha values. The style embedding is obtained by merging two color vectors corresponding to text
and background colors. For elements without text, we use a
special learnable embedding instead of the text color vector.
The style estimation head is the module in the final output
part that maps intermediate vectors representing elementby-element styles to discrete RGB and alpha probabilities
for text and background colors, respectively.

4.3. Core Generation Models
The role of the core generation model is to generate a discrete color style for each element {X̂} based on the given
conditions. Here, we implement this model using the Transformer network and train it with two different schemes:
maximum likelihood estimation (MLE) and conditionalvariational autoencoder (CVAE). The model learned with
the MLE is further subdivided into two variants, autoregressive and non-autoregressive models, resulting in a total ofthree variants of the core model to be considered.
4.3.1

MLE-based Model

The models are trained by maximizing the log-likelihood.
Let θ be the model parameters, the objective is written as
max E [log pθ (X | C, T )] .
θ

(5)Autoregressive Model: In the autoregressive model, theconditional probability of the color styles is modeled aspθ (X | C, T ) :=NY

p (Xn | X1 , X2 , · · · , Xn−1 , C, T ) .

n=1

(6)
The order of the elements is defined by the pre-order tree
traversal.
Non-autoregressive Model: In the non-autoregressivemodel, the conditional probability is assumed to be not conditioned on the previous estimates and is modeled aspθ (X | C, T ) :=NY

p (Xn | C, T ) .

(7)

We implement this model with both the Transformer encoder and decoder as shown in the bottom right of Fig. 3.
The Transformer encoder takes content and style embeddings as input and produces the estimated mean and the estimated variance for each element. They are concatenatedfor all elements and treated as the returned vectors of µ(·)
and σ(·) in Eq. (9), respectively. The latent variables Z
are then sampled from the Gaussian distribution using thereparametrization trick and partitioned into a set of latentvectors equal to the number of elements. The Transformer
decoder takes the latent vectors and embeddings as input
and estimates the color styles of all elements. Note thatthe latent vectors are applied the positional encoding so thatthe decoder can identify which latent vector corresponds towhich element.

n=1

We implement these models with the Transformer networks as illustrated in the upper right of Fig. 3. The content
and style embeddings and the style estimation head used arethose described in Section 4.2.
4.3.2

CVAE-based Model

In general, the non-autoregressive model is capable of faster
inference than the autoregressive model, but is unable tocreate a diverse colorizations for fixed input values. We introduce latent variables into the non-autoregressive modeland extend it to the formulation of conditional variational
autoencoder, allowing for the generation of diverse outputs
from a single input.
Let us denote latent variables as Z ∈ RN d , the conditional generative distribution as pθ (X | Z, C, T ), the posterior as qϕ (Z | X , C, T ), and the prior as p (Z | C, T ), respectively, the learning objective of CVAE is expressed as:maxϕ,θ

Eqϕ (Z | X ,C,T ) [log pθ (X | Z, C, T )]− λKL (qϕ (Z | X , C, T ) ∥ p (Z | C, T )) ,(8)

where ϕ and θ are the model parameters and λ is a hyperparameter to balance the two terms. We set λ = 0.1 for all
experiments.
Using multivariate Gaussian distributions with diagonalcovariance matrices, we model the conditional distributions
as follows:qϕ (Z | X , C, T ) :=N (µ (X , C, T ) , Iσ (X , C, T )) ,pθ (X | Z, C, T ) :=NY

p (Xn | Z, C, T ) ,

(9)
(10)

4.4. Color Upsampler
The color upsampler takes the discrete color styles X as
input and generates the color styles in full resolution Ŷ. To
force the upsampled colors to stay in their original quantization bins, we estimate the proportions in the bins insteadof directly estimating the full-resolution colors. We train theTransformer-based model to minimize the mean squared error between the predicted proportions and the ground-truth
proportions.

5. Experiments
We evaluate our methods against additional baseline
methods in terms of the quality and diversity of the generated color styles.

5.1. Methods
We employ baseline methods based on statistics of thedataset and image colorization.
Statistics-based Coloring: We use simpler methods
based on statistics of color styles. Specifically, we firstcollect the frequencies of the discrete color for each pairof HTML tags and CSS properties in the training set.
The color is then determined by mode selection (mode) or
frequency-weighted sampling (sampling) and upsampled in
the same way as our methods. When sampling, we set thesame color for the same pair of tags and properties, encouraging consistent color styling within a single web page. For
pairs that appear in the test set but not in the training set, theglobal frequencies across all the tags are referenced.

n=1

p (Z | C, T ) :=N (0, I) ,(11)

where µ(·) and σ(·) are the functions that return parameters
corresponding to the mean and the variance of the Gaussian
distribution, respectively.

Image Colorization [19]: We adapt the image colorization technique to our task as an additional baseline. The
main consideration of adaptation is that both input and output are images rather than structural color information. We
use a screenshot of ground-truth web page converted to agrayscale image as the input. Obtaining structural colors
from the output colorized image is not trivial, as it requires
knowing the correspondences between the pixel colors andthe element color styles. Inspired by chroma key compositing, we assign an unused color to a target element, render
the web page, and consider the pixels with that color to be
the corresponding pixels. The colorized image is then referenced and the most dominant color in the corresponding
pixels is taken for the target element. We repeat this process
for all the elements. Note that in this process, all colors aretreated as opaque colors, i.e., the alpha value is 1.0. For the
specific method, we employ ColTran [19], a state-of-the-art
method based on Transformer, whose official implementation is publicly available1 .
Our Methods: We use three methods with different
core models: the autoregressive model (AR), the nonautoregressive model (NAR), and the conditional variational autoencoder (CVAE) described in Section 4. Eachmethod uses the same color upsampler and the same hyperparameters of the Transformer network, the details canbe found in the supplemental material. To generate with
the AR model, we use the same trained model with threedifferent decoding strategies: greedy decoding, top-p sampling [13] with p=0.8 and p=0.9. Note that during training,
we exclude the text color of elements without text content
from the loss calculation because it does not affect the appearance.

5.2. Evaluation Metrics
Accuracy and Macro F-score: As proxy metrics for thequality of the generated results, the reproducibility of theground-truth data is measured. We compute the accuracy
and the Macro F-score using the discrete RGB and alphavalues. The text color of elements without text content is
excluded from the computation. For the Macro F-score, theaverage of class-wise F-scores is used.
Fréchet Color Distance: The diversity of the generated
results is another important measure. We devise a new metric, named Fréchet Color Distance (FCD), with reference
to FID [12], which is widely used in the image generationand colorization tasks. In the FID, the distance between
the distributions of the generated data and the real data iscomputed using intermediate features of the pre-trained Inception Network. These high-level image features may not
represent well low-level color information, thus in the FCD,
histograms of the discrete RGB colors are used as the features to compute the distribution distance. The histograms
are normalized by the number of elements, and their statistics are calculated for background colors, text colors, and1 https://github.com/google-research/google-research/tree/master/coltran

pixels, respectively, so that the diversity of the different perspectives can be measured. Note that the text color of elements without text content is excluded from the corresponding histogram.
Contrast violation: The quality of the generated results
is evaluated from another perspective: accessibility. To investigate the accessibility of the generated results, we use
Lighthouse [10], a commonly used auditing tool in practice,and consider the contrast audit, which is greatly affected bycolor styles. The contrast audit is based on Web Content
Accessibility Guidelines 2.1 [18] and tests whether the contrast ratio between the background and text colors meets thecriteria for all the text on a page. We report the percentageof pages and the average number of elements that violate
the contrast audit.

5.3. Color Style Generation
We summarize the quantitative results in Table 1 and thequalitative results in Fig. 4. In the quantitative results, wereport the mean and standard deviation of one evaluationfor each of the three models trained with different random
seeds, and for Stats (sampling), we report them for threeevaluations. The qualitative results show screenshots ofweb pages where the generated color styles are applied. For
those with multiple results, we first generate 20 variations
with a single model, then the one with the largest color distance is greedily selected, starting from a random selection.
The Fréchet Color Distances (FCDs) are calculated between two halves of the test set, one from the generated
results and the other from the ground-truth data. The FCDs
for real data use the other ground-truth data instead of thegenerated results, allowing for reasonable comparisons withother methods. The real data shows a large percentage
of web pages violating the contrast criterion, but this isroughly consistent with the public statistics [2]. We consider the metrics of contrast violation only for reference,and consider them good if they do not differ significantly
from the real statistics.
The image colorization baseline ColTran [19] performs
well for the Pixel-FCD and poorly for the other metrics,
which may indicate that by using the ground-truth grayscale
screenshots, ColTran can generate plausible color styles
only for those with many corresponding pixels, such as the
background color of the body element. In color style extraction from the colorized image, the extraction errors for
those with fewer corresponding pixels, such as the color forsmaller text as in the right example in Fig. 4, may be thecause of contrast violation. Due to not considering the alpha composition in the extraction, RGB values obtained differently from the ground-truth may cause poor accuracies,
BG-FCD, and Text-FCD.
The statistical baselines of structural coloring showGround-truth

Stats (mode)

Stats (sampling)

Ground-truth

Stats (mode)

Stats (sampling)

Ground-truth(grayscale)

ColTran [19](colorized)

ColTran [19]Ground-truth(grayscale)

ColTran [19](colorized)

ColTran [19]AR (greedy)

AR (top-p, p=0.9)

AR (greedy)

AR (top-p, p=0.9)

NAR

CVAE

NAR

CVAE

Figure 4: Qualitative comparison of color style generation. NAR and CVAE successfully generate more plausible color styles
than the others, and CVAE can produce multiple variations. Best viewed in color and zoom.

Table 1: Quantitative comparison of color style generation. The methods with dagger symbols (†) generate in a deterministicmanner, while the others generate in a stochastic manner. The Fréchet Color Distances are multiplied by 1e-3 for clarity.
Note that only ColTran uses the ground-truth grayscale screenshot, i.e., partial color information per pixel.

MethodAccuracy
RGB ↑Alpha ↑

Macro F-score
RGB ↑Alpha ↑

ColTran [19]Stats (mode)†
Stats (sampling)
AR (greedy)†
AR (top-p, p=0.8)
AR (top-p, p=0.9)
NAR†
CVAE

.285±.000.717.621±.000.720±.002.717±.003.714±.003.773±.001.771±.001.411±.000.891.821±.000.886±.002.885±.002.883±.003.929±.001.929±.000.009±.001.003.004±.000.033±.002.032±.002.030±.002.076±.001.069±.001.101±.000.219.207±.001.405±.003.403±.002.402±.006.670±.002.665±.003

665.78±2.49
20.94
0.71±0.02
2.93±0.33
2.63±0.37
2.40±0.33
1.57±0.33
1.50±0.04

103.50±1.61
263.1182.22±0.31
39.19±2.61
33.79±3.59
33.00±3.81
21.81±2.20
28.14±1.13

3.14±0.58
7.43
169.80±1.05
22.98±3.2133.67±0.55
37.44±0.30
5.98±1.23
4.20±0.19

95.51±0.15
34.17
94.41±0.08
66.43±0.70
71.00±1.1273.42±1.01
74.02±0.9274.71±0.19

5.29±0.02
1.18
4.47±0.02
2.02±0.02
2.30±0.06
2.40±0.06
2.25±0.09
2.23±0.05

CVAE w/o message passing
CVAE w/o residual connection.762±.001.768±.001.918±.000.927±.000.062±.002.064±.000.620±.005.647±.009

2.16±0.03
1.41±0.25

33.02±2.82
29.94±2.58

10.58±1.29
5.40±1.17

75.10±0.7974.25±0.87

2.29±0.06
2.20±0.07

1.000

1.000

1.000

1.000

0.08

3.04

0.56

71.72

2.39

Real data

higher accuracies and lower macro F-scores, which indicates the imbalance of color styles where typical color
styles, such as black text and white background, appear
much more frequently than other color styles. The Stats(sampling) improves BG-FCD and Text-FCD and degrades
Pixel-FCD compared to Stats (mode). The former is as
expected, and the latter can be interpreted because typicalcolor styles are often assigned to those with many corresponding pixels in the ground-truth data. For contrast violation, the results indicate that assigning typical color styles
leads to fewer violations, while sampling background andtext colors independently leads to more violations.
Our Transformer-based methods generally perform better than the other baselines, and the trends in contrast violations are similar to the real data. In AR, accuracies and
FCDs show that sampling method can control the typicality of the output. Both NAR and CVAE outperform the AR
variants with comparable scores and show better visual results than the others.
Ablation study: We perform an ablation study to investigate the performance contribution of the design of the hierarchical contextualization. The results using CVAE as thebase model are summarized in the lower part of Table 1
and show that removing message passing degrades performance, suggesting the importance of usage of hierarchicalmodeling. We can also see that the performance is degradedwithout using the residual connection, indicating that they
may help propagate only the minimum information in complex message passing.

6. Limitations and Discussion
While our methods succeed in generating relatively plausible color styles, they are still far from perfect. A typical

BG ↓Fréchet Color Distance
Text ↓
Pixel ↓

Contrast violation% Pages# Elements

failure case is the functional coloring that informs the product color or the current slide in a slideshow (i.e., carousel),as can be found in Fig. 4. The challenge with the functional coloring stems from the fact that necessary information is removed by the simplification or is not included on
the page in the first place. The necessary information is therepresentative color to distinguish variations for the productcolor, and the order of the displayed slide for the slideshow.
To address this issue, explicit information needs to be added
as additional content or the user needs to modify it in postprocessing.

Elements with certain styles are inevitably associatedwith specific color styles. For example, an element witha “round” style should have a background color. However,
our methods can fail to handle such styling rules. Addingcolor-related styles could improve the situation, but one
needs to consider which and how much to add. It may benecessary to find a way to handle all the complex CSS properties in a unified and efficient manner, which may also leadto the extension of CSS property generation beyond color.

There is still room for improvement regarding thedataset. The e-commerce mobile web pages used in ourstudy have low relevance between individual product images and color styles, making them unsuitable for evaluating the harmony between images and colors. Also, it has
not been tested whether our methods can handle more elements, such as full-page view on a mobile-sized screen oron a laptop-sized screen. To facilitate these studies, we believe that the development of stable data collection tool andthe construction of large-scale datasets using such a tool arepromising future directions.

References[1] Saeed Anwar, Muhammad Tahir, Chongyi Li, Ajmal Mian,
Fahad Shahbaz Khan, and Abdul Wahab Muzaffar. Image
Colorization: A Survey and Dataset. arXiv:2008.10774v3,
2022.
[2] The HTTP Archive.
HTTP Archive: Accessibility.
https://httparchive.org/reports/
accessibility#a11yColorContrast. (accessed2022-08-27).
[3] Tony Beltramelli. Pix2code: Generating code from a graphical user interface screenshot. In Proceedings of the ACM
SIGCHI Symposium on Engineering Interactive Computing
Systems, EICS’18, 2018. doi: 10.1145/3220134.3220135.
[4] Murtuza Bohra and Vineet Gandhi. ColorArt: Suggesting colorizations for graphic arts using optimal color-graphmatching. In Proceedings of Graphics Interface, GI’20,
2020. doi: 10.20380/GI2020.11.
[5] Huiwen Chang, Ohad Fried, Yiming Liu, Stephen DiVerdi, and Adam Finkelstein. Palette-based photo recoloring. ACM Transactions on Graphics, 34(4), 2015. doi:
10.1145/2766978.
[6] Xingyu Chen, Zihan Zhao, Lu Chen, JiaBao Ji, Danyang
Zhang, Ao Luo, Yuxuan Xiong, and Kai Yu. WebSRC: A
Dataset for Web-Based Structural Reading Comprehension.
In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP’21, 2021. doi:
10.18653/v1/2021.emnlp-main.343.
[7] Dianne Cyr, Milena Head, and Hector Larios. Colour
appeal in website design within and across cultures:
A multi-method evaluation.
International Journal of Human-Computer Studies, 68(1), 2010.
doi:
10.1016/j.ijhcs.2009.08.005.
[8] Justin Gilmer, Samuel S. Schoenholz, Patrick F. Riley, Oriol
Vinyals, and George E. Dahl. Neural message passing for
Quantum chemistry. In Proceedings of the 34th International
Conference on Machine Learning, ICML’17, 2017.
[9] Google. Google Chrome - Download the Fast, Secure
Browser from Google. https://www.google.com/
chrome/. (accessed 2022-08-29).
[10] Google.
Lighthouse.
https://github.com/
GoogleChrome/lighthouse, 2022. (accessed 202208-29).
[11] Zhenyu Gu and Jian Lou. Data driven webpage color
design.
Computer-Aided Design, 77, 2016.
doi:
10.1016/j.cad.2016.03.001.
[12] Martin Heusel, Hubert Ramsauer, Thomas Unterthiner,
Bernhard Nessler, and Sepp Hochreiter. GANs trained by a
two time-scale update rule converge to a local nash equilibrium. In Proceedings of the 31st International Conference on
Neural Information Processing Systems, NeurIPS’17, 2017.
[13] Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin
Choi. The curious case of neural text degeneration. In International Conference on Learning Representations, ICLR’20,
2020.
[14] Alexandra Hotti, Riccardo Sven Risuleo, Stefan Magureanu, Aref Moradi, and Jens Lagergren. The Klarna Product

Page Dataset: A Realistic Benchmark for Web Representation Learning. arXiv:2111.02168v2, 2021.
[15] Naz Kaya and Helen H. Epps. Relationship between colorand emotion: A study of college students. College Student
Journal, 38(3), 2004.
[16] Kotaro Kikuchi, Mayu Otani, Kota Yamaguchi, and Edgar
Simo-Serra. Modeling Visual Containment for Web Page
Layout Optimization. Computer Graphics Forum, 40(7),2021. doi: 10.1111/cgf.14399.
[17] Suzi Kim and Sunghee Choi. Dynamic closest color warpingto sort and compare palettes. ACM Transactions on Graphics, 40(4), 2021. doi: 10.1145/3450626.3459776.
[18] Andrew Kirkpatrick, Joshue O Connor, Alastair Campbell,and Michael Cooper. Web Content Accessibility Guidelines (WCAG) 2.1. https://www.w3.org/TR/2018/
REC-WCAG21-20180605/, 2018. (accessed 2022-0514).
[19] Manoj Kumar, Dirk Weissenborn, and Nal Kalchbrenner.
Colorization transformer. In International Conference on
Learning Representations, ICLR’21, 2021.
[20] Ranjitha Kumar, Arvind Satyanarayan, Cesar Torres, Maxine
Lim, Salman Ahmad, Scott R. Klemmer, and Jerry O. Talton.
Webzeitgeist: Design mining the web. In Proceedings of the
SIGCHI Conference on Human Factors in Computing Systems, CHI’13, 2013. doi: 10.1145/2470654.2466420.
[21] Sharon Lin, Daniel Ritchie, Matthew Fisher, and Pat Hanrahan. Probabilistic color-by-numbers: Suggesting pattern colorizations using factor graphs. ACM Transactions on Graphics, 32(4), 2013. doi: 10.1145/2461912.2461988.
[22] Peter O’Donovan, Aseem Agarwala, and Aaron Hertzmann.
Color compatibility from large datasets. ACM Transactions
on Graphics, 30(4), 2011. doi: 10.1145/2010324.1964958.
[23] Arnold Overwijk, Chenyan Xiong, and Jamie Callan.
ClueWeb22: 10 Billion Web Documents with Rich Information. In Proceedings of the 45th International ACM SIGIR
Conference on Research and Development in Information
Retrieval, SIGIR’22, 2022. doi: 10.1145/3477495.3536321.
[24] Gabriel Peyré and Marco Cuturi. Computational OptimalTransport: With Applications to Data Science. Foundations and Trends in Machine Learning, 11(5-6), 2019. doi:
10.1561/2200000073.
[25] Qianru Qiu, Mayu Otani, and Yuki Iwazaki. An Intelligent
Color Recommendation Tool for Landing Page Design. In
27th International Conference on Intelligent User Interfaces,
IUI’22 Companion, 2022. doi: 10.1145/3490100.3516450.
[26] Selenium. WebDriver — Selenium. https://www.
selenium.dev/documentation/webdriver/.
(accessed 2022-08-29).
[27] Gapsy Studio. How to Choose UI Colors for Mobile and Web
Design Wisely. https://gapsystudio.com/blog/
ui-design-colors/, 2020. (accessed 2022-05-17).
[28] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Łukasz Kaiser, and Illia
Polosukhin. Attention is all you need. In Proceedings of the
31st International Conference on Neural Information Processing Systems, NeurIPS’17, 2017.

[29] Polina Volkova, Soheila Abrishami, and Piyush Kumar. Automatic Web Page Coloring. In Advances in Visual Computing, Lecture Notes in Computer Science, 2016. doi:
10.1007/978-3-319-50835-1 9.
[30] Nanxuan Zhao, Ying Cao, and Rynson W.H. Lau. Modelingfonts in context: Font prediction on web designs. Computer
Graphics Forum, 37(7), 2018. doi: 10.1111/cgf.13576.
[31] Tianming Zhao, Chunyang Chen, Yuanning Liu, and Xiaodong Zhu. GUIGAN: Learning to Generate GUI Designs
Using Generative Adversarial Networks. In Proceedings ofthe 43rd International Conference on Software Engineering,
ICSE’21, 2021. doi: 10.1109/ICSE43902.2021.00074.