Colorful Image Colorizations
Supplementary Material
Richard Zhang, Phillip Isola, Alexei A. Efros{rich.zhang, isola, efros}@eecs.berkeley.edu
University of California, Berkeley

1Overview

This document is divided into three sections. Section 2 adds some clarifications
regarding filtering grayscale images from the dataset, along with additional details about the network architecture. Section 3 contains a discussion of our algorithm in comparison to Cheng et al. [1]. Section 4 contains a more detailed
explanation of the VGG category analysis presented in Section 4.1 and Figure 8of the paper, along with an additional analysis on common category confusions
after recolorization.

2Clarifications

3

Comparison to Cheng et al. [1]3.1

Network architecture

Figure 3 in the paper showed a diagram of our network architecture. Table
1 in this document thoroughly lists the layers used in our architecture during
training time. During testing, the temperature adjustment, softmax, and bilinear
upsampling are all implemented as subsequent layers in a feed-forward network.
Note the column showing the effective dilation. The effective dilation is thespacing at which consecutive elements of the convolutional kernel are evaluated,relative to the input pixels, and is computed by the product of the accumulated
stride and the layer dilation. Through each convolutional block from conv1 toconv5, the effective dilation of the convolutional kernel is increased. From conv6
to conv8, the effective dilation is decreased.
3.2

Filtering Grayscale Images

Some of the images in the Imagenet [2] dataset are in grayscale and were filtered out of training, validation, and testing sets. An image was considered tobe grayscale if no pixel had a value of ab above 5 and was withheld from training and testing. The threshold was set to be conservative. A more aggressive
threshold would remove more grayscale images at the expense of color images
that happened to contain a very desaturated palette.

2Zhang, Isola, Efrosdata
conv1 1
conv1 2conv2 1
conv2 1
conv3 1
conv3 2
conv3 3
conv4 1
conv4 2
conv4 3
conv5 1
conv5 2
conv5 3conv6 1
conv6 2
conv6 3conv7 1
conv7 2conv7 3conv8 1
conv8 2conv8 3

Our Network ArchitectureXCS D Sa De224
3224641111112642111112 128 1
12256128 2
12256256 1
14456
256 1
14428256 2
14428
512 1
18828
512 1
18828
512 1
18828
512 1281628
512 1281628
512 1281628
512 1281628
512 1281628
512 1281628
256 1
18828
256 1
18828
256 1
18856
128 .514456128 1
14456128 1
144BN
X
X
XXX
XX-LX
XX
X

Table 1. Our network architecture. X spatial resolution of output, C number of channels of output; S computation stride, values greater than 1 indicate downsampling
following convolution, values less than 1 indicate upsampling preceding convolution;
D kernel dilation; Sa accumulated stride across all preceding layers (product over all
strides in previous layers); De effective dilation of the layer with respect to the input(layer dilation times accumulated stride); BN whether BatchNorm layer was used afterlayer; L whether a 1x1 conv and cross-entropy loss layer was imposed

3.3

Training, Validation, and Testing Splits

The full 1.3M images, minus the grayscale images were used for training. The
first 2000 images in the Imagenet [2] validation set were used for validation. For
the aggregated quantitative testing results shown in Table 1 of the paper, weused the last 10,000 images in the validation set, 9803 of that were color. For thequalitative VGG classification results shown in Figure 8, we used the last 48,000
images in the validation set were used, 47,023 of that contained color. Since wesorted on classification performance across 1000 categories for the VGG analysis,
we used the full validation set for testing to maximize the number of samples
per category.
Quantitative comparisons to Cheng et al. [1] are not possible, as the authors
have not released their code or test set results. We provide qualitative comparisons to the 23 test images in [1] on the attached website, which we obtained by

Colorful Image Colorization

Algorithm

Learning

DatasetRun-time

Cheng et al. [1](1) Extract feature sets(a) 7x7 patch (b) DAISY(c) FCN on 47 categories(2) 3-layer NN regressor(3) Joint-bilateral filter
Extract features. Train FCN [4]on pre-defined categories.
Train 3-layer NN regressor.
2688/1344 images fromSUN [3] for train/test.
Limited variety withonly scenes.
4.9s/image onMatlab implementation

3Ours

Feed-forward CNN

Train CNN from pixels tocolor distribution. Tune single
parameter on validation.
1.3M/10k images from
ImageNet [2] for train/test.
Broad and diverse
set of objects and scenes.
100ms/image in Caffe
on K40 GPU

Table 2. Comparison to Cheng et al. [1]manually cropping from the paper. Our results are about the same qualitativelevel as [1]. Note that Cheng et al. [1] has several advantages in this setting:(1) the test images are from the SUN dataset [3], which we did not train onand (2) the 23 images were hand-selected from 1344 by the authors, and is notnecessarily representative of algorithm performance. We were unable to obtainthe 1344 test set results through correspondence with the authors.
Additionally, we compare the methods on several important dimensions inTable 2: algorithm pipeline, learning, dataset, and run-time. Our method isfaster, straightforward to train and understand, has fewer hand-tuned parameters and components, and has been demonstrated on a broader and more diverse
set of test images than Cheng et al. [1].

4VGG Evaluation

4.1

Classification Performance

In Section 4.1, we investigated the grayscale and re-colored images using theVGG classifier [5] for the last 48,000 images in the Imagenet validation set. For
each category, we computed the top-5 classification performance on grayscaleand recolorized images, agray , arecolor ∈ [0, 1]C , where C = 1000 categories. We
sorted the categories by arecolor − agray and plotted examples of some selected
top and bottom classes in Figure 8 of the paper. The re-colored vs grayscale performance per category is shown in Figure 1, with top and bottom 50 categories
highlighted. For the top example categories, the individual images are sorted byascending rank of the correct classification of the recolored image, with tiebreakers on descending rank of the correct classification of the grayscale image. For
the bottom example categories, the images are sorted in reverse, in order to highlight the instances when recolorization results in an errant classification relativeto the grayscale image.

4Zhang, Isola, Efros

Fig. 1. Performance of VGG top-5 classification on recolorized images vs grayscale
images per category. Test was done on last 48,000 images in Imagenet validation set.

4.2

Common Confusions

To further investigate the biases in our system, we look at the common classconfusions that often occur after image recolorization but not with the original
ground truth image. We compute the rate of top-5 confusion Corig , Crecolor ∈[0, 1]C×C , with ground truth colors and after recolorization. A value of Cc,d = 1
means that every image in category c was classified as category d in the top-5. We
find the class-confusion added after recolorization by computing A = Crecolor −Corig , and sort the off-diagonal entries. Figure 2 shows all C ×(C −1) off-diagonal
entries of Crecolor vs Corig , with the top 100 entries from A highlighted.
For each category pair (c, d), we extract the images that contained the confusion after recolorization but not with the original colorization. We then sortthe images in descending order of the classification score of the confused category. Examples for some top categories are shown in Figure 3. An image of a“minibus” is often colored yellow, leading to a misclassification as “school bus”.
Animal classes are sometimes colored differently than ground truth, leading tomisclassification to related species. Note that the colorizations are often visuallyrealistic, even though they lead to a misclassification.

Colorful Image Colorization

5Fig. 2. Top-5 confusion rates with recolorizations and original colors. Test was doneon last 48,000 images in Imagenet validation set [2].

6Zhang, Isola, Efros

Fig. 3. Examples of some most-confused categories. Top rows show ground truth image. Bottom rows show recolorized images. Rank of common confusion in parentheses.
Ground truth and confused categories after recolorization are labeled.

Colorful Image Colorization

7References
1. Cheng, Z., Yang, Q., Sheng, B.: Deep colorization. In: Proceedings of the IEEE
International Conference on Computer Vision. (2015) 415–423
2. Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z.,
Karpathy, A., Khosla, A., Bernstein, M., et al.: Imagenet large scale visual recognition challenge. International Journal of Computer Vision 115(3) (2015) 211–252
3. Patterson, G., Hays, J.: Sun attribute database: Discovering, annotating, and recognizing scene attributes. In: Computer Vision and Pattern Recognition (CVPR),
2012 IEEE Conference on, IEEE (2012) 2751–2758
4. Long, J., Shelhamer, E., Darrell, T.: Fully convolutional networks for semanticsegmentation. In: Proceedings of the IEEE Conference on Computer Vision andPattern Recognition. (2015) 3431–3440
5. Simonyan, K., Zisserman, A.: Very deep convolutional networks for large-scale imagerecognition. arXiv preprint arXiv:1409.1556 (2014)