Learning Representation in Colour Conversion
Authors: Arash Akbarinia
Raquel Gil-Rodriguez
Alban Flachot
Matteo Toscani
Venue:  (2021)
Keywords: Color representation
VAE
Color space
Unsupervised learning

Abstract: Colours can be represented in an infinite set of spaces highlighting distinct features. Here, we investigated the impact of colour spaces on the encoding capacity of a visual system that is subject to information compression, specifically variational autoencoders (VAEs) where bottlenecks are imposed. To this end, we propose a novel unsupervised task: colour space conversion (ColourConvNets). We trained several instances of VAEs whose input and output are in different colour spaces, e.g. from RGB to CIE L*a*b* (in total five colour spaces were examined). This allowed us to systematically study the influence of input-output colour spaces on the encoding efficiency and learnt representation. Our evaluations demonstrate that ColourConvNets with decorrelated output colour spaces produce higher quality images, also evident in pixel-wise low-level metrics such as colour difference ($\Delta E$), peak signal-to-noise ratio (PSNR) and structural similarity index measure (SSIM). We also assessed the ColourConvNets' capacity to reconstruct the global content in two downstream tasks: image classification (ImageNet) and scene segmentation (COCO). Our results show a 5-10% performance boost for decorrelating ColourConvNets with respect to the baseline network (whose input and output are RGB). Furthermore, we thoroughly analysed the finite embedding space of Vector Quantised VAEs with three different methods (single feature, hue shift and linear transformation). The interpretations reached with these techniques are in agreement suggesting that (i) luminance and chromatic information are encoded in separate embedding vectors, and (ii) the structure of the network's embedding space is determined by the output colour space.

Under review as a conference paper at ICLR 2021
LEARNING REPRESENTATION IN COLOUR CONVER- SION
A Anonymous authors
Paper under double-blind review
ABSTRACT
Colours can be represented in an infinite set of spaces highlighting distinct fea- tures. Here, we investigated the impact of colour spaces on the encoding capacity of visual system that is subject to information compression, specifically varia- tional autoencoders (VAEs) where bottlenecks are imposed. To this end, we pro- pose novel unsupervised task: colour space conversion (ColourConvNets) We trained several instances of VAEs whose input and output are in different colour spaces, e.g. from RGB to CIE L* t (in total five colour spaces This allowed us to systematically study the influence of input-output colour spaces the encoding efficiency and learnt representation. Our evaluations demonstrate that ColourConvNets with decorrelated output colour spaces produce higher qual- ity images, also evident in pixel-wise low-level metrics such colour difference (AE), peak signal-tor to-iise ratio (PSNR) and structural similarity index measure (SSIM). We also assessed the ColourConwNets'' capacity to reconstruct the global content in two downstream tasks: image classification (ImageNet) and scene seg- mentation (COCO). Our results show 5-10% performance boost for decorrelating ColourConvNets with respect to the baseline network (whose input and output are RGB). Furthermore, we thoroughly analysed the finite embedding space of Vee- tor Quantised VAEs with three different methods (single feature, hue shift and linear transformation.. The interpretations reached with these techniques are agreement suggesting that (i) luminance and chromatic information are encoded in separate embedding vectors, and (ii) the structure of the network's 's embedding space is determined by the output colour space.
examined).
INTRODUCTION
Colour is inseparable component of our conscious visual perception and its objective utility spans
over a large set of tasks such as object recognition and scene segmentation (Chirimuuta et al. 2015]
Gegenfurtner & Rieger 22000) Wichmann et (2002). Consequently, colour is an ubiquitous feature
in many applications: colour transfer (Reinhard et al. 2001), colour constancy (Chakrabarti 2015),
style transfer (Luan et al. (017)), computer graphics (Bratkova al. 2009), image denoising Dabov et al. 2007), quality assessment (Preiss et al., (2014). to name a few. Progress in these lines requires better understanding of colour representation and its neural encoding in deep networks. To this end, we present a novel unsupervised task: colour conversion.
In our proposed framework the input-output colour space is imposed on deep autoencoders (referred to ColourConvNets) that learn to efficiently compress the visual information (Kramer 1991b while transforming the input to output. Essentially. the output y for input image generated on the fly by transformation y T(x), where 7 maps input to output colour space. This task offers a fair comparison of different colour spaces within a system that learns to minimise loss function in the context of information bottleneck principle (Tishyy & Zaslavsky 2015)). The quality of output images demonstrates whether the representation of input-output colour spaces impacts networks' encoding power. Furthermore, the structure of internal representation provides insights on how colour transformation is performed within a neural network.
In this work, we focused Vector Quantised Variational Autoencoder (VQ-VAE) van den Oord ct al. (017)) due to the discrete nature of its latent space that facilitates the analysis and interpretabil- ity of the learnt features. We thoroughly studied five commonly used colour spaces by training

---

Under review as a conference paper at ICLR 2021
ColourConvNets for all combinations of input-ouiput spaces. First, show that ColourConwNetss with a decorrelated output colour space (e.g. CIE L)) convey information more efficiently in their compressing bottleneck, in line with the presence of colour opponency in the human visual system. This is evident qualitatively (Figures and A. and quantitatively (evaluated with three low-level and two high-level metrics). Next, we present the interpretation of ColourConwNets' latent space by means of three methods reaching consensus interpretation: sentation in the VQ-VAEs' latent space is determined by the output colour space, suggesting the transformation T occurs at the encoder, (ii) each embedding vector in VQ- VA-V encodes a specific part of the colour space, e.g. the luminance or chromatic information, which can be modelled by parsimonious linear transformation.
(i) the colour repre-
Original
rgb2rgb
rgb2dkl
rgb2lab
Figure 1: Qualitative comparison of three ColourConvNets IVQ-VAE of K=8 and D=128). The first column is the networks' input and the other columns their corresponding outputs. The output images of rgb2dkl and rgb2lab have been converted to the RGB colour space for visualisation purposes. The artefacts in rgb2rgh are clearly more visible in comparison to the other ColourConvNets.
1.1 RELATED WORK
The effectiveness of different colour spaces have been investigated in a few empirical studies of deep neural networks (DNNs). Information fusion over several colour spaces improved retinal medical imaging (Fu et al. 2019). A similar strategy enhanced the robustness of face (Li et al. 2014, Larbi et al., 2018) and traffic light recognition (Cireçan al. 2012] Kim et al.] 2018). This also effective in predicting eye fixation (Shen et al.] 2015). Opponent colour spaces have been explored for applications such as style transfer (I Luan et al. 2017 Gatys et al., (017) and picture colourisation (Cheng et al./ 2015; Larsson et al., 2016)) Most of these works are within the domain of supervised learning. The most similar approach our proposed ColourConwNets
image colourisation
pretext task for unsupervised visual feature learning (Larsson et al. 2017).
Initial works on colour representation in DNNs revealed object classification networks learn to decorrelate their input images (Rafegas & Vanrell [018 Flachot & Gegenfurtner 2018 Harri ct al. 2019). This is a reminiscence of horizontal and ganglion cells that decorrelate retinal sig- nal into colour-opponency before transmitting it to the visual cortex (Schiller & Malpeli 1977
Derrington et af.) 1984 Gegenfurtner Kiper 2003). Another set of works reported existence of hue-sensitive units (Engilberge et al.) (017]) that mainly emerge in early layers (Bau et al.) (017)) Representation of colours deep networks at intermediate and higher layers is rather understudied. In this article, we specifically focus the intermediate representation that emerges at the latent space of autoencoders, which to the best of our knowledge has not been reported in the literature.
2 COLOUR CONVERSION AUTOENCODERS
In this article, we propose novel unsupervised task of colour conversion: the network's output colour space is independent of its input (see Figure ( A colour space is an arbitrary definition of colours' organisation in the space (Koenderink & van Doorn 2003). Thus, the choice of transfor-
2

---

Under review as a conference paper at ICLR 2021
Colour Space Conversion
Colour Conversion Autoencoders
RGB
eabbldding space Ru
enmnder
drooder
RGB
DKL
LMS
Input
cntpert
Fe
Zq (x)
Figure 2: Left: exemplary conversions across different colour spaces, Right: the schematic view of VQ-VAE NolourConvNets..
mation matrix T in ColourConvNets is perfectly flexible to model any desired space,
T Couts
Cin
(1)
where Cin and Cout are the input and output colour spaces. This framework offers controlled environment to compare colour spaces within complex visual system. Here, we studied their ef- fectiveness in information encoding constrained to bottleneck. This can be extended to encompass other constraints (such as entropy, energy, wiring, etc.) relevant to understanding colour represen- tation in complex visual systems. We further used this structure to compare autoencoder's 's latent space across colour spaces aiming to decipher the intermediate colour representation within these networks. The proposed framework can also be employed in applications, e.g., add-on optimi- sation capsule to any computer vision application (Mosleh et al. 2020), or as proxy task for visual understanding (Larsson al. 2017).
2.1 NETWORKS
We studied a particular class of (van den Oord et al. 2017) due the discrete nature of its latent embedding space that facilitates the analysis and interpretability of the learnt features, which distinguishes it from others (Kingma & Welling 2013). VQ-VAF consists of three main blocks: 1) an encoder that processes the input data (x:); 2) latent embedding space {e} E K with K vectors of dimensionality D, that maps ze (x) onto zi (I) by estimating the nearest vector e to ze (x): 3) a decoder that reconstructs the final output x' with distribution p(r) (x))) over the input data (see the right panel in Figure
Quantised Variational Autoencoder (VQ-VAB)
The loss function defined as follows,
log p(z) (2 + || (2)) - el12 + (x) sgle]12.
(2)
where sg denotes the stop gradient computation that is defined as the identity during the forward- propagation, and with zero partial derivatives during the back-propagation refrain its update. The first term in Eq. corresponds to the reconstruction loss incorporating both encoder and decoder; the second term updates the embedding vectors; and the third term harmonies the encoder and embedding vectors. The parameter 3 E IR is set to 0.5 in all our experiments.
2.2 COLOUR SPACES
We explored five colour spaces: RGB, LMS, CIE L*a*b*, DKL and HSV. The standard space in electronic imaging is RGB that represents colours by three additive primaries in a cubic shape. The LMS colour space corresponds to the response of human cones (long-, middle-, and short- wavelengths) (Gegenfurtner & Sharpe 1999). The CIE *b* colour space (luminance, red-green and yellow-blue axes) is designed to be perceptually uniform (CIE 1978). The DKL colour space (Derrington- Krartton-KKuusoo---Lenn sual system (Derrington et al. 1984). The HSV colour space (hue, saturation, value) a cylindrical representation of RGB cube designed by computer graphics.
models the opponent responses of rhesus monkeys in the early vi-
The onput-output to our networks can be in any combination of these colour spaces. Effectively, VQ-VAE VAE models, in addition learning efficient representation, must learn the transformation
3

---

Under review as a conference paper at ICLR 2021
function from their input to output colour space. It is worth considering that the original images in explored datasets are in the RGB format. Therefore, one might expect a slight positive bias towards this colour space given its gamut defines the limits of other colour spaces.
EXPERIMENTS
We trained several instances of VQ-VAEs with distinct sizes of embedding space {e) E R The training procedure was identical for all networks: trained with Adam optimiser (Kingma & Ba 2014) (Ir 2 10=+) for 90 epochs. To isolate the influence of random variables, all networks were initialised with the same set of weights and an identical random seed was used throughout all experiments. We used ImageNet dataset (Deng et al., 2009) for training. This is a visual database of object recognition in real-world images, divided into one thousand categories. The training set contains 1.3 million images. At every epoch, We exposed the network to 100K images of size 224 x 224 of three colour channels. Figure [. reports the progress of loss function for various ColourCnnVNets. A similar pattern of convergence can be observed for all trained networks.
To increase the generalisation power of our findings, we evaluated all networks on the validation- set of three benchmark datasets: ImageNet (50K images), COCO (5K images), and CelebA (-20K images). COCO is large-scale object detection and segmentation dataset (Lin et al. 2014). CelebA contain facial attributes of celebrities (Liu et al. 2015). We relied on two classes of evaluation low-level (Thess et al.] 2013), capturing the local statistics of an image; high-level (Borji 2019) assessing the global content an image.
Low-level evaluation We computed three commonly used metrics to measure the pixel-wise per- formance of networks: (i) the colour difference CIE A F-2000 (Sharma et al. (2005). (ii) peak signal- to-noise ratio (PSNR), and (iii) structural similarity index measure (SSIM) (Wang et al. 2004).
High-Jevel evaluation Pixel-wise measures are unable to capture the global content of an image and whether semantic information remains perceptually intact. To account for this limitation, we performed procedure similar to the standard Inception Score (Salimans et al., 2016 Borji 2019) by feeding the reconstructed images to two pretrained networks (without tine-tuming) that perform the task of object classification, ResNet50 (He et al. 2016), and scene segmentation, Feature Pyra- mid Network-FPN (Kirillov et al., 2019). ResNet50 and FPN expect RGB inputs, thus non-RGB reconstructions were converted to RGB. The evaluation for ResNet50 the classification accuracy on ImageNet dataset. The evaluation for FPN the intersection over union (IoU) on COCO dataset.
3.1 EMBEDDING SIZE
We first evaluated the influence of embedding size for four regimes of ColourConvNetts whose input colour space is the original RGB images. The low-level evaluation for ImageNet reported in Figure 3 and COCO Figure C.I Across three metrics, the poor performance of rgb2hsv pops up at low-dimensionality of the embedding vector (D = 8). This might be due to the circular nature of huee For the smallest and the largest embedding space, we observe no significant differences between the four networks. However, for embedding spaces of 8 x 8 and 8 x 128 an advantage appears for networks whose outputs opponent colour spaces (DKL and CIE L L))
Colour Difference +
PSNR 1
SSIM 1
POHA
1011 arbbedding (ICx Di
15/175
1784778
Fal
Fir
of umbedding
enbedddnng 38:000
Figure Low-level evaluation for embedding spaces of different size (ImageNet validation-set).
The corresponding high-level evaluation is reported in Figure The overall trend is much alike for both tasks. The lowest performance occurs for rgb2hsv across all embedding spaces. Colour-
For reproduction, the source code and all experimental data are available in the supplementary materials.
4

---

Under review as a conference paper at ICLR 2021
ConvNets with opponent output colour space systematically perform better than rgb2rgb, with an exception for the largest embedding space (128 128) where networks perform equally (de- spite the substantial compression, 70% top-1 accuracy ImageNet and 60% IoU COCO). The comparison of low- and high-level evaluation for the smallest embedding space (4 x 128) (Figure demonstrates the importance of high-level evaluation. Although no differ- ence emerges for the low-level measure, the classification and segmentation metrics are substantially influenced by the quality of the reconstructed images in those four VQ-VAEs.
versus annd
ImageNet (Accuracy 1)
COCO (IoU f)
Segmentation lou (%)
Accuracy (%)
rgb2lab
rgh2dkl
rgb2rgb
r9b2hev
rgb2lab
rgh2dkl
rgb2rgb
rgh2hav
4x126
ExB
Exi2a
128x129
4x128
BxB
Ex12a
128x128
Size of the embadding space (K x D)
Size of the embadding space (K D)
Figure 4: High-level visual task evaluation. Left: ResNet50's classification accuracy on recon- structed images of ImageNet. Right: FPNS's segmentation IoU on reconstructed images of COCO.
3.2 PAIRWISE COMPARISON
For the two embedding spaces with the largest differences (8 x 8 and 8 128) we conducted an ex- haustive pairwise comparison across two regimes of colour spaces: sensory (RGB and LMS) versus opponency (DKL and CIE L)) HSV is excluded in these analysis due to the aforementioned reason. Figure 5 presents the low-level evaluation results for ImageNet (COCO Figure C.2 and CelebA Figure
There is clear tendency of better performance for ColourConyNets with an opponent output colour space across all measures and datasets. Overall, the rgb2lab reconstructs the highest quality images. In comparison to the bascline (i.c. rgb2rgb) both rgb2lab and rgb2dkl obtain substantially lower colour differences, and higher PSNRs and SSIMs.
Colour Difference +
PSNR 1
SSIM Output dkd
Output Ims
Output
dkl
lab
Ims
dki
lab
rgb
Ims
rgb
30.0
reb
rgb
0.63
0.60
0.66
0.66
06
:90
06
Input lms
10.3
7.2
6.6
lms
22.8
21.2
22.8
23.0
Input dkd Ims
0.61
0.59
0.61
0.62
Input del
:80
dki
7.2
9.9
6.5
6.2
22.4
21.3
23.0
23.0
22.0
0.61
0.64
06
lab
6.3
9.6
6.2
6.0
lab
23.2
21.4
23.2
23.0
22.5
lab
0.65
0.62
0.66
0.64
Figure 5: Low-level pairwise comparison of two groups of input-output colour spaces (ImageNet validation-set). Figures are averaged over two embedding spaces x 8 and 8 x 128.
The high-level evaluation results are reported in Figure 6 agreement to previous findings, rgb2lab performs best across both datasets and embedding spaces. Overall, ColourConvNets with oppo- nent output space show a clear advantage: rgb2lab and rgb2dkl obtain 5-7% higher accuracy and IoU with respect to the baseline rgb2rgb.
4 PERFORMANCE ADVANTAGE
The main difference between two regimes of colour spaces (sensory versus opponency) is their intra-axes correlation. The intra-axes correlation for RGB and LMS very high, hence referred to correlated colour spaces. On the contrary, the intra-axes correlations for CIE * and DKI.
5

---

Under review as a conference paper at ICLR 2021
ImageNet (Accuracy 1)
COCO (loU 1)
{e) E R2x8
{e) E R 1288
{e} E
R 8888
{e} E
RR 8
Durtput
Output
Oulgut
Output
rgb
Ims
d
rgB
Ims
leb
rgb
rgb
2015
d1
1ab
50,9
48
4
55.0
57.7
12.0
+9.4
59.5 39.7
5L3
41%
45
41.9
47.1 50.2 50.7
20
Import
53.1
45 2
34.2
56.4
24
24
57.1
50.0
57.4
50.5
46.0
+6.3
40. 1
68.3
Iaport
48.7
47.6
48.6
50.4
404
(6)
12.4
42
56.1
57.7
57.8
68.6
59.2
59.8
60.1
43.0
48.8
04.7
49.0
45.6
30.2
Figure 6: High-level pairwise comparison of sensory (RGB and LMS) versus opponency (DKL and
CIE L*)*b) input-output colour spaces of VQ-VAES.
are very low, hence referred to as decorrelated colour spaces In biological visual systems. the retinal signal is transformed to opponency before transmitted the visual cortex through the LGN bottleneck (Zhaoping (2066)). This transformation has been argued to boost the efficiency of in- formation coding (Buchshaum & Allan 1983, Ruderman et al. 1998 Lee et al.) 200T). Here, our results show a similar phenomenon in deep autoencoders that compress information in their bottle- neck. Contrary to this, the ImageNet classification performance was reported unaltered when input images converted from RGB to CIE L (Mishkin et al. 2017). This might be explained by the lack of bottleneck constraint in their examined architecture, thus decorrelating colour representa- tion leads to extra advantage. Interestingly, we can observe this with ColourConvNets of largest embedding space (128 x 128), suggesting decorrelation of colour signal become beneficial when system is constrained in its information flow.
2
Previous works in the literature (Foster et al. 2008 Malo 2019) have measured the decorrelation characteristics of colour opponent spaces in information theoretical analysis and demonstrated their effectiveness in encoding natural images. The understanding of how complex visual system, driven by error minimisation strategy (Laparra al. 2012), might utilise these properties the system level is of great interest (Lillicrap & Kording 2019). We hypothesised that an efficient system distributes its representation across all resources instead of heavily relying on few components (Laughlin) 1981). To measure this, the histogram of embedding vectors across all images of ImageNet (50K) and COCO (5K) were computed. zero standard deviation the frequency of selected vectors means embedding vectors are equally used by the network. Figure 7 reports the error rate function of this measure. A significant correlation emerges in both datasets, suggesting a more uniform contribution of embedding vectors enhances visual encoding in VQ-VAEs. This matches the neural model of histogram equalisation (Pratt 2007 Bertalmíoo 2014) and is consistent with the efficient coding theory for the biological visual system (Barlow] 1961] Zhaoping 2006a).
INTERPRETING THE EMBEDDING SPACE
Comprehension of the features learnt by a DNN remains a great challenge the entire community (Lillicrap & Kording, 2019). Generative models and in particular variational autoencoders are no exceptions. Strategies on the interpretation of the latent space structure include interpolation in latent space arithmetic operations on learnt features (Radford et al. 20155] Bojanowski et al.) 2017]] Kim et al., 2018). In practice, however, these approaches require explicit human supervision, a cumbersome task due to the often large dimensionality of the latent space. Here, we borrowed the "lesion" " technique, commonly practised in the neuroscience community (Vaidya et al.) 2019). and applied it the embedding space by silencing one vector time (i.e. setting its weights to zero). This procedure is referred to as "ablation" " in the learning community and it has been useful in dissecting classification DNNs (Sandler et al. 2018) and GANs (Bau al. 2020). To measure the consequences of vectors' lesion, we analysed the ColourConvNets' embedding space with three distinct methods: (i) single features, (ii) linear transformation and (iii) hue-shift.
We computed these correlations in all images of ImageNet dataset houndred-randon pixels per image). RGB: ,. x 0.90. ,B ze 0.77, GB as 0.89; LMS: ,LM xs 1.00. , S x 0.93, MS xs 0.93; L**ab*: Leav -0.14, Lebe
/itbe 0.13, r n- 0.34; DKL DK 0.01, y 0.14, p. 0.61.

---

Under review as a conference paper at ICLR 2021
ImageNet
COCO
0.50
0.68, p 2E-05
T- 0.07, p 2E-05
0735-
034
0.23
Error rate (h)
Error zate (W)
0.22
0275
0.
K=8 D-128
K-8 D-
0225
corr-cofT
corr-corT
decort-corr
decorr-corr
corr-decorr
corr-decorr
637
decorr-decort
decorr-decort
oi1
o12
515
oja
Bis
019
$10
aio ois
0ie
Standard deviation frequency of embeddling vectors
Standard devistion in frequency embedding vectors (+)
Figure 7: Error rate function of the difference in frequency of selected vectors in the embedding space. value of zero the z-axis indicates all embedding vectors are equally used by the model. Higher values of indicate that the model relies heavily on certain vectors.
5.1 SINGLE FEATURES
To visualise the encoded representation by each embedding vector, we sampled from the embedding space an example of spatial size 2 2 with all cells set the same vector index. Figure 8 shows the reconstructed images for all network combinations with embedding space {e} 188112 (Figure D.1 for {e} 8x88)). The input colour space is the same in each row, and the output space is the same in each column. An interesting column-wise feature appears. Networks with an identical output colour space share similar set of hues arranged in different order. The order within the embedding space of VQ-VAEs is arbitrary and changing it does not impact the network's output. the colour representation in network's embedding
This is an interesting phenomenon suggesting: space is an attribute of its output colour space, and (ii) the colour transformation T' is performed by encoder before reaching the embedding space. This is an exciting line of investigation for feature studies to systematically explore whether the concept of unique hues and colour categories (Witzel & Gegenfurtner 2018
Siuda- Suuda-Krzywicke et al. 2019) emerge in machine colour representation.
rgb2rgb
rgb2lms
rgb2dkl
rgb2lab
eg
E3 C4
e7
E1 E2 E3
Es
E6
e7
eo G1 62 C1 E5 eD
eo E3 E1
ET
Ims2rgb
Ims2lms
Ims2dkl
Ims2lab
eg
61 62 C3 64 E5
Gb e7
6o 61 62 6g 64
65 66 e7
en e1 e2 63 64 es CB 67
60 61 62 63
64
65 66
67
dkl2rgb
dkl2lms
dkd2dkl
dkl2lab
eg é1
e2 0g
64
E5
66 e7
60 61 62 E3
64
67
ep
el 62 ég
64
Es 6B 67
60 E1 62 Eg
64
65
66 67
lab2rgb
lab2lms
lab2dkl
lab2lab
Cp
C5
Co c1
Figure 8: The reconstruction output by selecting single vector of the entire embedding space. All models are VQ-VAE of K=8 and D=128.
The samples reconstructed with single embedding vector are not perfectly uniform (some small spatial variation is visible in Figure B). To better understand the spatio-chromatic aspect of the encoded information, we again drew sample of spatial size x 2 from the embedding space; this time instead of setting all elements to single vector, we combined two vectors in different spatial directions. The resulting reconstruction for the rgb2lab is illustrated in Figure 9 The embedding space spatial direction is relayed to the networks reconstructed images although the degree of it depends on the pair of embedding vectors. For instance, the horizontal combination of e0 - e7 results in two stripes of colour, while e0 e2 turn into three stripes. This is naturally due to the fact that embedding vectors encode beyond chromatic information, but also the distinct nature of spatio-chromatic combination the decoder learns.
7

---

Under review as a conference paper at ICLR 2021
Horizontal
Diagonal
Vertical
00
ex
e
e
e:
e4
e.
e ez
The reconstruction by pairwise combination of embedding vectors in different spatial directions. ColourConvNet rgb2lab with K=8 and D=128. In all cases a sample of spatial size x was drawn from the embedding space. Horizontal: the top elements set to vector 6; and bottom 6j- Diagonal: the principal diagonal 6 and off-diagonal ey. Vertical: the left elements e; and right ej.
Figure 9:
5.2 LINEAR TRANSFORMATION
Three exemplary reconstructions by rgb2dkl network are illustrated in Figure I0 (for other Colour- ConvNets refer to Sec. D.2). Panel A corresponds the full embedding space and B-D show examples of reconstructions with distinct vector lesions causing clear visible effects. In B, only the lightness of bright pixels is reduced (attend to pixels outside the window and around light bulbs). In C & D, lesioning en and 62, turns reddish and blueish pixels into achromatic. This is in agreement to colour of rgb2dkl eo and e2 in Figure 8
We hypothesised that the changes induced by lesion could be approximated by linear transfor- mation mapping the pixel distribution of the full reconstruction onto the lesion image. To compute these transformations, we used a multi-linear regression finding the best linear fit for the 1% of most affected pixels. The resulting 3 x 3 matrix is a linear transformation in CIE L*abb colour space. We have illustrated the result of applying these linear transformations on the right side of Figure 10 Panel E corresponds to the full RGB cube (essentially the CIE **aa#h* planes limited by RGB gamut). In F-H the very same points are plotted transformed by the model of lesioned vector.
Overall, lesions are closely approximated by a linear transformation: on average accounting for 97% of the total variance in the lesion effect (the lowest bound was 86%). This visualisation offers an intuitive interpretation of the learnt representation within the embedding space. In the images of the second row (panel B), contrast in bright pixels is reduced and colour is little modified. We can observe this in its corresponding CIE L planes (e.g. attend the b** plane in F where the overall chromaticity structure is retained). In C. red pixels turn grey also evident in its corresponding CIE bb* planes (panel G) where red coordinates are collapsed.
The geometrical properties of transformation can be captured by the relative norms of its eigen- values. For instance, zero-value eigenvalues indicate the extreme case of singular matrix, corre- sponding to a linear transformation projecting three-dimensional space onto lower dimensions. We quantified this by defining a singularity index (Philipona & O' O'regan| 2006). Consider a transforma- tion matrix T approximating the lesion effect on the image colour distribution. Let X1, X and As be the three eigenvalues of T,, such that || > ||al|| > ||aal. The singularity index is defined as: SI = 1 ANz This index captures the essence of these transformations. On the one hand, the low value of SI in F suggests the global shape of colour space is retained while its volume is reduced. On the other hand, high values of SI in panels G and H indicate the near collapse of a dimension.
5.3 HUE SHIFT
We further quantified the impact of vector lesion by computing the difference in CIE L*a*b* be- tween the full reconstructed image and lesioned one. The average difference over all pixels for rgb2dkl is illustrated in Figure (refer to Sec. D.3 for other ColourConvNets). The results of hue shift analysis restate the interpretation of learnt representation. For instance, the direction of shift in eg is limited to the first quadrant of the chromaticity plane (red pixels). The e vector largely
8

---

Under review as a conference paper at ICLR 2021
Reconstructed Image
CIE = Planes
Fhill model
100
100
100
0.09
B
F
Lesion
1000
100
100
100
100
R²
0.99 91
0.99
C
100
Lesion 6a
100
100
100
100
R²
0.99
0.93
D
100
Lesion 6/2
L*
L*
Figure 10: The lesion effect visualisation for the rgb2dkl {e) 8x182 Left, reconstructed images by A: the full model; B-D: the lesion embedding space. Right, scatter plots of pixels in CIE *** coordinates of E: the entire RGB cube; F-H: after applying the linear model to the RGB cube.
encodes the low-luminance information (the negative direction in the I* axis). The e2 vector pre- dominantly influences the blue pixels (the negative direction in the b* axis). Similar colours emerge for rgb2dkl eo. and e2 in Figure
Luminance (L') shit
Red-green (ar) shift
Yellow blue shift
100
C.
Index
vector
Index
vector
Index of lasioned vector
Figure 11: Average hue shifts for rgb2dkl {e) 8 in CIE L coordinates. Black- and
red-bars indicate significant impact on the luminance- or chromatic-channels respectively.
6 CONCLUSION
We proposed the unsupervised colour conversion task to investigate colour representation in deep networks. We studied the impact of colour on the encoding capacity of autoencoders, specifically VQ-VAEs whose feature representation is constrained by a discrete bottleneck. The comparison of several NolourConv/Nts exhibits advantage for decorrelated output colour space. This is evident qualitatively and measured quantitatively with five metrics. We discussed this benefit within the framework of efficient coding and histogram equalisation. These findings might contribute to our understanding of why the brain's 's natural network has developed the opponent representation. We further explored the networks' internal representation by means three methods. Our analyses suggest: (i) the colour transformation is performed at the encoding stage prior reaching the em- bedding space, (ii) despite the spatio-chromatic nature the constituent vectors, many manifest a clear effect along one colour direction that can be modelled by parsimonious linear model.

---

Under review as a conference paper at ICLR 2021
ACKNOWLEDGEMENTS
Use unnumbered third level headings for the acknowledgments. All acknowledgments, including those funding agencies, go at the end of the paper.
REFERENCES
Horace Barlow. Possible principles underlying the transformation of sensory messages. In Sensory communication, pp. 217-234, Cambridge, UK, 1961. MIT Press.
David Bau, Bolci Zhou, Aditya Khosla, Aude Oliva, and Antonio Torralba. Network dissection: Quantifying interpretability of deep visual representations. Proceedings the IEEE conference computer vision and pattern recognition, pp. 1144-6549, 2017.
David Bau, Jun-Yan Zhu, Hendrik Strobelt, Agata Lapedriza, Bolei Zhou, and Antonio Torralba. Understanding the role of individual units in deep neural network. Proceedings of the National Academy of Sciences, 2020.
Marcelo Bertalmío. From image processing to computational neuroscience: neural model based on histogram equalization. Frontiers in computational neuroscience, 8.71,2014
Piotr Bojanowski, Armand Joulin, David Lopez-Paz, and Arthur Szlam. Optimizing the latent space of generative networks. arXiv preprint arXiv: #rXivv/707.057700 2017.
Ali Borji. Pros and cons of gan evaluation measures. Computer Vision and Image Understanding, 179:41-65 65, 2019.
M. Bratkova, S. Boulos, and P. Shirley, orgb: A practical opponent color space for computer graph- ics. IEEE Computer Graphics and Applications, 29(1):42-55 55, 2009.
Gershon Buchsbaum and Gottschalk Allan. Trichromacy, opponent colours coding and optimum colour information transmission in the retina. Proceedings of the Royal society of London. Series B. Biological sciences, 220(1218):89-113) 113, 1983.
Ayan Chakrabarti. Color constancy by learning to predict chromaticity from luminance. In Advances in Neural Information Processing Systems, pp. 163-171, 2015.
Zezhou Cheng, Qingxiong Yang, and Bin Sheng. Deep colorization. In Proceedings of the IEEE International Conference on Computer Vision, pp. pp. 415-423, 2015.
M Chirimuuta et al. The uses of colour vision: Ornamental, practical, and theoretical. Minds and Machines, 25(2):213-229, 2015.
CIE. Recommendations on uniform color spaces, color-difference equations, psychometric color terms. Paris:CIE, 1978.
Dan Cireşan, Ueli Meier, Jonathan Masci, and Jürgen Schmidhuber. Multi-column deep neural network for traffic sign classification. Neural networks, 32:333-338, 2012.
Kostadin Dabov, Alessandro Foi, Vladimir Katkovnik, and Karen Egiazarian. Image denoising by sparse 3-d transform-donet collaborative filtering. IEEE transactions on image processing : publication the IEEE Signal Processing Society, 16:2080-95, 09 2007.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hi- erarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Andrew M Derrington, John Krauskopf, and Peter Lennie. Chromatic mechanisms in lateral genic- ulate nucleus of macaque. The Journal of physiology, 357(():241 1984.
Martin Engilberge, Edo Collins, and Sabine Süsstrunk. Color representation in deep neural net- works. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 7786-2790. IEFE, 2017.
10

---

Under review as a conference paper at ICLR 2021
Alban Flachot and Karl R Gegenfurtner. Processing of chromatic information in deep convolu- tional neural network. JOSA A, (5(4):B334-B3444 2018.
David H Foster, Iván Frann- Sérgio Nascimento, and Kinjiro Amano. Coding efficiency of cie color spaces. In Color and Imaging Conference, volume 2008, pp. 285-288. Society for Imaging Science and Technology, 2008.
Huazhu Fu, Boyang Wang. Jianbing Shen, Shanshan Cui, Yanwu Xu, Jiang Liu, and Ling Shao. Evaluation of retinal image quality assessment networks in different color-spaces. In Interna- fional Conference on Medical Image Computing and Computer-Assiste Intervention, pp. 48-56. Springer, 2019.
Leon A Gatys, Alexander S Ecker, Matthias Bethge, Aaron Hertzmann, and Eli Shechtman. Con- trolling perceptual factors in neural style transfer. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3985-3993, 2017.
Karl Gegenfurtner and Daniel C Kiper. Color vision. Annual review of neuroscience, 181-260, 2003.
Karl R Gegenfurtner and Jochem Rieger. Sensory and cognitive contributions of color to the recog- nition of natural scenes. Current Biology, 10(13):805 808, 2000.
Karl R Gegenfurtner and Lindsay Sharpe. Color vision. Cambridge University Press, Cambridge, UK, 1999.
Ethan Harris, Daniela Mihai, and Jonathon Hare. Spatial and colour opponency in anatomically constrained deep networks. arXiv preprint arXiv:1910.11080 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of CVPR, pp. 770-778, 2016.
Hyun-Koo Kim, Ju H Park, and Youl Jung. An efficient color space for deep-learning based traffic light recognition. Journal of Advanced Transportation, 2018, 2018.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6988 62,80000, 2014.
Diederik P Kingma and Max Welling. Auto-encoding variational bayes. 13126113 2013.
arXiv preprint
Alexander Kirillov, Kaiming He, Ross Girshick, Carsten Rother, and Piotr Dollár. Panoptic segmen- tation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 9404-9413, 2019.
JJ Koenderink and Andrea I van Doom. Perspectives colour space. Oxford University, 2003.
Mark Kramer. Nonlinear principal component analysis using autoassociative neural networks.
AIChE journal, 37(2):233-243 1991.
Calls-Valls, Valero Laparra, Sandra Jiménez, Gustavo Camps- Valls, and Jesús Malo. Nonlinearities and adap- tation of color vision from sequential principal curves analysis. Neural Computation, 24(10):
2751-2788, 2012.
Kaouthar Larbi, Wael Ouarda, Hassen Drira, Boulbaba Ben Amor, and Chokri Ben Amar. Deep- colorfasd: Face anti spoofing solution using multi channeled color spaces enn. In 2018 IEEE International Conference Systems, Man, and Cybernetics (SMC), pp. 401 1011-0016. IEEE, 2018.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Learning representations for auto- matic colorization. In European Conference on Computer Vision, pp. 577-593, Springer, 2016.
Gustav Larsson, Michael Maire, and Gregory Shakhnarovich. Colorization as proxy task for visual understanding. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 6874-6883, 2017.
11

---

Under review as a conference paper at ICLR 2021
Simon Laughlin. A simple coding procedure enhances neuron's information capacity. Zeitschrift für Naturforschung c, 36(9-10):910-99) 1981.
Te-Won Lee, Thomas Wachtler, and Terrence J Sejnowski. Color opponency constitutes sparse representation for the chromatic structure of natural scenes. In Advances in Neural Information Processing Systems, pp. 866-872, 2001.
Wei Li, Rui Zhao, Tong Xiao, and Xiaogang Wang. Deepreid: Deep filter pairing neural network for person re-identification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 152-159, 2014.
Timothy Lillicrap and Konrad P Kording. What does it mean to understand neural network? arXiv preprint arXiv:1907.06777 2019.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pp. 740-755. Springer, 2014.
Ziwei Liu, Ping Luo, Xiaogang Wang, and Xiaoou Tang. Deep learning face attributes in the wild. In Proceedings of the IEEE international conference on computer vision, pp. 3730-3738, 2015.
Fujun Luan, Sylvain Paris, Eli Shechtman, and Kavita Bala. Deep photo style transfer. In Pro- ceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4990-4998, 2017.
Jesus Malo. Information flow in color appearance neural networks. arXiv:1912.1209,, 2019.
arXiv preprint
Dmytro Mishkin, Nikolay Sergievskiy, and Jiri Matas. Systematic evaluation of convolution neural network advances on the imagenet. Computer Vision and Image Understanding, 161:11-19 2017.
Ali Mosleh, Avinash Sharma, Emmanuel Onzon, Fahim Mannan, Nicolas Robidoux, and Felix Heide. hardware-innthe-lou end-to-end optimization camera image processing pipelines. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp.
7529-7538, 2020.
David L Philipona and J Kevin O'regan. Color naming. unique hues, and hue cancellation predicted from singularities reflection properties. Visual neuroscience, 23(3-4):331-333 2006.
W Pratt. Digital image processing iiley-interscience, 2007.
J. Preiss, Fernandes, and P. Urban. Color-image quality assessment: From prediction to optimiza- tion. IEEE Transactions on Image Processing, 23(3):1366-1378, 2014.
Alec Radford, Luke Metz, and Soumith Chintala. Unsupervised representation learning with deep convolutional generative adversarial networks. arXiv preprint
2015.
Ivet Rafegas and Maria Vanrell. Color encoding in aetooicaall--nsspree convolutional neural net- works. Vision research, 15:77-17,2015
E. Reinhard, M. Adhikhmin, B. Gooch, and P. Shirley. Color transfer between images. IEEE Computer Graphics and Applications, 21(5):34 2001.
Daniel L Ruderman, Cronin W Thomas, and Chiao Chuan Statistics of cone responses to natural images: implications for visual coding. JOSA A. 15(8):0036-000 2045, 1998.
Tim Salimans, Ian Goodfellow, Wojciech Zaremba, Vicki Cheung, Alec Radford, and Xi Chen. Improved techniques for training gans. In Advances in neural information processing systems, pp. 2234-2242, 2016.
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang- Chiih Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 4510 4510-4520, 2018
12

---

Under review as a conference paper at ICLR 2021
Peter H. Schiller and Joseph G Malpeli. Properties and tectal projections of monkey retinal ganglion cells. Journal of Neurophysiologyy, 40(2):428 445, 1977.
Gaurav Sharma, Wencheng Wu, and Edul N Dalal. The ciede2000 color-difference formula: Im- plementation notes, supplementary test data, and mathematical observations. Color Research & Application, 30(1):21-30, 2005.
Chengyao Shen, Xun Huang, and Qi Zhao. Predicting eye fixations on webpage with an ensem- ble of early features and high-level representations from deep network. IEEE Transactions on Multimedia, 17(11):2044-299, 2015.
Katarzyna Studa-Krzywice Marianna Boros, Paolo Bartolomeo, and Christoph Witzel. The bio- logical bases of colour categorisation: From goldfish to the human brain. Cortex, 2019.
Lucas Theis, Aäron van den Oord. and Matthias Bethge. A note on the evaluation of generative models. arXiv preprint arXiv:1511.01880 2015.
Naftali Tishby and Noga Zaslavsky. Deep learning and the information bottleneck principle. In 2015 IEEE Information Theory Workshop (ITW), pp. 1-5. IEFE, 2015.
Avinash R Vaidya, Maia S Pujara, Michael Petrides, Elisabeth A Murray, and Lesley K Fellows. Lesion studies in contemporary neuroscience. Trends in cognitive sciences, 2019.
Aaron van den Oord, Oriol Vinyals, et al. Neural discrete representation learning. In Advances in Neural Information Processing Systems, pp. 6306-6315, 2017.
Zhou Wang, Bovik, Sheikh, and E. Simoncelli. Image quality assessment: from error visibility to structural similarity. IEEE Transactions on Image Processing, 13(4):600-612, 2004.
Felix A Wichmann, Lindsay A Sharpe, and Karl R Gegenfurtner. The contributions of color to recognition memory for natural scenes. Journal of Experimental Psychology: Learning. Memory, and Cognition, 28(3):509, 2002.
Christoph Witzel and Karl R Gegenfurtner. Color perception: Objects, constancy, and categories. Annual Review of Vision Science, 2018.
Li Zhaoping. Theoretical understanding of the early visual processes by data compression and data selection. Network: computation in neural systems, 17(4):301-334, 2006a.
Li Zhaoping. Theoretical understanding of the early visual processes by data compression and data selection. Network: computation in neural systems, 17(4):301-334 2006b.
13

---

Under review as a conference paper at ICLR 2021
A QUALITATIVE COMPARISON
Along with the quantitative evaluations reported the manuscript, the benefits of uttilising decorre- lated colour space for the network's 's output can be appreciated qualitatively (see Figure A.I). These are representative samples from the COCO dataset Lin el. (2014). The Jupyter-Notebook scripts in the supplementary materials provide more example Overall, the rgb2dkl and rgb2lab VQ-VA VA-VAEs generate more coherent images. For instance, in the first row of Figure [A.] the rgb2rgb output contains a large amount of artefacts on walls and ceilings. In contrast, the output of rgb2dkl and rgb2lab are sharper.
Original
rgb2rgb
rgb2dkl
rgb2lab
Figure A.1: Qualitative comparison of three VQ-VABs of K=8 and D=128.
'The weights of all trained networks and image outputs of lesion study exceed the 100MB upload limit, but they publicly availahle for interested readers under this link https: //www dropbox. com/ sh/ e 913p3uot94400F3 httpa//www.....././o/.///////naa
14

---

Under review as a conference paper at ICLR 2021
B LOSS FUNCTION
The loss function (Eq.) is computed in ColourConvNets' output colour space between the ground- truth and network's output. Figure B.1 reports the evolution of losses for all VQ- of K-8 and D=128. The convergence of losses comparable across all networks regardless of their input-output colour space.
kma7rgh
24-
Ima2lims
rgb2mms
rgb 2old
Ima
Ima2lab
rga2iabb
3.5
dki2rgh
2.2
die2ilins
1.7
1.6
0.8-
0.0
14-
0.6
1.5
tpoch
Epoch
Epoch
Epach
Figure B.1: Evolution losses for VQ- VAEs of K=8 and D=128. In each panel, the ColourConvnets have the same output space. Across panels, curves of the same colour have the same input space.
C LOW-LEVEL EVALUATION
Colour Difference
PSNR t
SSIM
48225
D-120
100120
6:123
Eb:20
1206130
66:28
2484128
Figure C.I: Low-level evaluation for embedding spaces different size (COCO validation-set),
Colour Difference +
PSNR 1
SSIM
Output Ims
Output
Output
rgb
dki
lab
rgb
Ims
dkl
lab
90
06
rgb
6.8
9.1
5.9
5.3
85
22.1
21.3
22.8
23.0
rgb
0.60
0.62
0.63
0.63
06
-80
Inpurt
Ims
-15
Input
lms
22.7
21.5
22.6
22.9
22.2
Input Ims
-10
06
dkl
6.8
9.0
6.1
okl
22.0
21.3
22.5
22.5
dkl
0.59
0.61
0.62
0.61
06
22.6
006
lab
6.0
8.8
5.8
5.7
lab
22.7
21.3
22.7
22.6
lab
0.62
0.61
0.63
0.62
Figure C.2: Low-level pairwise comparison of two groups of input-output colour spaces (COCO validation-set). Figures are averaged over two embedding spaces 8 x 8 and 8 128.
D INTERPRETING THE EMBEDDING SPACE
D.1 SINGLE FEATURES
The hues obtained from single vectors of VQ-VAE with K=8 and D=8 is reported in Figure D.I The effect observed for larger ColourConvNets (i.e. networks with an identical output colour space sharing similar set of hues arranged in different order) is less evident here. This might be due to the dimensionality the embedding space. This regime consists vectors of 8 elements, whereas in the previous regime (Figure the dimensionality of vectors is 128.
15

---

Under review as a conference paper at ICLR 2021
Colour Difference Output
PSNR 1
SSIM Output dki
Output
rgb
Ims
dki
rgb
Ims
dki
lab
lms
lab
rgb
6.0
8.9
5.1
rgb
25.9
23.3
27.3
27.9
rgb
0.79
0.76
0.83
0.82
-08
40
Input Ims
6.0
9.4
5.9
5.0
Imput Ims
26.6
23.2
26.4
27.1
Input Ims
0.76
0.73
0.76
0.77
10
dki
6.1
9.1
5.1
4.9
25.7
27.1
dkl
0.77
0.77
0.81
0.80
-60
lab
8.9
5.2
lab
lab
0.78
0.81
0.81
Figure C.3: Low-level pairwise comparison of two groups of input-output colour spaces (CelebA validation-set). Figures are averaged over two embedding spaces 8 and 128.
rgb2rgb
rgb2lms
rgb2dkl
rgb2lab
Eg
E3 E4 E5 Ims2rgh
e7
E1
E2 E3 eg
E1 E2 64 E5 Ims2dki
E6
6o
E3 E4
Ims2lms
Ims2lah
Cp C1 C2 C3 64
6
C6 67
co 61 62 C3
64
65 66 e7
Cn C1 C2 C3 C4 C5 c6 C7
Co 62 63 64 c5 66 67
dkd2rgb
dkl2lms
dkd2dkl
dkl2lab
eg 61 C2 E3 E1 C5 66 e7
eo 61 62 E3
E1
Eg
e6 eT
eo 61 62 63 61
EG 6p 67
6o E1 E2 E3 6.1 Es 66 E7 lab2lab
lab2rgb
lab2lms
lab2dkl
CD
C4
02
C6
C1
C4
C5 C6
6o
E3
Figure D.I: The reconstruction output by selecting single vector of the entire embedding space. All models are VQ-VAE of K=8 and D=8.
D.2 LINEAR MODELLING VECTOR LESIONS
In order to understand the features learnt by the colour conversion networks, we exercised the "le- sion" technique. consists of silencing the embedding space's vectors one at time. We explored whether vector lesion can be modelled by simple linear transformation. We estimated the trans- formation matrix that maps the pixel distribution of the full reconstruction onto the lesion image (refer to Section 4 in the manuscript). To our surprise, this simple parsimonious modelling can capture large portion of the vector's 's encoding. We present qualitative results for VQ-VAEs with K and D 128 in the colour conversion networks rgb2dkl (Figure D.2). rgb2lab (Figure D.3) and rgb2rgb (Figure D.4).
The top of Figure D.2 illustrated three examples from the COCO dataset reconstructed with the full model of rgb2dkl. The following rows depict the reconstruction output of lesion technique exercised on each embedding vector e; (the "Lesion output" column), alongside with the linear modelling estimation (the "Linear model" column) obtained by applying the linear transformation the full reconstructed image. On the bottom right corner, we have reported the fitness of the mode, correlation in the CIE L*a#b* colour coordinates (r) between the colour pixels of the lesion output and the linear model. The overall fitness is very high for such a simple model. Even cases with lower correlations, output. For instance, the indoor scene for en obtains r 0.81, however, can appreciated that the linear model accounts well for the disappearance of red pixels in the lesion output. This is also evident in the kite picture of E2 where blue pixels have vanished or the bench picture of es with green pixels.
can observe that the model captures well the characteristics of the lesion
Naturally, there are limits to this linear modelling. For instance, the excess of chromaticity (pink and blue colours) in the indoor scene of eg is not fully captured by its linear model. The most extreme can be observed in the kite picture of eg for the rgb2lab model (Figure D33) where the non-linear nature of lesion output is not accounted for in the linear model. Nevertheless, these parsimonious
16

---

Under review as a conference paper at ICLR 2021
Full model
Lesion output
Linear model
Lesion output
Linear model
Lesion output
Linear model
Veater di
r=
m0.95
r=0.7
Vectorr es
T=9.8
0099
r=9.7
Vector eog
0=99
Vector 63
F
0.93
r=
Vector
Vectar CG
0992
-0.6
r=
Vector CG
P=.78
00.77
P=0.9
Vector or
Figure D.2: The linear modelling of vector lesion for rgb2dkl VQ-VAE of K=8 and D=128. The denoted r on the bottom right corner of an image is the measure of transformations" fitness.
transformations reveal great details about the information encoded by each vector deserving more thorough investigation future studies.
we have illustrated the impact of each linear transformation applied to the entire RGB cube. This gives an intuitive idea of what each vector performs simple glance. Absence some vectors results in the collapse of chromatic direction. Others shear, shrink or expand the colour space.
In Figure D.5
D.3 HUE SHIFT
We quantified the chromatic shifts (in CIE L$a*b*) between the reconstructed image of full embed- ding space and lesion one. The differences computed for all pixels over hundred random images from the COCO dataset are illustrated in Figure D.6
17

---

Under review as a conference paper at ICLR 2021
Full mode))
Lesion output
Linear model
Lesion output
Linear model
Lesion output
Linear model
Vector do
0.94
00.96
Vector eg
r=9.88
00996
r=0,88
Vector eg
0
0093
r=
Vector
Veater 44
7=
m=..4
0=099
Vector es
7=997
0=998
0=
Vector CG
0=
Vector Cr
r=1.00
1.00
r=
Figure D.3: The linear modelling of vector lesion for rgh2lab VQ-VAE of K=8 and D=128. The denoted the bottom right corner of image the measure of transformations fitness.
18

---

Under review as a conference paper at ICLR 2021
Full modde)
Lesion output
Linear model
Lesion output
Linear model
Lesion output
Linear model
Vector do
0-98
00.97
Vector eg
r=9.9
00998
r=
Vector eg
0=97
0090
r=
Vector
Veater 64
7=,,
mm..79
r=
Vector es
0=997
0
Vector CG
0=
0=,99
Vector Or
r=1.00
0099
r=
Figure D.4: The linear modelling of vector lesion for rgb2rgb VQ-VAE of K=8 and D=128. The denoted the bottom right corner of image the measure of transformations' fitness.
19

---

Under review as a conference paper at ICLR 2021
RGB Cube in CIE L **
-33
100
100
rgh2rgb
rgb2dkl
rgb2lab
Vector Co
100
200
189
200
20
-5
Vector e3
106
130
200
Voctor
Vector 0g
200
Vector 64
Vector CG
100
15
Vector 6G
10c
176
16
Vector 07
Figure D.5: The impact of vector lesion on RGB cube plotted in CIE *b* coordinate,
20

---

Under review as a conference paper at ICLR 2021
Luminance (L*) shift
rgb2dkl
rgb2lab
Lominance (L*) shift
5t.
1gb22gb
Index lesioned
Rod green (37) shit.
green (ar) shhtt
Rod green (an) shitt
0
20
10
-10
2
4
7
3
5
6
2
7
Index lesioned
Index lesioned vector
Index lesioned
Yellow-biue (6+) shift
blue (6-1) shitt
Yellow blue (6*) shitt
5-
15
Index of lesioned vector
Index of lesioned vector
Index of lesioned vector
Figure D.6: Average hue shifts CIE = colour space for each vector lesion. Numbers in the r-axis denote the indices of the embedding vectors. All networks are of {e} E 888128 Black- and red-bars indicate significant impact on the luminance- or chromatic-channeis respectively.
21


Reviews:
[{"review_id": "uQtXmjYkaIp", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The motivation for this paper is quite hard to understand. A VQ-VAE is directly applied to convert an image from one colour space to another one. However, the colour space transform is human-defined, usually involving linear and a few non-linear (like selecting the maximum value is HSV) procedures. In this case, the latent space of VQ-VAE should be collapsed into this simple equation easily. The analysis of this paper does not teach us any additional knowledge.\nThe motivation of finding a better embedding space of colour is admirable, unfortunately, the analysis and methodology does not support the motivation. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "7oTh550JlUS", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 7, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary\n-------------\n \nThe authors analyze the quality of known color representations (non-opponent and opponent) in terms of the quality of the reconstructed images when spatio-chromatic information is constrained by a bottleneck of a discrete (quantized) representation of reduced dimensionality. \n \nThe quantized representation and encoding-decoding transforms are defined by a loss function optimized through an autoencoding tool (in particular a Vector-Quantized Variational Autoencoder). The quality of the reconstructed images is measured in terms of (1) low-level similarity metrics (some of them with perceptual meaning), and (2) performance in higher-level tasks such as classification and segmentation. \n \nThe conclusion is that perceptually meaningful opponent representations such as DKL and CIELab are better suited to the imposed bottleneck as opposed to color representations related to retinal sensors such as RGB or LMS. \n\nGeneral opinion and recommendation\n-----------------------------------------------------\n\nI think the metodology and findings are really interesting to understand why the brain may have developed the opponent \nrepresentations that have better performance in the presented experiments. The use of an autoencoding tool to enforce the minimization of the loss suggests that comparison between the color representations is fair.  \n\nHowever, this message (pointing out the advantages of opponent representations wrt more trivial non-opponent representations) is not clearly stated. The biological meaningfulness of the selected constraints is not discussed either. Presentation is confusing at many points (see specific list below), but this can be fixed. Therefore,  I think the work should be accepted after proper clarifications and removing some misconceptions.\n\nMajor Points\n----------------\n\nThe goal of the paper (in my view, a fair comparison of different color spaces in a bottleneck context using an appropriate optimization tool) is not clearly stated.\n\nInstead some sentences in the abtract and introduction of the paper suggests that color representation is learnt. For instance, in the abstract and intro it is said \"We propose a novel unsupervised task —colour conversion— to explicitly examine the colour representation learnt by deep networks (referred to as ColourConvNets).\" and \"the structure of internal representation provides insights on how this [color] transformation is performed within a neural network\". At this point the reader may think that the proposed autoencoder will learn a specific color representation well suited for certain goal(s). However, the autoencoder is not learning color representations (they are imposed at input and output), the autoencoder is only imposing certain bottleneck and hence, it is a controlled way to assess the suitability of the considered color representations in constrained settings. In fact, spatial and chromatic parts of visual information are mixed in the vectors of the inner representation of the considered autoencoders. This (hard to interpret) mixture necessarily comes from the reduced dimension of the vectors. Therefore, strictly speaking, there is no \"pure color representation learnt\" in the inner representation of the autoencoder. This misunderstanding about \"learning a color representation\" when it is actually a spatio-chromatic representation imposed by the constraints and the selected input-output color spaces happened to me, and only disappeared at the end of page 3 and 4. This misunderstanding should be avoided in abstract and intro.\n\nAnother confusing description is talking about the \"correlation\" and \"decorrelation\" properties when presenting the considered spaces (in section 2.2) just after talking about learning efficient representations through the autoencoder. Mentions to the \"efficiency\" of color space through citations to [Buchsbaum83,Ruderman98,Lee01] should appear later in the discussion (not as early as in section 1.1).\nReaders aware of Barlow's efficient coding hypothesis that leads to transforms that favour decorrelation and equalization\n(which in color lead to PCA-like transforms [Buchsbaum83], and nonlinear equalizations that explain chromatic adaptation [Laparra12]) may wonder why the cost function did not include decorrelation or independence measures. I would suggest talking about the decorrelation properties only in the discusion (do not mention in section 1.1 and remove the left part of fig 2 -devoted to highlight correlation and decorrelation-, and make these points in an expanded \"performance advantage\" section). Actually, decorrelation and equalization properties of these color spaces has been measured in information theoretic units by Foster et al. 2008 and by Malo 2020. I think these references [Buchsbaum83,Ruderman98,Lee01,Foster08,Laparra12,Malo20] should be included in this decorrelation-equalization discussion.\n\nAnother confusing statement is the apparent contradiction between this statement \"the conversion of RGB images\ninto other colour spaces yield to no performance improvement in ImageNet (Mishkin et al., 2017)\" and the interesting \nfindings done in this work. It is important to stress that maybe results in (Mishkin et al., 2017) were not subject to big enough dimensionality constraints and then proper representation of color was not that relevant.\n\nIt is unclear how Fig 8 and Fig. C.1 were computed. To me this is really important to clarify the non-trivial mixture of spatial and chromatic information in the vectors. From the text, I guess, a single codevector was used to decode the image.\nBut, given the dimension of the codevectors (bigger than 3), they encode not only chromatic information but also spatial information. Then, how is it possible to obtain uniform color (as in figs 8 and C.1) with no spatial variation from a single codevector? \n\nMinor Points\n------------\n\n* it is important to stress that the advantage of the proposed metodology to compare color spaces is that additional constraints can be included (such as entropy, energy, wiring, etc...). This framework able to encompass additional constraints is relevant to understand why the considered representations could have emerged in the brain.\n \n* Correlation = 1 between L and M seems like too much. Is this correct?\n\n* The value of the loss function is comparable in the different cases after training? This would be necessary for a fair comparison, isnt it?\n\nReferences:\n----------------\n\n[Foster08] D.H. Foster, I. Marin-Franch, and S.M.C. Nascimento. 2008. Coding efficiency of CIE color spaces. In Proc. 16th Color Imag. Conf. Soc. Imag. Sci. Tech., 285–288\n\n[Laparra12] V. Laparra, S. Jiménez, G. Camps and J. Malo. 2012. Nonlinearities and adaptation of color vision from sequential principal curves analysis. Neural Computation 24, 10 (2012), 2751–2788\n\n[Malo20] J. Malo. Information Flow in Color Appearance Neural Networks. Accepted in Entropy Conference 2020 https://arxiv.org/abs/1912.12093", "extra_scores": {}, "extra_text": {}}, {"review_id": "1FdVdkkNqs", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes to study an interesting problem of how color informaiton is structured in the variational autoencoders (VAEs). Several instances of VAEs are trained in an unsupervised manner to perform color space conversion. Both low-level and high-level evaluations are performed to study the local statistics and global content of converted images. Several interesting conclusions are drawn from the experiments that help interpret the encoding process of autoencoders.\n\nOverall, this paper studies an interesting problem and presents several insightful conclusions. My only concern is how significant the proposed method/task is, and how significant insights these conclusions could provide. To address this, it may show some applications based on the proposed task/conclusions.\n\n\nI only have some minor issues.\n\n1. It could be better to include results of state of the art methods on, e.g., object classification and scene segmentation. This could further show the potential application of proposed method.\n\n2. encode -> encodes, Line 7, Page 2\n\n3. Figure 1 is not referred to in the main text. Besides, it could be better to prodive more details in the caption.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper proposes a novel unsupervised task of colour conversion. In this respect, the task
becomes more like a regression problem -- rather than autoencoding the decoder needs to reconstruct the pixels in a different color system. 
While the idea is potentially interesting, there are fundamental problems with the paper:

* The motivations of the paper are obscure, (understanding colour representation in complex visual systems? Learning better representations? Disentangling color related information from the rest)
* No analysis is provided to highlight what the novel objective is achieving

The answers of the authors to AnonReviewer1 are not very convincing. As AnonReviewer1 has pointed out, the mapping between the color spaces is typically a simple invertible map so any conclusion that the authors arrive about ‘substantial impact’ could be simply the artefact of the particular architecture choice. The other claim, that ‘the proposed framework is able to encompass additional constraints relevant in understanding why the considered representations could have emerged in the brain’ quite far fetched and speculative at best.

The authors have a point in their reply ii) to AnonReviewer1 but if the claim is about the particular color coding schemata, it would be natural to include simple experiments where some arbitrary 3x3 invertible mapping (e.g. rgb in spherical or cylindrical coordinates) next to other color schemata to make a stronger point.

In point iii) the authors refer to tasks without being very explicit about what the tasks are. Colorization is a known proxy pretext task for learning representations when the downstream classification task is not known a-priori. The paper would have been much more easy to motivate if the authors could demonstrate the merit of the proposed objective using a more extensive and careful representation learning evaluation methodology.

In light of the above points, I feel that the paper needs further iterations to be presented at ICLR.

Decision: Reject