Learning Representation in Colour Conversion

1 INTRODUCTION . Colour is an inseparable component of our conscious visual perception and its objective utility spans over a large set of tasks such as object recognition and scene segmentation ( Chirimuuta et al. , 2015 ; Gegenfurtner & Rieger , 2000 ; Wichmann et al. , 2002 ) . Consequently , colour is an ubiquitous feature in many applications : colour transfer ( Reinhard et al. , 2001 ) , colour constancy ( Chakrabarti , 2015 ) , style transfer ( Luan et al. , 2017 ) , computer graphics ( Bratkova et al. , 2009 ) , image denoising ( Dabov et al. , 2007 ) , quality assessment ( Preiss et al. , 2014 ) , to name a few . Progress in these lines requires a better understanding of colour representation and its neural encoding in deep networks . To this end , we present a novel unsupervised task : colour conversion . In our proposed framework the input-output colour space is imposed on deep autoencoders ( referred to as ColourConvNets ) that learn to efficiently compress the visual information ( Kramer , 1991 ) while transforming the input to output . Essentially , the output y for input image x is generated on the fly by a transformation y = T ( x ) , where T maps input to output colour space . This task offers a fair comparison of different colour spaces within a system that learns to minimise a loss function in the context of information bottleneck principle ( Tishby & Zaslavsky , 2015 ) . The quality of output images demonstrates whether the representation of input-output colour spaces impacts networks ’ encoding power . Furthermore , the structure of internal representation provides insights on how colour transformation is performed within a neural network . In this work , we focused on Vector Quantised Variational Autoencoder ( VQ-VAE ) ( van den Oord et al. , 2017 ) due to the discrete nature of its latent space that facilitates the analysis and interpretability of the learnt features . We thoroughly studied five commonly used colour spaces by training ColourConvNets for all combinations of input-output spaces . First , we show that ColourConvNets with a decorrelated output colour space ( e.g . CIE L * a * b ) convey information more efficiently in their compressing bottleneck , in line with the presence of colour opponency in the human visual system . This is evident qualitatively ( Figures 1 and A.1 ) and quantitatively ( evaluated with three low-level and two high-level metrics ) . Next , we present the interpretation of ColourConvNets ’ latent space by means of three methods reaching a consensus interpretation : ( i ) the colour representation in the VQ-VAEs ’ latent space is determined by the output colour space , suggesting the transformation T occurs at the encoder , ( ii ) each embedding vector in VQ-VAEs encodes a specific part of the colour space , e.g . the luminance or chromatic information , which can be modelled by a parsimonious linear transformation . 1.1 RELATED WORK . The effectiveness of different colour spaces have been investigated in a few empirical studies of deep neural networks ( DNNs ) . Information fusion over several colour spaces improved retinal medical imaging ( Fu et al. , 2019 ) . A similar strategy enhanced the robustness of face ( Li et al. , 2014 ; Larbi et al. , 2018 ) and traffic light recognition ( Cireşan et al. , 2012 ; Kim et al. , 2018 ) . This was also effective in predicting eye fixation ( Shen et al. , 2015 ) . Opponent colour spaces have been explored for applications such as style transfer ( Luan et al. , 2017 ; Gatys et al. , 2017 ) and picture colourisation ( Cheng et al. , 2015 ; Larsson et al. , 2016 ) . Most of these works are within the domain of supervised learning . The most similar approach to our proposed ColourConvNets is image colourisation as a pretext task for unsupervised visual feature learning ( Larsson et al. , 2017 ) . Initial works on colour representation in DNNs revealed object classification networks learn to decorrelate their input images ( Rafegas & Vanrell , 2018 ; Flachot & Gegenfurtner , 2018 ; Harris et al. , 2019 ) . This is a reminiscence of horizontal and ganglion cells that decorrelate retinal signal into colour-opponency before transmitting it to the visual cortex ( Schiller & Malpeli , 1977 ; Derrington et al. , 1984 ; Gegenfurtner & Kiper , 2003 ) . Another set of works reported existence of hue-sensitive units ( Engilberge et al. , 2017 ) that mainly emerge in early layers ( Bau et al. , 2017 ) . Representation of colours in deep networks at intermediate and higher layers is rather understudied . In this article , we specifically focus on the intermediate representation that emerges at the latent space of autoencoders , which to the best of our knowledge has not been reported in the literature . 2 COLOUR CONVERSION AUTOENCODERS . In this article , we propose a novel unsupervised task of colour conversion : the network ’ s output colour space is independent of its input ( see Figure 2 ) . A colour space is an arbitrary definition of colours ’ organisation in the space ( Koenderink & van Doorn , 2003 ) . Thus , the choice of transfor- mation matrix T in ColourConvNets is perfectly flexible to model any desired space , Cin T−−−−−→ Cout , ( 1 ) where Cin and Cout are the input and output colour spaces . This framework offers a controlled environment to compare colour spaces within a complex visual system . Here , we studied their effectiveness in information encoding constrained to a bottleneck . This can be extended to encompass other constraints ( such as entropy , energy , wiring , etc . ) relevant to understanding colour representation in complex visual systems . We further used this structure to compare autoencoder ’ s latent space across colour spaces aiming to decipher the intermediate colour representation within these networks . The proposed framework can also be employed in applications , e.g. , as an add-on optimisation capsule to any computer vision application ( Mosleh et al. , 2020 ) , or as a proxy task for visual understanding ( Larsson et al. , 2017 ) . 2.1 NETWORKS . We studied a particular class of VAEs—Vector Quantised Variational Autoencoder ( VQ-VAE ) ( van den Oord et al. , 2017 ) —due to the discrete nature of its latent embedding space that facilitates the analysis and interpretability of the learnt features , which distinguishes it from others ( Kingma & Welling , 2013 ) . VQ-VAE consists of three main blocks : 1 ) an encoder that processes the input data x to ze ( x ) ; 2 ) a latent embedding space { e } ∈ RK×D , with K vectors of dimensionality D , that maps ze ( x ) onto zq ( x ) by estimating the nearest vector ei to ze ( x ) ; 3 ) a decoder that reconstructs the final output x′ with a distribution p ( x|zq ( x ) ) over the input data ( see the right panel in Figure 2 ) . The loss function is defined as follows , L = log p ( x|zq ( x ) ) + ‖sg [ ze ( x ) ] − e‖22 + β‖ze ( x ) − sg [ e ] ‖22 , ( 2 ) where sg denotes the stop gradient computation that is defined as the identity during the forwardpropagation , and with zero partial derivatives during the back-propagation to refrain its update . The first term in Eq . 2 corresponds to the reconstruction loss incorporating both encoder and decoder ; the second term updates the embedding vectors ; and the third term harmonies the encoder and embedding vectors . The parameter β ∈ R is set to 0.5 in all our experiments . 2.2 COLOUR SPACES . We explored five colour spaces : RGB , LMS , CIE L * a * b * , DKL and HSV . The standard space in electronic imaging is RGB that represents colours by three additive primaries in a cubic shape . The LMS colour space corresponds to the response of human cones ( long- , middle- , and shortwavelengths ) ( Gegenfurtner & Sharpe , 1999 ) . The CIE L * a * b * colour space ( luminance , red-green and yellow-blue axes ) is designed to be perceptually uniform ( CIE , 1978 ) . The DKL colour space ( Derrington-Krauskopf-Lennie ) models the opponent responses of rhesus monkeys in the early visual system ( Derrington et al. , 1984 ) . The HSV colour space ( hue , saturation , value ) is a cylindrical representation of RGB cube designed by computer graphics . The input-output to our networks can be in any combination of these colour spaces . Effectively , our VQ-VAE models , in addition to learning efficient representation , must learn the transformation function from their input to output colour space . It is worth considering that the original images in explored datasets are in the RGB format . Therefore , one might expect a slight positive bias towards this colour space given its gamut defines the limits of other colour spaces . 3 EXPERIMENTS . We trained several instances of VQ-VAEs with distinct sizes of embedding space { e } ∈ RK×D . The training procedure was identical for all networks : trained with Adam optimiser ( Kingma & Ba , 2014 ) ( lr = 2 × 10−4 ) for 90 epochs . To isolate the influence of random variables , all networks were initialised with the same set of weights and an identical random seed was used throughout all experiments . We used ImageNet dataset ( Deng et al. , 2009 ) for training . This is a visual database of object recognition in real-world images , divided into one thousand categories . The training set contains 1.3 million images . At every epoch , we exposed the network to 100K images of size 224 × 224 of three colour channels . Figure B.1 reports the progress of loss function for various ColourConvNets . A similar pattern of convergence can be observed for all trained networks . To increase the generalisation power of our findings , we evaluated all networks on the validationset of three benchmark datasets : ImageNet ( 50K images ) , COCO ( 5K images ) , and CelebA ( ~20K images ) . COCO is a large-scale object detection and segmentation dataset ( Lin et al. , 2014 ) . CelebA contain facial attributes of celebrities ( Liu et al. , 2015 ) . We relied on two classes of evaluation1 : low-level ( Theis et al. , 2015 ) , capturing the local statistics of an image ; high-level ( Borji , 2019 ) , assessing the global content of an image . Low-level evaluation – We computed three commonly used metrics to measure the pixel-wise performance of networks : ( i ) the colour difference CIE ∆E-2000 ( Sharma et al. , 2005 ) , ( ii ) peak signalto-noise ratio ( PSNR ) , and ( iii ) structural similarity index measure ( SSIM ) ( Wang et al. , 2004 ) . High-level evaluation – Pixel-wise measures are unable to capture the global content of an image and whether semantic information remains perceptually intact . To account for this limitation , we performed a procedure similar to the standard Inception Score ( Salimans et al. , 2016 ; Borji , 2019 ) by feeding the reconstructed images to two pretrained networks ( without fine-tuning ) that perform the task of object classification , ResNet50 ( He et al. , 2016 ) , and scene segmentation , Feature Pyramid Network—FPN ( Kirillov et al. , 2019 ) . ResNet50 and FPN expect RGB inputs , thus non-RGB reconstructions were converted to RGB . The evaluation for ResNet50 is the classification accuracy on ImageNet dataset . The evaluation for FPN is the intersection over union ( IoU ) on COCO dataset .

Summary: This paper proposes to study an interesting problem of how color informaiton is structured in the variational autoencoders (VAEs). Several instances of VAEs are trained in an unsupervised manner to perform color space conversion. Both low-level and high-level evaluations are performed to study the local statistics and global content of converted images. Several interesting conclusions are drawn from the experiments that help interpret the encoding process of autoencoders.