Learning Representation in Colour Conversion

1 INTRODUCTION . Colour is an inseparable component of our conscious visual perception and its objective utility spans over a large set of tasks such as object recognition and scene segmentation ( Chirimuuta et al. , 2015 ; Gegenfurtner & Rieger , 2000 ; Wichmann et al. , 2002 ) . Consequently , colour is an ubiquitous feature in many applications : colour transfer ( Reinhard et al. , 2001 ) , colour constancy ( Chakrabarti , 2015 ) , style transfer ( Luan et al. , 2017 ) , computer graphics ( Bratkova et al. , 2009 ) , image denoising ( Dabov et al. , 2007 ) , quality assessment ( Preiss et al. , 2014 ) , to name a few . Progress in these lines requires a better understanding of colour representation and its neural encoding in deep networks . To this end , we present a novel unsupervised task : colour conversion . In our proposed framework the input-output colour space is imposed on deep autoencoders ( referred to as ColourConvNets ) that learn to efficiently compress the visual information ( Kramer , 1991 ) while transforming the input to output . Essentially , the output y for input image x is generated on the fly by a transformation y = T ( x ) , where T maps input to output colour space . This task offers a fair comparison of different colour spaces within a system that learns to minimise a loss function in the context of information bottleneck principle ( Tishby & Zaslavsky , 2015 ) . The quality of output images demonstrates whether the representation of input-output colour spaces impacts networks ’ encoding power . Furthermore , the structure of internal representation provides insights on how colour transformation is performed within a neural network . In this work , we focused on Vector Quantised Variational Autoencoder ( VQ-VAE ) ( van den Oord et al. , 2017 ) due to the discrete nature of its latent space that facilitates the analysis and interpretability of the learnt features . We thoroughly studied five commonly used colour spaces by training ColourConvNets for all combinations of input-output spaces . First , we show that ColourConvNets with a decorrelated output colour space ( e.g . CIE L * a * b ) convey information more efficiently in their compressing bottleneck , in line with the presence of colour opponency in the human visual system . This is evident qualitatively ( Figures 1 and A.1 ) and quantitatively ( evaluated with three low-level and two high-level metrics ) . Next , we present the interpretation of ColourConvNets ’ latent space by means of three methods reaching a consensus interpretation : ( i ) the colour representation in the VQ-VAEs ’ latent space is determined by the output colour space , suggesting the transformation T occurs at the encoder , ( ii ) each embedding vector in VQ-VAEs encodes a specific part of the colour space , e.g . the luminance or chromatic information , which can be modelled by a parsimonious linear transformation . 1.1 RELATED WORK . The effectiveness of different colour spaces have been investigated in a few empirical studies of deep neural networks ( DNNs ) . Information fusion over several colour spaces improved retinal medical imaging ( Fu et al. , 2019 ) . A similar strategy enhanced the robustness of face ( Li et al. , 2014 ; Larbi et al. , 2018 ) and traffic light recognition ( Cireşan et al. , 2012 ; Kim et al. , 2018 ) . This was also effective in predicting eye fixation ( Shen et al. , 2015 ) . Opponent colour spaces have been explored for applications such as style transfer ( Luan et al. , 2017 ; Gatys et al. , 2017 ) and picture colourisation ( Cheng et al. , 2015 ; Larsson et al. , 2016 ) . Most of these works are within the domain of supervised learning . The most similar approach to our proposed ColourConvNets is image colourisation as a pretext task for unsupervised visual feature learning ( Larsson et al. , 2017 ) . Initial works on colour representation in DNNs revealed object classification networks learn to decorrelate their input images ( Rafegas & Vanrell , 2018 ; Flachot & Gegenfurtner , 2018 ; Harris et al. , 2019 ) . This is a reminiscence of horizontal and ganglion cells that decorrelate retinal signal into colour-opponency before transmitting it to the visual cortex ( Schiller & Malpeli , 1977 ; Derrington et al. , 1984 ; Gegenfurtner & Kiper , 2003 ) . Another set of works reported existence of hue-sensitive units ( Engilberge et al. , 2017 ) that mainly emerge in early layers ( Bau et al. , 2017 ) . Representation of colours in deep networks at intermediate and higher layers is rather understudied . In this article , we specifically focus on the intermediate representation that emerges at the latent space of autoencoders , which to the best of our knowledge has not been reported in the literature . 2 COLOUR CONVERSION AUTOENCODERS . In this article , we propose a novel unsupervised task of colour conversion : the network ’ s output colour space is independent of its input ( see Figure 2 ) . A colour space is an arbitrary definition of colours ’ organisation in the space ( Koenderink & van Doorn , 2003 ) . Thus , the choice of transfor- mation matrix T in ColourConvNets is perfectly flexible to model any desired space , Cin T−−−−−→ Cout , ( 1 ) where Cin and Cout are the input and output colour spaces . This framework offers a controlled environment to compare colour spaces within a complex visual system . Here , we studied their effectiveness in information encoding constrained to a bottleneck . This can be extended to encompass other constraints ( such as entropy , energy , wiring , etc . ) relevant to understanding colour representation in complex visual systems . We further used this structure to compare autoencoder ’ s latent space across colour spaces aiming to decipher the intermediate colour representation within these networks . The proposed framework can also be employed in applications , e.g. , as an add-on optimisation capsule to any computer vision application ( Mosleh et al. , 2020 ) , or as a proxy task for visual understanding ( Larsson et al. , 2017 ) . 2.1 NETWORKS . We studied a particular class of VAEs—Vector Quantised Variational Autoencoder ( VQ-VAE ) ( van den Oord et al. , 2017 ) —due to the discrete nature of its latent embedding space that facilitates the analysis and interpretability of the learnt features , which distinguishes it from others ( Kingma & Welling , 2013 ) . VQ-VAE consists of three main blocks : 1 ) an encoder that processes the input data x to ze ( x ) ; 2 ) a latent embedding space { e } ∈ RK×D , with K vectors of dimensionality D , that maps ze ( x ) onto zq ( x ) by estimating the nearest vector ei to ze ( x ) ; 3 ) a decoder that reconstructs the final output x′ with a distribution p ( x|zq ( x ) ) over the input data ( see the right panel in Figure 2 ) . The loss function is defined as follows , L = log p ( x|zq ( x ) ) + ‖sg [ ze ( x ) ] − e‖22 + β‖ze ( x ) − sg [ e ] ‖22 , ( 2 ) where sg denotes the stop gradient computation that is defined as the identity during the forwardpropagation , and with zero partial derivatives during the back-propagation to refrain its update . The first term in Eq . 2 corresponds to the reconstruction loss incorporating both encoder and decoder ; the second term updates the embedding vectors ; and the third term harmonies the encoder and embedding vectors . The parameter β ∈ R is set to 0.5 in all our experiments . 2.2 COLOUR SPACES . We explored five colour spaces : RGB , LMS , CIE L * a * b * , DKL and HSV . The standard space in electronic imaging is RGB that represents colours by three additive primaries in a cubic shape . The LMS colour space corresponds to the response of human cones ( long- , middle- , and shortwavelengths ) ( Gegenfurtner & Sharpe , 1999 ) . The CIE L * a * b * colour space ( luminance , red-green and yellow-blue axes ) is designed to be perceptually uniform ( CIE , 1978 ) . The DKL colour space ( Derrington-Krauskopf-Lennie ) models the opponent responses of rhesus monkeys in the early visual system ( Derrington et al. , 1984 ) . The HSV colour space ( hue , saturation , value ) is a cylindrical representation of RGB cube designed by computer graphics . The input-output to our networks can be in any combination of these colour spaces . Effectively , our VQ-VAE models , in addition to learning efficient representation , must learn the transformation function from their input to output colour space . It is worth considering that the original images in explored datasets are in the RGB format . Therefore , one might expect a slight positive bias towards this colour space given its gamut defines the limits of other colour spaces . 3 EXPERIMENTS . We trained several instances of VQ-VAEs with distinct sizes of embedding space { e } ∈ RK×D . The training procedure was identical for all networks : trained with Adam optimiser ( Kingma & Ba , 2014 ) ( lr = 2 × 10−4 ) for 90 epochs . To isolate the influence of random variables , all networks were initialised with the same set of weights and an identical random seed was used throughout all experiments . We used ImageNet dataset ( Deng et al. , 2009 ) for training . This is a visual database of object recognition in real-world images , divided into one thousand categories . The training set contains 1.3 million images . At every epoch , we exposed the network to 100K images of size 224 × 224 of three colour channels . Figure B.1 reports the progress of loss function for various ColourConvNets . A similar pattern of convergence can be observed for all trained networks . To increase the generalisation power of our findings , we evaluated all networks on the validationset of three benchmark datasets : ImageNet ( 50K images ) , COCO ( 5K images ) , and CelebA ( ~20K images ) . COCO is a large-scale object detection and segmentation dataset ( Lin et al. , 2014 ) . CelebA contain facial attributes of celebrities ( Liu et al. , 2015 ) . We relied on two classes of evaluation1 : low-level ( Theis et al. , 2015 ) , capturing the local statistics of an image ; high-level ( Borji , 2019 ) , assessing the global content of an image . Low-level evaluation – We computed three commonly used metrics to measure the pixel-wise performance of networks : ( i ) the colour difference CIE ∆E-2000 ( Sharma et al. , 2005 ) , ( ii ) peak signalto-noise ratio ( PSNR ) , and ( iii ) structural similarity index measure ( SSIM ) ( Wang et al. , 2004 ) . High-level evaluation – Pixel-wise measures are unable to capture the global content of an image and whether semantic information remains perceptually intact . To account for this limitation , we performed a procedure similar to the standard Inception Score ( Salimans et al. , 2016 ; Borji , 2019 ) by feeding the reconstructed images to two pretrained networks ( without fine-tuning ) that perform the task of object classification , ResNet50 ( He et al. , 2016 ) , and scene segmentation , Feature Pyramid Network—FPN ( Kirillov et al. , 2019 ) . ResNet50 and FPN expect RGB inputs , thus non-RGB reconstructions were converted to RGB . The evaluation for ResNet50 is the classification accuracy on ImageNet dataset . The evaluation for FPN is the intersection over union ( IoU ) on COCO dataset .

Summary: The motivation for this paper is quite hard to understand. A VQ-VAE is directly applied to convert an image from one colour space to another one. However, the colour space transform is human-defined, usually involving linear and a few non-linear (like selecting the maximum value is HSV) procedures. In this case, the latent space of VQ-VAE should be collapsed into this simple equation easily. The analysis of this paper does not teach us any additional knowledge.