Planckian jitter: enhancing the color quality of self-supervised visual representations

1 INTRODUCTION . Self-supervised learning enables the learning of visual representation without the need for any labeled data ( Doersch et al. , 2015 ; Dosovitskiy et al. , 2014 ) . Several recent works learn representations that are invariant with respect to a set of data augmentations , and have obtained spectacular results ( Grill et al. , 2020 ; Chen & He , 2021 ; Caron et al. , 2020 ) , significantly narrowing the gap with supervised learned representations . These works vary in their architecture , learning objective , and optimization strategy , however , they are similar in applying a common set of data augmentations to generate the various image views . The algorithms , while learning to map these different views to the same latent representation , learn complex semantic representation for visual data . The set of transformations ( data augmentation ) that are considered induce a set of invariances that characterizes the learned visual representation . Before deep learning revolutionized the way visual representations were computed , separate features were hand-designed to represent its various properties , leading to research on shape ( Lowe , 2004 ) , texture ( Manjunath & Ma , 1996 ) , and color features ( Finlayson & Schaefer , 2001 ; Geusebroek et al. , 2001 ) . Color features were typically designed to be invariant with respect to a set of scene accidental events , such as shadows , shading , illuminant and viewpoint changes . With the rise of deep learning , feature representations that exploit simultaneously color , shape and texture are learned implicitly and the invariances are a byproduct of the end-to-end training ( Krizhevsky et al. , 2009 ) . As discussed above , the current set of self-supervised learning methods explicitly define a set of invariances ( related to the applied data augmentations ) that are to be learned . In this work , we focus on the current de-facto choice for color augmentations . We argue that they seriously cripple the color quality of these representations and we propose an alternative color augmentation . Figure 1 ( left ) illustrates the currently applied color transformation for a sample image , depicted in the middle of the left-most grid . It is clear that the applied color transformation significantly alters the colors of the original image , both in terms of hue and saturation . One of the justifications in literature for such strong color augmentations is that without large color changes , mapping images to the same latent representation can be purely done based on color and no complex shape features are learned , therefore the best results when using only two transformations are obtained when applying image cropping with a color augmentation ( Chen et al. , 2020a ) . However , looking at the reported example it is evident that a representation that maps these images to the same latent representation can not rely on the object color , and as a results the quality of the color representation learned with such algorithms is expected to be inferior . Therefore , in this paper , we propose another set of color augmentations , shown on the right side of Figure 1 . In addition to introducing more natural variations in the image chromaticity , the proposed color augmentation also affects neutral regions such as the petals shown in the figure , which are left unvaried with the original color transformations . We draw on existing color imaging literature , that aimed to design features that were invariant with respect to illuminant changes that were commonly encountered in real-world scenes ( Finlayson & Schaefer , 2001 ) . Our augmentation , called Planckian jitter , applies a physically realistic illuminant variation to the images . We consider the illuminants that are described by Planck ’ s Law for black body radiation and that are known to be similar to illuminants encountered in real-life ( Tominaga et al. , 1999 ) . In the experimental section , we show that self-supervised features learned with Planckian jitter yield superior features , leading to gains of over 5 % on several downstream color classification tasks . However , since our color augmentation is less extreme , the learned shape features are of lower quality than with the original color jitter . A simple combination of both feature representations leads to huge performance gains with respect to default color jitter of between 10-15 % on several color downstream tasks . In addition , we show that our augmentation method can be applied to several state-of-the-art self-supervised learning methods . Finally , we analyze the color sensitivity of the learned color representations . 2 BACKGROUND AND RELATED WORKS . 2.1 SELF-SUPERVISED LEARNING AND CONTRASTIVE LEARNING . Recent improvements in self-supervision learn a semantically rich feature representation without the need of any labels . In SimCLR ( Chen et al. , 2020a ) similar samples are created by augmenting an input image , while dissimilar are chosen by random . To make contrastive training more efficient MoCo method ( He et al. , 2020 ) and its improved version ( Chen et al. , 2020b ) use a memory bank for learned embeddings which helps for efficient sampling . This memory is kept in sync with the rest of the network during training time by using a momentum encoder . Several methods do not have any explicit contrastive pairs . BYOL ( Grill et al. , 2020 ) propose an asymmetric network by introducing an additional MLP predictor between the outputs of the two branches . One of the branches is kept ” offline ” - updated by a momentum encoder . SimSiam ( Chen & He , 2021 ) goes even further and presents a simplified solution without a momentum encoder . Moreover , it obtains similar high-quality results and does not require a large mini-batch size . We will use the SimSiam method to verify our proposed color augmentation ( we also apply our approach to SimCLR ( Chen et al. , 2020a ) and Barlow Twins ( Zbontar et al. , 2021 ) in the experiments ) . The main part is a CNN-based encoder , learned end-to-end in an asymmetric siamese architecture , where one branch has an additional predictor ( Multi-Layer Perceptron , or MLP ) whose output aims to be as close as possible to the other branch ( see Figure 2 ) . The second branch is not updated during backward propagation . A negative-cosine loss function is used , defined as : L = D ( p1 , z2 ) /2 +D ( p2 , z1 ) /2 ( 1 ) D ( pA , zB ) = − pA ‖pA‖2 · zB ‖zB‖2 , ( 2 ) where z1 , z2 are encoded values respectively for two different augmented versions x1 and x2 of the same image x . Note that in Eq . 1 they are alternated between the two branches , but it is always only the first branch that uses a Multilayer Perceptron , producing either p1 or p2 . Additionally , no contrastive term is present : only the similarity is enforced during learning . 2.2 DATA AUGMENTATION . Data augmentation plays an important part in the learning process of self-supervised methods . In the works by Chen et al . ( 2020a ) and Zbontar et al . ( 2021 ) , authors discussed the importance of the different data augmentations . A set of well-defined transformations was proposed within the SimCLR method . This set is commonly used in later works in the self-supervision field . The augmentations include : rotation , cut , flip , color jitter , blur and gray scale . These operations are randomly applied to an image to generate different views ( x1 , x2 ) . Applied to the same image contrastive-like selfsupervised methods learn representation invariant for such distortions . The multiple view creation is a task-related procedure ( Tian et al. , 2020 ) . However , keeping in mind usefulness of learned representations for downstream tasks , color jittering is one of the most important ones ( Chen et al. , 2020a ; Zbontar et al. , 2021 ) , operating on hue , saturation , brightness and contrast . However , color jitter is expected to induce a certain level of color-invariance ( invariance to hue , saturation , brightnesss and contrast ) which are consequently transferred to the downstream task as well . As a consequence , we expect these learned features to underperform on downstream tasks for which color is crucial . Color imaging literature has long researched color features that were invariant with respect to scene accidental events , such as shading , shadows , and illuminant changes ( Geusebroek et al. , 2001 ; Finlayson & Schaefer , 2001 ) . Having features with invariance with respect to these events was found to be beneficial for object recognition . Having invariance with respect to hue and saturation changes ( which are induced by the currently used color jitter operation ) is detrimental for object recognition , especially for those objects were these characteristics are fundamental . Therefore , in this work , we aim to revisit early theory on illuminant invariance ( Finlayson & Schaefer , 2001 ) to design an improved color augmentation function that induces invariances common in the real-world and that does not deteriorate the color quality of the learned features . 3 PLANCKIAN JITTER . The range of image transformations introduced by traditional color jitter creates a variability in training data that indiscriminately explores all hues at various levels of saturation . The resulting invariance can be useful for downstream tasks where chromatic variations are indeed irrelevant ( such as car body color in vehicle recognition ) , but will be detrimental for downstream applications where the color information is known to be critical ( such as flowers , birds , vegetables classification ) . On the other hand , completely removing color invariance risks producing a model with little generalization capability , unable to handle the common variations in illumination conditions due to various sources of indoor and outdoor lighting . 3.1 PHYSICS-BASED PLANCKIAN JITTER . Here we describe an alternative data augmentation procedure , called Planckian jitter , that exploits the physical description of a black-body radiator to re-illuminate the training images within a realistic distribution ( Finlayson & Schaefer , 2001 ; Tominaga et al. , 1999 ) . The resulting augmentations are more realistic than those of the default color jitter ( see Fig . 1 ) . The resulting learned selfsupervised feature representation is thus expected to be robust to illumination changes that can be commonly observed in real-world images , while simultaneously maintaining the ability to discriminate the image content based on color information . Given an input RGB training image I , our physics-based Planckian jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions : 1 . We sample a new illuminant spectrum σT ( λ ) from the distribution of a black body radiator . 2 . We transform the sampled spectrum σT ( λ ) into its sRGB representation ρT ∈ R3 . 3 . We create a jittered image I ′ by reilluminating I with the sampled illuminant ρT . 4 . We introduce brightness and contrast variation , producing a Planckian-jittered image I ′′ . A radiating black body at temperature T can be synthesized using Planck ’ s Law ( Andrews , 2010 ) : σT ( λ ) = 2πhc2 λ5 ( e hc kTλ − 1 ) W/m3 , ( 3 ) where c = 2.99792458×108 m/s is the speed of light , h = 6.626176×10−34 Js is Planck ’ s constant , and k = 1.380662× 10−23 J/K is Boltzmann ’ s constant . For our experiments , we sampled T in the interval between 3000K and 15000K which is known to result in a set of illuminants that can be encountered in real life ( Tominaga et al. , 1999 ) . Then , we discretized wavelength λ in 10nm steps ( ∆λ ) in the interval between 400nm and 700nm . The resulting spectra are visualized in Figure 4 ( left ) in Appendix A.1 . The conversion from spectrum into sRGB is obtained through a series of intermediate steps ( Wyszecki & Stiles , 1982 ) : 1 . We first map the spectrum into the corresponding XYZ stimuli , using the 1931 CIE standard observer color matching functions c { X , Y , Z } ( λ ) , in order to bring the illuminant into a standard color space that represents a person with average eyesight . 2 . We normalize this tristimulus by its Y component , convert it into CIE 1976 L * a * b color space , and fix its L component to 50 in a 0-to-100 scale . This allows us to constrain the intensity of the represented illuminant in a controlled manner as a separate task . 3 . We then convert the resulting values in sRGB , obtaining ρT = { R , G , B } . The resulting distribution of illuminants is visualized with the Angle-Retaining Chromaticity diagram ( Buzzelli et al. , 2020 ) in Figure 4 ( right ) in Appendix A.1 . All color space conversions assume a D65 reference white , which means that a neutral surface illuminated by average daylight conditions would appear achromatic . Once the new illuminant has been converted in sRGB , it is applied to the input image I by resorting to a Von-Kries-like transform ( von Kries , 1902 ) , given by the following channel-wise scalar multiplication : I ′ { R , G , B } = I { R , G , B } · { R , G , B } / { 1 , 1 , 1 } , ( 4 ) where we assume the original scene illuminant to be white ( 1,1,1 ) . Finally , brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination : I ′′ = cB · cC · I ′ + ( 1− cC ) · µ ( cB · I ′ ) , ( 5 ) where cB = 0.8 and cC = 0.8 represent , respectively , brightness and contrast coefficients , and µ is a spatial average function .

Summary: This paper first examines that typical color jittering augmentation is harmful to feature representation learning. Then the authors proposed a physics-based color augmentation, called Planckian jitter to improve the performance. The proposed Planckian jitter performs better with the recent contrastive and self-supervised learning schemes.