Planckian jitter: enhancing the color quality of self-supervised visual representations

1 INTRODUCTION . Self-supervised learning enables the learning of visual representation without the need for any labeled data ( Doersch et al. , 2015 ; Dosovitskiy et al. , 2014 ) . Several recent works learn representations that are invariant with respect to a set of data augmentations , and have obtained spectacular results ( Grill et al. , 2020 ; Chen & He , 2021 ; Caron et al. , 2020 ) , significantly narrowing the gap with supervised learned representations . These works vary in their architecture , learning objective , and optimization strategy , however , they are similar in applying a common set of data augmentations to generate the various image views . The algorithms , while learning to map these different views to the same latent representation , learn complex semantic representation for visual data . The set of transformations ( data augmentation ) that are considered induce a set of invariances that characterizes the learned visual representation . Before deep learning revolutionized the way visual representations were computed , separate features were hand-designed to represent its various properties , leading to research on shape ( Lowe , 2004 ) , texture ( Manjunath & Ma , 1996 ) , and color features ( Finlayson & Schaefer , 2001 ; Geusebroek et al. , 2001 ) . Color features were typically designed to be invariant with respect to a set of scene accidental events , such as shadows , shading , illuminant and viewpoint changes . With the rise of deep learning , feature representations that exploit simultaneously color , shape and texture are learned implicitly and the invariances are a byproduct of the end-to-end training ( Krizhevsky et al. , 2009 ) . As discussed above , the current set of self-supervised learning methods explicitly define a set of invariances ( related to the applied data augmentations ) that are to be learned . In this work , we focus on the current de-facto choice for color augmentations . We argue that they seriously cripple the color quality of these representations and we propose an alternative color augmentation . Figure 1 ( left ) illustrates the currently applied color transformation for a sample image , depicted in the middle of the left-most grid . It is clear that the applied color transformation significantly alters the colors of the original image , both in terms of hue and saturation . One of the justifications in literature for such strong color augmentations is that without large color changes , mapping images to the same latent representation can be purely done based on color and no complex shape features are learned , therefore the best results when using only two transformations are obtained when applying image cropping with a color augmentation ( Chen et al. , 2020a ) . However , looking at the reported example it is evident that a representation that maps these images to the same latent representation can not rely on the object color , and as a results the quality of the color representation learned with such algorithms is expected to be inferior . Therefore , in this paper , we propose another set of color augmentations , shown on the right side of Figure 1 . In addition to introducing more natural variations in the image chromaticity , the proposed color augmentation also affects neutral regions such as the petals shown in the figure , which are left unvaried with the original color transformations . We draw on existing color imaging literature , that aimed to design features that were invariant with respect to illuminant changes that were commonly encountered in real-world scenes ( Finlayson & Schaefer , 2001 ) . Our augmentation , called Planckian jitter , applies a physically realistic illuminant variation to the images . We consider the illuminants that are described by Planck ’ s Law for black body radiation and that are known to be similar to illuminants encountered in real-life ( Tominaga et al. , 1999 ) . In the experimental section , we show that self-supervised features learned with Planckian jitter yield superior features , leading to gains of over 5 % on several downstream color classification tasks . However , since our color augmentation is less extreme , the learned shape features are of lower quality than with the original color jitter . A simple combination of both feature representations leads to huge performance gains with respect to default color jitter of between 10-15 % on several color downstream tasks . In addition , we show that our augmentation method can be applied to several state-of-the-art self-supervised learning methods . Finally , we analyze the color sensitivity of the learned color representations . 2 BACKGROUND AND RELATED WORKS . 2.1 SELF-SUPERVISED LEARNING AND CONTRASTIVE LEARNING . Recent improvements in self-supervision learn a semantically rich feature representation without the need of any labels . In SimCLR ( Chen et al. , 2020a ) similar samples are created by augmenting an input image , while dissimilar are chosen by random . To make contrastive training more efficient MoCo method ( He et al. , 2020 ) and its improved version ( Chen et al. , 2020b ) use a memory bank for learned embeddings which helps for efficient sampling . This memory is kept in sync with the rest of the network during training time by using a momentum encoder . Several methods do not have any explicit contrastive pairs . BYOL ( Grill et al. , 2020 ) propose an asymmetric network by introducing an additional MLP predictor between the outputs of the two branches . One of the branches is kept ” offline ” - updated by a momentum encoder . SimSiam ( Chen & He , 2021 ) goes even further and presents a simplified solution without a momentum encoder . Moreover , it obtains similar high-quality results and does not require a large mini-batch size . We will use the SimSiam method to verify our proposed color augmentation ( we also apply our approach to SimCLR ( Chen et al. , 2020a ) and Barlow Twins ( Zbontar et al. , 2021 ) in the experiments ) . The main part is a CNN-based encoder , learned end-to-end in an asymmetric siamese architecture , where one branch has an additional predictor ( Multi-Layer Perceptron , or MLP ) whose output aims to be as close as possible to the other branch ( see Figure 2 ) . The second branch is not updated during backward propagation . A negative-cosine loss function is used , defined as : L = D ( p1 , z2 ) /2 +D ( p2 , z1 ) /2 ( 1 ) D ( pA , zB ) = − pA ‖pA‖2 · zB ‖zB‖2 , ( 2 ) where z1 , z2 are encoded values respectively for two different augmented versions x1 and x2 of the same image x . Note that in Eq . 1 they are alternated between the two branches , but it is always only the first branch that uses a Multilayer Perceptron , producing either p1 or p2 . Additionally , no contrastive term is present : only the similarity is enforced during learning . 2.2 DATA AUGMENTATION . Data augmentation plays an important part in the learning process of self-supervised methods . In the works by Chen et al . ( 2020a ) and Zbontar et al . ( 2021 ) , authors discussed the importance of the different data augmentations . A set of well-defined transformations was proposed within the SimCLR method . This set is commonly used in later works in the self-supervision field . The augmentations include : rotation , cut , flip , color jitter , blur and gray scale . These operations are randomly applied to an image to generate different views ( x1 , x2 ) . Applied to the same image contrastive-like selfsupervised methods learn representation invariant for such distortions . The multiple view creation is a task-related procedure ( Tian et al. , 2020 ) . However , keeping in mind usefulness of learned representations for downstream tasks , color jittering is one of the most important ones ( Chen et al. , 2020a ; Zbontar et al. , 2021 ) , operating on hue , saturation , brightness and contrast . However , color jitter is expected to induce a certain level of color-invariance ( invariance to hue , saturation , brightnesss and contrast ) which are consequently transferred to the downstream task as well . As a consequence , we expect these learned features to underperform on downstream tasks for which color is crucial . Color imaging literature has long researched color features that were invariant with respect to scene accidental events , such as shading , shadows , and illuminant changes ( Geusebroek et al. , 2001 ; Finlayson & Schaefer , 2001 ) . Having features with invariance with respect to these events was found to be beneficial for object recognition . Having invariance with respect to hue and saturation changes ( which are induced by the currently used color jitter operation ) is detrimental for object recognition , especially for those objects were these characteristics are fundamental . Therefore , in this work , we aim to revisit early theory on illuminant invariance ( Finlayson & Schaefer , 2001 ) to design an improved color augmentation function that induces invariances common in the real-world and that does not deteriorate the color quality of the learned features . 3 PLANCKIAN JITTER . The range of image transformations introduced by traditional color jitter creates a variability in training data that indiscriminately explores all hues at various levels of saturation . The resulting invariance can be useful for downstream tasks where chromatic variations are indeed irrelevant ( such as car body color in vehicle recognition ) , but will be detrimental for downstream applications where the color information is known to be critical ( such as flowers , birds , vegetables classification ) . On the other hand , completely removing color invariance risks producing a model with little generalization capability , unable to handle the common variations in illumination conditions due to various sources of indoor and outdoor lighting . 3.1 PHYSICS-BASED PLANCKIAN JITTER . Here we describe an alternative data augmentation procedure , called Planckian jitter , that exploits the physical description of a black-body radiator to re-illuminate the training images within a realistic distribution ( Finlayson & Schaefer , 2001 ; Tominaga et al. , 1999 ) . The resulting augmentations are more realistic than those of the default color jitter ( see Fig . 1 ) . The resulting learned selfsupervised feature representation is thus expected to be robust to illumination changes that can be commonly observed in real-world images , while simultaneously maintaining the ability to discriminate the image content based on color information . Given an input RGB training image I , our physics-based Planckian jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions : 1 . We sample a new illuminant spectrum σT ( λ ) from the distribution of a black body radiator . 2 . We transform the sampled spectrum σT ( λ ) into its sRGB representation ρT ∈ R3 . 3 . We create a jittered image I ′ by reilluminating I with the sampled illuminant ρT . 4 . We introduce brightness and contrast variation , producing a Planckian-jittered image I ′′ . A radiating black body at temperature T can be synthesized using Planck ’ s Law ( Andrews , 2010 ) : σT ( λ ) = 2πhc2 λ5 ( e hc kTλ − 1 ) W/m3 , ( 3 ) where c = 2.99792458×108 m/s is the speed of light , h = 6.626176×10−34 Js is Planck ’ s constant , and k = 1.380662× 10−23 J/K is Boltzmann ’ s constant . For our experiments , we sampled T in the interval between 3000K and 15000K which is known to result in a set of illuminants that can be encountered in real life ( Tominaga et al. , 1999 ) . Then , we discretized wavelength λ in 10nm steps ( ∆λ ) in the interval between 400nm and 700nm . The resulting spectra are visualized in Figure 4 ( left ) in Appendix A.1 . The conversion from spectrum into sRGB is obtained through a series of intermediate steps ( Wyszecki & Stiles , 1982 ) : 1 . We first map the spectrum into the corresponding XYZ stimuli , using the 1931 CIE standard observer color matching functions c { X , Y , Z } ( λ ) , in order to bring the illuminant into a standard color space that represents a person with average eyesight . 2 . We normalize this tristimulus by its Y component , convert it into CIE 1976 L * a * b color space , and fix its L component to 50 in a 0-to-100 scale . This allows us to constrain the intensity of the represented illuminant in a controlled manner as a separate task . 3 . We then convert the resulting values in sRGB , obtaining ρT = { R , G , B } . The resulting distribution of illuminants is visualized with the Angle-Retaining Chromaticity diagram ( Buzzelli et al. , 2020 ) in Figure 4 ( right ) in Appendix A.1 . All color space conversions assume a D65 reference white , which means that a neutral surface illuminated by average daylight conditions would appear achromatic . Once the new illuminant has been converted in sRGB , it is applied to the input image I by resorting to a Von-Kries-like transform ( von Kries , 1902 ) , given by the following channel-wise scalar multiplication : I ′ { R , G , B } = I { R , G , B } · { R , G , B } / { 1 , 1 , 1 } , ( 4 ) where we assume the original scene illuminant to be white ( 1,1,1 ) . Finally , brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination : I ′′ = cB · cC · I ′ + ( 1− cC ) · µ ( cB · I ′ ) , ( 5 ) where cB = 0.8 and cC = 0.8 represent , respectively , brightness and contrast coefficients , and µ is a spatial average function .

Summary: **Overview.** This paper proposes a novel type of image colour augmentation to be used during self-supervised learning (SSL).  **Background.** In a typical SSL setting, similar samples are generated by randomly augmenting an image in a variety of different ways: random cropping, colour jittering, random rotations, etc. These similar images are then passed through a neural network, and the predicted features for similar samples are trained to be close to each other based on some similarity metric (ignoring the contrastive term description here as it is not directly related to the paper).  **Motivation.** In this paper, the authors address the issue of using colour jittering as augmentation during SSL. Specifically, using colour jittering pushes the network towards invariance to image colours, while relying more on the shape and texture of objects when making predictions. The authors point out that despite the benefits of this for many general detection tasks, this will be a detrimental property when dealing with more colour-dependant tasks.  **Method.** Thus, they propose to use Planckian jittering instead of colour jittering. Planckian jittering is a physics based augmentation method proposed by the authors (although the formulations for it come from existing literature) to re-illuminate the training images within a realistic distribution which leads to more realistic and constrained colour augmentations than colour jittering. The paper claims that Planckian jittering still helps improve network's dependence on shape and texture of objects (although less than colour jittering), while limiting the network's invariance to image colours.  **Experiments.**  6 SSL models were trained independently on CIFAR-100: 3 used different variants of the Planckian jittering, 2 used different variants of colour jittering (w/ and w/o Random GrayScale), and 1 used no augmentations. Linear classifiers for CIFAR-100 and FLOWERS-102 classification tasks were then trained on top of each of the SSL models' features, where FLOWERS-102 is the task that is claimed to be more heavily colour-dependant. Moreover, an extra linear classifier was trained on the concatenation of features from a Planckian jittering and a colour jittering model (called the "Latent space combination" model). Based on accuracy, "Latent space combination" outperforms all other models by a significant margin on both datasets. Planckian jitter seems to outperform other augmentations on FLOWERS-120 (Table 1), which supports the claim of the authors. On CIFAR-100, Planckian jitter performs slightly worse than colour jittering, which the authors attribute to the reduction of colour invariance in the features.  A very similar experiment was also done with different datasets (SSL training on tiny-imagenet, linear classifier trained on FLOWERS-102, CUB200, VEGFRU, T1K+), which also obtained similar results and conclusions. Moreover, in another similar experiment, the SSL models were trained with different SSL configurations (SimSiam, SimCLR, Barlow Twins) to indicate the generality of Planckian jitter for different types of SSL configurations. In another experiment, the robustness of the different models on augmented images using Planckian jittering was evaluated. Lastly, the colour sensitivity was analyzed to inspect the impact of colour information in neuron activations for each model.