Planckian jitter: enhancing the color quality of self-supervised visual representations
Authors: Simone Zini
Marco Buzzelli
Bartłomiej Twardowski
Joost van de weijer
Venue: ICLR 2022 Submitted (2022)
Keywords: Contrastive Learning
Self-Supervised Learning
Color Features
Illuminant Invariance

Abstract: Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The set of used data augmentations is of crucial importance for the quality of the learned feature representation. We analyze how the traditionally used color jitter negatively impacts the quality of the color features in the learned feature representation. To address this problem, we replace this module with physics-based color augmentation, called Planckian jitter, which creates realistic variations in chromaticity, producing a model robust to llumination changes that can be commonly observed in real life, while maintaining the ability to discriminate the image content based on color information. 
We further improve the performance by introducing a latent space combination of color-sensitive and non-color-sensitive features. 
These are found to be complementary and the combination leads to large absolute performance gains over the default data augmentation on color classification tasks, including on Flowers-102 (+15%), Cub200 (+11%), VegFru (+15%), and T1K+ (+12%). Finally, we present a color sensitivity analysis to document the impact of different training methods on the model neurons and we show that the performance of the learned features is robust with respect to illuminant variations. 

Under review as a conference paper at ICLR 2022
PLANCKIAN JITTER: ENHANCING THE COLOR QUAL- ITY OF SELE-SUPERVISED VISUAL REPRESENTATIONS
Anonymous authors Paper under double-blind review
ABSTRACT
Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The set of used data augmentations is of crucial importance for the quality of the learned fea- ture representation. We analyze how the traditionally used color jitter negatively impacts the quality of the color features in the learned feature representation. To address this problem, replace this module with soysics-based color augmen- tation, called Planckian jitter, which creates realistic variations in chromaticity, producing model robust to llumination changes that can be commonly observed in real life, while maintaining the ability to discriminate the image content based on color information. We further improve the performance by introducing la- tent space combination of color-sensitive and coon-cllor-sensiiie features. These are found to complementary and the combination leads large absolute per- formance gains over the default data augmentation including on Flowers- 102 (+15%), Cub200 (+11%), VegFru (+15%), and TIK+ (+ (+12%). Finally, we present a color sensitivity analysis to document the impact different training methods on the model neurons and we show that the perfor- mance of the learned features is robust with respect to illuminant variations.
color classification tasks,
INTRODUCTION
Self-supervised learning enables the learning of visual representation without the need for any la- beled data (Doersch et al. 2015 Dosovitskly et al. 2014). Several recent works learn representa- tions that are invariant with respect to a set of data augmentations, and have obtained spectacular results (Grill et al. 2020 Chen & He 2021 |021| Caron et al. 2020), significantly narrowing the gap with supervised learned representations. These works vary in their architecture, learning objective, and optimization strategy, however, they are similar in applying common set of data augmentations to generate the various image views. The algorithms, while learning to map these different views the same latent representation, learn complex semantic representation for visual data. The set of trans- formations (data augmentation) that are considered induce a set of invariances that characterizes the learned visual representation.
Before deep learning revolutionized the way visual representations were computed, separate fea- tures were hand-designed to represent its various properties, leading research shape (Lowe) 2004], texture (Manjunath & Ma 1996), and color features (Finlayson Schaefer 20001: Geuse- broek et al. 2001). Color features were typically designed to be invariant with respect set of scene accidental events, such as shadows, shading, illuminant and viewpoint changes. With the rise of deep learning, feature representations that exploit simultaneously color, shape and texture are learned implicitly and the invariances are a byproduct of the end-to-end training (Krizhevsky et al.) 2009) As discussed above, the current set of self-supervised learning methods explicitly define set of invariances (related to the applied data augmentations) that are to be learned. In this work, we focus on the current de-facto choice for color augmentations. We argue that they seriously cripple the color quality of these representations and we propose an alternative color augmentation.
Figure [ (left) illustrates the currently applied color transformation for a sample image, depicted in the middle of the left-most grid. It is clear that the applied color transformation significantly alters the colors of the original image, both in terms of hue and saturation. One of the justifications in literature for such strong color augmentations is that without large color changes, mapping images to the same latent representation can be purely done based on color and no complex shape features are learned, therefore the best results when using only transformations obtained when applying

---

Under review as a conference paper at ICLR 2022
Figure I: Default color jitter (left) and Planckian jitter (right). Augmentations based on default color jitter lead to unrealistic images, while Planckian augmentations lead to a set of realistic augmenta- tions. We also provide the ARC chromaticity diagrams. Here we sample a number of initial RGB values, and map them into the range of possible outputs given by the respective data augmentation technique. These diagrams show that Planckian jitter transforms colors along chromaticity lines that occur in nature when changing the illuminant, whereas the default color jitter transfers colors throughout the whole chromaticity plane. In both cases, only chromaticity effects have been shown, disregarding brightness and contrast variations, to simplify the visualization.
image cropping with a color augmentation (Chen et al.) 2020a). However, looking the reported example it is evident that representation that maps these images to the same latent representation cannot rely on the object color, and as results the quality of the color representation learned with such algorithms is expected to be inferior.
Therefore, in this paper, we propose another set of color augmentations, shown on the right side of Figure In addition to introducing more natural variations in the image chromaticity, the proposed color augmentation also affects neutral regions such as the petals shown in the figure, which are left unvaried with the original color transformations. We draw on existing color imaging literature, that aimed to design features that were invariant with respect to illuminant changes that were commonly encountered in real- world scenes (Finlayson & Schaefer (001). Our augmentation, called Planckian jitter, applies a physically realistic illuminant variation to the images. We consider the illuminants that described by Planck's Law for black body radiation and that are known to be similar to illuminants encountered in real-life (Tominaga et al. 1999). In the experimental section, we show that sulf-supe features learned with Planckian jitter yield superior features, leading to gains of over 5% on several downstream color classification tasks. However, since our color augmentation less extreme, the learned shape features are of lower quality than with the original color jitter. A simple combination of both feature representations leads to huge performance gains with respect to default color jitter of between 10-15% on several color downstream tasks. In addition, we show that our augmentation method can be applied to several state-of-ther of-e--oftthearrr art supersupe learning methods. Finally, we analyze the color sensitivity of the learned color representations.
2
BACKGROUND AND RELATED WORKS
2.1 SELF-SUPERVISED LEARNING AND CONTRASTIVE LEARNING
Recent improvements in self-supervision lear semantically rich feature representation without the need of any labels. In SimCLR (Chen et al., 2020a) similar samples are created by augmenting an input image, while dissimilar are chosen by random. To make contrastive training more efficient MoCo method (He et al. 2020) and its improved version (Chen et al.) (2020b) use a memory bank for learned embeddings which helps for efficient sampling. This memory is kept in syne with the rest of the network during training time by using a momentum encoder. Several methods do not have any explicit contrastive pairs. BYOL (Grill et al., (020) propose an asymmetric network by introducing an additional MLP predictor between the outputs of the two branches. One of the branches is kept "offline" updated by a momentum encoder. SimSiam (Chen & He, 202T) goes further and presents simplified solution without momentum encoder. Moreover, it obtains similar high-quality results and does not require a large mini-batch size.
We will use the SimSiam method verify our proposed color augmentation (we also apply our ap- proach to SimCLR (Chen et al.) (220aa) and Barlow Twins (Zbontar et al.) (202)) in the experiments). The main part is CNN-based encoder, learned end-to-end in an asymmetric siamese architecture,
2

---

Under review as a conference paper at ICLR 2022
[Resnet-18]
MLP
P)
Encoder [Resnet 18)
Dusa augrnimation Plandiinn litee
Similarity
Shop
[Resnet-18 18]
Linear Clessiter
SimSlam Training
Soopgrad
Class
Linear Cassilier training
Figure 2: SimSiam training procedure exploiting Planckian- data augmentation (left), and fine-tuning the linear classifier using the trained encoder (right).
where one branch has an additional predictor (Multi-Layer Perceptron, or MLP) whose output aims to be close possible to the other branch (see Figure 2). The second branch is not updated during backward propagation. A negative-cosine loss function is used, defined as:
D(p1 22 ((2,, 21)/2
P(AA.B)) = PA
2B
(2)
TPAll2 |
where 21, 22 are encoded values respectively for two different augmented versions 21 and 22 of the same image Note that in Eq. [ they alternated between the two branches, but it is always only the first branch that uses Multilayer Perceptron, producing either P1 or p2. Additionally, no contrastive term is present: only the similarity enforced during learning.
2.2
DATA AUGMENTATION
Data augmentation plays important part in the learning process of self-supervisee methods. In the works by byyh et al. (2020a) and Zbontar et al. (20TT) authors discussed the importance of the dif- ferent data augmentations. A set of well-defined transformations method. This set is commonly used in later works the self-supervision field. The augmentations rotation, cut, flip, color jitter, blur and gray scale. These operations randomly applied to an image to generate different views (I1, x2). Applied to the same image contrastive-like self- supervised methods learn representation invariant for such distortions. The multiple view creation is task-related procedure (Tian et al.) 2020). However, keeping in mind usefulness of learned repre- sentations for downstream tasks, color jittering is one of the most important ones (Chen et al., [2020a] Zbontar et al. 2021), operating hue, saturation, brightness and contrast. However, color jitter is expected to induce a certain level color-invariance (invariance to hue, saturation, brightnesss and contrast) which are consequently transferred to the downstream task well. As a consequence, expect these learned features to underperform on downstream tasks for which color crucial.
proposed within the SimCLR
include:
Color imaging literature has long researched color features that invariant with respect to scene accidental events, such as shading, shadows, and illuminant changes (Geusebroek et al.) 2001] Fin- layson & Schaefer 2001). Having features with invariance with respect to these events found to be heneficial for object recognition. Having invariance with respect to hue and saturation changes (which are induced by the currently used color jitter operation) is detrimental for object recognition, especially for those objects were these characteristics are fundamental. Therefore, in this work, aim revisit early theory on illuminant invariance (Finlayson Schaefer 2001) design an improved color augmentation function that induces invariances common in the real-world and that does not deteriorate the color quality of the learned features.
PLANCKIAN JITTER
The range image transformations introduced by traditional color jitter creates a variability in training data that indiscriminately explores all hues various levels of saturation. The resulting invariance can be useful for downstream tasks where chromatic variations are indeed irrelevant (such car body color in vehicle recognition), but will be detrimental for downstream applications where
3

---

Under review as a conference paper at ICLR 2022
the color information is known to be critical (such flowers, birds, vegetables classification). On the other hand, completely removing color invariance risks producing a model with little generalization capability, unable to handle the common variations in illumination conditions due to various sources of indoor and outdoor lighting.
3.1 PHYSICS-BASED PLANCKIAN JITTER
Here we describe an alternative data augmentation procedure, called Planckian jitter, that exploits the physical description of black-body radiator re-illuminate the training images within real- istic distribution (Finlayson & Schaefer, 2001; Tominaga et al., 1999). The resulting augmentations more realistic than those of the default color jitter (see Fig. Dp. The resulting learned self- supervised feature representation is thus expected to be robust to illumination changes that can be commonly observed in real-world images, while simultaneously maintaining the ability discrim- inate the image content based on color information.
Given an input RGB training image L, our physics-based Planckian jitter procedure applies a chro- matic adaptation transform that simulates realistic variations in the illumination conditions:
1. We sample new illuminant spectrum (A) from the distribution of black body radiator.
2. We transform the sampled spectrum (A)) into its sRGB representation pr IRS
3. We create jittered image I' by reilluminating I with the sampled illuminant pr-
4. We introduce brightness and contrast variation, producing a Planckian-jtttred image I".
A radiating black body at temperature T can be synthesized using Planck's Law (Andrews, 2010):
2xhc² W/m"
(A(A)= (e - 1)
is Planck's constant, and & = 1.380662 x 10-23 23 J/K is Boltzmann's 's constant, For our experiments, we sampled T the interval between 3000/ and 15000K which is known to result in set of illuminants that can be encountered in real life (Tominaga et al. 1999). Then, we discretized wavelength in 10nm steps (AA) in the interval between 400nm and 700mm. The resulting spectra visualized in Figure (left) in Appendix[A.1
where 2.99792458 x 10 m/s is the speed of light, h = 6.666.222 2 10-34 is
The conversion from spectrum into sRGB is obtained through series of intermediate steps Wyszecki & Stiles 1982):
1. We first map the spectrum into the corresponding XYZ stimuli, using the 1931 CIE stan- dard observer color matching functions ((X.Y.Z) (A), in order to bring the illuminant into standard color space that represents person with average eyesight.
2. We normalize this tristimulus by its Y component, convert it into CIE 1976 L color space, and fix its L component to 50 0-to-100 scale. This allows us to constrain the intensity of the represented illuminant in a controlled manner as a separate task.
3. We then convert the resulting values in sRGB, obtaining PT = {R, GJ. The result- ing distribution of illuminants is visualized with the Angle-Retaining Chromaticity dia- gram (Buzzelli et al. 2020) in Figure 4 (right) in Appendix A.I
All color space conversions assume a D65 reference white, which means that a neutral surface illuminated by average daylight conditions would appear achromatic. Once the new illuminant has been converted in sRGB, it is applied to the input image / by resorting to Von-Kries-like transform (yon Kries 1902). given by the following channel- wise scalar multiplication:
(R.G.B = (R.G.B) {R, G, 1),(1,111},
(4)
where we assume the original scene illuminant to be white (1,1,1). Finally, brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination:
I" CB co I' - (1 cc)cc) (CB -1),
(5)
where en = 0.8 and cc = 0.8 represent, respectively, brightness and contrast coefficients, and is spatial average function.
4

---

Under review as a conference paper at ICLR 2022
3.2 LATENT SPACE COMBINATION
The self-supervised learning paradigm involves a pretraining phase that relies on data augmentation to produce a set of features with certain invariance properties. These features are then used as the representation for a second phase, where we learn a given supervised downstream task.
Default color jitter configurations generate features that are strongly invariant to color information, thus relying mainly on shape and texture for the downstream task. Constraining the color jitter con- sequently reduces the role of shape favor of color in the decision process of the second phase. As stated by (Chen et al. 2020a), in fact, completely removing color jitter creates features that focus mainly on the color, and ignore the more difficult problem of learning a good shape representa- tion. Our Planckian jitter is designed to operate in middle ground, constraining the type of color transformation that should be given invariance to, and thus leaving space for shape information. Nonetheless, the relative role of color and non-color in the feature representation can be better tuned by creating a concatenation of features learned in different setups. For this reason, we propose a configuration where two pretraining phases tion, which is expected to exploit mostly shape information, and one with Planckian jitter, which expected to exploit color information. The two resulting features are then concatenated and used as the representation to learn downstream tasks (in our experiments each representation has 512 dimensions and the latent space combination 1024).
in parallel: one with default color augmentar
4 EXPERIMENTAL RESULTS
In this section we verify the superiority of the proposed color data augmentation method against the default color jitter and we ablate its components. We evaluate the impact on downstream classifica- tion tasks, and analyze the color sensitivity of the learned backbone networks. We have additional results on computational time of the proposed Planckian augmentation in Appendix A.4
4.1 TRAINING AND EVALUATION SETUP
We train a 18 He et al. 2 backbone encoder in the SimSiam framework and then eval- uate it on a classification downstream task with linear classifier. To analyze the impact of the augmentation operations on the learned features, we train the encoder with different data augmenta- for complete overview of the training procedure.
tions before the evaluation. See Figure 2
We perform the unsupervised training with two different image resolutions: 32 x 32 and 64 x 64 pixel<"I To train the model at the lowest resolution, we adopted the CIFAR- 1 (0 dataset (Krizhevsky et al. 2009), which composed of 50000 32 32 images, divided into 100 classes. Since the training has been done with images of size 32 32, the ResNet18 architecture has been slightly modified: the first convolutional layer's kernel size has been modified from 7 x to 3 and the first Max Pooling layer has been removed. Adopting the same modification to the architecture, we also trained a higher resolution version of the model, using the which contains 100,000 images of 200 classes (500 for each class) resolution 64 64 pixels. The linear classifier training at resolution 32 x 32 has been performed with two different datasets: the CIFAR- 100 and the FL.WWERS-I 102 datasets (Nilsback & Zisserman 2008). The CIFAR-100
dataset (Le & Yang 2015),
has been used in order to define a baseline for the classification task in generic conditions, since it has been used also in the original work on SimCLR. The FLOWERS- 102 dataset, which contains 8189 photos of flowers, divided in 102 classes, has been selected to assess the quality of the features extracted in scenarios where color information covers an important role. The FLOWERS- 102 dataset has been resized to dimension 32 x 32 to fit the pre-trained model input dimensions. For the training of the linear classifier at resolution 64 X 64, we used five different datasets:
FLOWERS- 102 dataset, the VEGERU dataset by [Sahuii Hou & Wang (2017), the CUB-200 dataset by Welinder et al. (2000) and the TIK+ dataset by Cusano et al. (002)). More details about these datasets are provided in Appendix A.2 In the case of CUB-200, each image has been cropped using the bounding boxes given in the annotations of the dataset. These five datasets have been resized at resolution 64 x 64 and used to train the linear classifier. For the T T+K+ dataset we adopted the 266 class labeling to train the linear classifier and to test our models.
The model for supervised and unsupervised training has been written in Pytorch v1.7.0 and run on an NVIDIA Titan V 12 GB GPU, The SimSiam training has been performed using Stochastic Gradient
I Limited hardware resources hindered verification standard ImageNet resolution (224 224)
5

---

Under review as a conference paper at ICLR 2022
Table 1: Analysis of data augmentation. Self-supervised training is performed on CIFAR- 100, whereas the learned features are evaluated at (32 x2) on CIFAR- 100 and FLOWERS-102. Augmen- tation techniques include variations in hue and saturation (H&S), brightness and contrast (B&C), Planckian-baseee chromaticity (P), and random grayscale conversions (G). The accuracy refers to the results of the linear classifiers trained with features extracted from different backbones.
DATA AUGMENTATION
H&S
B&C
G
P ACCURACY
None
41.93%
LSC CIFAR- 100
Default Color Jitter
59.93%
Default Color Jitter w/o Random GrayScale
41.96%
Default Color Jitter (chromaticity only)
32.46%
Planckian litter (chromaticity only)
30%
Planckian Jitter (brighteesscoontrust only)
31.78%
Planckian Jitter
47.31 %
None + Default Color Jitter w/o Random GrayScale
44.87%
Default Color Jitter + Default Color Jitter w/o Rundom GrayScule
62.27%
Default Color Jitter + Planckian Jitter
63.54%
LSC FLOWERS 102
None
36.47%
Default Color Jitter
30.00%
Default Color Jitter w/o Random GrayScale
36.96%
Default Color Jitter (chromaticity only)
39. 1
Planckian Jitter (chromaticity only)
39.51
Planckian Jitter (brightness/contrrast only)
41.96%
Planckian Jitter
42.75%
None Default Color Jitter w/o Random GrayS
43.33%
Default Color Jitter Default Color Jitter w/o Random GrayScale
47.65%
Default Color Jitter Planckian Jitter
51.66%
Descent, with starting learning rate 0.03, cosine annealing learning rate scheduler, and mini-batch size 512 (like in original work Chen & He (221I). The linear classifier training for CIFAR- 100 is done with exactly same settings, except a staring learning rate set to 0.1 and a train process to 500 epoch. For other datasets we used Adam optimizer with initial learning rate set to 0.03.
4.2 EXPERIMENTAL RESULTS- PLANCKIAN JITTER AUGMENTATION
4.2. ABLATION STUDY
To assess the impact of color data augmentations we define six different configurations:
None: no color jittering operations (Random Color Jitter and Random Grayscale) are used.
Default Color Jitter: the default configuration, as used in SimSiam and SimCLR, uses both Random Color Jitter and Random Grayscale operations.
Default Color Jitter w/o Random GrayScale: same as Default without the Random Grayscale operation.
Planckian Jitter: uses the complete proposed Planckian Jitter operation, operating chro- maticy and brightness and contrast aspects of the images. No Random Grayscale is applied.
Planckian Jitter (chromaticity only): applies Planckian jitter without modifying brightness and contrast values in the image.
Planckian Jitter (brighnness/contress only): this configuration only modifies images bright- ness and contrast values of the input images.
In all experiments these augmentations are combined with the other default augmentations, namely crop, horizontal and vertical flip and blur. Given these six configurations, six different models have been trained and have been fine tuned for image classification. For resolution 32 x 32 the model has been fine tuned with the datasets CIFAR-1O0 100 and FLOWERS- The results in terms of accuracy reported in Table
Looking at the results, we can identify basically two different trends for the two datasets. Con- sidering CIFAR-100. as can be observed, removing the augmentations related to the colors makes the model less powerful, due to the loss of color invariance in the features extracted by the encoder, trained with contrastive learning approach. This behaviour is coherent to what was reported by Chen

---

Under review as a conference paper at ICLR 2022
et al. (2020a). As can be seen in Table I if color augmentations (i.e. brighmess/contraast and Random GrayScale) are removed completely (None configuration), the accuracy drops of 18%. Furthermore, on the FLOWERS- 10 02 the behaviour is the opposite with respect to CIFAR- 100: removing the color augmentations helps the model to better classify images, obtaining an improvement of 6, 93% of accuracy. This behaviour confirms how the color invariance, learned in contrastive learning environ- ment, negatively impacts downstream tasks where the color information has a more relevant role. For these scenarios a solution that takes into consideration the color information preferred.
Considering the introduction of realistic color transformations, we can observe how this kind of augmentation operations positively impacts the contrastive training, producing models that achieve even better results with respect to the configuration without any kind of image color manipulation. As can be seen from Table []] by simply reducing the jittering operation to influence brightness and contrast, leaving hue and saturation unchanged, yields another boost in accuracy of 5.49%. When we start modifying chromaticity values using a realistic approach, moving to Planckian Jitter augmentation, the final result is a boost of 6. 28% in accuracy with respect to the None configuration, for a total boost of 12.75% in accuracy with respect to the Default configuration. Also on CIFAR- 100 dataset improvement of 5.38% is given by the of Planckian Jitter with respect to the configuration that does not use any color augmentation operation. Despite this improvement, in this scenario the contrastive training with the realistic augmentation does not give better results with respect to the Default contiguration.
Given the results obtained using the data augmentations reported in Table I and given the con- siderations made in Section 3.2, we finally evaluate the impact of combining latent spaces from different backbones. The combination analyzed is obtained by concatenating the features extracted by the models trained with different configurations (e.g. Default with Planckian Jitter). We tested three different Latent Space Combinations (LSC): the None configuration with the Default Color Jitter w/o Random GrayScale, the Default configuration with the Default Color Jitter w/o Random GrayScale and the Default Color Jitter with the proposed Planckian Jitter. The results have been reported in Table 2 of using the features coming from a non realistic ones. On both datasets, the Latent space com- bination with Default and Planckian Jitter configuration achieves the best results. On the original task, represented by the CIFAR- 100 dataset, this combination achieves a total accuracy of 63.54%, 3.61% more with respect to the Default configuration and 16.23% more with respect to the Planckian Jitter configuration. Comparing to the LSC using the Default ColorJitter w/o Random Grayscale, the version with the proposed Planckian Jitter achieves small improvement of 1.27% in classification accuracy. On the downstream task, represented by the FLOWERS- 102 dataset, the Latent space combination reaches an accuracy value of 51.66%: an improvement of 21.66% and 8.91% in accuracy respectively for the two original configurations. Comparing here with the LSC using the Default ColorJitter w/o Random Grayscale, the combination with Planckian Jitter achieve an higher result, with bigger gap in terms of accuracy with respect to the CIFAR- 100 scenario. Here the use of Planckian Jitter brings in an improvement of 4.01%, confirming the impact of using realistic augmentation in color related classification tasks. In both of the cases the LSC using the None configuration with the Default Color Jitter w/o Random GrayScale achieve the worst results out of the three LSC versions.
We can see the benefit coming from the use of realistic augmentation instead
4.2.2 EVALUATION ON DOWNSTREAM TASKS
Given the results obtained from the ablation study, we performed the analysis of the proposed config- urations on other downstream task, using the backhone trained on higher resolution images (64 x 64 pixels). We reported here a subset of the previously tested configurations: None, Default Color Jitter, Default Color Jitter without Random GrayScale, Planckian Jitter (brighiness/contrass only) and Planckian Jitter. We also reported the Latent space combination of the Default Color Jitter combined with the proposed Planckian Jitter.
Since the backbone for resolution 64 x 64 has been trained on TINY-IMAGENET dataset, we first fine-tuned and tested the behaviour of the different models on this dataset, then on the other ones representing the downstream tasks. As can be seen from the results in Table the behaviour of the three models on the TINY-IMAGENET is the same we have seen on the CIFAR- 100. This is expected since the tasks represented by these two datasets are similar in terms of relation between the color information and class discrimination. Looking at the other datsets, we can see that the introduction of the Planckian Jitter augmentation improves the accuracy on the test sets, and an even higher accuracy can be achieved by applying Latent space combination.
7

---

Under review as a conference paper at ICLR 2022
Table Comparison training different contrastive learning models. Self-supervised training is per- formed on CIFAR- 100,, whereas the learned features are evaluated at (32 32) on CIFAR-100 1000 and FLOWERS- 102. We reported the best configurations obtained on SimSiam model, and retrained other two models, SimCLR and Barlow Twins, with those selected configurations.
FRAMEWORK
DATA AUGMENTATION
CIFAR-100
FLOWERS-102
Default ColorJitter
59,93%
30.00%
SimSiam
Planckian Jitter
47.31 %
42.75%
Latent space combination
63.54%
51.66%
Default ColorJitter
56.99%
35.29%
SimCLR
Planckian Jitter
47.75%
45.00%
Latent space combination
61.07%
55.78%
Default ColorJitter
56.60%
40.78%
Barlow Twins
Planckian Jitter
52.71% %
54.50%
Latent space combination
62.85%
62.55%
Table 3: Analysis on downstream tasks. Self-supervised training is performed on
(64 x 64). The comparison has been done considering multiple datasets as the downstream task.
DATA AUGMENTATION
TINY-IMAGENET Fl.owEss-BE 102
CUB200
VEGFRU
TIK+
None
27.06%
37.65%
18.76%
24.07%
35 35.82%
Default Color Jitter
33.12%
46.27%
19.36%
23.92%
26.01%
Default Color Jitter w/o Random GraySeale
31.62%
40.39%
21.90%
27.39%
32.50%
Planckian Jitter (hrightness/cntrrast only)
29.97%
39.60%
21.35%
27.06%
31.42%
Planckian Jitter
30.95%
52.35%
25.12%
28.94%
32.51%
LSC (Default Color Jitter + Planckian Jitter)
39.23%
61.57%
30.45%
39.65%
38.20%
These experiments on different downstream tasks related to color and at higher resolution prove the effectiveness and the generalization capability of the proposed data augmentation and latent spaces combination. On the different tasks, the proposed data augmentation offers reliable way to train backbone feature extractor more effective with respect to the original set of augmentations. More- with the Latent space combination spaces helps in the training more general model for image classification, since we obtain im- provements not only color related datasets, but also the original general content datasets.
showed how the combination of the different feature
4.2.3 GENERALITY OF PLANCKIAN JITTER
To show that the method is generally applicable to self-supervise methods which exploit color jitter augmentation, we performed experiments on two other popular self-supervised models: the SimCLR and the Barlow Twins models. This comparison is reported in Table [ As can be seen, for all of the three models tested, the application of the Planckian Jitter improves the accuracy on the downstream task. Independently from the model used, the Default Color Jitter configuration of data augmentation gives the worst results on the FLOWERS- dataset. This difference from the Planckian Jitter configuration is mainly related to the model sensitivity to color information: since the default configuration makes the different models invariant to color information, the clas- sification the downstream color-related task suffer, independently from the contrastive learning model adopted. An analysis on the color sensitivity of the different models is reported in section The Latent space combination (Default Color Jitter with Planckian Jitter) configuration con- sistently achieves better results on both of the datasets. On FLOWERS- 102 the final results improve considerably by over 20% for all three models.
4.3
4.3
COLOR SENSITIVITY ANALYSIS
In order to better understand the final models obtained by training the backbone architectures in contrastive learning framework, we perform a robustness analysis on the FLOWERS- 102 dataset with realistic illuminant variations and analyze the models sensitivity color information.
Given the different backbones trained on CIFAR-100, we train the linear classifier on CIFAR- 100 and FLOWERS- 102 datasets, and test the models with different versions of these two datasets. As- suming as reference point the D65 illuminant, which for the purpose of this test considered the default illuminant condition of each image in the datasets used for training and fine tuning,
8

---

Under review as a conference paper at ICLR 2022
Defech tutuu Dicple Ponensee you wntorrattttiioo andd
02.03
4000
10
03.00
200
200
Dedsonn Cate pitsn
Dechal 2014 wit pemtee
Gropeneet moeeeiee ine Aeceeae doo
Color sensitivity analysis. On the left, the results of the robustness analysis: here are reported the accuracies at different illuminant achieved by the backbones trained with the different data augmentation configurations. On the right, the color sensistivity indexes computed for the different configurations used for training the backbone network.
Figure 3:
change the illuminant of the two considered datasets. Using the 25 illuminants derived from the Planckian Jitter, we create different versions of each dataset, one for every illuminant. We are now interested in evaluating the robustness of the learned representations with respect to these illumi- nant changes. The results on Flowers- 102 can be seen in Figure 3 Another plot containing also the accuracies on CIFAR-100 is provided in Figure from Appendix A.3 The Planckian Jitter obtains stable performance from around 4000- 14000K. On the other hand, the Default Color Jitter is more sensitive to the illumination color and the classification accuracy decreases when the scene illuminant is moving away from the white illuminant.
Finally, we conducted a color sensitivity analysis to inspect the impact of the color information for each neuron in the trained models. In order to perform the analysis adopted the color selectivity index presented by |Rafeaas & Vanrell (2018). This index is defined as the property of a neuron to highly activate when color is present the imput images, and not activate when color absent. We computed the index for the last layer of the different backbones. More details on the color selectivity index can be found in Appendix A3.3 The results are shown in Figure and represent the amount of color sensitive neurons for each of the considered models. As can be seen, a major difference can be observed between the Default Color Jitter and the other proposed solutions: this difference in the sensitivity of the models is mainly related to the presence of the Random GrayScale augmentation the Default Color Jitter configuration. This result confirms the hypothesis that models trained in this way color invariant, a property that negatively affects the model in scenarios where color information has an important role, as demonstrated by our experiments. We can see also how the Planckian Jitter (chromaticity only) configuration is slightly more color sensitive with respect the other configurations, due to the augmentation related only to realistic color variations in the scene.
5 CONCLUSION
Existing research on self-supervised learning mainly focuses on tasks where color is not a decisive component, and involuntarily exploits data augmentation procedures that negatively affect color- sensitive tasks, as we showed in this paper. We presented an alternative data augmentation technique based on the physical properties of light, called Planckian Jitter, and showed its positive effects on a wide variety of tasks where the intrinsic color of the objects (related to their reflectance) is crucial for discrimination, while the illumination source not relevant, and as such the model should learn to be invariant to it. The general applicability of the proposed strategy for data augmentation is shown by experimenting with three different popular self-supervised models, demonstrating consistent im- provements. We also proposed solution that manages to exploit both color and shape information, by concatenating features learned with different modalities of self-supervision, achieving even su- perior performance. Finally, presented an in-depth analysis the sensitivity of trained neurons to color information, to demonstrate the difference in models trained with and without our Planckian Jitter. This produces more general model for image classification, since obtain improvements also on con-color-sensitiin general datasets.
9

---

Under review as a conference paper at ICLR 2022
REPRODUCIBILITY STATEMENT
The code of the Planckian Jitter data augmentation procedure, written in MATLAB and PyTorch 1.7.0, will be made available upon acceptance.
The training runs have been performed using Pytorch in combination with Pytorch Lightning Bolt framework, which provides implementation of SimSiam methodology for backbone contrastive training. The model has been trained using CUDA deterministic, and random seed set to 1234.
All datasets used for the training and fine-tuning are publicly available. Only the CUB200 dataset has been pre-processed, by cropping each image using the given bounding box values, available alongside each image in the annotations files.
REFERENCES
David G Andrews. An introduction to atmospheric physics. Cambridge University Press, 2010.
Marco Buzzelli, Simone Bianco, and Raimondo Schettini. Arc: Angle-retaining chromaticity dia- gram for color constancy error analysis. JOSA A, 37(11):1721-17333 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Floria Balcan, and Hsuan- Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Anmual Conference Neural Information Pro- cessing Systems 2020, NeurlPS 2020, December 6-12, 2020, virtual, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Claudio Cusano, Paolo Napoletano, and Raimondo Schettini. Tik+: A database for benchmarking color texture classification and retrieval methods. Sensors, 2113):1010, 2021.
Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp- 1422-1430, 2015.
Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances neural information processing systems, 27:766-774, 2014.
Graham D Finlayson and Gerald Schaefer. Solving for colour constancy using a constrained dichro- matic reflection model. International Journal of Computer Vision, 12(3):127-144, 2001.
J-M Geusebroek, Rein Van den Boomgaard, Arnold W. M. Smeulders, and Hugo Geerts. Color invariance. IEEE Transactions on Pattern analysis and machine intelligence, 23(12):1138-1135) 2001.
lean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar,
Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284. Curran Associates, Inc., 2020. URL https: httpttpppppprocceeeeee ngs. neur /c//ppper/202////eeeeeaaadaeeeed
0ee144b1775 9 Paper. pdf
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
10

---

Under review as a conference paper at ICLR 2022
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729-9738. 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 231N, 7(7)3,, 2015.
David G Lowe. Distinctive image features from ccale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Bangalore S Manjunath and Wei-Ying Ma. Texture features for browsing and retrieval of image data. IEEE Transactions on paltern analysis and machine intelligence, 18(8):837-842, 1996.
Maria-Eles Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Ivet Rafegas and Maria Vanrell. Color encoding in biologically-inspired convolutional neural net- works. Vision research, 151:7-17, 2018.
Yushan Feng Saihui Hou and Zilei Wang- Vegfru: A domain-specific dataset for fine-grained visual categorization. In IEEE International Conference on Computer Vision, 2017.
Yonglong Tian, Chen Sun, Ben Poole. Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? Larochelle, M. Ranzato, Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp- 6827-6839. Curran Associates, Inc., 2020.
Shoji Tominaga, Satoru Ebisui, and Brian A Wandell. Color temperature estimation of scene illumi- nation. In Color and Imaging Conference, volume 1999, pp. 42-47. Society for Imaging Science and Technology, 1999.
J von Kries. Theoretische studien über die umstimmung des sehorgans. Festschrift der Albrecht- Buddigs-Unnierssiee pp. 145-158, 1902.
Welinder, Branson, Mita, C. Wah, Schroff, Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010- Californi Institute of Technology, 2010.
Gunter Wyszecki and Walter Stanley Stiles. Color science, volume 8. Wiley New York, 1982.
Jure Zhontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning, ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 22310-12320 PMLR, 2021.

---

Under review as a conference paper at ICLR 2022
A APPENDIX A
A.1 PLANCKIAN JITTER
Figure illustrates the illuminants sampled from the distribution of black body radiator, with related color temperature 7 in the interval between 3000 and 15000K. The resulting spectra are visualized on the left and in the middle, while the resulting distribution of illuminants visualized with the Angle-Retaining Chromaticity diagram on the right.
12 S
Nomalized spectral radiance
Speciral radiance (Wimm))
8
0.8
0.4
4
0.2
2
400
700
400
500
600
"
Wavelength (nm)
Wavelength (nm)
Figure 4: Spectral power distributions (left) and corresponding ARC chromaticities (right) of the sampled black body radiator, used to generate Planckian jittering.
Figure 5 shows comparison between default color jitter (left) and Planckian jitter (right), replicat- ing Figure in xy chromaticity.
A.2 DATASETS DETAILS
In section 4.2.2 we analyze the impact of our data augmentation, when using the features extracted by the backbone trained on TINY-IMAGENET dataset, on new datasets. The datsets used in the fine-tuning step
FLOWERS- 102 dataset by Nilsback Zisserman (2008): dataset consisting of 102 flower categories, commonly occuring the United Kingdom. Each class consists of between 40 and 258 images, for a total amount of 8.189 images.
VEGFRU dataset by (aahui Hou Wang 2017): this dataset consists of more than 160,000
images of vegetables and fruits, divided in 292 classes.
CUB-200 dataset by Welinder et al. 2010 an image dataset made 6033 photos of 200 bird species.
TIK+ dataset by Cusano et al. (2021): a dataset of textures divided 1129 classes, orga- nized in 5 groups of 266 super classes. We adopted the 266 class labeling fine tune and test our models.
Few example images for each of the color tasks datasets reported in Figure 6
A.3 COLOR SELECTIVITY INDEX
Color selectivity is defined as the property neuron that activates highly when a specific color appears in the input image, and do not when the color is not present. The color selectivity index of neuron is computed by estimating the ratio between its global activation with input color images and the global activation with corresponding grayscale images. By definition, the color selectivity index correspond to the complement to of the ratio between the area under the activation curve given by the N-top grayscale images and the area under the activation curve given given by the N-top colored images.
w/
Lin
(6)
Wj..h
12

---

Under review as a conference paper at ICLR 2022
Figure 5: Default color jitter (left) and Planckian jitter (right) in xy chromaticity.
Figure 6: Example of images taken from the datasets used for the downstream classification task. From left to right the images are taken from FLOWERS- 102 dataset, CUB-200 dataset, VEGFRU dataset and Tl K+ dataset.
In the equation, (Wj.i.))==::n the neuron activation values from the N-top colored images, while ("',,,));=1:N are the neuron activation values, by the same neurons, the N-top grayscale images. We can distinguish between neurons that color blind neurons that highly relies color infor- mation by looking at the alpha value obtained: an alpha value more than 0.25 means that a specific neuron high color selective, while an alpha value less than 0.1 means that the neuron is basically color blind. These thresholds have been selected in relation to the work presented by Rafegas & Vanrell (2018) Using the definition of color selectivity index shown above, we collected alpha values for the neurons on the last layer of the encoders trained with different data augmentation configurations, in order to compare the models sensitivity to color and how it change in relation to the training procedure adopted.
To analyze the learned features robustness different realistic illuminants, we tested the models with different versions of the CIFAR- 100 and FLOWERS-102 datasets. We applied the Planckian Jitter on the two datasets, generating 25 different versions of each datasets, one for each illuminant sampled. Using these different versions of the datasets we then test the models for each illuminant and collect the classification accuracies. The results on both CIFAR- 100 and FL.OWERS- 102 can be seen in Figure 7
A.4 EXECUTION TIME COMPARISON
Here reported execution time comparison performed to assess the usability of the proposed data augmentation, with respect
already existent Color Jitter data augmentation algorithm.
In order to analyse such aspect of the algorithms, executed the two algorithms, the Color Jitter image transform from Py Torch Torchvision package (respectively at versions v1.8.1 and v0.9. and the proposed Planckian Jitter different image resolutions. For each resolution, 40 times and averaged the execution time. Results can be seen in Figure g The exccution has been performed in CPU using a Intel i7-8700 processor. As be seen, the proposed augmentation algorithm is faster with respect to the Color Jitter algorithm.
the code
13

---

Under review as a conference paper at ICLR 2022
CPw100
Foweri52
3
Detuut wn Random Grrabcae
Mancdan fiter brngrnnetccettee onty)
Rancdan, fier (dromatictty
.2200
9922
39322
12000
12000
LE3C0
0220
9022
30332
10000
12000
L4C00
iRumirant
Fuuminant
Figure 7: Illumimant robustness analysis. To assess the features invariance to realistic color changes in the images, for each method has been tested the classification accuracy with 25 different versions of the datsets. The images of the two datasets (CIFAR- 100 on the left and FLOWERS-102 on the right) have been modified with the illumimants from temperature 3000 K 15000 K, using the Planckian Jitter transform.
1.4
1.3431
1.0
0.7906
0.6
Planckian Jitter
0.3362
0.2
0.1903
0.0204
0.0031
0.0053
0.048
0.0019
0.003
0.0116
128x128
256x256
512x512
1024x1024
2048x2048 4096x1096
Resolution
Figure 8: Execution time comparison between the proposed data augmentation algorithm imple- mentation and the Color Jitter implementation in Pytorch Torchvision v0.9.1. For each resolution we executed both the algorithms 40 times.
14


Reviews:
[{"review_id": "WC6YmpNdEsq", "reviewer": "Reviewer_wAfa", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Realistic data augmentation is important in self-supervised learning because this is the sort of data variability that one wants to ignore and be invariant to. \nOverall, this is an example of physics-aware machine learning (or in simpler words, do things just right ;-) ). The proposed \"Planck+VonKries\" color augmentation transform is more than 100 years old, but it may be interesting for different audiences for different reasons: (a) people with color science background will not find it surprising, but they may be interested to see the quantitative gain that can be obtained from doing the classical right thing, and (b) the average computer scientist facing this problem for the first time will find a good visual motivation to avoid random augmentation (fig. 1) and a physically founded recipe to get a 10% improvement in classification.\n\nAdditionally, authors make an interesting observation: they argue that unrealistic (random) transforms may induce overinvariance (insensitivity) to certain aspects of the data, thus reducing the performance of the model or shifting the focus to other features of the data. \nI think their results illustrate this general point: in their specific example, augmentation through unrealistic color jitter reduces the sensitivity to color (as confirmed by the number of color-sensitive neurons) and leads to extra consideration of spatial features. \nA similar argument could be made for other augmentation strategies: if the database lacks natural variability, the discovered features are going to be unbalanced.\n\nOverall I find this work as an interesting exercise in the context of data augmentation, although the technique is not novel and the general consequences are not extracted.\n\nOther points to address:\n\n* While the general change-of-focus effect is true, the authors should be more specific about when this is going to be a benefit. As it is presented now, it depends on the database or on the relevance of color for class discrimination. \nMy opinion is that the proposed augmentation (as any natural data augmentation) is going to be more positive in too artificial (too restricted) databases where illumination is fixed or it has too low variability. These are the cases that require the introduction of a natural variability (in this case of the illumination). This is consistent with the fact that wide databases such as CIFAR-100 as opposed to Flower102 get no improvement from the proposed jitter (see performance numbers or figure 6). Maybe this is because CIFAR-100 already includes the variability of natural illumination and hence explicit augmentation with the Planckian variability is not necessary.   \n\nCan the authors provide some extra evidence about the fact that neurons insensitive to color (when conventional color jitter is introduced) capture better the shape/texture information?\n\n* Equation 4 for VonKries is confusing. \nVon Kries-like adaptation consists of dividing each LMS (I may admit each RGB) value by the value corresponding to the white (or illuminant) in situation 1, and multiply by the value corresponding to the white (or illuminant) in situation 2 (see for instance Chapt.9 in Fairchild's Color Appearance Models, 2013).\nThis implies the multiplication of each tristimulus vector by clarify Eq. 4.\n\n* The spectral radiances in Fig. 4 are confusing: using no normalization and a linear vertical axis implies that all the explored spectra seem \"blueish\" (with bigger contribution of short wavelengths). Decay in energy at low temperatures of the black body radiator make it difficult to see that between 3000 and 4000 K the contribution of long wavelengths is bigger ---> you have reddish illuminants too, but it is hard to see in that figure if you dont nromalize the energy or put a log scale in the vertical axis... Note that for instance the CIE A illuminant corresponds to 2856 K (almost the 3000K reported by the authors) and it is markedly reddish.\n\n* Using the novel ARC chromatic diagram does not add anything special with regard to classical CIE xy diagram. Why not using the standard diagram?. Incidentally, I was surprised to see that the Planck locus is straight in ARC, but the variations in the colors in Fig. 1 (diagram at the right) go in curves... is this right?.\n\n* Eqs. 1 and 2 seem to suggest that in Fig. 2 there is an additional Multilayer Perception in the lower branch of the training, isnt it?", "summary_of_the_paper": "The authors propose a physically sensible data augmentation that takes into account the actual variation of day light illumination in self-supervised contexts. \nThey analyze the impact on the classification performance, the sensitivity of the performance to illumination changes, and the emergence of neurons sensitive or insensitive to color. \n", "summary_of_the_review": "I find this work is an interesting exercise in the context of data augmentation (an illustration of how classical models from physics may benefit the learning), although the technique is not novel and the general consequences are not discussed in detail. \n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Tgdly0zLfW_", "reviewer": "Reviewer_7zG7", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n - The proposed physics-based color augmentation is easy to understand and implement.\n - The performance seems to be surprisingly better than typical random color jittering.\n - This color augmentation can be applied to a wide variety of tasks.\n\nWeaknesses:\n - Is the proposed physics-based color augmentation slower than typical color jittering. Many tasks need color augmentation performed on the fly during training. If the proposed color augmentation is much slower, the impact of this work will be marginal.", "summary_of_the_paper": "This paper first examines that typical color jittering augmentation is harmful to feature representation learning. Then the authors proposed a physics-based color augmentation, called Planckian jitter to improve the performance. The proposed Planckian jitter performs better with the recent contrastive and self-supervised learning schemes.", "summary_of_the_review": "This paper proposed a general color augmentation that performs better than the typical color jittering and demonstrates better performance in contrastive and self-supervised learning schemes. The only concern I have is the speed.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "0_N6xiMZH7W", "reviewer": "Reviewer_dWZ9", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "**Pros:**\n\n+ Well written and easy to follow manuscript \n+The idea of the physics-based Planckian jitter which transforms colours along chromaticity lines is interesting\n+ There does seem to be some support in the experiments for Planckian jitter improving results for colour-dependant domains (I.e. Table 1 when comparing \"Default Color Jitter w/o Random GrayScale\" and \"Planckian Jitter\")\n\n\n**Cons:**\n\n- Comparing the \"Latent space combination\" with other forms of augmentations seems very unfair since \"Latent space combination\" has double the capacity and requires double the training compute. To me \"Latent space combination\" seems more as an ensemble of models. To make it a fair comparison, it needs to be compared with other ensemble combinations (e.g. \"Default Color Jitter w/o Random GrayScale\"+\"None\" or \"Default Color Jitter w/o Random GrayScale\"+\"Random Crop\" seems like a reasonable baseline). Otherwise, I think \"Latent space combination\" should be removed altogether.\n- Since cropping is one of the most important transformations for SSL, for all practical purposes there should be two versions of Table 1, one with and one without random cropping applied. I can imagine random cropping may impact the results.\n- The main focus in current comparisons should be between \"Default Color Jitter w/o Random GrayScale\" and \"Planckian Jitter\", but this was left out entirely in Tables 2 and 3. Comparing \"Default Color Jitter\" (with Random GrayScale) against \"Planckian Jitter\" does not seem fair.\n- Adding to above point, even comparing \"Planckian Jitter\" against \"Default Color Jitter w/o Random GrayScale\" isn't really enough. Since, as clearly indicated in Table 1, Planckian jitter mainly modifies brightness and contrast, there should be a direct comparison with an augmentation which changes brightness, contrast, brightness+contrast, and hue.\n- There isn't any large scale datasets used in the experimentations, which raises the question of scalability and whether the Planckian augmentation would still be relevant with larger datasets.\n- Planckian jitter is an augmentation technique which can also be evaluated on supervised learning tasks where image colours are important (and similarly, for transfer learning afterwards). Adding supervised experiments can add to the comprehensiveness of the experiments.\n - Section 4.2.1 mentions \"As can be seen in Table 1, if color augmentations are removed completely (None configuration), the accuracy drops of 18%.\". This seems incorrect when comparing \"Default Color Jitter w/o Random GrayScale\" to \"None\".\n- The left plot of the colour sensitivity analysis (Fig 3.) seems unfair. The Planckian jitter models were specifically trained to ignore Planckian jitters, so one would naturally expect their results to be more robust with respect to Planckian jittering.  \n\n\n**Typos:**\n- Section 3.1 paragraph 1 states \"...default color jitter (see Fig. 3).\", I believe this was intended to reference Fig 1.\n\n**Questions for authors:**\n\n1. It seems very odd to me that in table 1, \"None\" performs so similar to \"Default Color Jitter w/o Random GrayScale\". Do you have any thoughts on why this happens?\n2. Were there any other augmentations used during training (cropping, rotations, etc)?\n\nEdits:\n\nNov 3 -- typo fix", "summary_of_the_paper": "**Overview.**\nThis paper proposes a novel type of image colour augmentation to be used during self-supervised learning (SSL).\n\n**Background.**\nIn a typical SSL setting, similar samples are generated by randomly augmenting an image in a variety of different ways: random cropping, colour jittering, random rotations, etc. These similar images are then passed through a neural network, and the predicted features for similar samples are trained to be close to each other based on some similarity metric (ignoring the contrastive term description here as it is not directly related to the paper).\n\n**Motivation.**\nIn this paper, the authors address the issue of using colour jittering as augmentation during SSL. Specifically, using colour jittering pushes the network towards invariance to image colours, while relying more on the shape and texture of objects when making predictions. The authors point out that despite the benefits of this for many general detection tasks, this will be a detrimental property when dealing with more colour-dependant tasks.\n\n**Method.**\nThus, they propose to use Planckian jittering instead of colour jittering. Planckian jittering is a physics based augmentation method proposed by the authors (although the formulations for it come from existing literature) to re-illuminate the training images within a realistic distribution which leads to more realistic and constrained colour augmentations than colour jittering. The paper claims that Planckian jittering still helps improve network's dependence on shape and texture of objects (although less than colour jittering), while limiting the network's invariance to image colours.\n\n**Experiments.** \n6 SSL models were trained independently on CIFAR-100: 3 used different variants of the Planckian jittering, 2 used different variants of colour jittering (w/ and w/o Random GrayScale), and 1 used no augmentations. Linear classifiers for CIFAR-100 and FLOWERS-102 classification tasks were then trained on top of each of the SSL models' features, where FLOWERS-102 is the task that is claimed to be more heavily colour-dependant. Moreover, an extra linear classifier was trained on the concatenation of features from a Planckian jittering and a colour jittering model (called the \"Latent space combination\" model). Based on accuracy, \"Latent space combination\" outperforms all other models by a significant margin on both datasets. Planckian jitter seems to outperform other augmentations on FLOWERS-120 (Table 1), which supports the claim of the authors. On CIFAR-100, Planckian jitter performs slightly worse than colour jittering, which the authors attribute to the reduction of colour invariance in the features.\n\nA very similar experiment was also done with different datasets (SSL training on tiny-imagenet, linear classifier trained on FLOWERS-102, CUB200, VEGFRU, T1K+), which also obtained similar results and conclusions. Moreover, in another similar experiment, the SSL models were trained with different SSL configurations (SimSiam, SimCLR, Barlow Twins) to indicate the generality of Planckian jitter for different types of SSL configurations. In another experiment, the robustness of the different models on augmented images using Planckian jittering was evaluated. Lastly, the colour sensitivity was analyzed to inspect the impact of colour information in neuron activations for each model.", "summary_of_the_review": "This paper proposes an interesting idea and is very well written. The issues (with colour jittering) pointed out in this paper were previously known. Thus I view this paper as more geared towards practical purposes. However, the experiments are not comprehensive enough to strongly support the practical claims. Thus, my vote is for the paper to be rejected in its current form.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers were in general lukewarm about the paper, not convinced by why realistic augmentation mean more robust features in SSL, had concerns over the szie of the datasets (up to ~100k), and the success depends on the relevance of color for classification. The AC agrees with the reviewers. While the paper sounds interesting, there are many questions remain unanswered -- it's unclear that the rebuttal addressed the concerns shared by the reviewers.

In addition to the comments by the reviewers, the AC also feels that the overall design is adhoc and it's unclear that the proposed augmentation can generalize to larger, more practical problems.
Decision: Reject