Planckian Jitter: countering the color-crippling effects of color jitter on self-supervised training
Authors: Simone Zini
Alex Gomez-Villa
Marco Buzzelli
Bartłomiej Twardowski
Andrew D. Bagdanov
Joost van de weijer
Venue: ICLR 2023 poster (2023)
Keywords: Contrastive Learning
Self-Supervised Learning
Color Features
Illuminant Invariance

Abstract: Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, physics-based color data augmentation - which we call Planckian Jitter - that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information.
Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets. 
In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations.
Official code available at: https://github.com/TheZino/PlanckianJitter

Published as a conference paper at ICLR 2023
PLANCKIAN JITTER: COUNTERING THE COLOR- CRIPPLING EFFECTS OF COLOR JITTER ON SELF-
SUPERVISED TRAINING
Simone Zini', Alex Gomez-Ville Marco Buzzelli', Barttomiej Twardowski Andrew D. Bagdanov", Joost de Weijer2
Department of Informatics Systems and Communication, University of Milano Bicocca, Milan, Italy {s [sinoee. 2 zini, marco.. buzzelli @@unimi mimb..ii.... it
*Computer Vision Center, Universitat Autònoma de Barcelona, Barcelona, Spain fagomezvi, biwardowski, joost @eve. uab.es
MMedia Integration and Communication Center, University of Florence, Florence, Italy andrew bagdanovvun. it IDDEAS NCBR, Warsaw, Poland
ABSTRACT
Several recent works self-supervised learning trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature rep- resentations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, ahysics-based color data augmentation which we call Planckian Jitter that creates realistic variations in chromaticity and produces model robust to illu- mination changes that can commonly observed real life, while maintaining the ability to discriminate image content based on color information. Experiments confirm that such representation is complementary to the representations learned with the currently-used color jitter augmentation and that simple concatenation leads to significant performance gains on a wide range of downstream datasets. In addition, we present color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features robust with respect illuminant variations. Official code available at:
comp The www.noo//laacckiii The/ino/Plaacccaaiittte annitte
INTRODUCTION
Self-supervised learning enables the learning of representations without the need for labeled data (Doersch et al. 2015 Dosovitskiy et al. 2014). Several recent works learn representations that are invariant with respect to a set of data augmentations and have obtained spectacular results (Grill cet al. 2020 Chen & He 2021 Caron et al. 2020), significantly narrowing the gap with supervised learned representations. These works vary in their architectures, learning objectives, and optimiza- tion strategies, however they similar in applying common set of data augmentations generate different image views. These algorithms, while learning to map these different views to the same latent representation, learn rich semantic representations for visual data. The set of transformations (data augmentations) used induces invariances that characterize the learned visual representation.
Before deep learning revolutionized the way visual representations are learned, features were hand- crafted to represent various properties, leading to research on shape (Lowe) 2004), texture (Man-) junath & Ma 1996). and color features (Finlayson & Schaefer, 2001 Geusebrock al., (001)) Color features were typically designed to be invariant to a set of scene-accidental events such as shadows, shading, and illuminant and viewpoint changes. With the rise of deep learning, feature
"Corresponding author

---

Published as a conference paper at ICLR 2023
Figure Default color jitter (left) and Planckian Jitter (right). Augmentations based on default color jitter lead to unrealistic images, while Planckian Jitter leads to a set of realistic ones. The ARC chromaticity diagrams for each type of jitter are computed by sampling initial RGB values and mapping them into the range of possible outputs given by each augmentation. These diagrams show that Planckian Jitter transforms colors along chromaticity lines occurring in nature when changing the illuminant, whereas default color jitter transfers colors throughout the whole chromaticity plane.
representations that simultaneously exploit color, shape, and texture are learned implicitly and the invariances are byproduct of end-to-end training (Krizhevsky et al. 2009). Current approaches to self-supervision learn a set of invariances implicitly related the applied data augmentations.
In this work, we focus on the currently de facto choice for color augmentations. We argue that they seriously cripple the color quality of learned representations and we propose an alternative, physics-based color augmentation. Figure [ (left) illustrates the currently used color augmentation on a sample image. clear that the applied color transformation significantly alters the colors of the original image, both in terms of hue and saturation. This augmentation results in representation that is invariant with respect to surface reflectance an invariance beneficial for recognizing classes whose surface reflectance varies significantly, for example many man-made objects such cars and chairs. However, such invariance is expected to hurt performance on downstream tasks for which color is important feature, like natural classes such as birds food. One of the justifications is that without large color changes, mapping images to the same latent representation can be purely done based color and complex shape or texture features are learned. However, as a result the quality of the color representation learned with such algorithms is inferior and important information on surface reflectance might be absent. Additionally, some traditional supervised learning methods propose domain-speciff variations of color augmentation Galdran et al. (017)) Xiao et al. (2019).
In this paper we propose an alternative color augmentation (Figure I right) and we assess its impact on self-supervised learning. We draw on the existing color imaging literature on designing features invariant to illuminant changes commonly encountered
the real world (Finlayson & Schaefer, 2001). Our augmentation, called Planckian Jitter, applies physically-realiste illuminant variations. We consider the illuminants described by Planck's Law for black-bodi radiation, that are known to be similar to illuminants encountered in real-life (Tominaga et al. 1999). The aim of our color aug- mentation is to allow the representation to contain valuable information about the surface reflectance of objects- feature that is expected to be important for wide range of downstream tasks. Com- bining such a representation with the already high-quality shape and texture representation learned with standard data augmentation leads to more complete visual descriptor that also describes color.
Our experiments show that self-supervised representations learned with Planckian Jitter are robust to illuminant changes. In addition. depending the importance of color in the dataset, the pro- posed Planckian jitter outperforms the default color jitter. Moreover, for all evaluated datasets the combination of features of our new data augmentation with standard color jitter leads to significant performance gains of over 5% on several downstream classification tasks. Finally, we show that Planckian Jitter can be applied to several state-of-the-ar self-supervised learning methods.
2 BACKGROUND AND RELATED WORK
Self-supervised learning and contrastive learning. Recent improvements in self-supervision learn semantically rich feature representations without the need for labelled data. In SimCLR (Chen) et al. 2020a) similar samples created by augmenting input image, while dissimilar are chosen
2

---

Published as a conference paper at ICLR 2023
Encoder [Rasnet- 18]
MLP
Encoder [Resnet-18]
ougmentation including Pranckian Jiter
Similariyy
Stop
Encoder
|Resnet-15
Linear Classifier
SimSiam Training
Stop-grad
Class
Linear Classifler training
Figure 2: SimSiam training procedure exploiting Planckian-baal based data augmentation (left), and fine- tuning the linear classifier using the trained encoder (right).
randomly (Chen et al. 2020a). To improve efficiency, MoCo (He al. 2020) and its enhanced version (Chen et al. 2020b) use a memory bank for learned embeddings which makes sampling efficient. This memory is kept in syne with the rest of the network during training via momentum encoder. Several methods do not rely on explicit contrastive pairs. BYOL uses an asymmetric network incorporating
additional MLP predictor between the outputs of the two branches (Grill) et arall,]2000 2020). One of the branches is kept "ofiline" and is updated by a momentum encoder. SimSiam defines simplified solution without momentum encoder (Chen He 202T). It obtains similar high-qualit results and does not require a large minibatch size, in contrast to other methods.
We use the SimSiam method verify our proposed color augmentation (we also apply it to Sim- CLR (Chen et al. (2020a) and Barlow Twins (Zbontar et al. (021)) in the experiments). The main component of the network a CNN-based asymmetric siamese image encoder. One branch has an additional MLP predictor whose output aims to be as close as possible to the other (Figure 2 The second branch is not updated during backpropagation. A negative cosine loss function is used:
C
[P(pi, stopgrad (2)) + D(p2, stopgrad (21))
PA
2B
D(PA, ZB)
[aAl2
|
where 21, 22 are representations for two different augmented versions, #1 and æ2, the same image The MLP is applied in alternation to either 2 or #2, producing respectively Pi P2. Note that Figure [ouure]]n]] only shows an instance for æ1 and does not show p2. The stopgrad(.) operation blocks the gradient during the backpropagation. In SimSiam no contrastive term is used and only similarity is enforced during learning.
Data augmentation. Data augmentation plays central role in the self-supervised learning pro- cess. Authors (hen et al. (2020a) and /Zbontar et al. (202T) diseuss the importance of the different data augmenaations.
of well-defined transformations was proposed for SimCLR (Chen al.) 2020a). This set is commonly accepted and used in several later works. The augmentations include: rotation, cutout, flip, color jitter, blur and Grayscale. These operations image to generate the different views 21, from which are extracted the features and 22 used in the self-supervision loss in Eq. Applied the same image, contrastive-lik self-supervision learns representations invariant such distortions.
randomly applied to an
This multiple view creation is task-related (Tian et al. 2020). however color jittering operating on hue, saturation, brightness and contrast, is one of the most important ones in terms of overall useful- ness the learned representation for downstream tasks (Chen et al. 2020a; Zbontar et al.) 2021). Color jitter induces certain level of color invariance (invariance to hue, saturation, brightnesss and contrast) which are consequently transferred to the downstream task. As a consequence, we expect these learned features to underperform (2000) were the first to point out that the imposed invariances might not be beneficial for downstream tasks. As solution, they propose to learn different embedding spaces in parallel that capture each of
downstream tasks for which color is crucial. Xiao et al.

---

Published as a conference paper at ICLR 2023
the invariances. Differently than them, focus the color distortion and propose a chyyicc-bbssel a
color augmentation that allows learning invariance to physically realistic color variations.
Color imaging has long tradition in research on color features invariant to scenc-accidental events such as shading, shadows, and illuminant changes (Geusebrock et al., 2001 Finlayson & Schae- fer 2001). Invariant features were found be extremely beneficial for object recognition. The invariance to hue and saturation changes induced by color jitter, however, is detrimental to object recognition for classes in which color characteristies are fundamentally discriminative. Therefore, in this work we revisit early theory on illuminant invariance (Finlayson & Schaefer 2001) to design an improved color augmentation that induces invariances common in the real world and that, when used during self-supervised learning, does not damage the color quality of the learned features.
3 METHODOLOGY
The image transformations introduced by default color jitter creates variability in training data that indiscriminately explores all hues at various levels of saturation. The resulting invariance is useful for downstream tasks where chromatic variations are indeed irrelevant (e.g. car color in vehicle recognition), but is detrimental downstream tasks where color information is critical (e.g. natural classes like birds and vegetables). The main motivation for applying strong color augmentations that this it leads to very strong shape and texture representations. Indiscriminately augmenting color information in the image requires that the representation solve the matching problem using shape (Chen al. 2020al
As an alternative to color jitter, we propose a physics- color augmentation that mimics color variations due illuminant changes commonly encountered in the real world. The aim is to reach a representation that does not have the color crippling effects of color jitter, and that better describes classes for which surface reflectance is a determining feature. When combined with default color jitter, this representation should also provide a high-quality shape/texture and color representation.
3.1 PLANCKIAN JITTER
We call our color data augmentation procedure Planckian Jitter because it exploits the physical description of black-body radiator re-illuminate training images within a realistic illuminant distribution (Finlayson & Schaefer [001] Tominaga et al. 1999). The resulting augmentations are more realistic than those of the default color jitter (see Fig. D. The resulting learned, self-supervised
feature representation is thus expected to be robust to illumination changes commonly observed in real-world images, while simultaneously maintaining the ability to discriminate the image content based on color information.
Given an input RGB training image 1, our Planckian Jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions. The data augmentation procedure is as follows:
1. we sample a new illuminant spectrum (x from the distribution of a black-body radiator;
2. we transform the sampled spectrum or into its sRGB representation pr E TR3
3. we create a jittered image I' by reilluminating with the sampled illuminant pr; and
4. we introduce brightness and contrast variation, producing Planckian image I".
radiating black body at temperature T can be synthesized using Planck's Law (Andrews) 2010):
2xhc²
(T(() (e 1) W/m",
where 2.99792458 x 1 m/s is the speed of light, 6. 6.626176 10 33 Js is Planck's 's constant, and k = 1. 1.380662 x 10-23 J/K is Boltzmann's 's constant. We sampled T the interval between 3000 and 1 15000 which is known result in set of illuminants that can be encountered in real life (Tominaga et al. 1999). Then, we discretized wavelength À in 10nm steps (AA) in the interval between 400nm and 700nm. The resulting spectra are visualized in Figure 4 (left) in Appendix A.I
'This pointed in the discussion of Figure 5 in Chen al. 2020a
4

---

Published as a conference paper at ICLR 2023
The conversion from spectrum into sRGB is obtained according to Wyszecki & Stiles (1982):
1. we first map the spectrum into the corresponding XYZ stimuli, using the 1931 CIE stan- dard observer color matching functions etX,Y,Z) (A), in order to bring the illuminant into a standard color space that represents person with average eyesight;
2. we normalize this tristimulus by its Y component, convert it into the CIE 1976 L color space, and fix its L. component to 50 a 0-to-1000 scale, allowing us to constrain the intensity of the represented illuminant in controlled manner as separate task; and
3. we then convert the resulting values sRGB, applying a gamma correction and obtaining Pr {R. G, B); the resulting distribution of illuminants is visualized with the Angle- Retaining Chromaticity diagram (Buzzelli et al. (2020) in (righ))((right) in Appendix
All color space conversions assume a D65 reference white, which means that a neutral surface illuminated by average daylight conditions would appear achromatic. Once the new illuminant has been converted in sRGB, it is applied to the input image / by resorting to Von-Kries-like transform (von Kries 1902 given by the following Channel-wise scalar multiplication:
[(RRG.B) = flR,G,B). {R, G. 1,/11,1,1},
(4)
where we assume the original scene illuminant to be white (1,1,1). Finally, brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination:
I" CB cc I' + (1-cc) a (cB I'),
(5)
where cB = 0.8 and 0.8 represent, respectively, brightness and contrast coefficients, and is a spatial average function.
3.2 COMPLIMENTARITY OF SHAPE, TEXTURE AND COLOR REPRESENTATIONS
The supf-suue learning paradigm involves a pretraining phase that relies data augmentation to produce a set of features with certain invariance properties. These features are then used as the representation for a second phase, where learn given supervised downstream task. The default color jitter augmentation generates features that are strongly invariant to color information, resulting in high-qualit, representations of shape and texture, but that an inferior descriptor of surface reflectances (i.e. the color of objects). Our augmentation based on Planckian Jitter (see Figure 1 is based on transformations mimicking the physical color variations in the real world due to illuminant changes. As result, the learned representation yields high-quality color description of scene objects (this is also verified Appendix [A.8). However, it likely leads to a drop in the quality of the shape and texture representation (since color can be used to solve cases where previously shape/texture were required). To exploit the complimentarity the two representations, we propose to learn both one with color jitter and one with Planckian Jitter and then concatenate the results in a single representation vector (of 1024 dimensions, i.e. twice the original size of 512). We call this Latent space combination (LSC).
EXPERIMENTAL RESULTS
In this section. we analyze the color sensitivity of the learned backbone networks, verify the superi- ority of the proposed color data augmentation method compared to the default color jitter on color datasets, and evaluate the impact on downstream classification tasks. We report additional results on computational time of the proposed Planckian augmentation in Appendix A.5
4.1 TRAINING AND EVALUATION SETUP
We perform unsupervised training on two datasets: CIFAR-100 (Krizhessky et al. 2009 (32 x 32) and ImageNet (224 224). We slightly modify the ResNet18 architecture accommodate 32 32 images: the kernel size of the first convolutional was reduced from 7 x 7 to 3 x 3 and the first max
We conduct the investigative part of our research in an agile manner on low-resolution images, then transfer the significant configurations hegohenre,,lutton, ether confirm or refute the initial hypotheses.
5

---

Published as a conference paper at ICLR 2023
pooling layer removed. SimSiam training was performed using Stochastic Gradient Descent with a starting learning rate of 0.03, a cosine annealing learning rate scheduler, and mini-batch size of 512 (as in original SimSiam work by Chen & He (202T). For the training on the 1000-class ImageNet training set, we follow the same procedure as Chen & He (2021) with ResNet50.
The linear classifier training resolution 32 x 32 was performed on CIFAR-100 and FLOWERS- 102 (Nilsback & Zisserman 2008). CIFAR-100 is used as a baseline for the classification task. The linear classifier training for CIFAR-100 is done with Stochastic Gradient Descent for 500 epochs with a starting learning rate 0.1, a cosine annealing learning rate scheduler, and mini-batch size of 512. The FLOWERS- 102 dataset with 102 classes was selected to assess the quality of the features extracted in scenarios where color information plays an important role. Images from FLOWERS- 102 are resized 32 x 32 pixels to match the input dimensions of the pretrained model. Here we used the Adam optimizer with initial learning rate of 0.03.
For training linear classifiers at resolution 224 224 for downstream tasks we follow the evaluation protocol of Chen & (2021) We use six different datasets: IMAGENET, FLOWERS- 102, the fine- grained VEGFRU (Saihui Hou & Wang 2017). CUB-200 (Welinder et al., 2010), T (Cusano er al. )021), and USED (Ahmad et al. 2016). all resized to 224 x 224 pixels. More details about the datasets are provided in Appendix [A.2] In the case of CUB-200, each image was cropped using the bounding boxes given in the dataset annotations. For TIK+, we used the 266 class labeling to train and test the linear classifier.
To assess the impact of color data augmentations we define six different configurations:
Default Color Jitter (CJ): the default configuration, as used in SimSiam and SimCLR, uses both Random Color Jitter and Random Grayscale operations.
Default Color Jütter w/o Grayscale (CJ-): same as Desault, without Random Grayscale.
Planckian Jitter (PJ): uses the complete proposed Planckian Jitter operating on chromaticy, brightness, and contrast aspects of the images. No Random Grayscale is applied.
LSC Default Color Jitter + Planckian Jitter ([CJ,PJ]: This latent space combination (sim- ple concatenation of representations) combines the default color jitter with our Planckian jitter. It allows evaluation of the complimentary nature of the representations.
LSC Default Color Jitter + Default Color Jitter w/o Grayscale (/CJ,CI-)): We combine the default color jitter with a version without the Grayscale augmentation, since this repre- sentation is also expected to result in a better color representation.
LSC of two Default Color Jitter Models (JJ): We also show results of simply concate- nating two independently trained models (trained from different seeds) with default color jitter (an ensemble of two models).
In all experiments, these are combined with the other default augmentations (crop, flip, and blur).
4.2 COLOR SENSITIVITY ANALYSIS
We perform a robustness analysis the VegFru and CUB-200 datasets with realistic illuminant variations, and analyzed sensitivity to color information. This experiment is driven by two moti- vations: verify that we obtain invariance to the transformation applied during training, and to characterize the degradation of different non-Planckian training modalities. We assume as reference point the D65 illuminant, which for the purpose of this test is considered the default illuminant in every image. Given the different backbones pretrained on IMAGENET, we then train linear classi- fier on this dataset (assumed be under white illumination). For testing we create different versions of the dataset, each illuminated by illuminants of differing color temperature. This allows us to evaluate the robustness of the learned representations with respect to these illuminant changes.
Results are given in Figure 3a) (more results are provided in Figure 7 from Appendix A.4). Planck- ian Jitter obtains a remarkably stable performance between 4000K and 14000K, while Default Color Jitter is more sensitive to the illumination color and the classification accuracy decreases when the scene illuminant moves away from white. We also see that the combination of default and Planckian Jitter obtains the best results for all illuminants and manages to maintain high-level of invariance with respect to the illuminant color. Among the non- non-Planckian curves, default color jätter (CJ) is the most invariant, followed by CJ+CI- (although showing better performance at D65), and finally CJ-.

---

Published as a conference paper at ICLR 2023
60
600
50
50
40
Mathod
400
01.22 32
0.11
(++)
60
30
02 3
09 391
00 3.7)
077 2.8)
20
100
4000 6200
10000 12000 14000
4030
10000 12000 14000 Iluminanc
Iluminant
CJ
CJ
PJ
(b)
Figure 3: Color sensitivity analysis. (a) Robustness to illuminant change: report the accura- cies by differentll-traineee backbones as a function of illuminant. (b) The color sensitivity indexes computed for the different configurations used for training the backbone.
Table I: Accuracy results with ablation on color augmentations. Self sulf-supe training is per-
formed on CIFAR-100 100 and the learned features are evaluated at (32 x 32) on CIFAR-100 and FLOWERS- 102. Augmentation techniques include variations in hue and saturation (H&S), bright- ness and contrast (B&C), Planckian- chromaticity (P), and random Grayscale conversions (G). Accuracy refers to the linear classifiers trained with features extracted from the different backbones.
AUGMENTATION H&S
B&C
G P
CIFAR-100
FLOWERS-100
None
41.93%
36.47%
Default Color Jitter
V
59.93%
30.00%
41.96%
36.96%
32.46%
39.11%
36. 10%
39.51%
31.78%
41.96%
Planckian Jitter
47.31%
42.75%
In order to understand the impact of the color information on each neuron in trained models, we conducted an analysis using the color selectivity index described by Rafegas Vanrell (2018). This index measures neuron activation when color is present absent in input images. We computed the index for the last layer of different backbones, and high values indicate color-sensitive neurons. See Appendix(A) [A.3 for more details color selectivity. The results shown in Figure and indicate the number of color-sensitive neurons for each of the considered models. It is clear that the default color jitter has far fewer neurons dedicated color description. This result confirms the hypothesis that models trained this way are color invariant, a property that negatively affects the model in scenarios where color information has an important role seen in our experiments. We have also analyzed the results for the default color jitter without Grayscale augmentation (CJ-). These results show that removing the Grayscale augmentation improves color sensitivity significantly. We therefore also consider this augmentation in future experiments.
4.3 ABLATION STUDY
Six different models were trained and evaluated with a linear classification for image classification. For resolution
32 the model evaluated on CIFAR-100 and FLOOWERS- The results in terms of accuracy are reported in Table and Table We identify two different trends when inter- preting these results. On CIFAR-100, 100, removing color augmentations makes the model less powerful, due to the loss of color invariance in the features extracted by the encoder. This behaviour consis- tent with what was reported by Chen et al. (2020a). We see Table that if color augmentations removed completely (the None configuration),
(i.e. brightness/contrast and Random Grayscale) the accuracy drops by 18%. On FLOWERS- 102 the behavior the opposite however: removing color augmentations helps the model to better classify images, obtaining an improvement of 12.75%

---

Published as a conference paper at ICLR 2023
Table 2: Accuracy results for self-supervised training on CIFAR- 100 and evaluated 32 x 32 on CIFAR- (1FAR-100 and FLOWERS-102. The reported accuracy refers to the results of the linear classifiers trained with features extracted from the different trained backbones.
AUGMENTATION
CIFAR-100
FLOWERS-102
Default Color Jitter (CJ)
59.93%
30.00%
Default Color Jitter w/o Grayscale (CJ-)
41.96%
36.96%
Planckian Jitter (PJ)
47.31%
42.75%
LSC: [CJ, CJ-]
62.27%
47.65%
LSC: [CJ, PJ]
63.54%
51.66%
of accuracy with respect to the default color jitter. This behavior confirms that color invariance negatively impacts downstream tasks where color information plays an important role.
Taking a closer look at the various augmentation on FLOWERS- 1 02, we see that introducing more re- alistic color augmentations positively impacts contrastive training and produces models that achieve even better results with respect to the configuration without any kind of image color manipulation. Removing all color augmentations (None) improves results already by over 6%. Then, by sim- ply reducing the jittering operation to influence brightness and contrast, leaving hue and saturation unchanged, yields another boost in accuracy of 5.49% (to 41.96). When we start modifying chro- maticity using a more realistic transformation (i.e Planckian Jitter), the final result is boost of 6.28% in accuracy with respect to the None configuration. Also, on CIFAR- 100 we see an improve- ment of 5.38% from Planckian Jitter with respect color augmentation. Despite this improvement, in this scenario the contrastive training with the realistic augmentation does not yield better results with respect to the Defauls configuration because color only plays minor role on this dataset.
Given the results obtained using the data augmentations reported in Table [ and given the con- siderations made in Section 3.2] we evaluate the complementarity of the learned representation by combining latent spaces from different backbones. Results for two different latent space combina- tions are given in Table 4 On both datasets the Latent space combination of Default and Planckian Jitter achieves the best results. On the original CIFAR- 100 task, this combination achieves a total accuracy of 63.54%, 3.61% improvement over Default and 16.23% more compared Planckian Jitter alone. Comparing to the LSC using the Default ColorJitter w/o Grayscale, the version with Planckian Jitter achieves a small improvement of 1.27% in classification accuracy.
On the downstream FLOWERS-- 102 task, Latent space combination reaches accuracy value of 51. 31.6 an improvement of 21. 21.66% and 8.91 % in accuracy respectively compared to the two origi- nal configurations. Compared to the LSC using Default ColorJitter w/o Grayscale, the combination with Planckian Jitter achieves a higher result, and a bigger gap in terms of accuracy with respect to the CIFAR- 100 scenario. Here the use of Planckian Jitter brings an improvement of 4.01%, confirm- ing the impact of using realistic augmentation on classification tasks for which color is important.
4.4 EVALUATION ON DOWNSTREAM TASKS
Given the ablation study results, we performed the analysis of the proposed configurations on other downstream tasks using the backbone trained on higher resolution images (224 224 pixels). We report in Tahl results for: Default Color Jitter, Planckian Jitter, and latent space combinations.
Looking at the results, we see that the Planckian Jitter augmentation outperforms default color jitter on three datasets (CUB-200, T1 K+, and USED). Comparing the results on FLOWERS- 102 with those reported above at (32 x 32) pixels, we see that default color jitter actually obtains good results. We hypothesize that for high-resolution images the shape/texture information is very discriminative, and the additional color information yields little gain (for further analysis, see also A.7). Table also contains results for latent space combination, which confirm that the two learned representations complementary. Their combination yields gains of up to 9% TIK+. As sanity check also include the latent space combination of two networks separately trained with Color Jitter. This provides small gain on some datasets, but yields significantly inferior results than LSC.
8

---

Published as a conference paper at ICLR 2023
Table 3: Evaluation on downstream tasks. Self-supervised training was performed on IMAGENET at (224 224) and testing performed on the downstream datasets resized to (224 224).
AUGMENTATION
CUB-200
VEGFRU
T1K+
USED
FLOWERS-102
Default Color Jitter (CJ)
54.52%
67.63%
71.44%
59.90%
93.16%
Planckian Jitter (PJ)
56.28%
65.84%
77.42%
60.03%
90.29%
LSC [CJ.P]]
60.70% %
74.73%
80.49%
64.07% %
93.99%
LSC [CJ,C]]
56.16%
70.59%
73.47%
61.07%
93.13%
LSC [CJ,CJ-]
53.14%
70.54%
78.32%
63.87%
93.47%
Table 4: Effect of Plackian Jitter different contrastive learning models. Self-supervised training was performed on CIFAR-100 1-100 and the learned features are evaluated at (32 x 32) on CIFAR- 100 and FLOWERS- 102. We report the best configurations obtained SimSiam model and retrained SimCLR and Barlow Twins with those selected configurations.
FRAMEWORK
AUGMENTATION
CIFAR-100
FLOWERS-102
Default Color Jitter (CJ)
59.93%
30.00%
SimSiam
Planckian Jitter (PJ)
47.31%
42.75%
LSC [CJ,PJ]
63.54%
51.66%
Default Color Jitter (CJ)
56.99%
35.29%
SimCLR
Planckian Jitter (PJ)
47.75%
45.00%
LSC [CJ,PJ]
61.07%
55.78%
Default Color Jitter (CJ)
56.60%
40.78%
Barlow Twins
Planckian Jitter (PJ)
52.71%
54.50%
LSC [CJ.PJ]
62.85%
62.55%
Default Color Jitter (CJ)
65.23%
49.50%
VicReg
Planckian Jitter (PJ)
59. 19.19%
50.90%
LSC [CJ.PJ]
68.95%
60.80%
4.5 GENERALITY AND LIMITATIONS OF PLANCKIAN JITTER
To show that our approach is generally applicable to sulf-supervised methods exploiting color aug- mentations, we report in Table E experiments using SimCLR, Barlow Twins, and the more recent VicReg [Bardes et al. (2021). Independently of the model, Latent Space Combination consistently achieves the best results on both datasets.
A drawback of Planckian Jitter is the quality reduction of shape and texture representations. because the extreme color transformation of the standard Color Jitter force the network to solve the con- trastive learning problem mainly using shape/texture information. As have shown, this problem can be addressed by exploiting their complimentary nature. Secondly, our current latent space com- bination requires the training of two separate backbones, which will also learn partially-overlapping features. A training scenario with both augmentations simultancously in a single network while reserving part of the latent space for each augmentation could be pursued to address this limitation. Finally, object-specific augmentations that also take into account shadows, the type of reflectance, secondary light sources, refeecioss,csions, shadows, etc, could lead further improvements.
CONCLUSION
Existing research on self-supervised learning mainly focuses on tasks where color is not a deci- sive feature, and consequently exploits data augmentation procedures that negatively affect color- sensitive tasks. We propose an alternative color data augmentation, called Planckian Jitter, that is based on the physical properties of light. Our experiments demonstrate its positive effects wide variety of tasks where the intrinsic color of the objects (related to their reflectance) crucial for discrimination, while the illumination source is not. We also proposed exploiting both color and shape information by concatenating features learned with different modalities of supervuperviionn leading to significant overall improvements in learned representations.. Planckian Jitter can be easily incorporated into any self-supervised learning pipeline based on data augmentations, as shown by results demonstrating improved performance for three self-supervised learning models.

---

Published as a conference paper at ICLR 2023
REPRODUCIBILITY STATEMENT
The code of the Planckian Jitter data augmentation procedure, written in MATLAB and PyTorch 1.7.0, is made available at httttt::/ggtth/t http://gttthhhoo/oo.a. con/ comm/TTeeihoo/laaaaaaaaeeeeeee.
The training runs have been performed using Pytorch in combination with Pytorch Lightning Bolt framework, which provides an implementation of SimSiam methodology for backbone contrastive training. The model has been trained using CUDA deterministic, and random seed set to 1234.
All datasets used for the training and fine-tuning are publicly available. Only the CUB200 dataset has been pre-processed, by cropping each image using the given bounding box values, available alongside each image in the annotations files.
ACKNOWLEDGMENTS
We acknowledge the support from the Spanish Government funding for projects P1D2019-1041744GBB-000 TEDD2001-132513B-000 the support from the European Com- mission under the Horizon 2020 Programme, grant RYC2021-032765-1 funded by CEII/AEE/10011003550000000
and by European Union NextGeneratiinEU//RRRR,, and grant number 951911 (AI4Media).
REFERENCES
Kashif Ahmad, Nicola Conci, Giulia Boato, and Francesco GB De Natale, Used: a large-scale social event detection dataset. In Proceedings of the 7th International Conference Multimedia Systems, pp. 1-6, 2016.
David G Andrews. An introduction to atmospheric physics. Cambridge University Press, 2010.
Adrien Bardes, Jean Ponce, and Yann LeCun. Vicreg: Variaaceiivvriaccc--vvvaaccc regularizauiiioinnnn for self-supervised learning. arXiv preprint arX1iv2105..04906, 2021.
Marco Buzzelli, Simone Bianco, and Raimondo Schettini. Are: Angle-retaining chromaticity dia- gram for color constancy error analysis. JOSA (7(11): 2020.
Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria- Florina Balcan, and Aisuan-Tien Lin (eds.), Advances in Neural Information Processing Systems 33: Annual Conference Neural Information Pro- cessing Systems 2020, NeurIPS 2020, December 6-12, 2020, virtual, 2020.
Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference machine learning, pp. 1597-1607. PMLR, 2020a.
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 15750-15758, 2021.
Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved basclines with momentum contrastive learning. arXiv preprint arXiv:2003. 04297, 2020b.
Claudio Cusano, Paolo Napoletano, and Raimondo Schettini. Tik+: A database for benchmarking color texture classification and retrieval methods. Sensors, 2013):1010, 2021.
Carl Doersch, Abhinav Gupta, and Alexei Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pp. 1422-1430, 2015.
Alexey Dasovitskiy, Jost Tobias Springenberg. Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27:766-774, 2014.
Graham D Finlayson and Gerald Schaefer. Solving for colour constancy using a constrained dichro- matic reflection model. International Journal of Computer Vision, 42(3):127-144, 2001.
10

---

Published as a conference paper at ICLR 2023
Adrian Galdran, Aitor Alvarez-Gila Maria Ines Meyer, Cristina L Saratxaga, Teresa Araújo, Es- tibaliz Garrote, Guilherme Aresta, Pedro Costa, Ana Maria Mendonça, and Aurélio Campilho. Data-driven color augmentation techniques for deep skin image analysis. arXiv preprint (7X3:0070 2017.
J-M Geusebrock, Rein Van den Boomgaard, Amold W. M. Smeulders, and Hugo Geerts. Color invariance. JEEE Transactions on Pattern analysis and machine intelligence, 23(12); 1338-1333335 2001.
Man-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 21271-21284 Curran Associates, Inc., 2020. URL https: htt.ttss:/ppocceee....t neurips.
Pacieeee-ppree.ee haper.pal pdf
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9729 9738, 2020.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Ya Le and Xuan Yang, Tiny imagenet visual recognition challenge. CS 231N, 7(7):3 2015.
David G Lowe. Distinctive image features from cale-invarriant keypoints. International journal of computer vision, 60(2):91-110, 2004.
Bangalore S Manjunath and Wei-Ying Ma. Texture features for browsing and retrieval of image data. IEEE Transactions on pattern analysis and machine intelligence, 18(8):837-842, 1996.
Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pp. 722-729. IEEE, 2008
Ivet Rafegas and Maria Vanrell. Color encoding in biologically-inspiee convolutional neural net- works. Vision research, 151:7-17, 2018
Yushan Feng Saihui Hou and Zilei Wang. Vegfru: A domain-specific dataset for fine-grained visual categorization. In IEEE International Conference on Computer Vision, 2017.
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin (eds.), Advances in Neural Information Processing Systems, volume 33, pp. 6827-6839. Curran Associates, Inc., 2020.
Shoji Tominaga, Satoru Ebisui, and Brian A Wandell. Color temperature estimation of scene illumi- nation. In Color and Imaging Conference, volume 1999, pp. 42-47, Society for Imaging Science and Technology, 1999.
J von Kries. Theoretische studien über die umstimmung des sehorgans. Festschrift der Albrecht- Uniivisi-nnivesiittt, pp. 145-155 1902.
P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech-UCSD Birds 200. Technical Report CNS-TR-2010-001, Califomia Institute of Technology, 2010.
Gunter Wyszecki and Walter Stanley Stiles. Color science, volume 8. Wiley New York, 1982.
Tete Xiao, Xiaolong Wang, Alexei Efros, and Trevor Darrell. What should not contrastive in contrastive learning. arXiv preprint (rXiv:200.05565) 2020.

---

Published as a conference paper at ICLR 2023
Yang Xiao, Etienne Decencière, Santiago Selasco-Forera, Hélène Burdin, Thomas Bornschlögl, Françoise Bernerd, Emilie Warrick, and Thérèse Baldeweck. A new color augmentation method for deep learning segmentation of histological images. In 2019 IEEE 16th International Sympo- sium on Biomedical Imaging (ISBI 2019), pp. 886-890. IEFE, 2019.
Jure Zbontar, Li Jing, Ishan Misra. Yann LeCun, and Stéphane Deny. Barlow twins: Self-supervised learning via redundancy reduction. In Marina Meila and Tong Zhang (eds.), Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pp. 12310-1 12320. PMLR, 2021.
12


Reviews:
[{"review_id": "edYR6d9Z_-c", "reviewer": "Reviewer_w1qF", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a color data augmentation method for self-supervised learning or contrastive learning, which applies physically illuminant variation to images and the illuminants described by Planck’s Law for black-body radiation. The experimental results show that the proposed color augmentation method outperforms existing methods in some cases.", "summary_of_the_review": "Applying physical and realistic color data augmentation to network training is valuable, but the signification, comparison and some other aspects of this work can be improved.", "strength_and_weaknesses": "Strength\n+ Color is important in image classification, especially for some fine-grained image classification, such as flower classification.\n+ Considering the physical and realistic procedure for color data augmentation is novel.\n\nWeakness\n- The proposed method assumes that the images are captured under the daylight D65. However, in the daily life, we capture image under various lights, such as LED light and other daylight D55 and so on.\n- The improvement of proposed method is not significant in some cases. According to Table 1, the proposed Planckian Jitter performs significantly worse than default color jitter on CIFAR-100 dataset. Besides, Planckian Jitter without brightness and contrast (B&C) or Planckian-based chromaticity performs even worse than without data augmentation. \n- The compared methods are not comprehensive. The proposed Planckian Jitter mainly performs on SimSiam (2021), and generally performs on SimCLR (2020) and Barlow Twins (2021). It is better to perform on more recently proposed contrastive learning methods. Besides, the goal of color data augmentation is to improve the performance of image classification network. Thus, why not directly perform the proposed method on advanced image classification network, such as ResNet-101, Swin-Transformer and so on.\n- The selection of some parameters is difficult to understand, such as C_B, C_C in Equation 5. The effectiveness of these parameters is unknown.\n- There are some typos. For examples, ‘We’ of term 4 in page 4.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "W9EcGl5qhe", "reviewer": "Reviewer_kWdv", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper proposes a physics-based color data augmentation method (called Planckian Jitter), to train deep models to be robust to illumination changes. Experiments show that this method can produce a representation that is complementary to the representations learned by current color jitter augmentation methods, which further boosts performances on a variety of downstream tasks. The paper also presents an analysis of the color sensitivity of deep models to training strategies. Experiments show that the proposed method is robust to illuminant changes and can be combined with existing self-supervised learning methods to further boost their performance.", "summary_of_the_review": "The paper presents an interesting method for color augmentation and shows its effectiveness in some scenarios. I do not find major issues but am concerned about its further applications. I hope the authors could discuss them, which should be inspiring for future research.", "strength_and_weaknesses": "Strong points.\n\n1. A new color data augmentation method is proposed, which is demonstrated to be robust to illumination changes, useful for recognizing objects based on color information, and can be combined with other augmentation methods.\n\n2. This work is motivated well. Figure 1 is a good visualization that illustrates the differences between the existing color jitter method and the proposed method.\n\n3. Claims are generally supported via justifications and experiments.\n\nWeak points.\n\nWhile this paper proposes to learn color representations for tasks in which the intrinsic color of the objects is crucial, I wonder whether the proposed method can work well on 1, scene-level image-based tasks where multiple objects with different colors co-exist, and 2, Fine-grained image categorization (Sub-category recognition).\n\nThe writing could be improved for better readability. Some issues are listed below.\n\n1. The paper does not use hyperlinks for figures/tables/references discussed in the main text, making it difficult to locate them or refer to the references for more information.\n2. Figure 2 does not have \"p_2\" on it.\n3. It is quite strange to suddenly describe the proposed network architecture in the related work section.\n4. The \"Data augmentation\" paragraph of Section 2 says \"x_1, x_2 used in the ... in Eq. 2\" but Eq. 2 does not contain \"x_1\" and \"x_2\".\n5. Figure 4 is mentioned before Figure 3 in the main text.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Vm0ErGVwa0", "reviewer": "Reviewer_HQBn", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper examines the use of color in data augmentation for training deep networks and how it impacts the embedding representation of the input. In particular, they compare the traditional random color augmentation strategy with a color augmentation based on simulating changing illuminants. The illuminants used are drawn from the distribution of Planckian illuminants. The authors train the network in an unsupervised manner to learn an embedding space and then build a linear classifier using the embedding to test the effectiveness of the embedding on two standard data sets. Using a combined embedding that makes use of a network trained using the traditional color augmentation and a network trained using the Planckian color augmentation provides the best performance.", "summary_of_the_review": "This is an important issue in network training. However, it might be challenging to integrate it into a standard architecture/classification problem given the utility of concatenating separate embeddings based trained on different types of color jitter.", "strength_and_weaknesses": "The strength of the paper is the intentional analysis of the color augmentation process used in deep network training. In particular, the author's hypothesize that the standard color augmentation process strongly reduces color as a useful feature for classification. While for some objects, color is not strongly indicative (though certain distributions may be), for other tasks, such as natural objects like flowers and birds, color may be a critical feature. The improved performance on the flowers database appears to support this hypothesis. Likewise, the evaluation on two other data sets shows that the embedding trained with the Planckian illuminants is robust across simulated illumination changes, while the embedding trained with traditional color augmentation is not.\n\nThere are other potential illuminants besides Planckian, but for the purpose of analyzing this question, it is a reasonable model to choose.\n\nThis is only a first step towards appropriate training using color augmentation. The whole image augmentation model means that there is no selectivity by surface or current illumination conditions. An object in shadow, for example, is in a different lighting condition than the same object that is fully lit, and simulating the spectral rotation of a new direct illuminant shouldn't actually modify the shadowed areas as  much or at all.  Likewise, there is still a need for color augmentation of objects for which color is not discriminative.  However, the use of the combined embedding acknowledges that. It could be more explicit in the paper that there are several valid types of color augmentation, and that whole image color augmentation is a pretty heavy hammer compared to an ideal selective augmentation process.\n\nGiven the need to train two separate embeddings, one for each type of jitter, how does this get implemented in standard network architectures?  Does it have utility beyond this particular network structure?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "9tMZRC2QvoQ", "reviewer": "Reviewer_9p9z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper is about color augmentation, which is sometimes used for network training. Instead of common color augmentation by changing  brightness, contrast, saturation, and hue randomly, the paper suggests Planckian-based color augmentation. ", "summary_of_the_review": "Please see my comment on novelty and quality above.", "strength_and_weaknesses": "* Strengths:\n1. The idea is straightforward and the paper is easy to read.\n\n* Weaknesses:\n1. I am sure the suggested Planckian-based augmentation is useful for a few applications that suffer from light color changes. However, I am not sure the importance of this in general settings. The existing common color augmentation is still needed for general applications (those that are not much affected by light colors, but suffer from the limited training data). Hence, proposing only the Planckian augmentation in a whole paper is too narrow and too little in terms of the contribution. \n\n2. A quick search on the internet shows there are a few papers proposing different color augmentation techniques for certain applications: PCA Color Augmentation, \"Data-Driven Color Augmentation Techniques for Deep Skin Image Analysis\", \"A New Color Augmentation Method for Deep Learning Segmentation of Histological Images\". However, none of these are discussed in the paper.\n\n3. The Planckian-based augmentation might be new, yet the significance of this is minor.\n\n4. The proposed pipeline/architecture has nothing novel.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster