Planckian Jitter: countering the color-crippling effects of color jitter on self-supervised training
Authors: Simone Zini
Alex Gomez-Villa
Marco Buzzelli
Bartłomiej Twardowski
Andrew D. Bagdanov
Joost van de weijer
Venue: NeurIPS 2022 Submitted (2022)

Abstract: Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. In this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose a more realistic, physics-based color data augmentation – which we call Planckian Jitter – that creates realistic variations in chromaticity and produces a model robust to illumination changes that can be commonly observed in real life, while maintaining the ability to discriminate image content based on color information. Experiments confirm that such a representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on a wide range of downstream datasets. In addition, we present a color sensitivity analysis that documents the impact of different training methods on model neurons and shows that the performance of the learned features is robust with respect to illuminant variations.

Planckian Jitter: countering the color-crippling effects of color jitter on self-supervised training
Anonymous Author(s)
Affiliation
Address
email
Abstract
Several recent works on self-supervised learning are trained by mapping different augmentations of the same image to the same feature representation. The data augmentations used are of crucial importance to the quality of learned feature representations. this paper, we analyze how the color jitter traditionally used in data augmentation negatively impacts the quality of the color features in learned feature representations. To address this problem, we propose more realistic, physics- physics-based color data augmentation which we call Planckian Jitter that creates realistic variations in chromaticity and produces a model robust illumi- nation changes that can be commonly observed in real life, while maintaining the ability discriminate image content based on color information. Experiments confirm that such representation is complementary to the representations learned with the currently-used color jitter augmentation and that a simple concatenation leads to significant performance gains on wide range of downstream datasets. addition, we present a color sensitivity analysis that documents the impact different training methods model neurons and shows that the performance of the learned features is robust with respect to illuminant variations.
10
11
12
14
16
Introduction
18
Self-supervised learning enables the learning of representations without the need for labeled data S Several recent works learn representations that invariant with respect and have obtained spectacular results [2] 6 B], significantly narrowing the gap with supervised learned representations. These works vary in their architectures, learning objectives, and optimization strategies, however they are similar in applying a common set of data augmentations to generate different image views. These algorithms, while learning map these different views the same
of data augmentations
20
21
22
24 latent representation, learn rich semantic representations for visual data. The set of transformations (data augmentations) used induces invariances that characterizes the learned visual representation.
25
26 Before deep learning revolutionized the way visual representations are learned, features were hand- crafted to represent various properties, leading research shape [15], texture [16), and color features [00 [1] Color features were typically designed to be invariant to set of sceme-accidental events such shadows. shading, and illuminant and viewpoint changes. With the rise of deep learning, feature representations that simultancously exploit color, shape, and texture are learned byproduct of end-to-end training (14). Current approaches to self-supervised learning learn a set of invariances implicitly related to the applied data augmentations.
28
30
31 implicitly and the invariances
32
In this work, we focus the currently de facto choice for color augmentations. We argue that 34 they seriously cripple the color quality of learned representations and we propose an alternative, (leff) illustrates the currently used color augmentation on 36 a sample image. It is clear that the applied color transformation significantly alters the colors of the
physics-based color augmentation.
Submitted to 36th Conference on Neural Information Processing Systems (NeurIPS 2022). Do not distribute.

---

Figure 1: Default color jitter (left) and Planckian Jitter (right). Augmentations based default color jitter lead to unrealistic images, while Planckian Jitter leads to a set of realistic ones. The ARC chromaticity diagrams for each type of jitter are computed by sampling initial RGB values and mapping them into the range of possible outputs given by each augmentation. These diagrams show that Planckian Jitter transforms colors along chromaticity lines occurring nature when changing the illuminant, whereas default color jitter transfers colors throughout the whole chromaticity plane.
37 original image, both in terms of hue and saturation. This augmentation results in representation that is invariant with respect to surface reflectance an invariance beneficial for recognizing classes whose surface reflectance varies signficantly, for example many man-made objects such cars and chairs. However, such invariance is expected to hurt performance on downstream tasks for which color is an important feature, like natural classes such as birds or food. One of the justifications for such strong color augmentations is that without large color changes, mapping images to the same latent representation can be purely done based on color and no complex shape features are learned. However, as a result the quality of the color representation learned with such algorithms is inferior and important information on surface reflectance might be absent.
38
39
40
41
42
43
44
45
46
In this paper we propose an alternative color augmentation (Figure right). We draw on the existing color imaging literature designing features invariant to illuminant changes commonly encountered in real-world scenes [10]. Our augmentation, which we called Planckian Jitter, applies physically realistic illuminant variation to images. We consider the illuminants described by Planck's Law for black-body radiation and that are known to be similar to illuminants encountered in real-life (21]. The aim of our color augmentation is to allow the representation to contain valuable information about the surface reflectance of objects a feature that is expected to be important for a wide range of downstream tasks. Combining such a representation with the already high-quality shape representation learned with standard data augmentation leads to a more complete visual descriptor that describes both shape and color.
48
49
50
51
52
53
54
55
56
Our experiments show that self-supervised representations learned with Planckian Jitter are robust to illuminant changes. In addition, depending on the importance of color in the dataset, the pro- posed Planckian jitter outperforms the default color jitter. Moreover, for all evaluated datasets the combination of features of our new data augmentation with standard color jitter leads to significant performance gains of over 5% on several downstream classification tasks. Finally, we show that Planckian Jitter can be applied to several state-of- of-the-rrt self-supervised learning methods.
58
59
60
61
2
Background and related work
82
63
Self-supervised learning and contrastive learning. Recent improvements in self-supervision learn semantically rich feature representations without the need for labelled data. In SimCLR [4] similar samples are created by augmenting an input image, while dissimilar are chosen by random [4]. To make contrastive training more efficient, MoCo [13] and its improved version (5] use memory bank for learned embeddings which makes sampling efficient. This memory is kept in sync with the rest of the network during training via a momentum encoder. Several methods do not rely on explicit
84
85
86
67
68
69
contrastive pairs. BYOL uses an asymmetric network incorporating an additional MLP predictor
70
between the outputs of the two branches [12]. One of the branches kept "offline" and is updated by
71
momentum encoder. SimSiam goes even further with simplified solution without a momentum
encoder (6. It obtains similar high-quality results and does not require a large minibatch size, in
72
contrast to other methods.

---

Encoder [Resnet-88]
MLP
Encoder [Resnet-18]
sugmentation [including Planckian Jitorr
Similarity
Stpp go
X2
Encoder [Reenet- 18]
>2
Linear Classifier
SimSiam Training
Stop-grad
Class
Linear Classifler training
Figure SimSiam training procedure exploiting Planckian- data augmentation (left), and fine-tuning the linear classifier using the trained encoder (right).
We the SimSiam method verify our proposed color augmentation (we also apply it Sim-
75 CLR [ and Barlow Twins [26] in the experiments). The main component of the network is
CNN-based image encoder, learned end-to-end in asymmetric Siamese architecture. One branch
77 has an additional MLP predictor whose output aims to be as close as possible to other (see Figure 2}. The second branch not updated during backpropagation. negative cosine loss function is used:
78
c
[D(p,, stopgrad + D(p2, stopgrad(2:)))
PA
2B
D(PA, BB)
|PAl2
|z
where 21, 22 representations for two different augmented versions, 2ı and T2, of the same image 80 J. An additional predictor applied on 21 and 22 produces P1 and P2, respectively. The stopgrad operation blocks the gradient during the backpropagation. In SimSiam no contrastive term used and only similarity is enforced during learning.
82
83
Data augmentation. Data augmentation plays an central role in the self-supervised learning process described above. The authors of 4 and [26) discuss the importance of the different data augmentations. A set of well-defined transformations was proposed for SimCLR [4]. This set is commonly accepted and used several later works. The augmentations include: rotation, cutout, flip, color jitter, blur and Grayscale. These operations are randomly applied to an image generate the different views 31, 22 used in the self-supervision loss Eq. 2 Applied to the same image, contrastive-lee like supervied methods learn representations invariant to such distortions.
84
85
86
86
90
This multiple view creation task-related (20), however color jittering operating on hue, saturation, brightness and contrast, is one of the most important ones in terms of overall usefulness of the learned representation for downstream tasks 4 26 Color jitter induces a certain level of color invariance (invariance to hue, saturation, brightnesss and contrast) which are consequently transferred to the downstream task. As consequence, we expect these learned features to underperform on downstream tasks for which color is crucial. Xiao et al. invariances might not be beneficial for downstream tasks. As solution, they propose to learn different embedding spaces in parallel that capture each of the invariances. Differently than them, we focus on the color distortion and propose physics-based color augmentation that allows learning invariance to physically realistic color variations.
91
92
93
94
25 were the first point out that the imposed
96
98
100
The color imaging literature has a long tradition in research on color features invariant scene- accidental events such as shading, shadows, and illuminant changes [1]] I0] Invariant features were found to be extremely beneficial for object recognition. The invariance to hue and saturation changes, induced by the color jitter operation, however, it detrimental to object recognition for those classes which color characteristics are fundamentally discriminative. Therefore, this work we revisit early theory on illuminant invariance [10] to design improved color augmentation that induces invariances common in the real world and that, when used during self-super learning, does not
102
103
104
106
damage the color quality of the learned features.

---

108
3
Methodology
109
The image transformations introduced by default color jitter creates variability in training data that indiscriminately explores all hues at various levels of saturation. The resulting invariance is useful for downstream tasks where chromatic variations are indeed irrelevant (e.g. car color in vehicle recognition), but is detrimental to downstream tasks where color information is critical (e.g. natural 113 classes like birds and vegetables). The main motivation for applying strong color augmentations is that this it leads to very strong shape representations. Indiscriminately augmenting color information 115 in the image requires that the representation solve the matching problem using shape
111
112
116
As an alternative to color jitter, we propose physics- based color augmentation that mimics color variations due to illuminant changes commonly encountered in the real world. The aim is to arrive at a representation that does not have the color crippling effects of color jitter and that can therefore better describe classes for which surface reflectance a determining feature. The aim learn a representation that, when combined with default color jitter, provides high-quality shape and color representation.
117
119
3.1
Planckian Jitter
122
We call our color data augmentation procedure Planckian Jitter because it exploits the physical description of black-body radiator to re-illuminate training images within a realistic illuminant distribution [10 (1). The resulting augmentations are more realistic than those of the default color
123
124
125
jitter (see Fig. D. The resulting learned, self-supervised feature representation is thus expected to be robust to illumination changes commonly observed in real-world images, while simultancously maintaining the ability discriminate the image content based on color information.
128
127
128
Given an input RGB training image I, our Planckian Jitter procedure applies a chromatic adaptation transform that simulates realistic variations in the illumination conditions. The data augmentation procedure is as follows:
129
130
131
1. we sample a new illuminant spectrum or (A) from the distribution of black-body radiator;
2. we transform the sampled spectrum σr (A) into its sRGB representation pr E RR3
132
133
3. we create a jittered image I' by reilluminating with the sampled illuminant pr; and
134
4. We introduce brightness and contrast variation, producing a Planckian-jittered image I"
A radiating black body at temperature T can be synthesized using Planck's Law M:
2xhe2
oT(2) (e - 1) W/m²,
136
where 2.99792458 10 m/s is the speed of light, = 6. 6.626176 x 10-34 Js is Planck's 's constant, and k 1.380662 x 10 23 J/K is Boltzmann's 's constant. For our experiments we sampled T in the interval between 3000 and 15000K which known result in a set of illuminants that can be encountered in real life (2I]. Then, we discretized wavelength À in 10nm steps (AA) in the interval between 400nm and 700nm. The resulting spectra are visualized in Figure (left) in the Supplementary Material.
137
138
139
140
142
The conversion from spectrum into sRGB is obtained through series of intermediate steps 24
143
1. we first map the spectrum into the corresponding XYZ stimuli, using the 1931 CIE standard observer color matching functions ctX,Y,2) in order to bring the illuminant into a standard color space that represents a person with average eyesight;
144
145
146
2. We normalize this tristimulus by its Y component, convert it into the CIE 1976 L color space, and fix its L component to 50 in a O-to- 1-to-100 scale, allowing us to constrain the intensity of the represented illuminant in a controlled manner as a separate task; and
147
148
149
3. we then convert the resulting values to sRGB, obtaining PT = (R,C. G, B}; the resulting distribution of illuminants is visualized with the Angle-Retaining Chromaticity diagram 2 in Figure (right) in the Supplementary Material.
150
151
'This is pointed out in the discussion of Figure 5 in 4

---

152
All color space conversions assume a D65 reference white, which means that a neutral surface illuminated by average daylight conditions would appear achromatic. Once the new illuminant
153
154
has been converted in sRGB, it is applied to the input image / by resorting to a Von-Kries-like transform [22 given by the following channel-wise scalar multiplication:
155
/RR.G.B) = ( JRR.G.B). {R. G, 1,1(,,,,1}
(4)
156
where we assume the original scene illuminant to be white (1,1,1). Finally, brightness and contrast perturbations are introduced to simulate variations in the intensity of the scene illumination:
157
I" CB ce + (1-cc) - (cB I')
(5)
158
where eg 0.8 and cc 0.8 represent, respectively, brightness and contrast coefficients, and şi is spatial average function.
159
160
3.2
Complimentarity of shape and color representations
The self-supervised learning paradigm involves a pretraining phase that relies data augmentation to produce a set of features with certain invariance properties. These features are then used as the representation for a second phase, where learn given supervised downstream task. The default color jitter augmentation generates features that are strongly invariant to color information, resulting in high-quality representations of shape and texture, but that is an inferior descriptor of surface reflectances (i.e. the color of objects). Our augmentation based on Planckian Jitter (see Figure I is based on transformations mimicking the physical color variations in the real world due to illuminant changes. As a result, the learned representation yields a high-quality color description of scene objects. However, likely leads to a drop in the quality of the shape representation (since color can be used to solve cases where previously shape was required). To exploit the complimentarity of the two representations, we propose to learn both - one with color jitter and one with Planckian Jitter- and to then concatenate the results in a single representation vector (of 1024 dimensions, i.e. twice the original size of 5 12). We call this Latent space combination (LSC).
162
163
184
165
186
167
168
169
170
171
172
173
Experimental results
174
175
In this section, we analyze the color sensitivity of the learned backbone networks, verify the superiority of the proposed color data augmentation method compared to the default color jitter on color datasets, and evaluate the impact downstream classification tasks. We report additional results on computational time of the proposed Planckian augmentation in the Supplementary Material.
176
178
179
4.1
Training and evaluation setup
We perform unsupervised training two datasets: CIFAR-100 14
(32 32) and ImageNet (224 x 224). We slightly modify the ResNet18 architccture to accommodate 32 x 32 images: the kernel size of the first convolutional was reduced from 7 x 7 to 3 x 3 and the first max pooling layer was removed. SimSiam training was performed using Stochastic Gradient Descent with a starting learning rate of 0.03, cosine annealing learning rate scheduler, and mini-batch size of 512 (as in original SimSiam work (6). For the training on the 1000-class ImageNet training set, we follow the same procedure as [6] with ResNet50.
181
189
185
196
187
The linear classifier training at resolution 32 x 32 was performed on CIFAR- 100 and FLOWERS- 102 17. CIFAR-100 100 is used as baseline for the classification task. The linear classifier training for 100 is done with Stochastic Gradient Descent for 500 epochs with a starting learning rate 0.1, cosine annealing learning rate scheduler, and mini- bati- size of 512. The FLOWERS- 102 dataset with 102 classes was selected to assess the quality of the features extracted in scenarios where color information plays an important role. Images from FLOWERS- -02 are resized to 32 x 32 pixels to
189
191
192
193
match the input dimensions of the pretrained model. Here we used the Adam optimizer with initial learning rate of 0.03.
195
For training linear classifiers at resolution 224 x 224 for downstream tasks we follow the evaluation protocol of [6. We use five different datasets: IM
F.OwERS-102 VEGFRU [19, CUB- 200 23,, and T1 K+ 77. These five datasets were resized to 224 x 224 pixels. More details about
197
these datasets provided in the Supplementary Material. In the case of CUB-200 each image was
5

---

70
60
000
50
500
-0
Method
01.32
0.31
accurecy
(1+P)
80
300
09 341
2-1
02 2.71
20
20
10
100
30
4000 6:00
8000
10000
12000
14000
4030 5000
5000 10000 12000 14000 illmmnant
iluminant
CJ-
(b)
Figure 3: Color sensitivity analysis. (a) Robustness to illuminant change: we report the accuracies by differently-traine trained backbones as function of illuminant. (b) The color sensitivity indexes computed for the different configurations used for training the backbone.
199
cropped using the bounding boxes given in the dataset annotations. For T1K+, we use the 266 class labeling to train and test the linear classifier.
201
To assess the impact of color data augmentations we define six different configurations:
Default Color Jitter (CJ): the default configuration, as used in SimSiam and SimCLR, uses both Random Color Jitter and Random Grayscale operations.
203
204
Default Color Jitter w/o Grayscale (CJ-): same Default without the Random Grayscale operation.
206
Planckian Jitter (PJ): uses the complete proposed Planckian Jitter operation operating on chromaticy, brightness, and contrast aspects of the images. No Random Grayscale is applied.
207
LSC Default Color Jitter + Planckian Jitter (JCJ,PJJ: This latent space combination (simple concatenation of representations) combines the default color jitter with our Planckian jitter. It allows evaluation of the complimentary nature of the representations.
209
210
LSC Default Color Jitter Default Color Jitter w/o Grayscale (ICJ,CJ-J): We combine the default color jitter with a version without the Grayscale augmentation, since this representa- tion is also expected to result in better color representation.
212
214
LSC of two Default Color Jitter Models ([CJ, CJJ): We also show results of simply concate- nating two independently trained models (trained from different seeds) with default color jitter (an ensemble of two models).
215
217
In all experiments these augmentations are combined with the other default augmentations (crop, horizontal flip, and blur).
4.2
Color sensitivity analysis
220
To verify if our Planckian data augmentation actually leads to illuminant invariance, we performed robustness analysis on the CUB-200 dataset with realistic illuminant variations and analyzed sensitivity to color information. We assume as reference point the D65 illuminant, which for the purpose of this test is considered the default illuminant in every image. Given the different backbones pretrained on IMAGENET, we then train a linear classifier on this dataset (assumed be under white illumination). For testing create different versions of CUB-20 200, each illuminated by illuminants of differing color temperature. This allows us to evaluate the robustness of the learned representations with respect to these illuminant changes. A similar experiment is performed on VEGFRU.
222
224
226
228
Results are given in Figure 3 (more results are provided in the Supplementary Material), Planckian Jitter obtains remarkably stable performance from around 4000-14000K, while Default Color Jitter more sensitive to the illumination color and the classification accuracy decreases when scene illuminant moves away from white. We also see that the combination of default and Planckian Jitter
229
230
232
obtains the best results for all illuminants and manages to maintain high-level of invariance with
respect to the illuminant color.

---

Table 1: Ablation color augmentations. Self-supervised training is performed CIFAR-100 and the learned features are evaluated at (32 x 32) on CIFAR-100 and "LOWERS-102 Augmentation techniques include variations in hue and saturation (H&S), brightness and contrast (B&C), Planckian- based chromaticity (P), and random Grayscale conversions (G). Accuracy refers to the results of the linear classifiers trained with features extracted from the different backbones.
AUGMENTATION H&S B&C
G P ACCURACY
None
41.93%
CIFAR-100 100
Default Color Jitter
59.93%
41.96%
32.46%
36. 36.10%
31.78%
Planckian Jitter
47.31%
FLOWERS- 102
None
36.47%
Default Color Jitter
30.00%
36.96%
39.11%
39.51%
41.96%
Planckian Jitter
42.75%
Table 2: Results for self- training on CIFAR- 100 and evaluated at 32 x 32 on CIFAR- C10AR-100 and FLOWERS- 102. Accuracy refers the results of the linear classifiers trained with features extracted from the different trained backbones.
AUGMENTATION
ACCURACY
Default Color Jitter (CJ)
59.93%
CIFAR- 100
Default Color Jitter w/o Grayscale (CJ-)
41.96%
Planckian Jitter (PJ)
47.31%
LSC [CJ,CJ-]
62.27%
LSC: [CJ.PJ]
63.54%
FLOWERS- 102
Default Color Jitter (CJ)
30.00%
Default Color Jitter w/o Random Grayscale (CJ-)
36.96%
Planckian Jitter (PJ)
42.75%
LSC: [CJ,CI-]
47.65%
LSC: [CJ,PJ]
51. 51.66%
In order understand the impact of the color information each neuron in trained models, conducted an analysis using the color selectivity index described in [18]. This index measures neuron activation when color is present absent in input images. We computed the index for the last layer of different backbones, and high values indicate color-sensitive neurons. See the Supplementary Material for more details color selectivity. The results are shown in Figure b)) and indicate the number of color-sensitive neurons for each of the considered models. It is clear that the default color jitter has far fewer neurons dedicated to color description. This result confirms the hypothesis that models trained in this way are color invariant, a property that negatively affects the model in scenarios where color information has an important role as seen in our experiments. We have also analyzed the results for the default color jitter without Grayscale augmentation (CJ-). These results show that removing the Grayscale augmentation improves color sensitivity significantly. We therefore also consider this augmentation in future experiments.
235
236
237
240
242
244
245
246
4.3
Ablation study
247
Six different models were trained and evaluated with linear classification for image classification. For resolution 32 x 32 the model is evaluated CIFAR-100 and FLOWERS- 102. The results in
7

---

249
terms of accuracy are reported in 2 We identify two different trends when interpreting these results. On CIFAR- 100, removing color augmentations makes the model less powerful, due to the loss of color invariance in the features extracted by the encoder. This behaviour is consistent with what was reported in 4 We see in Table Taale[ttarr that if color augmentations (i.e. brightness/contrast and
250
252
Random Grayscale) are removed completely (the None configuration), the accuracy drops by 18%.
253
254
On FLOWERS- the behavior is the opposite however: removing color augmentations helps the model to better classify images, obtaining improvement of 12.75% of accuracy with respect to the default color jitter. This behavior confirms that color invariance negatively impacts downstream tasks where color information plays important role.
296
258
Taking a closer look at the various augmentation on FLOWERS- 102, we see that introducing more realistic color augmentations positively impacts contrastive training and produces models that achieve even better results with respect to the configuration without any kind of image color manipulation. Removing all color augmentations (None) improves results already by over 6%. Then, by simply reducing the jittering operation to influence brightness and contrast, leaving hue and saturation un- changed, yields another boost in accuracy of 5.49% (to 41.96). When we start modifying chromaticity using a more realistic transformation (i.e Planckian Jitter), the final result is a boost of 6.28% in accuracy with respect to the None configuration. Also, on CIFAR- we see an improvement of 5.38% from Planckian Jitter with respect no color augmentation. Despite this improvement, in this scenario the contrastive training with the realistic augmentation does not yield better results with respect to the Default configuration because color only plays a minor role on this dataset.
259
260
262
264
286
288
269
Given the results obtained using the data augmentations reported in Table | and given the con- siderations made in Section 3.2 we evaluate the complementarity of the learned representation by combining latent spaces from different backbones. Results for two different latent space combinations are given in Table 4 On both datasets the Latent space combination of Default and Planckian Jitter configurations achieves the best results. On the original CIFAR- 100 task, this combination achieves total accuracy of 63. 63,54%, a 3.61% improvement over the Default configuration and 16.23% more compared to Planckian Jitter alone. Comparing to the LSC using the Default ColorJitter w/o Grayscale, the version with Planckian Jitter achieves a small improvement of 1 1.27% in classification accuracy.
270
272
274
276
278
On the downstream FLOWERS- 102 task, the Latent space combination reaches an accuracy value of 51. 16.6 an improvement of 21.66% and 8.91 % in accuracy respectively compared to the two original
279
280
configurations. Compared to the LSSC using Default ColorJitter w/o Grayscale, the combination with
Planckian Jitter achieves a higher result, with a bigger gap in terms of accuracy with respect to the
282
CIFAR- 100 scenario. Here the use of Planckian Jitter brings in an improvement of 4.01% confirming the impact of using realistic augmentation on classification tasks for which color is important.
284
4.4 Evaluation on downstream tasks
Given the results obtained from the ablation study, performed the analysis of the proposed configurations on other downstream tasks using the backbone trained on higher resolution images (224 x 224 pixels). We report in Table the results for: Default Color Jitter, Planckian Jitter, and several latent space combinations.
286
288
I. Looking at the results, we see that the Planckian Jitter augmentation outperforms default color jitter on two datasets (CUB-200 and TIK). Comparing the results on FLOWERS- 102 with those reported above at (32 x 32) pixels, we see that default color jitter actually obtains good results. We hypothesize that for high-resolution images the shape information is very discriminative, and the additional color information yields little gain.
290
292
294
Table 3 also contains results for latent space combination. The results confirm that the two learned representation are complimentary and that their combination leads to significant performance gains of up to 9% on TIK when compared to default color jitter. As a sanity check, we have also included the latent space combination of two networks separately trained with color jitter. This provides small ensemble performance gain on some datasets but yields significantly inferior results compared
295
296
298
to our proposed LSC.
8

---

Table Evaluation on downstream tasks. Self-supervised training was performed on IMAGENET at (224 x 224) and testing performed on the downstream datasets resized to (224 x 224).
AUGMENTATION
CUB-200
VEGFRU
T1K+
FLOWWER--102
Default Color Jitter (CJ)
54.52%
67.63%
71.44%
93.16%
Planckian Jitter (PJ)
56.28%
65.84%
77.42%
90.29%
LSC [CJ,PJ]
60.70%
74.73%
80.49%
93.99%
LSC [CJ,CJ]
56.16%
70.59%
73.47%
93.13%
LSC [CJ,CJ-]
53.14%
70.54%
78.32%
93.47%
Table 4: Effect of Plackian Jitter on different contrastive learning models. Self-supervised training was performed CIFAR- and the learned features are evaluated at (32 32) CIFAR- and FLOWERS- 102. We report the best configurations obtained on SimSiam model and retrained SimCLR and Barlow Twins with those selected configurations.
FRAMEWORK AUGMENTATION CIFAR- 100
FLOWERS-102
Default Color Jitter
59.93%
30.00%
SimSiam
Planckian Jitter
47.31%
42.75%
I.SC [CJ,PJ]
63.54%
51.66%
Default Color Jitter
56.99%
35.29%
SimCLR
Planckian Jitter
47.75%
45.00%
LSC [CJ,PJ]
61.07%
55.78%
Default Color Jitter
56.60%
40.78%
Barlow Twins
Planckian Jitter
52.71%
54.50%
LSC [CJ,PJ]
62.85%
62.55%
300
4.5
Generality of Planckian Jitter
To show that our approach is generally applicable to self-supervised methods which exploit color augmentations, we also performed experiments using SimCLR and Barlow Twins. This comparison given in Table 4 Independently of the model used, the Default Color Jitter configuration of data augmentation gives the worst results on the FLOWERS- 10NS--02222 102 dataset. The Latent space combination configuration consistently achieves better results both datasets.
302
304
Limitations
307
Firstly, A drawback of Planckian jitter is that it reduces the quality of the shape representation, because the extreme color transformation of the standard color jitter force the network to solve the contrastive learning problem mainly using shape information. As shown in this article, this problem can be addressed by exploiting their complimentary nature. Secondly, our current latent space combination requires the training of two separate backhones, which certainly will also learn partially overlapping features. A training scenario, with both augmentations simultaneously single network while reserving part of the latent space for each augmentation, could be pursued to address this limitation.
308
310
312
314
Conclusion
315
316
Existing research on self-supervised learning mainly focuses on tasks where color is not decisive feature, and subsequently exploits data augmentation procedures that negatively affect color-sensitive tasks. We propose is based on the physical properties of light. Our experiments demonstrate its positive effects on wide variety of tasks where the intrinsic color of the objects (related their reflectance) crucial for discrimination, while the illumination source is not. We also proposed solution that exploits both color and shape information by concatenating features learned with different modalities of self- supervision, leading to significant overall improvements in learned representations. Planckian Jitter
317
318
alternative color data augmentation technique, called Planckian Jitter, that
319
320
322
324
can be easily incorporated into any self-supervised learning pipeline based on data augmentations, as shown by results demonstrating improved performance for three self-supervised learning models.

---

References
328
327
[1] David G Andrews. An introduction to atmospheric physics. Cambridge University Press, 2010.
328
[2] Marco Buzzelli,, Simone Bianco, and Raimondo Schettini. Are: Angle-retaining chromaticity diagram for color constancy error analysis. JOSA A, 37(11)1121-11330, 2020.
330
[3] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsupervised learning of visual features by contrasting cluster assignments. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florin, Balcan, and Hsuan-Tier Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems 2020, NeurlPS 2020, December 6-12, 2020, viriual, 2020.
331
332
335
335
[4] Ting Chen, Simon Kornblith, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for contrastive learning of visual representations. In International conference on machine learning, pages 1597-1607. PMLR, 2020.
336
337
338
[5] Xinlei Chen, Haoqi Fan, Ross Girshick, and Kaiming He. Improved baselines with momentum contrastive learning. arXiv preprint arXiv:2003.0422 2020.
339
340
[6]
Xinlei Chen and Kaiming He. Exploring simple siamese representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 15750-15758, 2021.
341
342
343
[7] Claudio Cusano, Paolo Napoletano, and Raimondo Schettini. TIk+: A database for benchmark- ing color texture classification and retrieval methods. Sensors, 2 21(3):0010, 2021.
344
345
[8] Carl Doersch, Abhinav Gupta, and Alexei A Efros. Unsupervised visual representation learning by context prediction. In Proceedings of the IEEE international conference on computer vision, pages 1422-1430, 2015.
348
347
348
99] Alexey Dosovitskiy, Jost Tobias Springenberg, Martin Riedmiller, and Thomas Brox. Discrimi- native unsupervised feature learning with convolutional neural networks. Advances in neural information processing systems, 27:766-774, 2014.
349
350
351
[10] Graham D Finlayson and Gerald Schaefer. Solving for colour constancy using a constrained dichromatic reflection model. International Journal of Computer Vision, 42(3):127-144, 2001.
352
[11] J-M Geusebroek, Rein Van den Boomgaard, Arnold W. M. Smeulders, and Hugo Geerts. Color invariance. IEEE Transactions on Pattern analysis and machine intelligence, (1112):3338-1501 2001.
354
[12] Jcan-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre Richemond, Elena Buchatskaya, Carl Doersch, Bernardo Avila Pires, Zhaohan Guo, Mohammad Gheshlaghi Azar, Bilal Piot, koray kavukcuoglu, Remi Munos, and Michal Valko. Bootstrap your own latent- a new approach to self-supervised learning. In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 21271-21284 Curran Associates, Inc., 2020.
357
359
351
362
[13] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9729 9738, 2020.
363
364
365
[14] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
[15] David G Lowe. Distinctive image features from scale-invariant keypoints. International journal of computer vision, 60(2):91-110, 2004.
368
369
[16] Bangalore S Manjunath and YeigYing Ma. Texture features for browsing and retrieval of image data. IEEE Transactions pattern analysis and machine intelligence, 18(8):337-842, 1996.
117] Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over large number of classes. In 2008 Sixth Indian Conference on Computer Vision, Graphics & Image Processing, pages 722-729. IEEE, 2008
372
10

---

374
[18] Ivet Rafegas and Maria Vanrell. Color encoding in biologically inspiieed convolutional neural networks. Vision research, 151:7-17, 2018.
375
376
[19] Yushan Feng Saihui Hou and Zilei Wang. Vegfru: domain-speciffe dataset for fine-grained visual categorization. In IEEE International Conference on Computer Vision, 2017.
378
[0
Yonglong Tian, Chen Sun, Ben Poole, Dilip Krishnan, Cordelia Schmid, and Phillip Isola. What makes for good views for contrastive learning? In H. Larochelle, M. Ranzato, R. Hadsell, M. F. Balcan, and H. Lin, editors, Advances in Neural Information Processing Systems, volume 33, pages 2827-6839, Curran Associates, Inc., 2020.
379
380
382
[21] Shoji Tominaga, Satoru Ebisui, and Brian A Wandell. Color temperature estimation of scene illumination. In Color and Imaging Conference, volume 1999, pages 42-47. Society for Imaging Science and Technology, 1999.
385
[22] J von Kries. Theoretische studien über die umstimmung des sehorgans. Festschrift der Albrecht- Ludwigs- Uniibiiss--nivesiitt,, pages 145 158, 1902.
386
387
[23] P. Welinder, S. Branson, T. Mita, C. Wah, F. Schroff, S. Belongie, and P. Perona. Caltech- UCCDD Birds 200. Technical Report ENS-TR-2010-001 California Institute of Technology. 2010.
388
389
[24]
Gunter Wyszecki and Walter Stanley Stiles. Color science, volume 8. Wiley New York, 1982.
390
[25]
Tete Xiao, Xiaolong Wang, Alexei A Efros, and Trevor Darrell. What should not be contrastive in contrastive learning. arXiv preprint arXiv:2008.0566 2020.
392
[26]
Jure Zbontar, Li Jing, Ishan Misra, Yann LeCun, and Stéphane Deny. Barlow twins: Self- supervised learning via redundancy reduction. In Marina Meila and Tong Zhang. editors, Proceedings of the 38th International Conference on Machine Learning. ICML 2021, 18-24 July 2021, Virtual Event, volume 139 of Proceedings of Machine Learning Research, pages 12310-12320. PMLR, 2021.
393
394
385
396
Checklist
397
398
The checklist follows the references. Please read the checklist guidelines carefully for information on how to answer these questions. For each question, change the default [TODO] [Yes]. [No], or [N/A] You are strongly encouraged to include justification to your answer, either by referencing the appropriate section of your paper or providing a brief inline description. For example:
399
400
401
402
Did you include the license to the code and datasets? [Yes] See Section 4.1
403
Did you include the license to the code and datasets? [No] The code and the data are proprictary.
404
Did you include the license to the code and datasets? [N/A]
406
Please do not modify the questions and only use the provided macros for your answers. Note that the Checklist section does not count towards the page limit. In your paper, please delete this instructions block and only keep the Checklist section heading above along with the questions/answers below.
408
409
1. For all authors...
410
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
412
(b) Did you describe the limitations of your work? [Yes] See section
413
(c) Did you discuss any potential negative societal impacts of your work? [No]
414
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
417
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]

---

3. If you ran experiments...
420
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or URL)? [Yes] It will be included in the supplementary material.
422
428
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes]
424
425
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [No]
427
(d) Did you include the total amount of compute and the type of resources used (e.g.. type of GPUs, internal cluster, or cloud provider)? [Yes] It will be included in the supplementary material.
428
429
4. If you using existing assets (e.g., code, data, models) curating/releasing assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [N/A]
433
(c) Did you include any new assets either in the supplemental material or as URL? [No]
434
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A]
437
438
5. If you used crowdsourcing or conducted research with human subjects...
439
(a) Did you include the full text of instructions given participants and screenshots, applicable? [N/A]
440
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
442
443
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/A]
12


Reviews:
[{"review_id": "wc5BD_KSj9T", "reviewer": "Reviewer_fsUt", "summary": "This paper adds to the data augmentation transforms in the literature by proposing a more physics-based approach to generate more color-realistic augmentations compared to the standard more random color jitter.  This is well-demonstrated in Figure 1 for natural image classes such as flowers where although some classes may have strong color variations, in general the variations are more realistic when due to illumination, compared to man-made classes such as cars.  The approach is motivated by and follows a standard black-body radiation model, which again as shown in Figure 1 constrains the color jitter to be more realistic.\n\nThe experiments are performed in a self-supervised fashion as that nicely focuses the weight on data augmentation.  Both natural and man-made data sets are tested such as Flowers-102, CUB-200, and CIFAR.  Overall the results summarized in Figure 3, and Tables 1, 2, 3 show the improvements in downstream classification with Planckian jitter on natural image classes and the potential for combining Planckian and random color jitter in general.", "questions": "As mentioned above, the main question for the reader is the somewhat counter-intuitive results in Table 3, for the same data sets but with higher resolution at 224x224.  One would expect a similar trend as the low-resolution data, but the numbers show otherwise.  Also the explanation on better shape discrimination on higher resolution bears more analysis and detail, e.g., is it more about texture than shape at higher resolution?\n\nAlthough the LSC combination is a good start to fuse both types of color augmentation, what about a way to make it more class-specific?  In other words, for even a coarse dichotomy such as natural vs man-made images, apply some complementary weighting/distribution of Planckian/default color jitter to get the best of both worlds?", "limitations": "The paper has an explicit section on limitations regarding the overhead with the combined color augmentation approach and also mentions the incomplete investigation into how much and in what detail the proposed Planckian jitter improves color realism why discouraging shape discrimination.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is very direct in the exact problem and contribution, namely more realistic data color augmentation in a self-supervised setting, especially for natural image classes such as birds and flowers.  For the targeted data sets such as Flowers-102, at low resolution 32x32, downstream classification can be improved by 5%.  Also by combining Planckian jitter with default random color jitter, similar or even larger improvements are shown more generally across data sets such as CIFAR and T1K+.  Overall the paper keeps it simple and clear for the reader and the overall idea is sound and the experiments very directly justify the basic idea of more realistic color augmentation for data sets with natural images.\n\nFigure 3 analyzing the color sensitivity of the results and the general results showing where for man-made imagery data sets, Planckian Jitter can be worse is useful to point out the limitations and details of the approach.  Finally it's an added bonus to verify that this basic general idea is applicable to multiple self-supervised models as shown in Table 4.  Although this isn't particularly surprising, it does add to the completeness of the experiments to support the basic idea in the paper.\n\nAlthough the LSC combined augmentation results show that Planckian and random jitter are somewhat complementary, the higher-resolution, 224x244, results in Table 3 do cause the reader to question whether the low resolution case is somehow special with respect to color augmentation.  Here Planckian jitter can actually be worse for the target data set of Flowers-102 which isn't fully explained why this happens.  Although the idea seems ok for CUB-200 and better for T1K+, again for the natural image data set of VegFru, performance is worse or similar than default random color jitter.  The main explanation is a one sentence claim that for higher resolution shape is very representative and color may add little discrimination, but more investigation here is necessary.  For example, perhaps higher resolution is actually causing texture to be more discriminative, rather than shape?  Intuitively, shape seems reasonably preserved even at lower resolution, unless this is correlated with the actual self-supervised architecture of hyperparameters.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "fuIHpzs9vSr", "reviewer": "Reviewer_W5kb", "summary": "This paper proposes to use a realistic white balancing as a way to augment datasets for self-supervised learning approach. The balancing was based on Planckian blackbody radiation. The author used this to train a siamese model in conjunction with standard color jittering that involves hue and saturation perturbation, and showed that the combination of the two color jittering method works best. Furthermore, the planckian jittering seems to lead to better performance on tasks where color is an important features such as flower classification task.", "questions": "Please see my weakness section. I would like to see the response to my weakness section. particularly around the benefits of PJ, which is my main concern for this paper.", "limitations": "Not quite. The author mentioned that their PJ reduces the quality of shape representation, but I do not see a convincing evidence of this in the text. No harm to the society is anticipated for the current manuscript.", "rating": 3, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 1, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n- Realistic image processing is complicated and is often skipped in ML literature. The color model used in this paper is largely correct and well-grounded in the color-conversion theory\n\nWeakness:\n- My main concern for this paper is that I am remain unconvinced of the benefit of the planckian jittering as presented:\n  - Fig 3 shows a more stable result, but it seems that the testing dataset also undergoes the simulated jittering rather than a real captures of different illumination. Such artificial manipulation of the test set is going to yield the same result regardless of how poorly the jittering performs.\n  - Planckian jitter do not always perform the best, and can underperform by a significant margin (see Table 1).\n  - The motivation for resizing images to 32x32 is unclear. In most practical application, such small patch sizes are unlikely to contain enough information to be useful. More investigation is needed on more realistic patch size of 200+pixel. In the present study, only Table 3 shows this result, and the performance of PJ is mixed on different dataset.\n- The color jittering model missed one crucial fact about the sRGB space, which is that it is gamma-encoded. Gamma compression is lossy, and resulted in far inferior color correction result compared to images captured in RAW. Nonetheless, I do not see the mentioning of gamma correction prior to color correction (and then reapplication to get back into the original color space).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "9OHoJ9chkuR", "reviewer": "Reviewer_eWq2", "summary": "The paper examines the color data augmentation used to train many deep networks for object classification tasks. It proposes that the existing color augmentation method does not permit the network to make effective use of color information when appropriate. They propose using a color augmentation that follows Planckian illuminants, which are physically-realistic. The paper shows that using the combination of a network trained using the standard color jitter and a network using the Plankcian jitter perform significantly better on classification tasks that can take advantage of color information.", "questions": "Are there other color distributions, or ways of obtaining color distributions, that might be better than the Planckian illuminants?  There are illumination situations, such as light filtering through trees, that are non-Planckian but still non-zero in terms of likelihoods.", "limitations": "There is a difference between color augmentation of materials and color augmentation of illumination. Illumination color is one color augmentation that can be done easily through a color rotation of the pixels. While on a jpg-compressed/color-enhanced image the effect is not perfect, it's close enough that the augmentation can allow the network to learn color-based rules that follow a more typical distribution of colors.  Regardless of whether color is a useful feature for an object, this type of augmentation matches what is likely to occur in the real world.\n\nColor augmentation for materials, on the other hand, is challenging to do properly as an augmentation. First, it is class-based, as every class will have a different material color distribution. Second, it is not appropriate to change the colors of the entire image in order to modify the material color. The uniformly random color augmentation seems like it is intended to try and cover both types of color changes (illumination and material).\n\nIt would be nice to have some discussion of this in the paper, though not necessary as it is not directly relevant to the author's contribution but an an additional limitation of the standard color augmentation methods.  But it would give a little more context to the issue in the introduction.", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The ablation study and analysis of the results support the hypothesis that a physics-based color jitter enables the network to more effectively use color information. However, there is no direct analysis of the network's learned features/structure to confirm that. The author's assume that color is a useful feature on the flower data set--but is not necessarily a useful feature on other data sets--and use that assumption to make the claim. It is clear that the Planckian color jitter is responsible for the bump in performance on the flower data set, and the assumption is reasonable.\n\nIt's important to look at data augmentation from a physics-based perspective, because uniformly randomized data augmentation ignores the fact that illumination does have typical distributions. The existing data sets usually do not sample extensively from illumination distributions, especially given that most of the images are have white-balancing applied by default.\n\nIt would be nice to see more extensive use of physics-based principles, however, since almost all standard data sets use jpg-compressed and color enhanced images (designed for human viewing), most of the physical rules that govern illumination and materials are lost.  The Plackian jitter at least replaces the uniformly random color data augmentation with expected results on a data set where color is likely to play a strong role in classification. Other examples of data sets where that might be the case are the bird and animal identification data sets.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Reject