Venue: ICLR 2022 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The authors propose a physically sensible data augmentation that takes into account the actual variation of day light illumination in self-supervised contexts. 
They analyze the impact on the classification performance, the sensitivity of the performance to illumination changes, and the emergence of neurons sensitive or insensitive to color. 

main_review: Realistic data augmentation is important in self-supervised learning because this is the sort of data variability that one wants to ignore and be invariant to. 
Overall, this is an example of physics-aware machine learning (or in simpler words, do things just right ;-) ). The proposed "Planck+VonKries" color augmentation transform is more than 100 years old, but it may be interesting for different audiences for different reasons: (a) people with color science background will not find it surprising, but they may be interested to see the quantitative gain that can be obtained from doing the classical right thing, and (b) the average computer scientist facing this problem for the first time will find a good visual motivation to avoid random augmentation (fig. 1) and a physically founded recipe to get a 10% improvement in classification.

Additionally, authors make an interesting observation: they argue that unrealistic (random) transforms may induce overinvariance (insensitivity) to certain aspects of the data, thus reducing the performance of the model or shifting the focus to other features of the data. 
I think their results illustrate this general point: in their specific example, augmentation through unrealistic color jitter reduces the sensitivity to color (as confirmed by the number of color-sensitive neurons) and leads to extra consideration of spatial features. 
A similar argument could be made for other augmentation strategies: if the database lacks natural variability, the discovered features are going to be unbalanced.

Overall I find this work as an interesting exercise in the context of data augmentation, although the technique is not novel and the general consequences are not extracted.

Other points to address:

* While the general change-of-focus effect is true, the authors should be more specific about when this is going to be a benefit. As it is presented now, it depends on the database or on the relevance of color for class discrimination. 
My opinion is that the proposed augmentation (as any natural data augmentation) is going to be more positive in too artificial (too restricted) databases where illumination is fixed or it has too low variability. These are the cases that require the introduction of a natural variability (in this case of the illumination). This is consistent with the fact that wide databases such as CIFAR-100 as opposed to Flower102 get no improvement from the proposed jitter (see performance numbers or figure 6). Maybe this is because CIFAR-100 already includes the variability of natural illumination and hence explicit augmentation with the Planckian variability is not necessary.   

Can the authors provide some extra evidence about the fact that neurons insensitive to color (when conventional color jitter is introduced) capture better the shape/texture information?

* Equation 4 for VonKries is confusing. 
Von Kries-like adaptation consists of dividing each LMS (I may admit each RGB) value by the value corresponding to the white (or illuminant) in situation 1, and multiply by the value corresponding to the white (or illuminant) in situation 2 (see for instance Chapt.9 in Fairchild's Color Appearance Models, 2013).
This implies the multiplication of each tristimulus vector by clarify Eq. 4.

* The spectral radiances in Fig. 4 are confusing: using no normalization and a linear vertical axis implies that all the explored spectra seem "blueish" (with bigger contribution of short wavelengths). Decay in energy at low temperatures of the black body radiator make it difficult to see that between 3000 and 4000 K the contribution of long wavelengths is bigger ---> you have reddish illuminants too, but it is hard to see in that figure if you dont nromalize the energy or put a log scale in the vertical axis... Note that for instance the CIE A illuminant corresponds to 2856 K (almost the 3000K reported by the authors) and it is markedly reddish.

* Using the novel ARC chromatic diagram does not add anything special with regard to classical CIE xy diagram. Why not using the standard diagram?. Incidentally, I was surprised to see that the Planck locus is straight in ARC, but the variations in the colors in Fig. 1 (diagram at the right) go in curves... is this right?.

* Eqs. 1 and 2 seem to suggest that in Fig. 2 there is an additional Multilayer Perception in the lower branch of the training, isnt it?
summary_of_the_review: I find this work is an interesting exercise in the context of data augmentation (an illustration of how classical models from physics may benefit the learning), although the technique is not novel and the general consequences are not discussed in detail.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: **Overview.**
This paper proposes a novel type of image colour augmentation to be used during self-supervised learning (SSL).

**Background.**
In a typical SSL setting, similar samples are generated by randomly augmenting an image in a variety of different ways: random cropping, colour jittering, random rotations, etc. These similar images are then passed through a neural network, and the predicted features for similar samples are trained to be close to each other based on some similarity metric (ignoring the contrastive term description here as it is not directly related to the paper).

**Motivation.**
In this paper, the authors address the issue of using colour jittering as augmentation during SSL. Specifically, using colour jittering pushes the network towards invariance to image colours, while relying more on the shape and texture of objects when making predictions. The authors point out that despite the benefits of this for many general detection tasks, this will be a detrimental property when dealing with more colour-dependant tasks.

**Method.**
Thus, they propose to use Planckian jittering instead of colour jittering. Planckian jittering is a physics based augmentation method proposed by the authors (although the formulations for it come from existing literature) to re-illuminate the training images within a realistic distribution which leads to more realistic and constrained colour augmentations than colour jittering. The paper claims that Planckian jittering still helps improve network's dependence on shape and texture of objects (although less than colour jittering), while limiting the network's invariance to image colours.

**Experiments.** 
6 SSL models were trained independently on CIFAR-100: 3 used different variants of the Planckian jittering, 2 used different variants of colour jittering (w/ and w/o Random GrayScale), and 1 used no augmentations. Linear classifiers for CIFAR-100 and FLOWERS-102 classification tasks were then trained on top of each of the SSL models' features, where FLOWERS-102 is the task that is claimed to be more heavily colour-dependant. Moreover, an extra linear classifier was trained on the concatenation of features from a Planckian jittering and a colour jittering model (called the "Latent space combination" model). Based on accuracy, "Latent space combination" outperforms all other models by a significant margin on both datasets. Planckian jitter seems to outperform other augmentations on FLOWERS-120 (Table 1), which supports the claim of the authors. On CIFAR-100, Planckian jitter performs slightly worse than colour jittering, which the authors attribute to the reduction of colour invariance in the features.

A very similar experiment was also done with different datasets (SSL training on tiny-imagenet, linear classifier trained on FLOWERS-102, CUB200, VEGFRU, T1K+), which also obtained similar results and conclusions. Moreover, in another similar experiment, the SSL models were trained with different SSL configurations (SimSiam, SimCLR, Barlow Twins) to indicate the generality of Planckian jitter for different types of SSL configurations. In another experiment, the robustness of the different models on augmented images using Planckian jittering was evaluated. Lastly, the colour sensitivity was analyzed to inspect the impact of colour information in neuron activations for each model.
main_review: **Pros:**

+ Well written and easy to follow manuscript 
+The idea of the physics-based Planckian jitter which transforms colours along chromaticity lines is interesting
+ There does seem to be some support in the experiments for Planckian jitter improving results for colour-dependant domains (I.e. Table 1 when comparing "Default Color Jitter w/o Random GrayScale" and "Planckian Jitter")


**Cons:**

- Comparing the "Latent space combination" with other forms of augmentations seems very unfair since "Latent space combination" has double the capacity and requires double the training compute. To me "Latent space combination" seems more as an ensemble of models. To make it a fair comparison, it needs to be compared with other ensemble combinations (e.g. "Default Color Jitter w/o Random GrayScale"+"None" or "Default Color Jitter w/o Random GrayScale"+"Random Crop" seems like a reasonable baseline). Otherwise, I think "Latent space combination" should be removed altogether.
- Since cropping is one of the most important transformations for SSL, for all practical purposes there should be two versions of Table 1, one with and one without random cropping applied. I can imagine random cropping may impact the results.
- The main focus in current comparisons should be between "Default Color Jitter w/o Random GrayScale" and "Planckian Jitter", but this was left out entirely in Tables 2 and 3. Comparing "Default Color Jitter" (with Random GrayScale) against "Planckian Jitter" does not seem fair.
- Adding to above point, even comparing "Planckian Jitter" against "Default Color Jitter w/o Random GrayScale" isn't really enough. Since, as clearly indicated in Table 1, Planckian jitter mainly modifies brightness and contrast, there should be a direct comparison with an augmentation which changes brightness, contrast, brightness+contrast, and hue.
- There isn't any large scale datasets used in the experimentations, which raises the question of scalability and whether the Planckian augmentation would still be relevant with larger datasets.
- Planckian jitter is an augmentation technique which can also be evaluated on supervised learning tasks where image colours are important (and similarly, for transfer learning afterwards). Adding supervised experiments can add to the comprehensiveness of the experiments.
 - Section 4.2.1 mentions "As can be seen in Table 1, if color augmentations are removed completely (None configuration), the accuracy drops of 18%.". This seems incorrect when comparing "Default Color Jitter w/o Random GrayScale" to "None".
- The left plot of the colour sensitivity analysis (Fig 3.) seems unfair. The Planckian jitter models were specifically trained to ignore Planckian jitters, so one would naturally expect their results to be more robust with respect to Planckian jittering.  


**Typos:**
- Section 3.1 paragraph 1 states "...default color jitter (see Fig. 3).", I believe this was intended to reference Fig 1.

**Questions for authors:**

1. It seems very odd to me that in table 1, "None" performs so similar to "Default Color Jitter w/o Random GrayScale". Do you have any thoughts on why this happens?
2. Were there any other augmentations used during training (cropping, rotations, etc)?

Edits:

Nov 3 -- typo fix
summary_of_the_review: This paper proposes an interesting idea and is very well written. The issues (with colour jittering) pointed out in this paper were previously known. Thus I view this paper as more geared towards practical purposes. However, the experiments are not comprehensive enough to strongly support the practical claims. Thus, my vote is for the paper to be rejected in its current form.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper first examines that typical color jittering augmentation is harmful to feature representation learning. Then the authors proposed a physics-based color augmentation, called Planckian jitter to improve the performance. The proposed Planckian jitter performs better with the recent contrastive and self-supervised learning schemes.
main_review: Strengths:
 - The proposed physics-based color augmentation is easy to understand and implement.
 - The performance seems to be surprisingly better than typical random color jittering.
 - This color augmentation can be applied to a wide variety of tasks.

Weaknesses:
 - Is the proposed physics-based color augmentation slower than typical color jittering. Many tasks need color augmentation performed on the fly during training. If the proposed color augmentation is much slower, the impact of this work will be marginal.
summary_of_the_review: This paper proposed a general color augmentation that performs better than the typical color jittering and demonstrates better performance in contrastive and self-supervised learning schemes. The only concern I have is the speed.
review_title: null

Meta-review: The reviewers were in general lukewarm about the paper, not convinced by why realistic augmentation mean more robust features in SSL, had concerns over the szie of the datasets (up to ~100k), and the success depends on the relevance of color for classification. The AC agrees with the reviewers. While the paper sounds interesting, there are many questions remain unanswered -- it's unclear that the rebuttal addressed the concerns shared by the reviewers.

In addition to the comments by the reviewers, the AC also feels that the overall design is adhoc and it's unclear that the proposed augmentation can generalize to larger, more practical problems.
Decision: Reject