Venue: NeurIPS 2022 Conference

Reviews:

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: The paper examines the color data augmentation used to train many deep networks for object classification tasks. It proposes that the existing color augmentation method does not permit the network to make effective use of color information when appropriate. They propose using a color augmentation that follows Planckian illuminants, which are physically-realistic. The paper shows that using the combination of a network trained using the standard color jitter and a network using the Plankcian jitter perform significantly better on classification tasks that can take advantage of color information.
strengths_and_weaknesses: The ablation study and analysis of the results support the hypothesis that a physics-based color jitter enables the network to more effectively use color information. However, there is no direct analysis of the network's learned features/structure to confirm that. The author's assume that color is a useful feature on the flower data set--but is not necessarily a useful feature on other data sets--and use that assumption to make the claim. It is clear that the Planckian color jitter is responsible for the bump in performance on the flower data set, and the assumption is reasonable.

It's important to look at data augmentation from a physics-based perspective, because uniformly randomized data augmentation ignores the fact that illumination does have typical distributions. The existing data sets usually do not sample extensively from illumination distributions, especially given that most of the images are have white-balancing applied by default.

It would be nice to see more extensive use of physics-based principles, however, since almost all standard data sets use jpg-compressed and color enhanced images (designed for human viewing), most of the physical rules that govern illumination and materials are lost.  The Plackian jitter at least replaces the uniformly random color data augmentation with expected results on a data set where color is likely to play a strong role in classification. Other examples of data sets where that might be the case are the bird and animal identification data sets.
questions: Are there other color distributions, or ways of obtaining color distributions, that might be better than the Planckian illuminants?  There are illumination situations, such as light filtering through trees, that are non-Planckian but still non-zero in terms of likelihoods.
limitations: There is a difference between color augmentation of materials and color augmentation of illumination. Illumination color is one color augmentation that can be done easily through a color rotation of the pixels. While on a jpg-compressed/color-enhanced image the effect is not perfect, it's close enough that the augmentation can allow the network to learn color-based rules that follow a more typical distribution of colors.  Regardless of whether color is a useful feature for an object, this type of augmentation matches what is likely to occur in the real world.

Color augmentation for materials, on the other hand, is challenging to do properly as an augmentation. First, it is class-based, as every class will have a different material color distribution. Second, it is not appropriate to change the colors of the entire image in order to modify the material color. The uniformly random color augmentation seems like it is intended to try and cover both types of color changes (illumination and material).

It would be nice to have some discussion of this in the paper, though not necessary as it is not directly relevant to the author's contribution but an an additional limitation of the standard color augmentation methods.  But it would give a little more context to the issue in the introduction.
review_title: null

final_score: rating: 7: Accept: Technically solid paper, with high impact on at least one sub-area, or moderate-to-high impact on more than one areas, with good-to-excellent evaluation, resources, reproducibility, and no unaddressed ethical considerations.
review_content: summary: This paper adds to the data augmentation transforms in the literature by proposing a more physics-based approach to generate more color-realistic augmentations compared to the standard more random color jitter.  This is well-demonstrated in Figure 1 for natural image classes such as flowers where although some classes may have strong color variations, in general the variations are more realistic when due to illumination, compared to man-made classes such as cars.  The approach is motivated by and follows a standard black-body radiation model, which again as shown in Figure 1 constrains the color jitter to be more realistic.

The experiments are performed in a self-supervised fashion as that nicely focuses the weight on data augmentation.  Both natural and man-made data sets are tested such as Flowers-102, CUB-200, and CIFAR.  Overall the results summarized in Figure 3, and Tables 1, 2, 3 show the improvements in downstream classification with Planckian jitter on natural image classes and the potential for combining Planckian and random color jitter in general.
strengths_and_weaknesses: The paper is very direct in the exact problem and contribution, namely more realistic data color augmentation in a self-supervised setting, especially for natural image classes such as birds and flowers.  For the targeted data sets such as Flowers-102, at low resolution 32x32, downstream classification can be improved by 5%.  Also by combining Planckian jitter with default random color jitter, similar or even larger improvements are shown more generally across data sets such as CIFAR and T1K+.  Overall the paper keeps it simple and clear for the reader and the overall idea is sound and the experiments very directly justify the basic idea of more realistic color augmentation for data sets with natural images.

Figure 3 analyzing the color sensitivity of the results and the general results showing where for man-made imagery data sets, Planckian Jitter can be worse is useful to point out the limitations and details of the approach.  Finally it's an added bonus to verify that this basic general idea is applicable to multiple self-supervised models as shown in Table 4.  Although this isn't particularly surprising, it does add to the completeness of the experiments to support the basic idea in the paper.

Although the LSC combined augmentation results show that Planckian and random jitter are somewhat complementary, the higher-resolution, 224x244, results in Table 3 do cause the reader to question whether the low resolution case is somehow special with respect to color augmentation.  Here Planckian jitter can actually be worse for the target data set of Flowers-102 which isn't fully explained why this happens.  Although the idea seems ok for CUB-200 and better for T1K+, again for the natural image data set of VegFru, performance is worse or similar than default random color jitter.  The main explanation is a one sentence claim that for higher resolution shape is very representative and color may add little discrimination, but more investigation here is necessary.  For example, perhaps higher resolution is actually causing texture to be more discriminative, rather than shape?  Intuitively, shape seems reasonably preserved even at lower resolution, unless this is correlated with the actual self-supervised architecture of hyperparameters.
questions: As mentioned above, the main question for the reader is the somewhat counter-intuitive results in Table 3, for the same data sets but with higher resolution at 224x224.  One would expect a similar trend as the low-resolution data, but the numbers show otherwise.  Also the explanation on better shape discrimination on higher resolution bears more analysis and detail, e.g., is it more about texture than shape at higher resolution?

Although the LSC combination is a good start to fuse both types of color augmentation, what about a way to make it more class-specific?  In other words, for even a coarse dichotomy such as natural vs man-made images, apply some complementary weighting/distribution of Planckian/default color jitter to get the best of both worlds?
limitations: The paper has an explicit section on limitations regarding the overhead with the combined color augmentation approach and also mentions the incomplete investigation into how much and in what detail the proposed Planckian jitter improves color realism why discouraging shape discrimination.
review_title: null

final_score: rating: 3: Reject: For instance, a paper with technical flaws, weak evaluation, inadequate reproducibility and incompletely addressed ethical considerations.
review_content: summary: This paper proposes to use a realistic white balancing as a way to augment datasets for self-supervised learning approach. The balancing was based on Planckian blackbody radiation. The author used this to train a siamese model in conjunction with standard color jittering that involves hue and saturation perturbation, and showed that the combination of the two color jittering method works best. Furthermore, the planckian jittering seems to lead to better performance on tasks where color is an important features such as flower classification task.
strengths_and_weaknesses: Strength:
- Realistic image processing is complicated and is often skipped in ML literature. The color model used in this paper is largely correct and well-grounded in the color-conversion theory

Weakness:
- My main concern for this paper is that I am remain unconvinced of the benefit of the planckian jittering as presented:
  - Fig 3 shows a more stable result, but it seems that the testing dataset also undergoes the simulated jittering rather than a real captures of different illumination. Such artificial manipulation of the test set is going to yield the same result regardless of how poorly the jittering performs.
  - Planckian jitter do not always perform the best, and can underperform by a significant margin (see Table 1).
  - The motivation for resizing images to 32x32 is unclear. In most practical application, such small patch sizes are unlikely to contain enough information to be useful. More investigation is needed on more realistic patch size of 200+pixel. In the present study, only Table 3 shows this result, and the performance of PJ is mixed on different dataset.
- The color jittering model missed one crucial fact about the sRGB space, which is that it is gamma-encoded. Gamma compression is lossy, and resulted in far inferior color correction result compared to images captured in RAW. Nonetheless, I do not see the mentioning of gamma correction prior to color correction (and then reapplication to get back into the original color space).
questions: Please see my weakness section. I would like to see the response to my weakness section. particularly around the benefits of PJ, which is my main concern for this paper.
limitations: Not quite. The author mentioned that their PJ reduces the quality of shape representation, but I do not see a convincing evidence of this in the text. No harm to the society is anticipated for the current manuscript.
review_title: null

Meta-review: metareview: Training representations in computer vision typically requires systematic augmentations of the input training set such as crops, reflections, translations including color jitter. As illumination invariance tends to be a desired property in visual object detection tasks, design of specific color augmentations is of great interest. In this context, this paper proposes to improve 
the arguably inferior color representation of standard Color Jitter by proposing 
a realistic and convincing physics-based color representation schema, – called Planckian Jitter -- and provides some experimental support. 

The discussion about the work concentrated around the overall effectiveness and experimental validation. The reviewers have mixed conclusions about the work. In light of the mixed opinions,
I tend to agree with the concerns raised by reviewer W5kb, who suggests to 

-- Repeat experiments with higher resolution as 32 x 32 seems to be rather small for a conclusion of effectiveness.

-- Including some obvious baselines (e.g., no augmentation)

-- Clarify the take home message and conclusions from ablations.

I feel that the paper would benefit from a further iteration and that the current manuscript is not ready yet for publication.

confidence: Certain
Decision: Reject