Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The paper examines the use of color in data augmentation for training deep networks and how it impacts the embedding representation of the input. In particular, they compare the traditional random color augmentation strategy with a color augmentation based on simulating changing illuminants. The illuminants used are drawn from the distribution of Planckian illuminants. The authors train the network in an unsupervised manner to learn an embedding space and then build a linear classifier using the embedding to test the effectiveness of the embedding on two standard data sets. Using a combined embedding that makes use of a network trained using the traditional color augmentation and a network trained using the Planckian color augmentation provides the best performance.
strength_and_weaknesses: The strength of the paper is the intentional analysis of the color augmentation process used in deep network training. In particular, the author's hypothesize that the standard color augmentation process strongly reduces color as a useful feature for classification. While for some objects, color is not strongly indicative (though certain distributions may be), for other tasks, such as natural objects like flowers and birds, color may be a critical feature. The improved performance on the flowers database appears to support this hypothesis. Likewise, the evaluation on two other data sets shows that the embedding trained with the Planckian illuminants is robust across simulated illumination changes, while the embedding trained with traditional color augmentation is not.

There are other potential illuminants besides Planckian, but for the purpose of analyzing this question, it is a reasonable model to choose.

This is only a first step towards appropriate training using color augmentation. The whole image augmentation model means that there is no selectivity by surface or current illumination conditions. An object in shadow, for example, is in a different lighting condition than the same object that is fully lit, and simulating the spectral rotation of a new direct illuminant shouldn't actually modify the shadowed areas as  much or at all.  Likewise, there is still a need for color augmentation of objects for which color is not discriminative.  However, the use of the combined embedding acknowledges that. It could be more explicit in the paper that there are several valid types of color augmentation, and that whole image color augmentation is a pretty heavy hammer compared to an ideal selective augmentation process.

Given the need to train two separate embeddings, one for each type of jitter, how does this get implemented in standard network architectures?  Does it have utility beyond this particular network structure?
clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and should have enough detail to reproduce the work. The analysis is original and asks an important question about something that is usually overlooked in network training.
summary_of_the_review: This is an important issue in network training. However, it might be challenging to integrate it into a standard architecture/classification problem given the utility of concatenating separate embeddings based trained on different types of color jitter.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a color data augmentation method for self-supervised learning or contrastive learning, which applies physically illuminant variation to images and the illuminants described by Planck’s Law for black-body radiation. The experimental results show that the proposed color augmentation method outperforms existing methods in some cases.
strength_and_weaknesses: Strength
+ Color is important in image classification, especially for some fine-grained image classification, such as flower classification.
+ Considering the physical and realistic procedure for color data augmentation is novel.

Weakness
- The proposed method assumes that the images are captured under the daylight D65. However, in the daily life, we capture image under various lights, such as LED light and other daylight D55 and so on.
- The improvement of proposed method is not significant in some cases. According to Table 1, the proposed Planckian Jitter performs significantly worse than default color jitter on CIFAR-100 dataset. Besides, Planckian Jitter without brightness and contrast (B&C) or Planckian-based chromaticity performs even worse than without data augmentation. 
- The compared methods are not comprehensive. The proposed Planckian Jitter mainly performs on SimSiam (2021), and generally performs on SimCLR (2020) and Barlow Twins (2021). It is better to perform on more recently proposed contrastive learning methods. Besides, the goal of color data augmentation is to improve the performance of image classification network. Thus, why not directly perform the proposed method on advanced image classification network, such as ResNet-101, Swin-Transformer and so on.
- The selection of some parameters is difficult to understand, such as C_B, C_C in Equation 5. The effectiveness of these parameters is unknown.
- There are some typos. For examples, ‘We’ of term 4 in page 4.

clarity,_quality,_novelty_and_reproducibility: The representation is clear, the quality can be improved, the novelty is good, and the reproducibility of this work seems to be good.
summary_of_the_review: Applying physical and realistic color data augmentation to network training is valuable, but the signification, comparison and some other aspects of this work can be improved.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper proposes a physics-based color data augmentation method (called Planckian Jitter), to train deep models to be robust to illumination changes. Experiments show that this method can produce a representation that is complementary to the representations learned by current color jitter augmentation methods, which further boosts performances on a variety of downstream tasks. The paper also presents an analysis of the color sensitivity of deep models to training strategies. Experiments show that the proposed method is robust to illuminant changes and can be combined with existing self-supervised learning methods to further boost their performance.
strength_and_weaknesses: Strong points.

1. A new color data augmentation method is proposed, which is demonstrated to be robust to illumination changes, useful for recognizing objects based on color information, and can be combined with other augmentation methods.

2. This work is motivated well. Figure 1 is a good visualization that illustrates the differences between the existing color jitter method and the proposed method.

3. Claims are generally supported via justifications and experiments.

Weak points.

While this paper proposes to learn color representations for tasks in which the intrinsic color of the objects is crucial, I wonder whether the proposed method can work well on 1, scene-level image-based tasks where multiple objects with different colors co-exist, and 2, Fine-grained image categorization (Sub-category recognition).

The writing could be improved for better readability. Some issues are listed below.

1. The paper does not use hyperlinks for figures/tables/references discussed in the main text, making it difficult to locate them or refer to the references for more information.
2. Figure 2 does not have "p_2" on it.
3. It is quite strange to suddenly describe the proposed network architecture in the related work section.
4. The "Data augmentation" paragraph of Section 2 says "x_1, x_2 used in the ... in Eq. 2" but Eq. 2 does not contain "x_1" and "x_2".
5. Figure 4 is mentioned before Figure 3 in the main text.
clarity,_quality,_novelty_and_reproducibility: The paper is generally written well but there are some minor issues as listed above. 

The paper presents an interesting method for color augmentation and shows its effectiveness in some scenarios. 

The paper explains detailed information about the proposed method and experiments and promises to release the code.
summary_of_the_review: The paper presents an interesting method for color augmentation and shows its effectiveness in some scenarios. I do not find major issues but am concerned about its further applications. I hope the authors could discuss them, which should be inspiring for future research.
review_title: null

final_score: rating: 3: reject, not good enough
review_content: summary_of_the_paper: The paper is about color augmentation, which is sometimes used for network training. Instead of common color augmentation by changing  brightness, contrast, saturation, and hue randomly, the paper suggests Planckian-based color augmentation. 
strength_and_weaknesses: * Strengths:
1. The idea is straightforward and the paper is easy to read.

* Weaknesses:
1. I am sure the suggested Planckian-based augmentation is useful for a few applications that suffer from light color changes. However, I am not sure the importance of this in general settings. The existing common color augmentation is still needed for general applications (those that are not much affected by light colors, but suffer from the limited training data). Hence, proposing only the Planckian augmentation in a whole paper is too narrow and too little in terms of the contribution. 

2. A quick search on the internet shows there are a few papers proposing different color augmentation techniques for certain applications: PCA Color Augmentation, "Data-Driven Color Augmentation Techniques for Deep Skin Image Analysis", "A New Color Augmentation Method for Deep Learning Segmentation of Histological Images". However, none of these are discussed in the paper.

3. The Planckian-based augmentation might be new, yet the significance of this is minor.

4. The proposed pipeline/architecture has nothing novel.

clarity,_quality,_novelty_and_reproducibility: The paper is lack of novelty and the significance of the problem.
summary_of_the_review: Please see my comment on novelty and quality above.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper re-examines color in the context of data augmentation.  The new augmentation is shown to have beneficial effects in some applications, and to additively combine with existing augmentations in others.

Reviewers concur: the paper's proposed augmentation is novel, and will be useful in some scenarios e.g. fine grained recognition.  The idea is simple, but reviewers consider it worthy of exposure to the community, so lean towards acceptance.

R1 expresses concern that table 1 shows the augmentation is not useful in general, but other reviewers, and the AC are convinced by table 3, showing that it is generally useful when combined with other augmentations.

Reviewers were concerned that the experiments were shown only for simple models, and that the augmentation might not help with more complex models, but the rebuttal's inclusion of the VicReg experiment mitigated this concern.

justification_for_why_not_higher_score: While useful, the paper's contribution is somewhat slight - a poster will be sufficient to expose the idea to other practitioners.
justification_for_why_not_lower_score: Reviewers are in agreement that it is timely to publish this work, and that ICLR is an appropriate venue.
Decision: Accept: poster