Learning Color Equivariant Representations

Authors: Yulong Yang, Felix O'Mahony, Christine Allen-Blanchette
Primary area: interpretability and explainable AI
Keywords: Equivariant Neural Network, Geometric Deep Learning, Group Convolution

Abstract:
In this paper, we introduce group convolutional neural networks (GCNNs) equivariant to color variation. GCNNs have been designed for a variety of geometric transformations from 2D and 3D rotation groups, to semi-groups such as scale. Despite the improved interpretability, accuracy and generalizability of these architectures, GCNNs have seen limited application in the context of perceptual quantities. Notably, the recent CEConv network uses a GCNN to achieve equivariance to hue transformations by convolving input images with a hue rotated RGB filter. However, this approach leads to invalid RGB values which break equivariance and degrade performance. We resolve these issues with a lifting layer that transforms the input image directly, thereby circumventing the issue of invalid RGB values and improving equivariance error by over three orders of magnitude. Moreover, we extend the notion of color equivariance to include equivariance to saturation and luminance shift. Our hue-, saturation-, luminance- and color-equivariant networks achieve strong generalization to out-of-distribution perceptual variations and improved sample efficiency over conventional architectures. We demonstrate the utility of our approach on synthetic and real world datasets where we consistently outperform competitive baselines.

Decision: accept (poster)
Reviewer scores: 6, 8, 6, 5, 3

Reviews and discussion:
<thread>
creation date:
1730324653680

thread_type:
official_review

<thread_object>
creator: reviewer_r9zc

<content>
summary:
The authors propose GCNNs which are equivariant to color variation. GCNNs have been widely used for geometric transformations while only a few research has attempted to apply GCNNs to color-equivariant networks to achieve generalization to out-of-distribution samples. Throughout the paper, the authors reasonably justified the reason for designing the color-equivariant networks and proposed a proper method for dealing with such situations. Both quantitative and qualitative evaluations are conducted in the paper.

strengths:
- Interpretable visual illustrations of the results
- Interesting motivation
- Decent paper writing

weaknesses:
Overall, it is clear to me that the proposed method is better than the baselines (ResNet50, Z2CNN) under the generalization testing setup. However, the improvements over CEConvs look marginal (performance-wise and method-wise). Other weaknesses I have found are:

1. Related Works: 
    - What is the connection/difference of the proposed setting with Domain Generalization?
2. Missing baselines to support the argument in L44-50.
    - converting input image to grayscale 
    - data augmentation

questions:
1. Fig. 1: Baseline comparison with CEConvs

2. Fig. 4
    - What is the “Model sample efficiency”? 
    - How can the error improvement be computed?
    - Fig. 4 shows that there is no improvement over Z2CNN if 10% or 100% of training example is used. Why does it happen?
    - On the other hand, the advantage of the proposed methods are shown effective when only a little portion of the training example is used. What does this mean?

3. Table 1: Baseline comparison with CEConvs

4. Fig 5: Baseline comparison with CEConvs

5. Fig 7: Baseline comparison with CEConvs

6. What is the difference between the global hue-shift out-of-distribution (A/B) and the local hue-shift out-of-distribution (A/C)?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730393175179

thread_type:
official_review

<thread_object>
creator: reviewer_7wsa

<content>
summary:
This paper builds on the approach “Color Equivariant Convolutional Networks” [16, Neurips 2023], modifying their lifting layer by operating on the input image rather than on the filters. Additionally, it generalizes the equivariance, previously only defined on the hue channel to the  saturation channel. Experiments on synthetic datasets (hue shift MNIST and hue shift 3D shape) and on CIFAR, Caltech, Camelyon17, Caltech, STL-101, Stanford Cars, and Oxford pets dataset show improved classification performance compared to the CE-conv3 approach [16] and to Resnet baselines with same number of parameters.

strengths:
* The paper is well written and illustrated.
* The approach is well motivated and addresses concrete tasks, for instance improving classification robustness in case of different imaging processes coming from different labs in the case of medical imagery. 
* The claims are supported by experiments. 
* Some limitations of the approach are presented.  
* The code will be open source.
* The appendix provides necessary information to reproduce results.
* The approach designs color equivariance in group CCNN models, which is an interesting property.

weaknesses:
* The claim of improved equivariance by order of magnitude over [16] on the 3D shape dataset is true but the error of [16] is already very low on this dataset, 0.05 at most for CEconv-3. As the error is very small, this difference in magnitude will not translate into drastic improvements in practice.
* [Comparison to stronger baselines] The comparison to [16] on non-synthetic datasets displays larger errors than in [16]. Why not use their experimental protocol that seems well detailed in their paper? 
[Incomplete comparison] Only results with CE-conv3 and CE-conv4 appear in the current paper, while [16] often shows better results with CE-conv2.
* [No comparison to SOTA] Further discussion on the Camelyon-17 results would be interesting: Looking at the leaderboard on this dataset, https://wilds.stanford.edu/leaderboard/ , we note a number of approaches are performing better than the proposed approach. What would be the pros and cons of using the proposed approach vs the SOTA ?       
* The cardinality N of the group used in the experiments is not discussed. In addition, it is only introduced on page 7, it would help the reader to define the notation before as all figures are using it.

questions:
1/ Why not use the experimental protocol of [16] that seems well detailed in their paper for Table 4?

2/ What would be the pros and cons of using the proposed approach vs the SOTA (or a strong baseline from the leaderboard such as Group DRO) on Camelyon17 ?

3/ Could you explain how the different cardinalities have been chosen for the different experiments?

4/ In the limitation section, the computation overhead of the approach is mentioned, without mentioning explicit numbers compared to the baseline’s training time, could this information be mentioned?
</content>
</thread_object>

</thread>
<thread>
creation date:
1730456717028

thread_type:
official_review

<thread_object>
creator: reviewer_wtm5

<content>
summary:
The paper presents a novel convolutional neural network (CNN) architecture tailored to handle color variations effectively. The authors introduce a color-equivariant group convolutional network (GCNN) that is robust to changes in hue and saturation, significantly improving performance in tasks involving perceptual color transformations. By employing a new lifting layer that operates directly on input images rather than filters, the model overcomes limitations of existing color-equivariant networks, achieving enhanced stability and reducing equivariance errors by orders of magnitude. This approach yields superior performance on both synthetic and real-world datasets, particularly excelling in generalizing to out-of-distribution color variations and enhancing sample efficiency.

strengths:
1, The paper presents a novel approach to color-equivariant convolutional neural networks (CNNs) by extending GCNNs to handle hue and saturation changes in the HSL color space. This is achieved with an inventive lifting layer that acts directly on input images, improving stability and reducing errors by avoiding invalid RGB values—a common issue in previous architectures.

2, The authors demonstrate a thorough experimental design using various synthetic and real-world datasets (e.g., Hue-shift MNIST, Camelyon17) known for color variations, showing the model’s robustness. Both quantitative and qualitative analyses, including sample efficiency and feature map visualizations, offer a detailed evaluation. Comparisons with competitive baselines like CEConv and ResNet confirm the model’s generalizability and efficiency.

3, The paper is well-structured, with a logical flow from problem statement to methodology and results. Clear explanations of the hue- and saturation-equivariant group actions, along with detailed figures, aid in understanding the technical concepts. Comprehensive appendices further enhance reproducibility and transparency by providing proofs and additional experiment details.

weaknesses:
1, The paper extends GCNNs to be equivariant to hue and saturation but does not address luminance variations, which are also important for color representation. Although the authors suggest that luminance invariance may be approximately preserved, this may not hold in real-world applications, such as medical imaging or scenes with varying lighting. Future work could integrate luminance into equivariant transformations or evaluate its impact on specific tasks.

2, The model is compared to standard CNNs (e.g., ResNet) and color-equivariant networks (e.g., CEConv), but not to other common approaches for handling color variations, such as color augmentation techniques or robust contrastive learning models. Adding these comparisons would strengthen the case for this method’s effectiveness in handling color shifts.

questions:
1, The additional computational load from group convolutions and lifting layers present a challenge for real-time applications. Therefore, it is essential to compare your method with common alternatives, such as color augmentation or robust contrastive learning approaches, which are known to enhance robustness to color and perceptual variations. These comparisons would be especially valuable for demonstrating the unique advantages of color equivariance over standard augmentation techniques.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730527209134

thread_type:
official_review

<thread_object>
creator: reviewer_jspt

<content>
summary:
The paper extends the concept of color equivariance to include saturation shifts, in addition to hue transformations, then they design GCNNs that are equivariant to color variation by leveraging the geometric structure of color in the hue-saturation-luminance (HSL) color space, additionally a novel lifting layer is proposed to transform input images directly, avoiding invalid RGB values and significantly improving equivariance error.

strengths:
1. The paper is very well written and easy to read.

2. The authors conducted extensive experiments using natural images (CIFAR10, CIFAR100, STL-10, etc.) as well as medical images (Camelyon-17) to validate the effectiveness of their algorithm. The experiments are thorough and detailed.

3. Color equivariant representations are an important concept in computer vision, yet they are often overlooked in contemporary research. The authors effectively claim the significance of color in the era of deep learning in this work.

weaknesses:
I believe this work has no significant drawbacks and meets the acceptance standards for ICLR. 

Beyond that, I am curious about two questions:

1. This paper uses the HSL color space as the basis for designing experiments, specifically designing groups and group actions for hue and saturation, as well as a hue-saturation group action. I am interested to know whether this approach would also be applicable to other chromacity-luminance color spaces, such as CIELAB or YUV.  Additionally, the experiments are designed around hue shifts in the HSI series (i.e. H in HSV. HSL, HSB...). Could other chromacity shifts in different color spaces be designed to further demonstrate the robustness of the model?

2. The experiments presented in this paper focus on classification tasks. I wonder if this design could be even more beneficial for higher-level tasks or those that rely more on semantic information, such as semantic segmentation. Do the authors have any plans to design similar experiments for such tasks?

questions:
See weakness part.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730783059735

thread_type:
official_review

<thread_object>
creator: reviewer_y47k

<content>
summary:
This paper introduces group convolutional neural networks designed to be equivariant to variations in hue, saturation, and color in RGB images. It proposes a lifting layer that directly transforms the input image to produce a color-equivariant descriptor, effectively addressing the challenge of invalid RGB values found in previous works. Experiments on synthetic and real-world datasets highlight the robustness of the proposed method to color variations.

strengths:
1.	The approach is technically compelling, as it enables convolutional neural networks to be equivariant to hue and saturation by design, leveraging the inherent geometric structure of these color components.

2.	Results across several benchmarks indicate that the proposed method offers stronger robustness to color variation compared to existing methods.

3.	The paper is well-structured and easy to understand.

weaknesses:
1.	Experiments are limited to small toy datasets with pronounced hue or saturation shifts, leaving it unclear how the method would perform on large datasets like ImageNet.

2.	The proposed method focuses on hue and saturation equivariance, but does not address other common variations in real-world data, such as 3D geometric transformations or lighting changes.

questions:
Would the proposed method improve image classification performance on large-scale datasets like ImageNet or on tasks beyond image classification?
</content>
</thread_object>

</thread>
<thread>
creation date:
1732159948554

thread_type:
official_comment

<thread_object>
creator: authors

<content>
comment:
We thank the reviewers for their thoughtful feedback. We are encouraged they found our work to be well motivated (7Wsa, r9ZC); technically compelling/novel/inventive (y47, jSPt, WTM5); effective in addressing the challenge of invalid RGB values (y47, jSPt, WTM5); and robust to color variation compared to existing methods (y47, WTM5, 7Wsa). We are also glad they found our claims to be well substantiated by the numerous experiments (y47, jSPt, WTM5, 7Wsa) and analyses (WTM5); and our paper to be well written, well illustrated, and easy to read (y47, jSPt, WTM5, 7Wsa, r9ZC). 

Reviewers y47 and WTM5 noted the prevalence of luminance variation in real-world data. To explore this, we evaluated the generalization performance of a conventional CNN and **our newly developed luminance equivariant GCNN** on the small NORB dataset [1]. The small NORB dataset consists of 48.6k 96x96 grayscale images captured under 6 different lighting conditions, 9 elevations, and 18 azimuths. When we trained and tested on the same luminance conditions, the conventional CNN and our new luminance equivariant GCNN performed comparably, around 92% accuracy. However, when we trained and tested on different luminance conditions, the performance of the conventional CNN dropped to 64% while our luminance equivariant GCNN maintained an accuracy of 78%. The details of how we define our luminance space, and luminance transformation are provided in Appendix E.1. Thank you for pointing out this opportunity to extend the notion of color equivariance to luminance, we will definitely include these results in our paper! 

Reviewers WTM5 and R9ZC suggested comparisons against baselines that use data augmentation or grayscale input images. **We ran additional experiments to compare the performance of our color equivariant approach to the suggested color invariant approaches** and found that our color equivariant approach performed on-par or better than the suggested color invariant approaches. We have included these results in Table 4. We’d like to point out that even though all methods perform similarly on classification, our color equivariant architecture is more interpretable by construction, has higher sample efficiency (which others have shown both theoretically and empirically [2,3,4,5], and we’ve shown Figure 4), and naturally allows for downstream tasks such as color based sorting (see Figure 7) and color alignment.

We address other reviewer comments below and will incorporate all feedback. We highlight changes to our original manuscript with blue text.

[1] LeCun, Yann, Fu Jie Huang, and Leon Bottou. "Learning methods for generic object recognition with invariance to pose and lighting." Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2004. CVPR 2004.. Vol. 2. IEEE, 2004.  
[2] Behboodi, Arash, Gabriele Cesa, and Taco S. Cohen. "A pac-bayesian generalization bound for equivariant networks." Advances in Neural Information Processing Systems 35 (2022): 5654-5668.  
[3] Elesedy, Bryn. "Group symmetry in pac learning." ICLR 2022 workshop on geometrical and topological representation learning. 2022.  
[4] Zhu, Sicheng, Bang An, and Furong Huang. "Understanding the generalization benefit of model invariance from a data perspective." Advances in Neural Information Processing Systems 34 (2021): 4328-4341.  
[5] Brehmer, Johann, et al. "Does equivariance matter at scale?." arXiv preprint arXiv:2410.23179 (2024).
</content>
</thread_object>

</thread>


Metareview:
metareview:
This submission introduces a colour-equivariant group convolutional network (GCNN) that generalises equivariance beyond the hue channel to saturation in RGB images. The proposed method employs a lifting layer capable of directly transforming input images and further leverages the geometric structure in hue-saturation-luminance (HSL) colour space. Experiments on synthetic and real-world datasets show improved classification performance over baseline approaches.

This paper was discussed at length with the SAC. After reviewing the paper, rebuttal and resulting discussions AC believes that the overall strengths outweigh the weaknesses and recommends acceptance. For the camera-ready version, the authors should incorporate all key rebuttal results and discussion points as suggested.