Title: Color Equivariant Convolutional Networks
Venue: NeurIPS 2023
Accepted: yes
Keywords: color equivariance

equivariance

color robustness

equivariant convolutions
Abstract: Color is a crucial visual cue readily exploited by Convolutional Neural Networks (CNNs) for object recognition. However, CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. Color invariance addresses this issue but does so at the cost of removing all color information, which sacrifices discriminative power. In this paper, we propose Color Equivariant Convolutions (CEConvs), a novel deep learning building block that enables shape feature sharing across the color spectrum while retaining important color information. We extend the notion of equivariance from geometric to photometric transformations by incorporating parameter sharing over hue-shifts in a neural network. We demonstrate the benefits of CEConvs in terms of downstream performance to various tasks and improved robustness to color changes, including train-test distribution shifts. Our approach can be seamlessly integrated into existing architectures, such as ResNets, and offers a promising solution for addressing color-based domain shifts in CNNs.

Color Equivariant Convolutional Networks

Abstract

Color is a crucial visual cue readily exploited by Convolutional Neural Networks (CNNs) for object recognition. However, CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. Color invariance addresses this issue but does so at the cost of removing all color information, which sacrifices discriminative power. In this paper, we propose Color Equivariant Convolutions (CEConvs), a novel deep learning building block that enables shape feature sharing across the color spectrum while retaining important color information. We extend the notion of equivariance from geometric to photometric transformations by incorporating parameter sharing over hue-shifts in a neural network. We demonstrate the benefits of CEConvs in terms of downstream performance to various tasks and improved robustness to color changes, including train-test distribution shifts. Our approach can be seamlessly integrated into existing architectures, such as ResNets, and offers a promising solution for addressing color-based domain shifts in CNNs.

1 Introduction

Color is a powerful cue for visual object recognition. Trichromatic color vision in primates may have developed to aid the detection of ripe fruits against a background of green foliage [38, 45]. The benefit of color vision here is two-fold: not only does color information improve foregroundbackground segmentation by rendering foreground objects more salient, color also allows diagnostics, e.g. identifying the type (orange) and ripeness (green) where color is an intrinsic property facilitating recognition [3], as illustrated in Fig. 1a. Convolutional neural networks (CNNs) too exploit color information by learning color selective features that respond differently based on the presence or absence of a particular color in the input [42].

Unwanted color variations, however, can be introduced by accidental scene recording conditions such as illumination changes [29, 48], or by low color-diagnostic objects occurring in a variety of colors, making color no longer a discriminative feature but rather an undesired source of variation in the data. Given a sufficiently large training set that encompasses all possible color variations, a CNN learns to become robust by learning color invariant and equivariant features from the available data [36, 37]. However, due to the long tail of the real world it is almost impossible to collect balanced training data for all scenarios. This naturally leads to color distribution shifts between training and test time, and an imbalance in the training data where less frequently occurring colors are underrepresented. As CNNs often fail to generalize to out-of-distribution test samples, this can have significant impact on many real-world applications, e.g. a model trained mostly on red cars may struggle to recognize the exact same car in blue.

Color invariance addresses this issue through features that are by design invariant to color changes and therefore generalize better under appearance variations [14, 17]. However, color invariance comes at the loss of discriminative power as valuable color information is removed from the model's

Class A

Class B

Grayscale

RGB

Low object

saliency

Intra-class

color variations

Color is a

discriminative

feature

(a)

150

100

50

0

50

100

150

Test-time hue shift

0.0

0.1

0.2

0.3

0.4

0.5

0.6

0.7

Test accuracy

Robustness to test-time hue shifts - Flowers-102

ResNet-18

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18-1 + jitter [ours]

(b)

internal feature representation [18]. We therefore propose to equip models with the less restrictive color equivariance property, where features are explicitly shared across different colors through a hue transformation on the learned filters. This allows the model to generalize across different colors, while at the same time also retaining important color information in the feature representation.

An RGB pixel can be decomposed into an orthogonal representation by the well-known hue-saturationvalue (HSV) model, where hue represents the chromaticity of a color. In this work we extend the notion of equivariance from geometric to photometric transformations by hard-wiring parameter sharing over hue-shifts in a neural network. More specifically, we build upon the seminal work of Group Equivariant Convolutions [7] (GConvs), which implements equivariance to translations, flips and rotations of multiples of 90 degrees, and formulates equivariance using the mathematical framework of symmetry groups. We introduce Color Equivariant Convolutions (CEConvs) as a novel deep learning building block, which implements equivariance to the H n symmetry group of discrete hue rotations. CEConvs share parameters across hue-transformed filters in the input layer and store color information in hue-equivariant feature maps.

CEConv feature maps contain an additional dimension compared to regular CNNs, and as a result, require larger filters and thus more parameters for the same number of channels. To evaluate equivariant architectures, it is common practice to reduce the width of the network to match the parameter count of the baseline model. However, this approach introduces a trade-off between equivariance and model capacity, where particularly in deeper layers the quadratic increase in parameter count of CEConv layers makes equivariance computationally expensive. We therefore investigate hybrid architectures, where early color invariance is introduced by pooling over the color dimension of the feature maps. Note that early color invariance is maintained throughout the rest of the network, despite the use of regular convolutional layers after the pooling operation. Limiting color equivariant filters to the early layers is in line with the findings that early layers tend to benefit the most from equivariance [5] and learn more color selective filters [37, 42].

We rigorously validate the properties of CEConvs empirically through precisely controlled synthetic experiments, and evaluate the performance of color invariant and equivariant ResNets on various more realistic classification benchmarks. Moreover, we investigate the combined effects of color equivariance and color augmentations. Our experiments show that CEConvs perform on par or better

than regular convolutions, while at the same time significantly improving the robustness to test-time color shifts, and is complementary to color augmentations.

The main contributions of this paper can be summarized as follows:

We show that convolutional neural networks benefit from using color information, and at the same time are not robust to color-based domain shifts.

We introduce Color Equivariant Convolutions (CEConvs), a novel deep learning building block that allows feature sharing between colors and can be readily integrated into existing architectures such as ResNets.

We demonstrate that CEConvs improve robustness to train-test color shifts in the input.

All code and experiments are made publicly available on https://github.com/Attila94/CEConv .

2 Related work

Equivariant architectures Translation equivariance is a key property of convolutional neural networks (CNNs) [23, 28]: shifting the input to a convolution layer results in an equally shifted output feature map. This allows CNNs to share filter parameters over spatial locations, which improves both parameter and data efficiency as the model can generalize to new locations not covered by the training set. A variety of methods have extended equivariance in CNNs to other geometric transformations [44], including the seminal Group Equivariant Convolutions [7] for rotations and flips, and other works concerning rotations [2, 30, 52], scaling [50, 53] and arbitrary Lie groups [32]. Yet to date, equivariance to photometric transformations has remained largely unexplored. Offset equivariant networks [9] constrain the trainable parameters such that an additive bias to the RGB input channels results in an equal bias in the output logits. By applying a log transformation to the input the network becomes equivariant to global illumination changes according to the Von Kries model [13]. In this work we explore an alternative approach to photometric equivariance inspired by the seminal Group Equivariant Convolution [7] framework.

Color in CNNs Recent research has investigated the internal representation of color in Convolutional Neural Networks (CNNs), challenging the traditional view of CNNs as black boxes. For example, [41, 42] introduces the Neuron Feature visualization technique and characterizes neurons in trained CNNs based on their color selectivity, assessing whether a neuron activates in response to the presence of color in the input. The findings indicate that networks learn highly color-selective neurons across all layers, emphasizing the significance of color as a crucial visual cue. Additionally, [43] classifies neurons based on their class selectivity and observes that early layers contain more class-agnostic neurons, while later layers exhibited high class selectivity. A similar study has been performed in [12], further supporting these findings. [36, 37] investigate learned symmetries in an InceptionV1 model trained on ImageNet [10] and discover filters that demonstrated equivariance to rotations, scale, hue shifts, and combinations thereof. These results motivate color equivariance as a prior for CNNs, especially in the first layers. Moreover, in this study, we will employ the metrics introduced by [42] to provide an explanation for several of our own findings.

Color priors in deep learning Color is an important visual discriminator [15, 19, 51]. In classical computer vision, color invariants are used to extract features from an RGB image that are more consistent under illumination changes [14, 17, 18]. Recent studies have explored using color invariants as a preprocessing step to deep neural networks [1, 33] or incorporating them directly into the architecture itself [29], leading to improved robustness against time-of-day domain shifts and other illumination-based variations in the input. Capsule networks [22, 47], which use groups of neurons to represent object properties such as pose and appearance, have shown encouraging results in image colorization tasks [39]. Quaternion networks [16, 54] represent RGB color values using quaternion notation, and employ quaternion convolutional layers resulting in moderate improvements in image classification and inpainting tasks. Building upon these advancements, we contribute to the ongoing research on integrating color priors within deep neural architectures.

3 Color equivariant convolutions

3.1 Group Equivariant Convolutions

A CNN layer Φ is equivariant to a symmetry group G if for all transformations g ∈ G on the input x the resulting feature mapping Φ( x ) transforms similarly, i.e., first doing a transformation and then the mapping is similar to first doing the mapping and then the transformation. Formally, equivariance is defined as

where T g and T ′ g are the transformation operators of group action g on the input and feature space, respectively. Note that T g and T ′ g can be identical, as is the case for translation equivariance where shifting the input results in an equally shifted feature map, but do not necessarily need to be. A special case of equivariance is invariance, where T ′ g is the identity mapping and the input transformation leaves the feature map unchanged:

We use the definition from [7] to denote the i -th output channel of a standard convolutional layer l in terms of the correlation operation ( ⋆ ) between a set of feature maps f and C l +1 filters ψ :

Here f : Z 2 → R C l and ψ i : Z 2 → R C l are functions that map pixel locations x to a C l -dimensional vector. This definition can be extended to groups by replacing the translation x by a group action g :

As the resulting feature map f ⋆ ψ i is a function on G rather than Z 2 , the inputs and filters of all hidden layers should also be defined on G :

Invariance to a subgroup can be achieved by applying a pooling operation over the corresponding cosets. For a more detailed introduction to group equivariant convolutions, please refer to [4, 7].

3.2 Color Equivariance

We define color equivariance as equivariance to hue shifts. The HSV color space encodes hue by an angular scalar value, and a hue shift is performed as a simple additive offset followed by a modulo operator. When projecting the HSV representation into three-dimensional RGB space, the same hue shift becomes a rotation along the [1 , 1 , 1] diagonal vector.

We formulate hue equivariance in the framework of group theory by defining the group H n of multiples of 360 /n -degree rotations about the [1 , 1 , 1] diagonal vector in R 3 space. H n is a subgroup of the SO (3) group of all rotations about the origin of three-dimensional Euclidean space. We can parameterize H in terms of integers k, n as

with n the total number of discrete rotations in the group, k the rotation, a = 1 3 -1 3 cos( 2 kπ n ) and b = √ 1 3 ∗ sin( 2 kπ n ) . The group operation is matrix multiplication which acts on the continuous R 3 space of RGB pixel values. The derivation of H n is provided in Appendix A.

Color Equivariant Convolution (CEConv) Let us define the group G = Z 2 × H n , which is a direct product of the Z 2 group of discrete 2D translations and the H n group of discrete hue shifts. We can then define the Color Equivariant Convolution (CEConv) in the input layer as:

We furthermore introduce the operator L g = L ( t,m ) including translation t and hue shift m acting on input f defined on the plane Z 2 :

Since H n is an orthogonal matrix, the dot product between a hue shifted input H n f and a filter ψ is equal to the dot product between the original input f and the inverse hue shifted filter H -1 n ψ :

Then the equivariance of the CEConv layer can be derived as follows (using C l = 1 for brevity):

Since input f and feature map [ f ⋆ ψ ] are functions on Z 2 and G , respectively, L ( t,k ) and L ′ ( t,k ) represent two equivalent operators acting on their respective groups. For all subsequent hidden layers the input f and filters ψ i are functions on G parameterized by x, k , and the hidden layer for CEConv is defined as:

where n is the number of discrete rotations in the group and % is the modulo operator. In practice, applying a rotation to RGB pixels will cause some pixel values to fall outside of the RGB cube, which will then have to be reprojected within the cube. Due to this discrepancy, Eq. (9) only holds approximately, though in practice this has only limited consequences, as we empirical show in Appendix D.

3.3 Implementation

Tensor operations We implement CEConv similarly to GConv [7]. GConv represents the pose associated with the added spatial rotation group by extending the feature map tensor X with an extra dimension G l to size [ C l , G l , H, W ] , denoting the number of channels, transformations that leave the origin invariant, and height and width of the feature map at layer l , respectively (batch dimension omitted). Similarly, a GConv filter ˜ F with spatial extent k is of size [ C l +1 , G l +1 , C l , G l , k, k ] . The GConv is then defined in terms of tensor multiplication operations as:

where (:) denotes tensor slices. Note that in the implementation, a GConv filter F only contains [ C l +1 , C l , G l , k, k ] unique parameters - the extra G l +1 dimension is made up of transformed copies of F .

As the RGB input to the network is defined on Z 2 , we have G 1 = 1 and ˜ F has size [ C l +1 , G l +1 , 3 , 1 , k, k ] . The transformed copies in G l +1 are computed by applying the rotation matrix from Eq. (6):

In the hidden layers ˜ F contains cyclically permuted copies of F :

Furthermore, to explicitly share the channel-wise spatial kernel over G l [30], filter F is decomposed into a spatial component S and a pointwise component P as follows:

F is precomputed in each forward step prior to the convolution operation in Eq. (12).

Input normalization is performed using a single value for the mean and standard deviations rather than per channel, as is commonly done for standard CNNs. Channel-wise means and standard deviations break the equivariance property of CECNN as a hue shift could no longer be defined as a rotation around the [1 , 1 , 1] diagonal. Experiments have shown that using a single value for all channels instead of channel-wise normalization has no effect on the performance.

Compute efficiency CEConvs create a factor | H n | more feature maps in each layer. Due to the decomposition in Eq. (15), the number of multiply-accumulate (MAC) operations increase by only a factor | H n | 2 k 2 + | H n | , and the number of parameters by a factor | H n | k 2 +1 . See Appendix C.3 for an overview of parameter counts and MAC operations.

4 Experiments

4.1 When is color equivariance useful?

Color equivariant convolutions share shape information across different colors while preserving color information in the group dimension. To demonstrate when this property is useful we perform two controlled toy experiments on variations of the MNIST [11] dataset. We use the Z2CNN architecture from [7], and create a color equivariant version of the network called CECNN by replacing all convolutional layers by CEConvs with three rotations of 120 ◦ . The number of channels in CECNN is scaled such as to keep the number of parameters approximately equal to the Z2CNN. We also create a color invariant CECNN by applying coset max-pooling after the final CEConv layer, and a color invariant Z2CNN by converting the inputs to grayscale. All experiments are performed using the Adam [24] optimizer with a learning rate of 0.001 and the OneCycle learning rate scheduler. No data augmentations are used. We report the average performance over ten runs with different random initializations.

Color imbalance is simulated by long-tailed ColorMNIST , a 30-class classification problem where digits occur in three colors on a gray background, and need to be classified by both number (0-9) and color (red, green, blue). The number of samples per class is drawn from a power law distribution resulting in a long-tailed class imbalance. Sharing shape information across colors is beneficial as a certain digit may occur more frequently in one color than in another. The train set contains a total of 1,514 training samples and the test set is uniformly distributed with 250 samples per class. The training set is visualized in Appendix B.1. We train all four architectures on the dataset for 1000 epochs using the standard cross-entropy loss. The train set distribution and per-class test accuracies for all models are shown in Fig. 2a. With an average accuracy of 91 . 35 ± 0 . 40% the CECNN performs significantly better than the CNN with 71 . 59 ± 0 . 61% . The performance increase is most significant for the classes with a low sample size, indicating that CEConvs are indeed more efficient in sharing shape information across different colors. The color invariant Z2CNN and CECNN networks, with an average accuracy of 24 . 19 ± 0 . 53% and 29 . 43 ± 0 . 46% , respectively, are unable to discriminate between colors. CECNN with coset pooling is better able to discriminate between foreground and background and therefore performs slightly better. We repeated the experiment with a weighted loss and observed no significantly different results. We have also experimented with adding color jitter augmentations, which makes solving the classification problem prohibitive, as color is required. See Appendix B.2 for both detailed results on both experiments.

G3 G7 B1 B7 B2 R7 B4 B9 G1 G9 R2 R8 G6 G0 B3 B0 R9 R3 R1 R5 G4 G5 G2 R6 R4 B6 B5 G8 B8 R0

Class

0.0

0.2

0.4

0.6

0.8

1.0

Test Accuracy

Z2CNN

Z2CNN (grayscale)

CECNN

CECNN (coset pool)

Class frequency

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Class frequency

ColorMNIST - longtailed

(a)

0

12

24

36

48

60

1e6

Dataset Standard Deviation

0.94

0.95

0.96

0.97

0.98

0.99

1.00

Test Accuracy

ColorMNIST - biased

Z2CNN

CECNN

CECNN (coset pool)

(b)

Color variations are simulated by biased ColorMNIST , a 10-class classification problem where each class c has its own characteristic hue θ c defined in degrees, distributed uniformly on the hue circle. The exact color of each digit x is sampled according to θ x ∼ N ( θ c , σ ) . We generate multiple datasets by varying σ between 0 and 10 6 , where σ = 0 results in a completely deterministic color for each class and σ = 10 6 in an approximately uniform distribution for θ x . For small σ , color is thus highly informative of the class, whereas for large σ the classification needs to be performed based on shape. The dataset is visualized in Appendix B.1. We train all models on the train set of 1.000 samples for 1500 epochs and evaluate on the test set of 10.000 samples. The test accuracies for different σ are shown in Fig. 2b. CECNN outperforms Z2CNN across all standard deviations, indicating CEConvs allow for a more efficient internal color representation. The color invariant CECNN network outperforms the equivariant CECNN model from σ ≥ 48 . Above this value color is no longer informative for the classification task and merely acts as noise unnecessarily consuming model capacity, which is effectively filtered out by the color invariant networks. The results of the grayscale Z2CNN are omitted as they are significantly worse, ranging between 89 . 89% ( σ = 0 ) and 79 . 94 ( σ = 10 6 ). Interestingly, CECNN with coset pooling outperforms the grayscale Z2CNN. This is due to the fact that a CECNN with coset pooling is still able to distinguish between small color changes and therefore can partially exploit color information. Networks trained with color jitter are unable to exploit color information for low σ ; see Appendix B.2 for detailed results.

4.2 Image classification

Setup We evaluate our method for robustness to color variations on several natural image classification datasets, including CIFAR-10 and CIFAR-100 [27], Flowers-102 [35], STL-10 [6], Oxford-IIIT Pet [40], Caltech-101 [31], Stanford Cars [26] and ImageNet [10]. We train a baseline and color equivariant (CE-)ResNet [20] with 3 rotations and evaluate on a range of test sets where we gradually apply a hue shift between -180 ◦ and 180 ◦ . For high-resolution datasets (all except CIFAR) we train a ResNet-18 architecture and use default ImageNet data augmentations: we scale to 256 pixels, random crop to 224 pixels and apply random horizontal flips. For the CIFAR datasets we use the ResNet-44 architecture and augmentations from [7], including random horizontal flips and translations of up to 4 pixels. We train models both with and without color jitter augmentation to separately evaluate the effect of equivariance and augmentation. The CE-ResNets are downscaled in width to match the parameter count of the baseline ResNets. We have also included AugMix [21] and CIConv [29] as baselines for comparison. Training is performed for 200 epochs using the Adam [25] optimizer with a learning rate of 0.001 and the OneCycle learning rate scheduler. All our experiments use PyTorch and run on a single NVIDIA A40 GPU.

Hybrid networks In our toy experiments we enforce color equivariance throughout the network. For real world datasets however, we anticipate that the later layers of a CNN may not benefit from enforcing parameter sharing between colors, if the classes of the dataset are determined by color

specific features. We therefore evaluate hybrid versions of our color equivariance networks, denoted by an integer suffix for the number of ResNet stages, out of a possible four, that use CEConvs.

Results We report both the performance on the original test set, as well as the average accuracy over all hue shifts in Table 1. For brevity we only show the fully equivariant and hybrid-2 networks, a complete overview of the performances of all hybrid network configurations and error standard deviations can be found in Appendix C.1. Between the full color equivariant and hybrid versions of our CE-ResNets, at least one variant outperforms vanilla ResNets on most datasets on the original test set. On most datasets the one- or two-stage hybrid versions are the optimal CE-ResNets, providing a good trade-off between color equivariance and leaving the network free to learn color specific features in later layers. CE-ResNets are also significantly more robust to test-time hue shifts, especially when trained without color jitter augmentation. Training the CE-ResNets with color jitter further improves robustness, indicating that train-time augmentations complement the already hard-coded inductive biases in the network. We show the detailed performance on Flowers-102 for all test-time hue shifts in Fig. 1b. The accuracy of the vanilla CNN quickly drops as a hue shift is applied, whereas the CE-CNN performance peaks at -120 ◦ , 0 ◦ and 120 ◦ . Applying train-time color jitter improves the CNN's robustness to the level of a CNN with grayscale inputs. The CE-CNN with color jitter outperforms all models for all hue shifts. Plots for other datasets are provided in Appendix C.2.

Color selectivity To explore what affects the success of color equivariance, we investigate the color selectivity of a subset of the studied datasets. We use the color selectivity measure from [42] and average across all neurons in the baseline model trained on each dataset. Fig. 3 shows that color selective datasets benefit from using color equivariance up to late stages, whereas less color selective datasets do not.

Feature representations of color equivariant CNNs We use the Neuron Feature [42] (NF) visualization method to investigate the internal feature representation of the CE-ResNet. NF computes a weighted average of the N highest activation input patches for each filter at a certain layer, as such representing the input patch that a specific neuron fires on. Fig. 4 shows the NF ( N = 50 ) and top-3 input patches for filters at the final layers of stages 1-4 of a CE-ResNet18 trained on Flowers-102.

0

1

2

3

4

6

4

2

0

2

4

6

Accuracy improvement

(equivariant - vanilla)

w/o color-jitter augmentation

0

1

2

3

4

6

4

2

0

2

4

6

Accuracy improvement

(equivariant - vanilla)

w/ color-jitter augmentation

Flowers102 (color sel. 0.70)

Caltech101 (color sel. 0.43)

STL10 (color sel. 0.38)

StanfordCars (color sel. 0.38)

Stage 1

Stage2

Stage3

Stage4

NF

8.168

7.255

NF

11.530

11.048

10.542

NF

8.022

7.263

7.234

NF

17,678

16,816

16.762

NF

8,070

7,706

7,436

NF

12,740

12,530

12,265

NF

7,845

7,768

NF

14,786

13.455

10,597

NF

9.549

9.101

9.092

NF

10.398

10.048

9.522

NF

8.387

7.283

NF

24.518

21.305

21.163

Different rows represent different rotations of the same filter. As expected, each row of a NF activates on the same shape in a different color, demonstrating the color sharing capabilities of CEConvs. More detailed NF visualization are provided in Appendix C.4.

Ablation studies We perform ablations to investigate the effect of the number of rotations, the use of group coset pooling, and the strength of train-time color jitter augmentations. In short, we find that a) increasing the number of hue rotations increases robustness to test-time hue shifts at the cost of a slight reduction in network capacity, b) removing group coset pooling breaks hue invariance, and c) hue equivariant networks require lower intensity color jitter augmentations to achieve the same test-time hue shift robustness and accuracy. The full results can be found in Appendix D.

5 Conclusion

In this work, we propose Color Equivariant Convolutions (CEConvs) which enable feature sharing across colors in the data, while retaining discriminative power. Our toy experiments demonstrate benefits for datasets where the color distribution is long-tailed or biased. Our proposed fully equivariant CECNNs improve performance on datasets where features are color selective, while hybrid versions that selectively apply CEConvs only in early stages of a CNN benefit various classification tasks.

Limitations CEConvs are computationally more expensive than regular convolutions. For fair comparison, we have equalized the parameter cost of all models compared, at the cost of reducing the number of channels of CECNNs. In cases where color equivariance is not a useful prior, the reduced capacity hurts model performance, as reflected in our experimental results.

Pixel values near the borders of the RGB cube can fall outside the cube after rotation, and subsequently need to be reprojected. Due to this clipping effect the hue equivariance in Eq. (9) only holds approximately. As demonstrated empirically, this has only limited practical consequences, yet future work should investigate how this shortcoming could be mitigated.

Local vs. global equivariance The proposed CEConv implements local hue equivariance, i.e. it allows to model local color changes in different regions of an image separately. In contrast, global equivariance, e.g. by performing hue shifts on the full input image, then processing all inputs with the same CNN and combining representations at the final layer to get a hue-equivariant representation, encodes global equivariance to the entire image. While we have also considered such setup, initial experiments did not yield promising results. The theoretical benefit of local over global hue equivariance is that multiple objects in one image can be recognized equivariantly in any combination of hues - empirically this indeed proves to be a useful property.

Future work The group of hue shifts is but one of many possible transformations groups on images. CNNs naturally learn features that vary in both photometric and geometric transformations [5, 37]. Future work could combine hue shifts with geometric transformations such as roto-translation [7] and scaling [49]. Also, other photometric properties could be explored in an equivariance setting, such as saturation and brightness.

Our proposed method rotates the hue of the inputs by a predetermined angle as encoded in a rotation matrix. Making this rotation matrix learnable could yield an inexact but more flexible type of color equivariance, in line with recent works on learnable equivariance [34, 46]. An additional line of interesting future work is to incorporate more fine-grained equivariance to continuous hue shifts, which is currently intractable within the GConv-inspired framework as the number multiplyaccumulate operations grow quadratically with the number of hue rotations.

Broader impact Improving performance on tasks where color is a discriminative feature could affect humans that are the target of discrimination based on the color of their skin. CEConvs ideally benefit datasets with long-tailed color distributions by increasing robustness to color changes, in theory reducing a CNN's reliance on skin tone as a discriminating factor. However, careful and rigorous evaluation is needed before such properties can be attributed to CECNNs with certainty.

References

Naif Alshammari, Samet Akcay, and Toby P. Breckon. On the impact of illumination-invariant image pre-transformation for contemporary automotive semantic scene understanding. In 2018 IEEE Intelligent Vehicles Symposium (IV) , pages 1027-1032, 2018.

Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen A. J. Eppenhof, Josien P. W. Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In Medical Image Computing and Computer Assisted Intervention - MICCAI 2018 , pages 440-448. Springer International Publishing, 2018.

Inês Bramão, Luís Faísca, Karl Magnus Petersson, and Alexandra Reis. The contribution of color to object recognition. In Ioannis Kypraios, editor, Advances in Object Recognition Systems , chapter 4. IntechOpen, Rijeka, 2012.

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR , abs/2104.13478, 2021.

Robert-Jan Bruintjes, Tomasz Motyka, and Jan van Gemert. What affects learned equivariance in deep image recognition models? arXiv preprint arXiv:2304.02628 , 2023.

Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics , pages 215-223. JMLR Workshop and Conference Proceedings, 2011.

Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48 , ICML'16, page 2990-2999. JMLR.org, 2016.

Ian R. Cole. Modelling CPV. 6 2015.

Marco Cotogni and Claudio Cusano. Offset equivariant networks and their applications. Neurocomputing , 502:110-119, 2022.

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009.

Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine , 29(6):141-142, 2012.

Martin Engilberge, Edo Collins, and Sabine Süsstrunk. Color representation in deep neural networks. In 2017 IEEE International Conference on Image Processing (ICIP) , pages 2786-2790, 2017.

G.D. Finlayson, M.S. Drew, and B.V. Funt. Diagonal transforms suffice for color constancy. In 1993 (4th) International Conference on Computer Vision , pages 164-171, 1993.

[14]

G.D. Finlayson, S.D. Hordley, Cheng Lu, and M.S. Drew. On the removal of shadows from images.

Transactions on Pattern Analysis and Machine Intelligence

, 28(1):59-68, 2006.

B.V. Funt and G.D. Finlayson. Color constant color indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence , 17(5):522-529, 1995.

Chase J. Gaudet and A. Maida. Deep quaternion networks. 2018 International Joint Conference on Neural Networks (IJCNN) , pages 1-8, 2018.

J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, and H. Geerts. Color invariance. IEEE Transactions on Pattern Analysis and Machine Intelligence , 23(12):1338-1350, 2001.

T. Gevers, A. Gijsenij, J. van de Weijer, and J. M. Geusebroek. Color in Computer Vision : Fundamentals and Applications . Series in Imaging Science and Technology. The Wiley-IS&T, 2012.

T. Gevers, A. Gijsenij, J. van de Weijer, and J. M. Geusebroek. Color in Computer Vision : Fundamentals and Applications . Series in Imaging Science and Technology. The Wiley-IS&T, 2012.

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778. IEEE, June 2016.

Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR) , 2020.

Geoffrey E Hinton, Sara Sabour, and Nicolas Frosst. Matrix capsules with EM routing. In International Conference on Learning Representations , 2018.

Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 14274-14285, 2020.

Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference on Learning Representations , 12 2014.

Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR , abs/1412.6980, 2014.

Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops , pages 554-561, 2013.

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.

Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE , volume 86, pages 2278-2324, 1998.

Attila Lengyel, Sourav Garg, Michael Milford, and Jan C. van Gemert. Zero-shot day-night domain adaptation with a physics prior. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , pages 4399-4409, October 2021.

Attila Lengyel and Jan van Gemert. Exploiting learned symmetries in group equivariant convolutions. In 2021 IEEE International Conference on Image Processing (ICIP) , pages 759-763, 2021.

Fei-Fei Li, Marco Andreeto, Marc'Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.

Lachlan E. MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 8183-8192, June 2022.

Bruce A. Maxwell, Casey A. Smith, Maan Qraitem, Ross Messing, Spencer Whitt, Nicolas Thien, and Richard M. Friedhoff. Real-time physics-based removal of shadows and shading from road surfaces. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW) , pages 1277-1285, 2019.

Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arnold Smeulders. Liegg: Studying learned lie group generators. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 25212-25223. Curran Associates, Inc., 2022.

Maria-Elena Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indian Conference on Computer Vision, Graphics and Image Processing , Dec 2008.

Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionv1. Distill , 2020. https://distill.pub/2020/circuits/early-vision.

Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally occurring equivariance in neural networks. Distill , 2020. https://distill.pub/2020/circuits/equivariance.

Daniel Osorio and Misha Vorobyev. Colour vision as an adaptation to frugivory in primates. Proceedings. Biological sciences / The Royal Society , 263:593-9, 06 1996.

Gokhan Ozbulak. Image colorization by capsule networks. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops , June 2019.

Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition , 2012.

IEEE

Ivet Rafegas and Maria Vanrell. Color representation in cnns: Parallelisms with biological vision. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW) , pages 2697-2705, 2017.

Ivet Rafegas and Maria Vanrell. Color encoding in biologically-inspired convolutional neural networks. Vision Research , 151:7-17, 2018. Color: cone opponency and beyond.

Ivet Rafegas, Maria Vanrell, Luís A. Alexandre, and Guillem Arias. Understanding trained cnns by indexing neuron selectivity. Pattern Recognition Letters , 136:318-325, 2020.

M. Rath and A. Condurache. Boosting deep neural networks with geometrical prior knowledge: A survey. ArXiv , abs/2006.16867, 2020.

B Regan, C Julliot, Bruno Simmen, Françoise Viénot, P.C. Charles-Dominique, and John Mollon. Fruits, foliage and the evolution of primate colour vision. Philosophical transactions of the Royal Society of London. Series B, Biological sciences , 356:229-83, 03 2001.

David W. Romero and Suhas Lohit. Learning partial equivariances from data. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 36466-36478. Curran Associates, Inc., 2022.

Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems , volume 30. Curran Associates, Inc., 2017.

Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertaintyaware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) , October 2019.

Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. Disco: accurate discrete scale convolutions. In Proceedings of the 32nd British Machine Vision Conference (BMVC) , 2021.

Ivan Sosnovik, Michał Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. In International Conference on Learning Representations , 2020.

Jan C Van Gemert. Exploiting photographic style for category-level image classification by generalizing the spatial pyramid. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval , pages 1-8, 2011.

Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , June 2018.

Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In Advances in Neural Information Processing Systems , volume 32, pages 7366-7378. Curran Associates, Inc., 2019.

Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In Proceedings of the European Conference on Computer Vision (ECCV) , September 2018.

A Derivation of H n

Rotation around an arbitrary unit vector u by angle θ can be decomposed into five simple steps [8]:

rotating the vector such that it lies in one of the coordinate planes, e.g. xz using M xz ;

rotating the vector such that it lies on one of the coordinate axes, e.g. x using M x ;

rotating the point around vector u on axis x using R x ;

reversing the rotation in step 2. using M -1 x = M T x ;

reversing the rotation in step 1. using M -1 xz = M T xz .

These operations can be combined into a single matrix:

Substituting u = [ 1 √ 3 , 1 √ 3 , 1 √ 3 ] yields

and lastly, rearranging and substituting θ = 2 kπ n results in

with n the total number of discrete rotations in the group, k the rotation, a = 1 3 -1 3 cos( 2 kπ n ) and b = √ 1 3 ∗ sin( 2 kπ n ) .

B ColorMNIST

B.1 Dataset visualization

Long-tailed ColorMNIST dataset The training samples of the Longtailed ColorMNIST dataset are depicted in Fig. 5, clearly indicating a class imbalance.

Biased ColorMNIST dataset A small subset of the samples of Biased ColorMNIST is shown in Fig. 6 for σ = 0 (a) and σ = 36 (b), respectively. Note that the samples in (a) have a deterministic color, whereas in (b) exhibit some variation in hue.

(a)

(b)

0

一

3

0

一

92

13

50

一

3

3

1

2

50 -

一

3

3

6

7

2

8

6

H

9

8

6

H

9

100

工

2

3

3

d

100

2

3

2

7

3

d

G

28

0

5

6

9

0

5

6

十

150

1

7

3

9

8

5

150

8

1

9

3

9

8

200

3

3

7

8

200

3

3

7

#

8

?

9

4

6

9

4

6

0

50

100

150

200

0

50

100

150

200

B.2 Additional experiments

Results with color jitter augmentation We performed both ColorMNIST experiments with color jitter augmentations. The results are shown in Fig. 7. (a) For long-tailed ColorMNIST, adding jitter makes solving the classification problem prohibitive, as color is required. Z2CNN and CECNN with jitter therefore perform no better than as the CECNN model with coset pooling. (b) For biased MNIST, performance decreases for small and improves for large σ , with CEConv still performing best.

G3 G7 B1 B7 B2 R7 B4 B9 G1 G9 R2 R8 G6 G0 B3 B0 R9 R3 R1 R5 G4 G5 G2 R6 R4 B6 B5 G8 B8 R0

Class

0.0

0.2

0.4

0.6

0.8

1.0

Test Accuracy

Z2CNN

Z2CNN - jitter

Class frequency

CECNN

CECNN - jitter

CECNN (coset pool)

CECNN (coset pool) - jitter

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Class frequency

ColorMNIST - longtailed

(a)

0

12

24

36

48

60

1e6

Dataset Standard Deviation

0.94

0.95

0.96

0.97

0.98

0.99

1.00

Test Accuracy

ColorMNIST - biased

Z2CNN

Z2CNN - jitter

CECNN

CECNN - jitter

CECNN (coset pool)

CECNN (coset pool) - jitter

(b)

Long-tailed ColorMNIST with weighted loss We performed the longtailed ColorMNIST experiment both with a uniformly weighted loss and a loss where classes are weighted inversely to their frequency according to w i = N c ∗ n i , where w i denotes the weight for class i , N the number of samples in the training set, c the number of classes, and n i the number of samples for class i . The results are shown in Fig. 8. We observed no significant difference between the two setups, with the CECNN without coset pooling outperforming the other models by a large margin in both.

G3 G7 B1 B7 B2 R7 B4 B9 G1 G9 R2 R8 G6 G0 B3 B0 R9 R3 R1 R5 G4 G5 G2 R6 R4 B6 B5 G8 B8 R0

Class

0.0

0.2

0.4

0.6

0.8

1.0

Test Accuracy

Z2CNN

Z2CNN (grayscale)

CECNN

CECNN (coset pool)

Class frequency

0.00

0.01

0.02

0.03

0.04

0.05

0.06

0.07

0.08

Class frequency

ColorMNIST - longtailed

C Classification experiments

C.1 Overview of all CE-ResNet configurations

Table 2 shows an overview of the classification accuracies of all baselines and equivariant architectures. CEConv-x denotes the number of ResNet stages with CE convolutions with CEConv-4 (3 for CIFAR) being a fully equivariant ResNet. In nearly all cases, early equivariance is beneficial for improving classification accuracy on both the original as well as the hue shifted test sets. In case of the Flowers-

102 dataset late equivariance seems to have a significant advantage, whereas for Caltech-101 and Stanford Cars the color equivariance bias does not seem to have much added value.

C.2 Test-time hue shift plots

Fig. 9 shows the test accuracies under a test time hue shift on all datasets in the paper. Each figure includes a regular ResNet, a color equivariant ResNet-x (CE-ResNet-x) and a ResNet-x with color equivariant convolutions in the first ResNet stage (CE-ResNet-x-1), trained with and without color jitter augmentation. Finally, the plot shows the accuracy of a ResNet-x trained on grayscale inputs. CEConv improves robustness to test-time hue shifts on all datasets.

150

100

50

0

50

100

150

Test-time hue shift (°)

80

82

84

86

88

90

92

94

Test accuracy (%)

CIFAR-10

ResNet-44

CE-ResNet-44 [ours]

CE-ResNet-44-1 [ours]

ResNet-44 - grayscale

ResNet-44 + jitter

CE-ResNet-44 + jitter [ours]

CE-ResNet-44-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

35

40

45

50

55

60

65

70

Test accuracy (%)

CIFAR-100

ResNet-44

CE-ResNet-44 [ours]

CE-ResNet-44-1 [ours]

ResNet-44 - grayscale

ResNet-44 + jitter

CE-ResNet-44 + jitter [ours]

CE-ResNet-44-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

0

10

20

30

40

50

60

70

Test accuracy (%)

Flowers-102

ResNet-18

CE-ResNet-18 [ours]

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18 + jitter [ours]

CE-ResNet-18-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

30

40

50

60

70

Test accuracy (%)

Oxford-IIIT-Pet

ResNet-18

CE-ResNet-18 [ours]

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18 + jitter [ours]

CE-ResNet-18-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

40

45

50

55

60

65

70

75

Test accuracy (%)

Caltech-101

ResNet-18

CE-ResNet-18 [ours]

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18 + jitter [ours]

CE-ResNet-18-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

45

50

55

60

65

70

75

80

Test accuracy (%)

Stanford Cars

ResNet-18

CE-ResNet-18 [ours]

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18 + jitter [ours]

CE-ResNet-18-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

60

65

70

75

80

85

Test accuracy (%)

STL-10

ResNet-18

CE-ResNet-18 [ours]

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18 + jitter [ours]

CE-ResNet-18-1 + jitter [ours]

150

100

50

0

50

100

150

Test-time hue shift (°)

50

55

60

65

70

Test accuracy (%)

ImageNet

ResNet-18

CE-ResNet-18 [ours]

CE-ResNet-18-1 [ours]

ResNet-18 - grayscale

ResNet-18 + jitter

CE-ResNet-18 + jitter [ours]

CE-ResNet-18-1 + jitter [ours]

C.3 CE-ResNet configurations

The configurations of the color equivariant ResNet with three hue rotations, as used in the classification experiment in Section 4.2, are shown in Table 3. CE stages 0 denotes a regular ResNet.

C.4 Neuron Feature visualizations

Fig. 10 shows the Neuron Feature [42] (NF) visualization with top-3 patches of two neurons at different stages in a CE-ResNet18 trained on Stanford Cars. As expected, each row of a NF activates on the same shape in a different color. We show neurons that are insensitive to color (top row) and neurons that are sensitive to color (bottom row).

Stage 1

Stage 2

Stage 3

Stage 4

20.455

20.409

20.279

20.582

20.517

NF

20.789

20.469

NF

20.554

20.446

NF

21.909

20.296

14.985

22.294

22.027

NF

21.391

21.127

NF

16.636

15.634

NF

18.125

17.695

17.495

NF

18.128

17.715

17.526

NF

18.136

17.616

17.485

NF

9.972

9.878

9.846

NF

8.535

8.445

8.157

NF

8.751

8.656

8.183

NF

9.425

8.289

7.960

NF

9.613

8.396

7.962

NF

8.790

8.196

7.965

NF

11.058

9.627

9.479

NF

12.976

12.580

12.576

NF

9.986

9.948

9.784

NF

25.032

24.372

23.664

NF

25.066

24.076

22.677

NF

23.940

23.344

23.326

NF

26.329

24.012

23.750

NF

30.466

30.158

29.863

NF

30.343

21.615

20.688

NF

1630

D Ablation studies

Strength of color jitter augmentations Fig. 11 shows the effect of hue jitter augmentation during training on both a color equivariant ResNet-18 with 3 rotations (a) and a regular ResNet-18 (b) trained on Flowers-102. All runs have been repeated 3 times and the mean performance is reported. As expected, the color equivariant network (a) without jitter augmentation is equivariant to rotations of multiples of 120 degrees, but performance quickly degrades. Applying slight (0.1) hue jitter during training both helps in an absolute sense, increasing performance over all rotations, and makes the network more robust to hue changes as shown by the increasing width of the peaks. Further increasing the strength of the augmentation results in a uniform performance over all hue shifts, indicated by the flat lines. There appears to be no significant difference for jitter strength > 0 . 2 . In comparison, the

regular ResNet (b) trained without hue augmentation shows a single peak around 0 degrees, which increases in width when applying more severe augmentation. Note that the increase in absolute performance is smaller compared to the color equivariant network. The reason for this is that the equivariant architecture only requires augmentation "between" the discrete rotations to which it is already robust, as opposed to the full scale of hue shifts for the baseline architecture. Augmentation and equivariance thus exhibit a remarkable synergistic interaction.

150

100

50

0

50

100

150

Test-time hue shift (°)

50

55

60

65

70

75

80

Test accuracy (%)

Effect of jitter augmentation - Flowers-102

CE-ResNet-18

CE-ResNet-18 - j=0.1

CE-ResNet-18 - j=0.2

CE-ResNet-18 - j=0.3

CE-ResNet-18 - j=0.4

CE-ResNet-18 - j=0.5

(a)

150

100

50

0

50

100

150

Test-time hue shift (°)

50

55

60

65

70

75

80

Test accuracy (%)

Effect of jitter augmentation - Flowers-102

ResNet-18

ResNet-18 - j=0.1

ResNet-18 - j=0.2

ResNet-18 - j=0.3

ResNet-18 - j=0.4

ResNet-18 - j=0.5

(b)

Group coset pooling We have removed the group coset pooling operation by flattening the feature map group dimension into the channel dimension in the penultimate layer, before applying the final classification layer. As shown in Fig. 12, the model without pooling layer is no longer invariant to hue shifts and behaves identically to the baseline model.

150

100

50

0

50

100

150

Test-time hue shift (°)

0

10

20

30

40

50

60

70

Test accuracy (%)

Effect of group coset pooling - Flowers-102

CE-ResNet

CE-ResNet w/o pooling

Regular

Number of color rotations We investigate the effect of the number of hue rotations in color equivariant convolutions by training CE-ResNets with 2-10 rotations on Flowers-102. Fig. 13 shows the test accuracies for rotations 1-5 (a) and 6-10 (b), respectively. Note that, for this particular dataset, more hue rotations not only lead to better robustness to test-time hue shifts, but also to better absolute performance. However, there is a trade-off between number of rotations and model capacity, as increasing the number of rotations increases the number of parameters in the model, and the model width needs to be scaled down to keep the number of parameters equal. Both the optimal number of color rotations and network width therefore depend on the amount of color vs. the complexity of the data, and therefore both need to be carefully calibrated per dataset.

As expected, the number of peaks increases with the number of hue rotations, though interestingly, the peaks do vary in height. This is an artifact due the way test-time hue shifts are applied to the input images. When RGB pixels are rotated about the [1,1,1] diagonal, values near the borders of the RGB cube tend to fall outside the cube and subsequently need to be reprojected. This reprojection is not modeled by the filter transformations in the CEConv layers, and subsequently

causes a discrepancy between the filter and the image transformations. Indeed, when the test-time hue shift is instead implemented through a rotation in RGB space without reprojecting into the cube, this artifact disappears and all peaks are of equal height, as shown in Fig. 13 (c-d). Note that rotations of multiples of 120 degrees always end up within the RGB cube, which is why this artifact does never occur at -120, 0 and 120 degrees. Future work should further investigate the extent to which this discrepancy is problematic in practice, and look into alternative solutions.

150

100

50

0

50

100

150

Test-time hue shift (°)

0

10

20

30

40

50

60

70

Test accuracy (%)

Effect of hue rotations - Flowers-102

CE-1

CE-2

CE-3

CE-4

CE-5

(a) Test-time hue shift with reprojection.

150

100

50

0

50

100

150

Test-time hue shift (°)

0

10

20

30

40

50

60

70

Test accuracy (%)

Effect of hue rotations - Flowers-102

CE-7

CE-8

CE-9

CE-10

(b) Test-time hue shift with reprojection.

150

100

50

0

50

100

150

Test-time hue shift (°)

0

10

20

30

40

50

60

70

Test accuracy (%)

Effect of hue rotations - Flowers-102

CE-1

CE-2

CE-3

CE-4

CE-5

150

100

50

0

50

100

150

Test-time hue shift (°)

0

10

20

30

40

50

60

70

Test accuracy (%)

Effect of hue rotations - Flowers-102

CE-7

CE-8

CE-9

CE-10

(c) Test-time hue shift without reprojection.

(d) Test-time hue shift without reprojection.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper questions the importance of color variations for neural classifiers and proposes to use color-equivariant architectures in the case of unbalanced datasets.\nTo demonstrate the validity of the presented approach, the authors conduct experiments both on synthetic controlled datasets and on common object recognition benchmarks.\nAs the experiments show, the injection of color-equivariant layers leads to a slight improvement on almost all common benchmarks when the performance is measured on the original test set but the advantage of the presented method becomes more evident when the test data is corrupted with hue shifts.\n\nSTRENGTHS:\nThis paper studies an interesting and underinvestigated question of the importance of color representation for neural networks.\nThe submission is easy to read, and the motivation is well explained in the example of the Flowers dataset. \nThe authors have conducted a significant number of experiments to support their claims.\nAdditional strength is that the demonstrated performance improvement is achieved without increasing the number of trainable parameters (line 238).\n\nWEAKNESSES:\n1. While the idea of extending equivariance from geometric to photometric transformations is definitely interesting, the submitted manuscript, unfortunately, focuses on the only type of such transformations, i.e. hue shifts. Despite the case of the Flowers dataset is a perfect fit for this transformation, the authors do not discuss other use cases when this type of equivariance may be interesting in practice and just mention \"accidental recording conditions\" (line 3). For other datasets, hue shifts seem less meaningful, and the better robustness of the proposed CE-ResNets to such shifts at test time is explained by the fact the architecture was just intentionally designed for this scenario. Taking this into account, I find the scope of the paper a bit limited.\n\n1. In addition to being limited in the number of considered photometric transformations, the paper also considers a single task of object recognition. I would encourage the authors to consider other tasks as well, e.g. unsupervised domain adaptation.\n\n1. While the authors claim their approach makes networks more robust to test time corruptions (Tab. 1), they do not demonstrate other baselines aiming to provide robust outputs, e.g. adversarially robust models.\n\nQUESTIONS:\n1. I ask the authors to discuss the weaknesses stated above. The main thing I am interested in is the usage of robust baseline models.\n\n1. While preserving the number of trainable parameters is valuable, I wonder if the network throughput remains the same for color-equivariant architectures in comparison with the baselines. If this is not the case, how significant is the decrease?\n\nLIMITATIONS:\nLimitations are addressed adequately.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors introduce a color equivariant convolutional neural network. To achieve this the authors represent the image in HSV format, and achieve hue equivariance using methods for rotational equivariance. This is possible since hue can be represented by an angle. The authors show that the proposed approach out performs standard CNNs and color invariant CNNs when there is a hue shift between the train and test set.\n\nSTRENGTHS:\n* Originality: The presented method of building a color equivariant CNN appears to be original. \n* Quality: The work appears to be of fairly good quality. \n* Clarity: The paper is well written. \n* Significance: The observation that color equivariance can be achieved by identifying hue with the rotation group is interesting. The results show the proposed approach leads to improved performance when there is a color based domain shift.\n\nWEAKNESSES:\n* Quality: I have some questions about the mathematical presentation, and experiment design (see questions).\n* Clarity: Some aspects were unclear to me, due to presentation or motivation (see questions)\n\nQUESTIONS:\n* regarding equation 3. I believe the correlation should be between the feature maps and $C^{l+1}$ filters [7]. I think it could be clarifying to write that $c$ is the input channel and $i$ is output channel.\n* regarding equation 6: I'm not sure I understand how does H_n(k) acts on \\psi_c? In the case of group 2D rotation equivariance, the filters are transformed by in plane rotations. It is not clear to me how one can perform a 3D rotation of a filter with an arbitrary number of channels.\n* It seems to me that the network is designed for local hue equivariance, can the authors clarify the benefit of this over global hue equivariance (i.e, performing hue shift on the input image then processing all inputs with the same cnn, and combining representations at the final layer to get a hue-equivariant representation)?\n* Does computational expense improve if input images are converted to HSV before being passed into the network? In this case, I expect hue equivariance could be achieved by discretization of the 2D rotation group rather than a 3D transformation.\n* Have the authors experimented with finer/coarser discretizations of the hue/rotation group\n\nLIMITATIONS:\n* Limitations -- in particular the issue of computational cost -- are communicated", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPaper proposes color-equivariant CNN layers by imposing equivariance to H_n (a discrete subgroup of SO(3)) in the RGB space which is imposes hue equivariance. Implementation follows the framework of Group-equivariant CNNs. Experiments show marginal improvements over standard CNNs for in-distribution test data but significant improvements when test data is hue-shifted.\n\nSTRENGTHS:\n1. Color equivariance in CNNs is a relatively less-studied but an important topic for robustness. The proposed idea of incorporating equivariance to hue transformations via rotations in the RGB space is novel. \n2. Experiments are setup well clearly showing when color equivariance is helpful vs color invariance vs no symmetry. Proposed approach shows improves over CNN even when in in-distribution test data, but major improvements come when test data is hue-shifted.\n\nWEAKNESSES:\n1. Definition of color equivariance considered in the paper seems to be restricted as it only considers the hue dimension. One of the motivations for incorporating color equivariance is for robustness to illumination changes which I do not think is guaranteed here. A general definition of color-equivariance should consider other dimensions. Maybe the claims are better justified if Hue-equivariance is emphasized in the title/introduction/method name, etc.\n2. The definition of hue-equivariance is not precise in the paper. Ideally, it should include all rotations in the RGB space (i.e., SO(3)), but also consider the fact that many of these rotations take the color values out of the RGB space (unit cube). In general, this issue occurs for the discrete subgroup $H_n$ as well. Simply projecting the color values back into the RGB space does not work as it breaks the invertibility property of these transformations. \n3. Experiments compare with a standard CNN (+grayscale) as baseline. Other baselines can be included, for example [1], that considers invariance to illumination color/intensity. \n4. Experiments in the main paper only consider the group $H_3$ (i.e., 3 rotations in the RGB space), which seems limited in robustness, as shown in Figure 1 without jitter augmentations. \n\n\nReferences:\n\n[1] Lengyel, Attila, et al. \"Zero-shot day-night domain adaptation with a physics prior.\" _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.\n\nQUESTIONS:\n1. How is the hyperparameter $s$ chosen for different datasets in Table 1, especially for hue-shifted test dataset (assumed not known during training/validation)?\n2. Is the invariance captured by Grayscale-Z2CNN same as that considered in the paper, i.e., invariance to hue? Would a better preprocessing step be only considering a canonical representation in the hue dimension?\n3. In the color-imbalance experiments, were the CNNs trained with loss weighted according to the class imbalance? \n4. Except being computationally expensive, are there other issues with using $H_n$ for higher n? Ablation in the Appendix seem to show better robustness with higher n, and also good in-distribution performance (at 0 hue shift). So I am not sure why $H_3$ alone was considered in the main paper.\n5. It would be helpful to add a few lines of summary on the ablation studies in Sec 4.2.\n6. Can experiments in Section 4.1 include CNN+jitter as it seems to provide competitive performance in Section 4.2?\n\n\n**AFTER REBUTTAL**\n\nAuthors have addressed most of my concerns, particularly, adjusted the claims regarding color vs hue equivariance and added relevant baselines. I am increasing my score as I think this paper can be an important addition to an understudied topic. \n\nHowever, I am still concerned about the mathematical definition of hue-equivariance (via H_n group) since these transformations can map a color outside the unit cube and projecting back loses invertibility requirement of a group. I accept that this may not reduce performance in practical tasks (except certain inconsistencies pointed out by the authors), but I believe it is not mathematically correct.\n\nLIMITATIONS:\nLimitations should discuss lack of robustness to other color dimensions (e.g., illumination).", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-12"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes color equivariant convolutional networks (CE-CNNs), a novel convolutional neural network architecture that achieves equivariance to hue changes.&#x20;\n\nThey introduce color equivariant convolutions that apply a discrete set of hue rotations to the convolution filters during the forward pass. This makes the network output invariant to corresponding hue changes in the input image.\nThey propose a group coset pooling layer that pools feature maps over the group of hue transformations to achieve invariance.\nThey evaluate CE-CNNs on several image classification datasets, showing improved robustness to hue changes at test time compared to regular CNNs. The method also improves absolute classification performance in some cases.\nOverall, the paper presents a novel and intuitive technique to build invariance to hue changes into CNNs. The evaluations demonstrate its advantages over standard networks, especially under shifted hue distributions between training and test.\n\nSTRENGTHS:\nThis paper introduces a clever yet intuitive technique to make convolutional neural networks invariant to hue changes in the input image. The core idea is to apply discrete hue rotations to the convolution filters during the forward pass, essentially \"baking in\" robustness to color changes.\n\nThe paper is clearly written and easy to follow. The authors motivate the problem well, explain their proposed method succinctly, and provide thorough experimentation across image datasets. The visualizations offer useful insights, confirming that the networks learn consistent features across hues.\n\nOverall, I found this to be an original and significant contribution. Invariance to hue shifts is a practical problem, and this paper tackles it through an elegant approach that outperforms regular CNNs. The concept of encoding transformations into convolutions seems powerful. While not the flashiest technique, the method is thoughtful, principled, and achieves strong results. The paper is presented clearly and comprehensively, making the ideas accessible. In summary, this is a high quality paper with both theoretical and practical value.\n\nWEAKNESSES:\n-   The method is demonstrated on image classification, but it's unclear how well it would generalize to other tasks like detection or segmentation. Additional experiments on other applications could strengthen the claims.\n-   The ablation study on number of hue rotations suggests performance varies across different shifts. It would be useful to dig deeper into why - is it an artifact of how shifts are applied? Better understanding this could improve results further.\n-   The approach encodes discrete hue rotations. An interesting extension could be supporting continuous rotations for finer-grained equivariance.\n-   The comparisons to \"grayscale\" networks should be interpreted carefully, as removing color information entirely handicaps models. Comparisons to networks pre-trained onImagenet may be more meaningful.\n-   The Flowers-102 experiments indicate the method doesn't help much on datasets without color bias. Analyzing when color equivariance helps or hurts could guide adoption.\n\nQUESTIONS:\n1.  The amount of color jitter augmentation during training seems to significantly impact performance. Could you do an ablation or analysis to tease apart the direct benefits of the color equivariant convolutions versus the data augmentation?\n2.  Have you evaluated the method on larger-scale datasets like ImageNet? Results on more complex data could better demonstrate the scalability.\n3.  The choice of 3 discrete rotations seems arbitrary. Can you analyze the impact of the group size to help guide selection? Are there benefits beyond 3 rotations?\n\nLIMITATIONS:\n1.  The comparison to only a ResNet baseline limits the conclusions on the benefits of the proposed method. Comparisons to other approaches could provide useful context.\n2.  The long-term impacts of building color equivariance into models is unclear. Discussion of downstream effects on fairness and interpretability could be beneficial.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-25"}