Color Equivariant Convolutional Networks
Authors: Attila Lengyel
Ombretta Strafforello
Robert-Jan Bruintjes
Alexander Gielisse
Jan van Gemert
Venue: NeurIPS 2023 poster (2023)
Keywords: color equivariance
equivariance
color robustness
equivariant convolutions
TL;DR: We propose Color Equivariant Convolutions (CEConv), which enables shape feature sharing across the color spectrum while retaining important color information.

Abstract: Color is a crucial visual cue readily exploited by Convolutional Neural Networks (CNNs) for object recognition. However, CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. Color invariance addresses this issue but does so at the cost of removing all color information, which sacrifices discriminative power. In this paper, we propose Color Equivariant Convolutions (CEConvs), a novel deep learning building block that enables shape feature sharing across the color spectrum while retaining important color information. We extend the notion of equivariance from geometric to photometric transformations by incorporating parameter sharing over hue-shifts in a neural network. We demonstrate the benefits of CEConvs in terms of downstream performance to various tasks and improved robustness to color changes, including train-test distribution shifts. Our approach can be seamlessly integrated into existing architectures, such as ResNets, and offers a promising solution for addressing color-based domain shifts in CNNs.

Color Equivariant Convolutional Networks
Attila Lengyel
Ombretta Strafforello
Robert-Jan Bruintjes
Alexander Gielisse Jan van Gemert
Computer Vision Lab
Delft University of Technology Delft, The Netherlands
Abstract
Color is a crucial visual cue readily exploited by Convolutional Neural Networks (CNNs) for object recognition. However, CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. Color invariance addresses this issue but does the cost of removing all color infor- mation, which sacrifices discriminative power. In this paper, we propose Color Equivariant Convolutions (CEConvs), novel deep learning building block that enables shape feature sharing across the color spectrum while retaining important color information. We extend the notion of equivariance from geometric to pho- tometric transformations by incorporating parameter sharing over hue-shifts in a neural network. We demonstrate performance to various tasks and improved robustness to color changes, including train-test distribution shifts. Our approach can be seamlessly integrated into exist- ing architectures, such as ResNets, and offers a promising solution for addressing color-based domain shifts in CNNs.
benefits of CEConvs in terms of downstream
Introduction
Color is a powerful cue for visual object recognition. Trichromatic color vision in primates may have developed to aid the detection of ripe fruits against a background of green foliage [38] 45). The benefit of color vision here is two-fold: not only does color information improve foreground- background segmentation by rendering foreground objects more salient, color also allows diagnostics, e.g. identifying the type (orange) and ripeness (green) where color an intrinsic property facilitating recognition [3], as illustrated in Fig, Ta Convolutional neural networks (CNNs) too exploit color information by learning color selective features that respond differently based on the presence or absence of particular color in the input [42]
Unwanted color variations, however, be introduced by accidental scene recording conditions such as illumination changes [29] 48), or by low color-diagnostic objects occurring in variety of colors, making color longer discriminative feature but rather undesired source of variation in the data. Given a sufficiently large training set that encompasses all possible color variations, CNN learns to become robust by learning color invariant and equivariant features from the available data [3 37]. However, due to the long tail of the real world it is almost impossible to collect balanced training data for all scenarios. This naturally leads color distribution shifts between training and test time, and an imbalance in the training data where less frequently occurring colors are underrepresented. As CNNs often fail to generalize dutto--distribution test samples, this can have significant impact on many real-world applications, e.g. model trained mostly on red cars may struggle to recognize the exact same car in blue.
Color invariance addresses this issue through features that are by design invariant to color changes and therefore generalize better under appearance variations [4 [7]. However, color invariance comes at the loss of discriminative power as valuable color information is removed from the model's
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

Grayscale
RGB
Robustness to test-time hue shifts Flowers-102
0.7
Class
0.5
Test accurncy
Color
diseriminative feature
Class B
Resheet-12
Rushart-18
0.1
CE CE-Reshitt-Ee sew-ettt----- tours]
CE Reshict:111 8-1 + jittrr [ours]
Restict 18 grayscale
Low object sallency
Intra-class
0.0
color variations
150
-00
50
Testitime tue shift
100
150
(a)
(h)
Figure 1: Color plays a significant role in object recognition. (a) The absence of color makes flowers less distinct from their background and thus harder to classify. The characteristic purple-blue color of the Monkshood (Class A) enables a clear distinction from the Snapdragon (Class B) [35].
On the
other hand, relying too much colors might negatively impact recognition to color variations within same flower class. (b) Image classification performance on the Flower- 102 dataset (35 under a gradual variation of the image hue. Test-time hue shifts degrade the performance of CNNs (ResNet- 18) drastically. Grayscale images and color augmentations result in invariance to hue variations, but fail to capture all the characteristic color features of flowers. Our color equivariant network (CE-ResNet-18-- enables feature sharing across the color spectrum, which helps generalise to underrepresented colors in the dataset, while preserving discriminative color information, improving classification for unbalanced color variations.
internal feature representation [18]. We therefore propose to equip models with the less restrictive color equivariance property, where features are explicitly shared across different colors through a hue transformation on the learned filters. This allows the model to generalize across different colors, while at the same time also retaining important color information in the feature representation.
An RGB pixel can be decomposed into an orthogonal representation by the well-known hue-saturation- value (HSV) model, where hue represents the chromaticity of a color. In this work we extend the notion of equivariance from geometric to photometric transformations by hard-wiring parameter sharing over hue-shifts in a neural network. More specifically, we build upon seminal work of Group Equivariant Convolutions 7 (GConvs), which implements equivariance to translations, flips and rotations of multiples of degrees, and formulates equivariance using the mathematical framework of symmetry groups. We introduce Color Equivariant Convolutions (CEConvs) as novel deep learning building block, which implements cquivariance to the H, symmetry group of discrete hue rotations. CEConvs share parameters across hue- nie-traasformed filters in the input layer and store color information in hue-equivariant feature maps.
CEConv feature maps contain an additional dimension compared to regular CNNs, and as result, require larger filters and thus more parameters for the same number of channels. To evaluate equivariant architectures, it is common practice to reduce the width of the network to match the parameter count of the baseline model. However, this approach introduces a trade-off between equivariance and model capacity, where particularly in deeper layers the quadratic increase in parameter count of CECony layers makes cquivariance computationally expensive. We therefore investigate hybrid architectures, where early color invariance is introduced by pooling over the color dimension the feature maps. Note that early color invariance is maintained throughout the rest of the network, despite the use of regular convolutional layers after the pooling operation. Limiting color equivariant filters to the early layers is in line with the findings that early layers tend to benefit the most from equivariance [5] and learn more color selective filters 37
We rigorously validate the properties of CEConvs empirically through precisely controlled synthetic experiments, and evaluate the performance of color invariant and equivariant ResNets on various more realistic classification benchmarks. Moreover, we investigate the combined effects of color equivariance and color augmentations. Our experiments show that CEConvs perform par or better

---

than regular convolutions, while at the same time significantly improving the robustness to test- test-time color shifts, and is complementary to color augmentations.
The main contributions of this paper can be summarized as follows:
We show that convolutional neural networks benefit from using color information, and at the same time are not robust to color-based domain shifts.
We introduce Color Equivariant Convolutions (CEConvs), novel deep learning building block that allows feature sharing between colors and can be readily integrated into existing architectures such as ResNets.
We demonstrate that CEConvs improve robustness to train-test color shifts in the input.
All code and experiments are made publicly available on htttp://ggiccabb..mm/tttiiieee
Related work
Equivariant architectures Translation equivariance is key property of convolutional neural networks (CNNs) [23 28] shifting the input to a convolution layer results in an equally shifted output feature map. This allows CNNs to share filter parameters over spatial locations, which improves both parameter and data efficiency as the model can generalize to new locations not covered by the training set. variety of methods have extended equivariance in CNNs to other geometric transformations 44 including the seminal Group Equivariant Convolutions 7 for rotations and flips, and other works concerning rotations 2 30 522, scaling [50, 53 and arbitrary Lie groups [32] Yet to date, equivariance to photometric transformations has remained largely unexplored. Offset equivariant networks [|] constrain the trainable parameters such that an additive bias to the RGB input channels results in an equal bias in the output logits. By applying log transformation to the input the network becomes equivariant to global illumination changes according to the Von Kries model [13] In this work explore an alternative approach to photometric equivariance inspired by the seminal Group Equivariant Convolution 7 framework.
Color in CNNs Recent research has investigated the internal representation of color in Convo- lutional Neural Networks (CNNs), challenging the traditional view of CNNs as black boxes. For example, 41 42 introduces the Neuron Feature visualization technique and characterizes neurons in trained CNNs based on their color selectivity, assessing whether neuron activates in response to the presence of color in the input. The findings indicate that networks learn highly color-selective neurons across all layers, emphasizing the significance of color as a crucial visual cue. Additionally, 43 classifies neurons based their class selectivity and observes that early layers contain more class-agnostic neurons, while later layers exhibited high class selectivity. A similar study has been performed in [12], further supporting these findings. [36] 37 investigate learned symmetries in an Inception V1 model trained on ImageNet (10) and discover filters that demonstrated equivariance to rotations, scale, hue shifts, and combinations thereof. These results motivate color equivariance as prior for CNNs, especially in the first layers. Moreover, in this study, we will employ the metrics introduced by [42 provide an explanation for several of our own findings.
Color priors in deep learning Color is an important visual discriminator 15 computer vision, color invariants are used extract features from an RGB image that are more consistent under illumination changes [14 18) Recent studies have explored using color invariants as a preprocessing step to deep neural networks I
19 51
In classical
33 or incorporating them directly into the architecture itself [29]. leading to improved robustness against time-of-day day domain shifts and other illumination- based variations in the input. Capsule networks 22 477, which use groups of neurons to represent object properties such as pose and appearance, have shown encouraging results in image colorization tasks 339. Quaternion networks [16] 54 represent RGB color values using quaternion notation, and employ quaternion convolutional layers resulting in moderate improvements in image classification and inpainting tasks. Building upon these advancements, we contribute to the ongoing research integrating color priors within deep neural architectures.
3

---

3 Color equivariant convolutions
3.1 Group Equivariant Convolutions
CNN layer + is equivariant to a symmetry group if for all transformations E G on the input x the resulting feature mapping (xx) transforms similarly, i... first doing transformation and then the mapping is similar to first doing the mapping and then the transformation. Formally, equivariance defined as
(U Vg
where Tg and T3 are the transformation operators of group action on the input and feature space, respectively. Note that T. and T can be identical, as is the case for translation equivariance where shifting the imput results in equally shifted feature map, but do not necessarily need to be. A special case of equivariance is invariance, where T is the identity mapping and the input transformation leaves the feature map unchanged:
(T = ((x), Vg E G.
(2)
We use the definition from 7 to denote the i-th output channel of a standard convolutional layer in terms of the correlation operation (+) between set of feature maps f and C++1 filters th:
c²
(()) = Σ fely) -x) I).
Here f: z² - C² and v/² :2 - R are functions that map pixel locations to a C -i-dimensional vector. This definition can be extended groups by replacing the translation x by group action g:
[ * (g) = Σ Σ falu)
DEZ2
As the resulting feature map f 2b is a function on G rather than Z the inputs and filters of all hidden layers should also be defined on G:
()))
Σ E (o(h))?(G NEG
Invariance to subgroup can be achieved by applying pooling operation over the corresponding cosets. For a more detailed introduction to group equivariant convolutions, please refer to 71.
3.2 Color Equivariance
We define color equivariance as equivariance to hue shifts. The HSV color space encodes hue by an angular scalar value, and hue shift is performed simple additive offset followed by a modulo operator. When projecting the HSV representation into three-dimensional RGB space, the same hue shift becomes a rotation along the [1, 1, 1] diagonal vector.
We formulate hue equivariance the framework of group theory by defining the group Hn of multiples of 360/n-ddoeee nndegree rotations about the [1, I, diagonal vector in R space. Hn subgroup of the SO(3) group of all rotations about the origin of three-dimensional Euclidean space. We can parameterize Hl in terms of integers k, n
cos( 2km +
b 2h cOS(
H (k)
+ b
b
+ b
cos(
2kE
+ a
with the total number of discrete rotations in the group, k the rotation, a 3 cos 24T and b = / * sin( 2kg ). The group operation is matrix multiplication which acts on the continuous g space of RGB pixel values. The derivation of H, is provided in Appendix A

---

Color Equivariant Convolution (CEConv) Let us define the group G = Z² x Hn, which is direct product of the z² group discrete 2D translations and the H,, group of discrete hue shifts. We can then define the Color Equivariant Convolution (CHConv) in the input layer as:
C²
If * ( k) = Σ f(l)) (k) z).
NE c
We furthermore introduce the operator b ()) including translation and hue shift acting input f defined the plane Z²:
(Zyf((z) (L) = (m) -)
(8)
Since H, is an orthogonal matrix, the dot product between hue shifted input H,f and a filter &b equal the dot product between the original input f and the inverse hue shifted filter H
is
Hn. 1o = (HnJ) = H H
H tv..
(9)
Then the equivariance of the CEConv layer can be derived as follows (using C = for brevity):
[[ ( k)
Σ Hn(m) - (k) -x)
yEza
Σ f(u) (m) HA. -0:--(x-000 1))
yE2?
(10)
Σ fly) Hn(k - m) (x t))
vEZZ
[/ x vi (x: 1, k m))
[C' (t,(mm)) + "1][[1] k)
Since input f and feature map [/ * bb are functions on 2 and G, respectively, L((,)) and L (1.)) represent two equivalent operators acting their respective groups. For all subsequent hidden layers the input and filters vpt are functions on G parameterized by x, k, and the hidden layer for CECony defined as:
8 * ΣΣΣ r=
felu.r) v. x. (r k))sn),
(11)
where n is the number of discrete rotations in the group and % is the modulo operator. In practice, applying a rotation to RGB pixels will cause some pixel values to fall outside of the RGB cube, which will then have to be reprojected within the cube. Due to this discrepancy, Eq. (9) only holds approximately, though in practice this has only limited consequences, as we empirical show in Appendix D
3.3 Implementation
Tensor operations We implement CEConv similarly to GConv (7]. GConv represents the pose associated with the added spatial rotation group by extending the feature map tensor X with an extra dimension G size [C*, G H. W), denoting the number of channels, transformations that leave the origin invariant, and height and width of the feature map at layer I, respectively (batch dimension omitted). Similarly, GConv filter F with spatial extent is of size [C++1, Gl+1 Cl. Gl. k. kj. The GConv then defined in terms of tensor multiplication operations as:
C
G' Fd
X
of ΣΣFd
Felgg
X! g.gg egga
(12)
where (:) denotes tensor slices. Note that in the implementation, a GConv filter only contains [O++1 Ci, Gf, k ki unique parameters the extra G1+1 dimension is made up of transformed copies of F.
5

---

As the RGB input to the network is defined on z². we have G¹ = 1 and F has size
[C++] G++1 3, 1. k. k). The transformed copies in G are computed by applying the rotation
matrix from Eq. 4
d' Fa(g') dall
(13)
In the hidden layers F contains eyclically permuted copies of F:
of s.g.uur
F "cl..oo-"")nn...
(14)
Furthermore, to explicitly share the channel-wise spatial kemne over G' 30.. filter F is decomposed into a spatial component S and a pointwise component P follows:
Fe f.g..g. ,,,,,,,,,." Pe
(15)
F' is precomputed in each forward step prior to the convolution operation in Eq. 12
Input normalization is performed using single value for the mean and standard deviations rather than per channel, as is commonly done for standard CNNs. Channel-wise means and standard deviations break the equivariance property of CECNN a rotation around the [1, 1, 1] diagonal. Experiments have shown that using a single value for all channels instead of channel-wis normalization has no effect on the performance.
hue shift could no longer be defined as
Compute elficiency CEConvs create factor [ more feature maps in each layer. Due the decomposition in Eq. (15). the number of multiply-accumuut (MAC) operations increase by only factor E overview of parameter counts and MAC operations.
| + l. and the number of parameters by factor [fr 1. See Appendix C.3 for an
Experiments
4.1 When is color equivariance useful?
Color equivariant convolutions share shape information across different colors while preserving color information in the group dimension. To demonstrate when this property is useful perform two controlled toy experiments on variations of the MNIST [1] dataset. We use the 72CNN architecture from 77 and create color equivariant version of the network called CECNN by replacing all convolutional layers by CEConvs with three rotations of 120 The number of channels in CECNN is scaled such as keep the number of parameters approximately equal to the Z2CNN. We also create a color invariant CECNN by applying coset max-pooling after the final CEConv layer, and a color invariant Z2CNN by converting the inputs grayscale. All experiments are performed using the Adam 24 optimizer with learning rate of 0,001 and the OneCycle learning rate scheduler. No data augmentations are used. We report the average performance over ten runs with different random initializations.
Color imbalance is simulated by long-tailed ColorMNIST, 30-class classification problem where digits occur in three colors on a gray background, and need to be classified by both number (0-9) and color (red, green, blue). The number of samples per class is drawn from a power law distribution resulting in long-tailed class imbalance. Sharing shape information across colors is beneficial as a certain digit may occur more frequently in one color than in another. The train set contains a total of 1,51 14 training samples and the test set is uniformly distributed with 250 samples per class. The training is visualized in Appendix B. We train all four architectures epochs using the standard cross-entropy loss. The train set distribution and per-class test accuracies for all models are shown in Fig. 2a With an average accuracy of 91.35 +0. 00.4%% the CECNN performs significantly better than the CNN with 71.59 + 0.61%. The performance increase is most significant for the classes with low sample size, indicating that CEConvs are indeed more efficient in sharing shape information across different colors. The color invariant Z2CNN and CECNN networks, with an average accuracy of 24.19 + 0.53% and 29.43 = 0.46%, respectively, are unable discriminate between colors. CECNN with coset pooling is better able to discriminate between foreground and background and therefore performs slightly better. We repeated the experiment with a weighted loss and observed no significantly different results. We have also experimented with adding color jitter augmentations, which makes solving the classification problem prohibitive, as color required. See Appendix B.2 for both detailed results on both experiments.
the dataset for 1000

---

ColorMNIST -longtailed
ColorMNIST biased
1.00
w
0.99
CEC (cosel poct)
0.98
0.97
Teet
0.98
0.2
-343
0.95
ZZCZN
Class Euquericy
0.3
ZZCHN (rraystale
CECNN ccoset poot
165
(b)
Figure 2: Color equivariant convolutions efficiently share shape information across different colors. CECNN outperforms vanilla network both long-tailed class imbalance setting (a), where MNIST digits are to be classified based on both shape and color, and color biased setting (b), where the color of each class is sampled according to tg N((,,
Color variations are simulated by biased ColorMNIST, 10-class classification problem where each class has own characteristic hue 0g defined in degrees, distributed uniformly on the hue circle. The exact color of each digit is sampled according to 0, (f... We generate multiple datasets by varying between O and 10 where a results in a completely deterministic color for each class and σ = 10 in an approximately uniform distribution for 0z. For small ø, color is thus highly informative of the class, whereas for large a the classification needs be performed based shape. The dataset is visualized in Appendix B.I] We train all models on the train set of 1.000 samples for 1500 epochs and evaluate on the test set of 10.000 samples. The test accuracies for different a are shown in Fig. 2b CECNN outperforms Z2CNN across all standard deviations, indicating CEConvs allow for a more efficient internal color representation. The color invariant CECNN network outperforms the equivariant CECNN model from 2 48. Above this value color is no longer informative for the classification task and merely acts as noise unnecessarily consuming model capacity, which is effectively filtered out by the color invariant networks. The results of the grayscale Z2CNN are omitted as they are significantly worse, ranging between 39.89% (a = 0) and 79.94 (a 109). Interestingly, CECNN with coset pooling outperforms the grayscale Z2CNN. This due to the fact that a CECNN with coset pooling is still able to distinguish between small color changes and therefore can partially exploit color information. Networks trained with color jitter are unable to exploit color information for low a; see Appendix B.2 for detailed results.
4.2 Image classification
Setup We evaluate our method for robustness to color variations on several natural image classifica- tion datasets, including CIFAR- 10 and CIFAR- 100 (27). Flowers- 102 [35, STL-10 (6. Oxford-IIIT Pet [40]. Caltech-10) 101 [B Stanford Cars (26] and ImageNet [10]. We train a baseline and color equivariant Res-) (20 with 3 rotations and evaluate on range of test
where we gradually apply a hue shift between 180 and 180 For high-resolution datasets (all except CIFAR) we train a ResNet-18 architecture and use default ImageNet data augmentations: we scale to 256 pixels, random crop to 224 pixels and apply random horizontal flips. For the CIFAR datasets we use the ResNet-44 architecture and augmentations from 77 including random horizontal flips and translations of up to pixels. We train models both with and without color jitter augmentation to separately evaluate effect of equivariance and augmentation. The CE-Re CEE-ReNeets are downscaled width to match the parameter count of the baseline ResNets. We have also included AugMix [21] and CIConv [29 as basclines for comparison. Training is performed for 200 epochs using the Adam (25 optimizer with learning rate of 0.001 and the OneCycle learning rate scheduler. All our experiments use Py Torch and run on a single NVIDIA A40 GPU.
Hybrid networks In our toy experiments we enforce color equivariance throughout the network. For real world datasets however, we anticipate that the later layers of a CNN may not benefit from enforcing parameter sharing between colors, if the classes of the dataset are determined by color
7

---

specific features. We therefore evaluate hybrid versions of our color equivariance networks, denoted by an integer suffxx for the number of ResNet stages, out possible four, that use CEConvs.
Original test set
Caltech
C-10
C-100
Flowers
Ox-Pet
Cars
STL10
ImageNet
Baseline
71.61
93.69
71.28
66.79
69.87
76.54
83.80
69.71
CIConv-W
72.85
75.26
38.81
68.71
61.53
79.52
80.71
65.81
CEConv
70.16
93.71
71.37
68.18
70.24
76.22
84.24
66.85
CEConv-2
71.50
93.94
72.20
68.38
70.34
77.06
84.50
70.02
Baseline + jitter
73.93
93.03
69.23
68.75
72.71
80,59
83,91
69.37
CIConv-Ve W jitter
74.38
77.49
42.27
75.05
64.23
81.56
81.88
65.95
CEConv jitter
73.58
93.51
71.12
74.17
73.29
79.79
84.16
65.57
CEConv-2 + jitter
72.61
93.86
71.35
71.72
72.80
80.32
84.46
69.42
Baseline + AugMix
94.13
72.64
75.49
76.02
82.32
84.99
CEConv + AugMix
70.74
94.22
72.48
78.10
75.90
80.81
85.46
Hue-shifted test set
Baseline CIConv-W
51.14
85.26
47.01
13.41
37.56
$5.59
67.60
54.72
71.92
74.88
37.09
59.03
60.54
78.71
79.92
64.62
CEConv
62.17
90,90
59.04
33.33
54.02
67.16
78.25
56.90
CEConv-2
64.51
91.43
62.11
33.32
51.14
68.17
77.80
62.26
Baseline + jitter
73.61
92.91
69.12
68.44
72.30
80.65
83.71
67.10
CIConv-W jitter
74.40
77.28
42.30
75.66
63.93
81.44
81.54
65.03
CEConv + jitter
73.57
93.39
71.06
73.86
72.94
79.79
84.02
64.52
CEConv- + jitter
73.03
93.80
71.33
71.44
72.58
80.28
84.31
68.74
Baseline + AugMix
51.82
88.03
51.39
15.99
48.04
68.69
72.19
CEConv + AugMix
62.29
91.68
60.75
41.43
62.27
73.59
80.17
Table 1: Classification accuracy in % of vanilla color equivariant (Cs-)ResNets, evaluated both on the original and hue-shifted test sets. Color equivariant CNNs perform on par with vanilla CNNs on the original test sets, but significantly
robust test-time hue shifts.
Results We report both the performance on the original test set, as well as the average accuracy over all hue shifts in Table I For brevity only show the fully equivariant and hybrid-2 networks, a complete overview of the performances of all hybrid network configurations and error standard deviations be found in Appendix C.T Between the full color equivariant and hybrid versions of our CE-ResNets, at least one variant outperforms vanilla ResNets on most datasets on the original test
On most datasets the one- or two-stage hybrid versions the optimal CE-ResNets, providing good trade-off between color equivariance and leaving the network free to learn color specific features later layers. CE-ResNets are also significantly more robust test-time hue shifts, especially when trained without color jitter augmentation. Training the CE-ResNets with color jitter further improves robustness. indicating that train-time augmentations complement the already hard-coded inductive biases in the network. We show the detailed performance on Flowers- 102 for all test- tist-time hue shifts Fig. lb The accuracy of the vanilla CNN quickly drops hue shift applied, whereas the CE-CNN performance peaks at -1 -200,, 0 120% Applying train-time color jitter improves the CNN's robustness to the level of CNN with grayscale inputs. The CE-CNN with color jitter outperforms all models for all hue shifts. Plots for other datasets are provided in Appendix
Color selectivity To explore what affects the success of color equivariance, we investigate the color selectivity of a subset of the studied datasets. We use the color selectivity measure from [42) and average across all neurons the baseline model trained on each dataset. Fig. shows that color selective datasets benefit from using color equivariance up to late stages, whereas less color selective datasets do not.
Feature representations of color equivariant CNNs We use the Neuron Feature 42 (NF)
visualization method to investigate the internal feature representation of the CE-ResNet. NI computes weighted average of the N highest activation input patches for each filter at certain layer, such
representing the input patch that specific neuron fires on. Fig. shows the NF (N = 50) and top-3 input patches for filters at the final layers of stages 1-4 of FE-ResNet18 trained on Flowers- 102.

---

Color Equivarian: stages accuracy improvement
colarjjiter augmentatin
w/ colorjjiter augme
Accurocy improveettt
Accuracy improvemeett
(oogehaa and valnilai
Caltech1 01 tcolor sel. 0.4)
0-
STL10 icolor sel. 0,38)
StanforeCars 0.38)
-6
-6
Color Equivvriance embedied up stage Color Equivariacc embedded up to stage
Figure 3: Color selective datasets benefit from using color equivariance up late stages, whereas less color selective datasets do not. We compute average color selectivity [42] of all neurons in the baseline CNN trained on each dataset, and plot the accuracy improvement of using color equivariance in hybrid and full models, coloring each graphed dataset for color selectivity.
Stage
Stage
Stage
ASD
TAT
Figure 4: Neuron Feature 42] (NF) visualization with top-3 patches at different stages of a CE- ResNet18 trained on Flowers- 102. Rows represent different rotations of the same filter. As expected, each row a NF activates on same shape in a different color.
Different represent different rotations of the filter. As expected, each row of NF activates on the same shape in a different color, demonstrating the color sharing capabilities of CEConvs. More detailed NF visualization are provided in Appendix C.4
Ablation studies We perform ablations to investigate the effect of the number of rotations, the use short, we find that
of group coset pooling, and the strength of train-time color jitter augmentations. a) increasing the number of hue rotations increases robustness to test-time hue shifts at the cost of slight reduction in network capacity, b) removing group coset pooling breaks hue invariance, and c) hue equivariant networks require lower intensity color jitter augmentations to achieve the same test-time hue shift robustness and accuracy. The full results can be found in Appendix D
Conclusion
In this work, we propose Color Equivariant Convolutions (CEConvs) which enable feature sharing across colors in the data, while retaining discriminative power. Our toy experiments demonstrate ben- efits for datasets where the color distribution is long-tailed or biased. Our proposed fully equivariant CECNNs improve performance on datasets where features are color selective, while hybrid versions that selectively apply CHConvs only in early stages of CNN benefit various classification tasks.
Limitations CEConvs are computationally more expensive than regular convolutions. For fair comparison, we have equalized the parameter cost of all models compared, at the cost of reducing the number of channels of CECNNs. In cases where color equivariance not a useful prior, the reduced capacity hurts model performance, reflected in experimental results.
Pixel values near the borders of the RGB cube can fall outside the cube after rotation, and subsequently need to be reprojected. Due to this clipping effect the hue equivariance in Eq. ( only holds approximately. As demonstrated empirically, this has only limited practical consequences, yet future work should investigate how this shortcoming could be mitigated.

---

Local vs. global equivariance The proposed CEConv implements local hue equivariance, i.c. allows to model local color changes in different regions of an image separately. In contrast, global equivariance, e.g. by performing hue shifts on the full input image, then processing all inputs with the same CNN and combining representations at the final layer to get a hue-equivariant representation, encodes global equivariance to the entire image. While we have also considered such setup, initial experiments did not yield promising results. The theoretical benefit of local over global hue equivariance is that multiple objects in one image can be recognized equivariantly in any combination of hues- empirically this indeed proves to be a useful property.
Future work The group of hue shifts is but one of many possible transformations groups on images. CNNs naturally learn features that vary in both photometric and geometric transformations [5] 37] Future work could combine hue shifts with geometric transformations such as oloottranslatinn and scaling [49]. Also, other photometric properties could be explored in an equivariance setting, such saturation and brightness.
Our proposed method rotates the hue of the inputs by a predetermined angle as encoded in a rotation matrix. Making this rotation matrix learnalee could yield inexact but more flexible type of color equivariance, in line with recent works on learnable equivariance 34 46]. An additional
line of interesting future work is to incorporate more fine-grained equivariance to continuous hue shifts, which is currently intractable within the GConv-inspired framework as the number multiply- accumulate operations grow quadratically with the number of hue rotations.
Broader impact Improving performance on tasks where color is a discriminative feature could affect humans that are the target of discrimination based on the color of their skin. CEConvs ideally benefit datasets with long-tailed color distributions by increasing robustness to color changes, in theory reducing a CNN's reliance on skin tone as a discriminating factor. However, careful and rigorous evaluation is needed before such properties can be attributed to CECNNs with certainty.
Acknowledgements
This project is supported in part by NWO (project VI. Vidi. 102..
References
[1] Naif Alshammari, Samet Akeay, and Toby Breckon. On the impact coumination-nnvaiiaan image pre- pre-transformation for contemporary automotive semantic scene understanding. In 2018 IEEE Intelligent Vehicles Symposium (/V), pages 1027-1032, 2018.
[2] Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen A. J. Eppenhof, Josien P. W. Pluim, and Remco Duits Roto-translation covariant convolutional networks for medical image analysis. In Medical Image Computing and Computer Assisted Intervention MICCAI 2018, pages 440-448. Springer International Publishing, 2018
[3] Inês Bramão, Luis Faísca, Karl Magnus Petersson, and Alexandra Reis. The contribution of color object recognition. In loannis Kypraios, editor, Advances in Object Recognition Systems, chapter 4. IntechOpen,
Rijeka, 2012.
[4] Michacl M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR, abs/2 104. 13478, 2021.
[5] Robert-Jan Bruintjes, Tomasz Motyka, and Jan van Gemert. What affects learned equivariance in deep image recognition models? arXiv preprint arXiv:2304..2260
2023.
[6] Adam Coates. Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[7] Taco S. Cohen and Max Welling, Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning Volume 48, ICML 16, page 2990 10000299..1111 JMLR. 2016.
[8] Ian R. Cole. Modelling CPV. 6 2015.
[9] Marco Cotogni and Claudio Cusano. Offiset equivariant networks and their applications. Neurucomputing,
502:110-119, 2022.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hicrarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248-255,
2009.
[11] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, (9(6):141-14 2012.
10

---

[12] Martin Engilberge, Edo Collins, and Sabine Süsstrunk. Color represeniation in deep neural networks. In 2017 IEEE International Conference on Image Processing (ICIP), pages 2786-2790, 2017.
[13] G.D. Finlayson, M.S. Drew, and B. Funt. Diagonal transforms suffice for color constancy. In 1993 (4th) International Conference on Computer Vision, pages 164-171, 1993.
[14] G.D. Finlayson. S.D. Hordley, Cheng Lu, and M.S. Drew. On the removal of shadows from images. IEEE Transactions on Puttern Analysis and Machine Intelligence, 28(1):59-68. 2006.
[15] B. Funt and G.D. Finlayson. Color constant color indexing. IEEE Transactions on Pattern Analysis and Machine Intelligence, 17(5):522-529 1995.
[16] Chase Gaudet and A. Maida. Deep quaternion networks. 2018 International Joint Conference on Neural
Networks (JJCNN) pages 1 8, 2018.
[17] J. M. Geusebroek, R. van den Boomgaard, A. W. M. Smeulders, and H. Geerts. Color invariance. IEFE
Transactions on Pattern Analysis and Machine Intelligence, 23(12):1338-1333 2001. [18] T. Gevers, A. Gijsenij. J. van de Weijer, and J. M. Geusebroek. Color in Computer Vision Fundamentals
and Applications. Series in Imaging Science and Technology. The Wiley-IS&T, 2012.
[19] T. Gevers, A. Gijsenij. J. van de Weijer, and J. M. Geusebroek. Color in Computer Vision Fundamentals
and Applications. Series in Imaging Science and Technology. The Wiley-IS&T. 2012.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. In Proceedings of 2016 IEFE Conference Computer Vision and Pattern Recognition, pages 770-778.
TEEE, June 2016.
[21] Dan Hendrycks, Norman Mu, Ekin D. Cubuk, Barret Zoph, Justin Gilmer, and Balaji Lakshminarayanan. AugMix: A simple data processing method to improve robustness and uncertainty. Proceedings of the International Conference on Learning Representations (ICLR), 2020.
[22] Geoffrey E Hinton, Sara Sahour, and Nicolas Frosst. Matrix capsules with EM routing. In International
Conference on Learning Representations, 2018.
[23] Osman Semih Kayhan and Jan C van Gemert. On translation invariance in cnns: Convolutional layers can exploit absolute spatial location. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 14274-1428 14285, 2020.
[24] Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization. International Conference
on Learning Representations, 12 2014.
[25] Diederik P. Kingma and Jimmy Ba. Adam: A method for stochastic optimization. CoRR, abs/1412.6980.
2014.
[26] Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine- grained categorization. 2013 IEEE International Conference on Computer Vision Workshops, pages 554-561,
2013.
[27] Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. Technical Report 0, University of Toronto, Toronto, Ontario, 2009.
[28] Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. In Proceedings of the IEEE, volume 86, pages 2224,4 1998.
[29] Attila Lengyel, Sourav Garg, Michacl Milford, and Jan C. van Gemert. Zero-shot day-night domain adaptation with physics prior. In Proceedings of the IEEE/CVF International Conference Computer Vision (ICCV), pages 4399-4409, October 2021.
[30] Attila Lengyel and Jan van Gemert. Exploiting learned symmetries in group equivariant convolutions. In
2021 IEEE International Conference on Image Processing (ICIP). pages 759-763, 2021.
[31] Fei-Fei Li, Marco Andrecto, Mare' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
[32] Lachlan E. MacDonald, Sameera Ramasinghe, and Simon Lucey. Enabling equivariance for arbitrary lie groups. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 8183-8192 8192, June 2022.
[33] Bruce A. Maxwell, Casey A. Smith, Maan Qraitem, Ross Messing, Spencer Whitt, Nicolas Thien, and Richard M. Friedhoff. Real-time physics-based removal of shadows and shading from road surfaces. In 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops (CVPRW), pages 1277-1285, 2019.
[34] Artem Moskalev, Anna Sepliarskaia, Ivan Sosnovik, and Arolld Smeulders. Liegg: Studying learned lie group generators. In S. Koyejo. S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 25212-25223. Curran Associates,
Inc., 2022.
[35] Maria-Elens Nilsback and Andrew Zisserman. Automated flower classification over a large number of classes. In Indion Conference on Computer Vision, Graphics and Image Processing, Dec 2008
[36] Chris Olah, Nick Cammarata, Ludwig Schubert, Gabriel Goh, Michael Petrov, and Shan Carter. An overview of early vision in inceptionvl. 1. Distill, 2020. https://disiill.ubbb
[37] Chris Olah, Nick Cammarata, Chelsea Voss, Ludwig Schubert, and Gabriel Goh. Naturally occurring equivariance in neural networks. Distill, 2020. http://distill, http:://ditill.pub/220000 pubb2000//icuuiss//uuiaaaatt
[38] Daniel Osorio and Misha Vorobyev. Colour vision as an adaptation to frugivory in primates. Proceedings. Biological sciences/ / The Royal Society, 263:593-99 9, 06 1996.
[39] Gokhan Ozbulak. Image colorization by capsule networks. In Proceedings of the IEEE/CVF Conference
on Computer Vision and Pattern Recognition (CVPR) Workshops, June 2019.
[40] Omkar M. Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. Cats and dogs. In IEEE Conference on Computer Vision and Pattern Recognition, 2012.
11

---

[41] Ivel Rafegas and Maria Vanrell. Color representation in cans: Parallelisms with biological vision. In 2017
IEEE International Conference Computer Vision Workshops (ICCVW), pages 2697-2705, 2017.
[42] Ivet Rafegas and Maria Vanrell. Color encoding in biologically-inspicee
convolutional neural networks.
Vision Research, 151:7-17, 2018. Color: cone opponency and beyond.
[43] Ivet Rafegas, Maria Vanrell, Luís A. Alexandre, and Guillem Arias. Understanding trained cnns by
indexing neuron selectivity. Pattern Recognition Letters, 136:318-3255 2020.
[44] M. Rath and Condurache. Boosting deep ncural networks with geometrical prior knowledge: A survey.
ArXiv, abs/2006. 16867, 2020.
[45] B Regan, Julliot, Bruno Simmen, Françoise Vićnot, P.C.
and John Mollon. Fruits, foliage and the evolution of primate colour vision. Philosophical transactions of the Royal Society of
London. Series B, Biological sciences. 356:229 83, 03 2001.
[46] David W. Romero and Suhas Lohit. learming partial equivariances from data. In S. Kayejo, S. Mohamed, A. Agarwal, D. Belgrave. K. Cho, and A. Oh, editors. Advances in Neurul Information Processing Systems, volume 35. pages 36466-36478. Curran Associates, 2022..
[47] Sara Sabour, Nicholas Frosst, and Geoftrey E Hinton. Dynamic routing between capsules. In I. Guyon,
U. Von Luxburg. S. Bengio, H. Wallach, R. Fergus, Vishwanathan, and R. Garnett, editors, Advances Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[48] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertainty- aware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF Interna- tional Conference on Computer Vision (ICCV), October 2019.
[49]
Ivan Sosnovik, Artem Moskalev, and Amold Smeulders. Disco: accurate discrete scale convolutions. In Proceedings of the 32nd British Machine Vision Conference (BMVC), 2021.
[50] Ivan Sosnovik, Michal Szmaja, and Amoll Smeulders. Scale-cquivariant steerable networks. In Interna- tional Conference
Learning Representations, 2020.
[51] Jan C. Van Gemert. Exploiting photographic style for category-level image classification by generalizing the spatial pyramid. In Proceedings of the 1st ACM International Conference Multimedia Retrieval, pages 1-8.200 2011.
[52] Maurice Weiler, Fred Hamprecht, and Martin Storath. Learning stecrable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[53] Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In Advances in Neural Information Processing Systems. volume 32, pages 7366-7378. Curran Associates, Inc., 2019.
[54] Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In P Proceedings of the European Conference Computer Vision (ECCV), September 2018.
12

---

A Derivation of Hn
Rotation around arbitrary unit vector by angle 0 can be decomposed into five simple steps 8
1. rotating the vector such that it lies in one of the coordinate planes, e.g. a2 using M.z;
2. rotating the vector such that it lies on one the coordinate axes, e.g. a using M,;
3. rotating the point around vector u on axis x using Rz:
4. reversing the rotation in step using M,-' M
5. reversing the rotation in step 1. using M,z' M
These operations can be combined into a single matrix:
Ru. = M (M. (Ra,
(16)
MT. M Ra
(17)
cos 0 + u cos 0)
Uy Mly (1
Ug sin 0
Ug (1 cos 0) + uy sin 0
Un (1 cos 0) + sin 0
Cos 0 + u (1 cos 0)
Uy (1 cos 0) sin 0
Uz (1 cos 0) - uy sin 0
Uy (1 cos 0) + sin 0
cos 0 + u (1 cos 0)
(18)
Substituting
1 V3 v3' v3 yields
cos 0 + 1/ (1 cos 0) (1 cos 0) + 73
cos 0)
v3 sin
(1 cos 0) sin 0
(1 cos 0) v/3
Ru.0
1 sin
cos 0 + 3 (1 cos 0)
sin (19)
cos 0) v3
sin
cos 0) V3 sin
cos 3 (1 cos
and lastly, rearranging and substituting 0 results
Tcos( 2km
- b
+ b - b
(k)
a + b
cos( 2km + a
(20)
b
++ b
2km cos
a
with n the total number of discrete rotations in the group, the rotation, a b = * sin 2km
cos(
2km and
n
13

---

B ColorMIISST
B.1 Dataset visualization
Long-tailed ColS dataset The training samples the Longtailed ColorMNIST dataset
depicted in Fig. 5 clearly indicating class imbalance.
Figure 5: Long-tailed ColorMNIST. Note the strong class imbalance the dataset. Best viewed in color.
Biased ColorMNIST dataset A small subset of the samples of Biased ColorMNIST is shown in Fig. 60 for a = 0 (a) and a = 36 (b), respectively. Note that the samples in (a) have a deterministic color, whereas in (b) exhibit some variation in hue.
(a)
Figure 6: Samples from Biased ColorMNIST for 0 (a) and 36 (b), respectively. Best viewed
color.
14

---

shown in Fig. [ We observed no significant difference between the two setups, with the CECNN Additional experiments
B.2
Results with color jitter augmentation We performed both ColorMNIST experiments with color jitter augmentations. The results are shown in Fig. 7 (a) For long-tailed ColorMNIST, adding jitter makes solving the classification problem prohibitive, as color required. Z2CNN and CECNN with jitter therefore perform no better than the CECNN model with coset pooling. (b) For biased MNIST, performance decreases for small and improves for large a, with CEConv still performing best.
ColorMNIST longtailed
ColorMNIST
1,00
Z2CNN
ZZCHN
jELer
ww
301
35t.Acuracy
0.98
(COSNt pool)
3/3
CECNN (Coset pooli jRter
0.97
72CHN
CEC
CONN trosel
rrequencyy
0.94
0
12
24
36
48
60
166
Class
Dataset Standard Devatton
(a)
(b)
Figure 7: Color equivariant convolutions efficiently share shape information across different colors. CECNN outperforms vanilla network in both long-tailed class imbalance setting (a), where MNIST digits are to be classified based on both shape and color, and color biased setting (b), where the color of each class c is sampled according to 0g (8,,,
Long-tailed ColorMNIST with weighted loss We performed the longtailed ColorMNIST experi- ment both with a uniformly weighted loss and loss where classes are weighted inversely to their frequency according w: cany in the training
N where wy denotes the weight for class N the number of samples the number of classes, and n, the number of samples for class i. The results
without coset pooling outperforming the other models by large margin in both.
ColorMNIST longtta iled
V
W 
Taat Accurecy
1.2
6.5
CEONN
Class
Figure 8: Per-class accuracy of various models trained with a loss function weighted by inverse class frequency. CECNN without coset pooling outperforms other models, with no significant differences compared to uniformly weighted loss function.
C Classification experiments
C.1 Overview of all CE-ResNet configurations
T an overview of the classification accuracies all baselines and equivariant architectures. CEConv-x denotes the number of ResNer stages with CE convolutions with CEConv-4 (3 for CIFAR) being a fully equivariant ResNet. nearly cases, early equivariance is beneficial for improving classification accuracy both the original as well the hue shifted test sets. In case of the Flowers-
15

---

102 dataset late equivariance seems to have significant advantage, whereas for Caltech-101 101 and Stanford Cars the color equivariance bias does not seem have much added value.
shows Shows the classification results for all network architectures, trained with and without color jitter.
Table
ImageNet
69.71
65.81
70.05
70.02
69.35
66.85
69.37
65.95
69.36
69.42
69.10
65.57
54.72
64.62
63.02
62.26
60.95
56.90
67.10
65.03
68.85
68.74
68.03
64.52
83,80 + 0.36
+ 0.27
84.34 1 0.38
84.50 1 0.31
83.61 1 0.35
- 0.49
83.91 + 0.38
81.88 + 0.24
84.31 + 0.47
84.46 1 0.39
84.08 1 0.44
14.1610. 0.10
84.99 + 0.24
85.46 : 0.30
67.60 + 0.56
79.92 + 0.25
78.83 1 0.43
77.80 1 0.58
76.23 1 0.37
78.25 + 0.52
+: 0.35
81.54 + 0.21
84.29 + 0.50
84.31 1 0.34
1 0.33
0.14
72.19 +: 0.45
+: 0.15
STL-10
80.71
84.24
84.16
83.71
84.00
84.02
80.17
Stanford Cars
76.54 + 0.10
79.52 + 0.42
78.00 78.03 1 0.29
0.38
75.33 +: 0.66
= 0.19
80.59 + 0.36
81.56 + 0.32
0.51
80.32 + 0.47
79.62 1 0.54
+ 0.37
82.32 + 0.07
80.81 + 0.27
55.59 + 0.74
78.71 +: 0.33
0.21
68.17 1: 0.86
65.82 + 1.34
67.16 + 0.58
+:55:0055 0.36
81.44 +. 0.26
18 4 0.50
80.28 +: 0.52
79.62 # 0.54
+ 0.34
68.69 + 0.73
73.59 1: 0.30
77.06
76.22
80.24
79.79
70.08
80.65
79.79
Oxford-IIT Pet
+ 0.57
+::00010 0.53
1.07
70.34 1: 0.67
68.92 1 0.46
0.79
+ 0.67
64.23 + 0.51
72.44 1 0.76
+ 0.87
0.67
+ 0.63
+ 0.51
75.90 + 0.22
+ 0.76
0.46
1.00
1 0.95
0.65
- 1.34
0.49
+ 0.42
0.58
0.86
72.32 1 0.64
72.94 + 0.56
+ 0.74
0.81
69.87
61.53
70.47
70.24
72.71
72.80
72.55
73.28
76.02
37.56
60.54
52.97
51.14
49.45
45
54.02
72.31
63.93
72.34
72.58
48.04
62.27
+ 0.89
68.71 + 0.29
67.29 T 0.57
68.38 1 0.55
69.42 :: 0.58
68.18 0.45
+ 1.50
+ 0.39
0.81
0.63
0.63
2.71 0.23
74.17 0.49
0.24
+ 0.50
13.41 + 0.34
59.03 + 0.62
0.83
33.32 1. 0.55
$ 0.38
0.38
+ 1.60
1.60
75.66 + 0.27
0.79
71.44 1 0.57
72.47 1 0.20
0.39
15.99 +: 0.28
0.97
41.43 0.97
Flowers-100
66.79
68.75
75.05
70.39
71.72
72.71
75.49
78.10
32.88
33.76
33.33
08.444
68.44
69.98
73.86
CIFAR-100
71.28 + 0.20
38.81 + 0.66
71.82 1 0.36
0.48
+ 0.24
123::000 0.44
42.27 + 0.56
0.27
0.20
+ 0.57
72.64 + 0.27
72.48 + 0.18
+ 0.38
0.74
0.17
62.11 1 0.43
+ 0.45
+ 0.47
42.30 + 0.48
0.22
0.19
+ 0.53
1 0.19
L 0.24
72.20
71.37
69.23
71.08
71.35
71.12
47.01
37.09
63.37
59.04
69.12
70.98
71.33
71.06
51.39
60.75
CIFAR-10
93.69 0.16
75.26 + 0.57
94.06 1 0.09
93.94 1 0.07
93.71 1 0.26
93.03 0.16
77.49 + 0.53
93,93 + 0.16
+ 0.22
1 0.10
94.13 94.13 0.22
94.22 + 0.16
85.26 + 0.56
74.88 + 0.54
91.93 1 0.14
91.43 1 0.18
0.25
92.91 0.17
77.28 + 0.54
93.86 + 0.20
93. + 0.14
93.39 1 0.08
+
0.26
1 0,21
93.03
93. 93.86
93.51
90.90
88.03
91.68
71.61 + 0.87
72.85 + 1.12
71.59 0.64
71.50 1 0.29
70.45 0.41
70.16 11:11.1 1.05
73.93 0.73
+. 0.43
73.43 0.59
72.61 + 0.95
73.21 + 0.87
0.68
71.92 + 0.95
70.74 + 1.12
51.14 0.71
71.92 1 1.11
65.60 0.47
64.51 1: 0.64
62.22 0.99
62.17 + 1.01
73.61 + 0.60
74.40 +. 0.55
73.34 0.96
73.03 + 0.97
73,26 + 0.74
0.75
51.82 + 0.60
62.29 +. 0.97
Caltech-101
74.38
73.58
73.57
CIConv- W + jitter
CEConv- + jitter
CEConv-2 + jitter
CEConv-34 + jitter
CEConv-4 + jitter
Baseline AugMix
CECony AugMix
Hue-shifted test set
CIConv- W + jitter
CEConv- + jitter
CEConv-2 + jitter
CEConv-3 3+ jitter
CEConv-4 + jitter
Baseline AugMix
CBCony AugMix
Original test set
Baseline + jitter
Baseline + jitter
CIConv-W W
CEConv-1
CEConv-2
3
CEConv-4
CIConv-W W
CEConv-1
CEConv-2
3
CEConv-4
Bascline
Bascline
Table 2: Classification accuracy on various datasets. CEConV-s denotes a ResNet with s color
equivariant stages. We report results for models trained with and without color jitter augmentation. (Hybrid) color equivariant networks improve performance over the baseline model on both the original well the hue-shifted test set.
16

---

C.2
Test-time hue shift plots
Fig. shows the test accuracies under test time hue shift on all datasets the paper. Each figure includes a regular ResNet, a color equivariant ResNet-x (CE-ResNet-xx and ResNet-x with color equivariant convolutions in the first ResNet stage (CE- (CE-ResNet-x-11 1). trained with and without color jitter augmentation. Finally, the plot shows the accuracy of ResNet-x trained on grayscale inputs. CHConv improves robustness test-time hue shifts on all datasets.
CIFAR-10
Test Scouracy
Test scourscy (36)
88
30
84
ResNet-44
Reshet-44 jiter
ResNet-44
Reshet-A4 jater
Resh [ours]
Rest 44 + jiter lours]
Restiet [ours]
CE ResNeet-40 + jitte lours)
CE-Resh st-4411 [ours]
ResNNtt44:11 jeber [ours]
CE-ResNet-44-1 [ours]
CE-ReNNet-44:1 jRter [ours]
ResNet-44 grayscale
ResNet-44 grayscale
150
100
50
50
100
150
150
100
50
0
50
100
150
Test-tmme hue (*)
Tist-time hue shitt
Flowers 102
Oxford IIT Pet
Test Sccursy (96)
Test sccurssy (66)
50
30
80
20
ResNee-18
BesNet-188 jitte
ResNet-18
ResNet-18 jittr
CE-Reshi RE-Resee--18 et-18 [ours]
CE ResNett + jitter lours]
CE ResNet-11 18 [ours)
CE ResNett-1 + jitter (ours)
CE-Resi et-18-1 [ours]
Resteet-188
CE-ResNee 18-1 [ours]
ResNect18-1
ResNet-18.-- grayicale
ResNet-18- grayscale
-150
-100
-50
50
100
150
-150
-100
-50
50
100
Test-lime hue shift (*)
Test-time hue shift (*)
Caltech-101
Stanford Cars
Test acurasy (861
03 1965
70-
80
ResNct:18
ResNcc-18
ResNet:1E
ResNet 18
CE-Resh [ours]
CE-ResNet-18 jittrr jours]
CE-RusNet-18 [surs]
CE-ResNet-18 Jitter [ours]
CE RosiN 18-18-1 [ours]
CE 18:11 jitte [ours]
CE Resteet-18 18-1 [ours]
CE ResNet 19:1 [ours]
ResNet-18- grayscale
RasNet-18 grayscale
-150
-100
5o
100
-150
-100
50
100
STL-10
ImageNet
sty Secures (N)
70
8
65
RusNet-18
Reskitt Jitter
ResNet-18 18
Reslict 18 Jitter
CE-Resh
CE EE- jittrr
CE-RestNet-1
CE RE--ReNNet-11 jitter [ours]
CF-Resh et-18-1 [ours]
Ct-Rest 18-1 jeter (ours]
CF-ResNeet18-- [nurs]
CF-Rusttei-18. jRter [curs]
RBsNet-18 grayscale
Reshiet-88 grayscale
-150
-100
-50
50
100
150
-150
-100
-50
0
50
100
150
Test-lime bue shift (*)
Test-lime hus shift
Figure 9: Test accuracy on various classification datasets under a test time hue shift.
17

---

C.3
CE-ResNet configurations
The configurations of the color equivariant ResNet with three hue rotations, as used the classification experiment in Section 4.2 are shown in Table CE stages 0 denotes a regular ResNet.
Model
CE stages
Width
Parameters (M)
MACs (G)
64
11.69
3.59
1
63
11.38
5.66
ResNet-18
2
63
11.57
7.37
3
61
11.54
8.80
55
11.79
10.32
32
2.64
0.78
1
31
2.51
1.23
ResNet-44
2
30
2.50
1.63
3
27
2.60
1.83
Table 3: Color equivariant ResNet configurations.
C.4 Neuron Feature visualizations
(NF) visualization with top-3 patches of two neurons at different stages in a CE-ResNet18 trained on Stanford Cars, As expected, each row of a NF activates on the same shape in different color. We show neurons that are insensitive to color (top row) and neurons that are sensitive to color (bottom row).
Fig. 10 shows the Neuron Feature 2
Stage
Stage
IEIT
XLI
Figure 10: Neuron Feature (NF) visualization with top-3 patches of two neurons at different stages in CE-ResNer18 trained on Stanford Cars. Rows represent different rotations of the same filter.
D Ablation studies
Strength of color jitter augmentations Fig. I1 shows the effect of hue jitter augmentation during training both color equivariant ResNet-18 with rotations (a) and regular ResNet-18 (b) trained on Flowers- 102. All runs have been repeated 3 times and the mean performance is reported. As expected, the color equivariant network (a) without jitter augmentation equivariant rotations of multiples of 120 degrees, but performance quickly degrades. Applying slight (0.1) l) hue jitter during training both helps in an absolute sense, increasing performance over all rotations, and makes network more robust to hue changes shown by the increasing width of the peaks. Further increasing the strength the augmentation results a uniform performance over all hue shifts, indicated by the flat lines. There appears to be significant difference for jitter strength > 0.2. In comparison, the
18

---

regular ResNet (b) trained without hue augmentation shows single peak around 0 degrees, which increases in width when applying more severe augmentation. Note that the increase in absolute performance is smaller compared to the color equivariant network. The reason for this is that the equivariant architecture only requires augmentation "between" the discrete rotations to which already robust, opposed to the full scale of hue shifts for the baseline architecture. Augmentation and equivariance thus exhibit a remarkable synergistic interaction.
Effect of jitter augmentation Flowers -022
Effect of jitter augmentation Flowers 102
75
30curss) 09
Test accurssy 05
j=0.3
CE Res
j-0.1
ResNeet 18
J-0.4
Net- j=0.1
Reshet--18 J-0.4
CE-ResNet-18
j-0.2
CB
j-0.5
Ras Not-11 --0.2
ResWat-18
j-0.5
-15
-100
-50
50
100
-150
-100
-50
50
100
Test umme (*)
Test time shitt
(b)
Figure 11: Effect of hue jitter augmentation color equivariant (a) and regular (b) ResNet-18.
Group coset pooling We have removed the group coset pooling operation by flattening the feature map group dimension into the channel dimension in the penultimate layer, before applying the final classification layer. As shown in Fig. 12] the model without pooling layer is no longer invariant to hue shifts and behaves identically to the baseline model.
group pooling Flowers-102
70
Test accosNyy (53
50
CB-ResNet CB-Resdee wlo pooling Reqular
10
-1.00
100
Test time shitt (?)
Figure 12: CE-ResNet without group coset pooling behaves similarly regular ResNet (average over 5 runs).
Number of color rotations We investigate the effect of the number of hue rotations in color equivariant convolutions by training CE-ResNets with 2-10 rotations Flowers- 102. Fig. 13 shows the test accuracies for rotations 1-5 (a) and 6-10 (b), respectively. Note that, this particular dataset, more hue rotations not only lead to better robustness test-time hue shifts, but also to better absolute performance. However, there is a trade-off between number of rotations and model capacity, as increasing the number of rotations increases the number of parameters in the model, and the model width needs to be scaled down to keep the number of parameters equal. Both the optimal number of color rotations and network width therefore depend on the amount of color the complexity of the data, and therefore both need to be carefully calibrated per dataset.
As expected, the number of peaks increases with the number of hue rotations. though interestingly. the peaks do vary in height. This is an artifact due the way test-time hue shifts are applied to the input images. When RGB pixels are rotated about the [1,1,1] diagonal, values near the borders of the RGB cube tend to fall outside the cube and subsequently need to be reprojected. This reprojection is not modeled by the filter transformations in the CEConv layers, and subsequently
19

---

causes discrepancy between the filter and the image transformations. Indeed, when the test-time hue shift instead implemented through a rotation in RGB space without reprojecting into the cube, this artifact disappears and all peaks are of equal height, as shown in Fig. 13 (c-d). Note that rotations of multiples of 120 degrees always end up within the RGB cube, which is why this artifact does never occur at - 120, 0 and 120 degrees. Future work should further investigate the extent to which this discrepancy problematic in practice and look into alternative solutions.
Effect of hue rotations Flowers-102 102
Effect of hue rotations Flowers-104 102
Text eccurerury 180)
50-
Text accurasy 1907
500
60-
40
20
20
CE-3
C2-2
C2-43
C2-5
C2-6
CE-7
C2-48
CB-9
CE-10
1150
-100
50
300
150
155
-100
-50
50
100
150
Testitime hue shift (?)
Tesh hue shirt
(a) Test-time huc shift with reprojection.
(h) Test-time huc shift with reprojection.
Effect of hue rotations Flowers-102
Effect of hue rotations Flowers-102
60
60-
Text accuarr (80
Text accuracy 192
30
30
-1.0
-1.00
-50
50
300
150
-150
-1.00
-50
50
100
150
Test-kime shift (*)
Twstdime shift (?)
(c) Test-time huc shift without reprojection.
(d) Test-time huc shift without reprojection.
Figure 13: The effect of the number of hue rotations in color equivariant convolutions downstream performance. More rotations increases robustness test-time hue shifts. Note that (a-b) the peaks not of equal height due to clipping effects near the boundaries of the RGB cube. This artifact disappears when the test-time hue shift is also applied without reprojection, resulting in peaks of equal height (c-d).
20


Reviews:
[{"review_id": "mHh8TnSeCH", "reviewer": "Reviewer_gSRU", "summary": "This paper proposes color equivariant convolutional networks (CE-CNNs), a novel convolutional neural network architecture that achieves equivariance to hue changes.&#x20;\n\nThey introduce color equivariant convolutions that apply a discrete set of hue rotations to the convolution filters during the forward pass. This makes the network output invariant to corresponding hue changes in the input image.\nThey propose a group coset pooling layer that pools feature maps over the group of hue transformations to achieve invariance.\nThey evaluate CE-CNNs on several image classification datasets, showing improved robustness to hue changes at test time compared to regular CNNs. The method also improves absolute classification performance in some cases.\nOverall, the paper presents a novel and intuitive technique to build invariance to hue changes into CNNs. The evaluations demonstrate its advantages over standard networks, especially under shifted hue distributions between training and test.", "questions": "1.  The amount of color jitter augmentation during training seems to significantly impact performance. Could you do an ablation or analysis to tease apart the direct benefits of the color equivariant convolutions versus the data augmentation?\n2.  Have you evaluated the method on larger-scale datasets like ImageNet? Results on more complex data could better demonstrate the scalability.\n3.  The choice of 3 discrete rotations seems arbitrary. Can you analyze the impact of the group size to help guide selection? Are there benefits beyond 3 rotations?", "limitations": "1.  The comparison to only a ResNet baseline limits the conclusions on the benefits of the proposed method. Comparisons to other approaches could provide useful context.\n2.  The long-term impacts of building color equivariance into models is unclear. Discussion of downstream effects on fairness and interpretability could be beneficial.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "This paper introduces a clever yet intuitive technique to make convolutional neural networks invariant to hue changes in the input image. The core idea is to apply discrete hue rotations to the convolution filters during the forward pass, essentially \"baking in\" robustness to color changes.\n\nThe paper is clearly written and easy to follow. The authors motivate the problem well, explain their proposed method succinctly, and provide thorough experimentation across image datasets. The visualizations offer useful insights, confirming that the networks learn consistent features across hues.\n\nOverall, I found this to be an original and significant contribution. Invariance to hue shifts is a practical problem, and this paper tackles it through an elegant approach that outperforms regular CNNs. The concept of encoding transformations into convolutions seems powerful. While not the flashiest technique, the method is thoughtful, principled, and achieves strong results. The paper is presented clearly and comprehensively, making the ideas accessible. In summary, this is a high quality paper with both theoretical and practical value.", "weaknesses": "-   The method is demonstrated on image classification, but it's unclear how well it would generalize to other tasks like detection or segmentation. Additional experiments on other applications could strengthen the claims.\n-   The ablation study on number of hue rotations suggests performance varies across different shifts. It would be useful to dig deeper into why - is it an artifact of how shifts are applied? Better understanding this could improve results further.\n-   The approach encodes discrete hue rotations. An interesting extension could be supporting continuous rotations for finer-grained equivariance.\n-   The comparisons to \"grayscale\" networks should be interpreted carefully, as removing color information entirely handicaps models. Comparisons to networks pre-trained onImagenet may be more meaningful.\n-   The Flowers-102 experiments indicate the method doesn't help much on datasets without color bias. Analyzing when color equivariance helps or hurts could guide adoption.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We highly appreciate your helpful comments and the recognition of our efforts. Please find a point-by-point response to your remarks below.\n\n---\n\n**Weaknesses:**\n> 1. The method is demonstrated on image classification, but it's unclear how well it would generalize to other tasks like detection or segmentation. Additional experiments on other applications could strengthen the claims.\n\nThis is true. yet, given the limited space we could only focus on image classification. Image classification models, including the tested ResNets, are commonly used as backbones for other applications. We will publish our code where CEConv and CE-ResNets are implemented as easy to use plug-and-play modules such as to accommodate for exploring other relevant tasks.\n\n> 2. The ablation study on number of hue rotations suggests performance varies across different shifts. It would be useful to dig deeper into why - is it an artifact of how shifts are applied? Better understanding this could improve results further.\n\nGreat question! We analyzed this inconsistency and found that they results from the clipping caused by occasionally rotating outside of the RGB cube and projecting it back. While the values are now not exact; the general trend for improvement still holds: albeit no longer exact, but approximately the same. See also our answer to Reviewer 81VY. \n\nWe will add this analysis and conclusions to the paper; and add a warning about \"color clipping\" in the source code. \n\n> 3. The approach encodes discrete hue rotations. An interesting extension could be supporting continuous rotations for finer-grained equivariance.\n\nDefinitely. Such “steerable” hue equivariance method would however require a quite different method from the proposed G-CNN-based approach. As this is an interesting direction to explore, we will add this idea to our discussion of future work.\n\n> 4. The comparisons to \"grayscale\" networks should be interpreted carefully, as removing color information entirely handicaps models. Comparisons to networks pre-trained on Imagenet may be more meaningful.\n\nExactly - This is what hue-equivariance aims to remedy. See \"Grayscale\" in Fig 1 of the paper. We will better emphasis this.\n\n> 5. The Flowers-102 experiments indicate the method doesn't help much on datasets without color bias. Analyzing when color equivariance helps or hurts could guide adoption.\n\nThis is indeed an important analysis, which we have looked into by investigating the color selectivity of CNN neurons when trained on different datasets (see Fig. 3 in the main paper). Color equivariance appears to be most beneficial whenever color selective neurons are learned - this is mostly the case for colorful datasets, such as Flowers-102. We will further clarify this in the text.\n\n---\n\n**Questions**\n\n> 1. The amount of color jitter augmentation during training seems to significantly impact performance. Could you do an ablation or analysis to tease apart the direct benefits of the color equivariant convolutions versus the data augmentation?\n\nCertainly - we provide an ablation study on the strength of color jitter augmentation in appendix D. Figure 5 shows that training with color jitter improves both the baseline and the hue equivariant models, however the equivariant model needs less jitter to achieve good generalization and achieves a higher accuracy overall. The reason for this is that the equivariant architecture only requires hue augmentation \"between\" the discrete rotations that it is already robust to, as opposed to the full scale of hue shifts for the baseline architecture. We will elaborate on this in the paper.\n\n> 2. Have you evaluated the method on larger-scale datasets like ImageNet? Results on more complex data could better demonstrate the scalability.\n\nWe have performed experiments on Imagenet; the results are reported in Table 1 in the main text, as well as in Table 1 in the supplementary material. We find that also on more complex datasets color equivariance is beneficial, though model capacity is of bigger importance. The hybrid color equivariant architectures where only the first $s$ stages are equivariant demonstrate the best of both worlds, i.e. equivariance to hue shifts, while also allowing a bigger network width for the same number of parameters compared to a fully equivariant architecture.\n\n> 3. The choice of 3 discrete rotations seems arbitrary. Can you analyze the impact of the group size to help guide selection? Are there benefits beyond 3 rotations?\n\nIn appendix D we provide an ablation study on the number of hue rotations. Figure 7 shows that increasing the number of rotations improves test-time hue shift performance. However, there is also a trade-off in model capacity as increasing the number of rotations increases the number of parameters in the model, as discussed in the Limitations section of the paper. The trade-off therefore depends on the amount of color vs. the complexity of the data - we will clarify this in the paper.\n\n---\n\n**Limitations**\n\n> 1. The comparison to only a ResNet baseline limits the conclusions on the benefits of the proposed method. Comparisons to other approaches could provide useful context.\n\nThank you for the suggestion. We have performed additional comparisons with two baselines, CIConv and AugMix, the results and discussion can be seen in the general author rebuttal, and will also be included in the paper.\n\n> 2. The long-term impacts of building color equivariance into models is unclear. Discussion of downstream effects on fairness and interpretability could be beneficial.\n\nImproving performance on tasks where color is a discriminative feature could affect humans that are the target of discrimination based on skin tone. CEConvs benefit datasets with long-tailed color distributions by increasing robustness to color changes, which could be useful in reducing a CNN's reliance on skin tone as a discriminating factor. We will add these considerations to the paper."}, {"review_id": "uevuIo5j5n", "reviewer": "Reviewer_81VY", "summary": "Paper proposes color-equivariant CNN layers by imposing equivariance to H_n (a discrete subgroup of SO(3)) in the RGB space which is imposes hue equivariance. Implementation follows the framework of Group-equivariant CNNs. Experiments show marginal improvements over standard CNNs for in-distribution test data but significant improvements when test data is hue-shifted.", "questions": "1. How is the hyperparameter $s$ chosen for different datasets in Table 1, especially for hue-shifted test dataset (assumed not known during training/validation)?\n2. Is the invariance captured by Grayscale-Z2CNN same as that considered in the paper, i.e., invariance to hue? Would a better preprocessing step be only considering a canonical representation in the hue dimension?\n3. In the color-imbalance experiments, were the CNNs trained with loss weighted according to the class imbalance? \n4. Except being computationally expensive, are there other issues with using $H_n$ for higher n? Ablation in the Appendix seem to show better robustness with higher n, and also good in-distribution performance (at 0 hue shift). So I am not sure why $H_3$ alone was considered in the main paper.\n5. It would be helpful to add a few lines of summary on the ablation studies in Sec 4.2.\n6. Can experiments in Section 4.1 include CNN+jitter as it seems to provide competitive performance in Section 4.2?\n\n\n**AFTER REBUTTAL**\n\nAuthors have addressed most of my concerns, particularly, adjusted the claims regarding color vs hue equivariance and added relevant baselines. I am increasing my score as I think this paper can be an important addition to an understudied topic. \n\nHowever, I am still concerned about the mathematical definition of hue-equivariance (via H_n group) since these transformations can map a color outside the unit cube and projecting back loses invertibility requirement of a group. I accept that this may not reduce performance in practical tasks (except certain inconsistencies pointed out by the authors), but I believe it is not mathematically correct.", "limitations": "Limitations should discuss lack of robustness to other color dimensions (e.g., illumination).", "rating": 6, "confidence": 4, "soundness": 2, "presentation": 4, "contribution": 3, "strengths": "1. Color equivariance in CNNs is a relatively less-studied but an important topic for robustness. The proposed idea of incorporating equivariance to hue transformations via rotations in the RGB space is novel. \n2. Experiments are setup well clearly showing when color equivariance is helpful vs color invariance vs no symmetry. Proposed approach shows improves over CNN even when in in-distribution test data, but major improvements come when test data is hue-shifted.", "weaknesses": "1. Definition of color equivariance considered in the paper seems to be restricted as it only considers the hue dimension. One of the motivations for incorporating color equivariance is for robustness to illumination changes which I do not think is guaranteed here. A general definition of color-equivariance should consider other dimensions. Maybe the claims are better justified if Hue-equivariance is emphasized in the title/introduction/method name, etc.\n2. The definition of hue-equivariance is not precise in the paper. Ideally, it should include all rotations in the RGB space (i.e., SO(3)), but also consider the fact that many of these rotations take the color values out of the RGB space (unit cube). In general, this issue occurs for the discrete subgroup $H_n$ as well. Simply projecting the color values back into the RGB space does not work as it breaks the invertibility property of these transformations. \n3. Experiments compare with a standard CNN (+grayscale) as baseline. Other baselines can be included, for example [1], that considers invariance to illumination color/intensity. \n4. Experiments in the main paper only consider the group $H_3$ (i.e., 3 rotations in the RGB space), which seems limited in robustness, as shown in Figure 1 without jitter augmentations. \n\n\nReferences:\n\n[1] Lengyel, Attila, et al. \"Zero-shot day-night domain adaptation with a physics prior.\" _Proceedings of the IEEE/CVF International Conference on Computer Vision_. 2021.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Many thanks for the helpful comments, please find our detailed response to your questions and remarks below:\n\n---\n**Weaknesses:**\n> 1. Definition of color equivariance considered in the paper seems to be restricted as it only considers the hue dimension. [...]\n\nYes! While hue is arguably the most fundamental property of color, others such as saturation and brightness are also important. We will refocus the title, introduction, and method sections to be on \"hue-equivariance\". In the conclusion and future work we will clarify limitations and possible extensions.\n\n> 2. The definition of hue-equivariance is not precise in the paper. Ideally, it should include all rotations in the RGB space (i.e., SO(3)), but also consider the fact that many of these rotations take the color values out of the RGB space (unit cube). [...]\n\nAgreed. To clarify, for an operation to be hue-equivariant, it should be equivariant only to all rotations around the [1,1,1] diagonal (not necessarily full SO(3)). It is indeed true that pixel values near the cube borders can fall outside of the RGB cube and need to be projected back. In practice, this caused some inconsistencies over varying hue (this explains inconsistent peaks in Fig 7 of supplemental material). We will better clarify this in the method and limitations sections. See also our answer to Reviewer gSRU.\n\n> 3. Experiments compare with a standard CNN (+grayscale) as baseline. Other baselines can be included, for example [1], that considers invariance to illumination color/intensity.\n\nAgreed. We added [1], as well as AugMix - please find the results above.\n\n> 4. Experiments in the main paper only consider the group  (i.e., 3 rotations in the RGB space), which seems limited in robustness [...].\n\nDue to space limitations we had to restrict our analyses in the main paper to 3 rotations - in appendix D we provide an ablation study on the number of hue rotations. Figure 7 shows that increasing the number of rotations improves test-time hue shift performance. However, there is also a trade-off in model capacity as increasing the number of rotations increases the number of parameters in the model. We will add this to the paper.\n\n---\n\n**Questions:**\n> 1. How is the hyperparameter chosen for different datasets in Table 1, especially for hue-shifted test dataset?\n\nDue to space limitations, Table 1 in the main text shows only the best networks. The same value for $s$ is used for both the original and hue-shifted test set and we provide the full results in Table 1 of supplementary material - we will better clarify this in the main paper text and caption. Note that for any fixed $s$ the CE networks outperform the baseline model on the hue-shifted test set of almost all datasets.\n\n> 2. Is the invariance captured by Grayscale-Z2CNN same as that considered in the paper, i.e., invariance to hue? Would a better preprocessing step be only considering a canonical representation in the hue dimension?\n\nInterestingly, the grayscale invariance makes objects less distinct from their background (see fig 1 of the original paper). A canonical representation in the hue dimension would indeed retain background distinction and be hue-shift invariant. Yet, such a mapping is non-trivial as it depends on the semantic object/background colors. Such a canonical representation approach is similar to white balancing for achieving color constancy (see also our answer to Reviewer w9HG) . As such, finding a \"canonical hue representation\" is orthogonal to our equivariance approach and reminds us of a spectral variant of \"spatial transformer networks\" (ref below). \n\nWe will better explain the relationship with color constancy and white-balancing in related work; and add this alternative approach to the conclusion and future work.\n\n- Jaderberg, Max, Karen Simonyan, and Andrew Zisserman. \"Spatial transformer networks.\" Advances in neural information processing systems 28 (2015).\n\n> 3. In the color-imbalance experiments, were the CNNs trained with loss weighted according to the class imbalance?\n\nThis was not the case in the original experiments. We have re-run the experiments with a weighted loss and observed no significant differences to an unweighted loss. We will include both results in the paper / supp. mat.\n\n> 4. Except being computationally expensive, are there other issues with using $H_n$ for higher $n$?\n\nIncreasing $n$ increases both computation cost as well as the model’s parameter count. As such, in fair experiments where the total number of parameters is to be equalized between tested models, there is a trade-off between increasing $n$ and increasing the number of channels. Increasing $n$ by too much therefore hurts the model capacity, as can be seen by the slightly decreasing performance at 0-deg. in Figure 7 of the supp. mat. We will further clarify this in the paper.\n\n> 5. It would be helpful to add a few lines of summary on the ablation studies in Sec 4.2.\n\nDefinitely! We will add the following to our camera-ready version of the paper: “In short, we find a) that hue equivariant networks require less intense color jitter augmentation to achieve the same test-time hue shift robustness and accuracy, b) that removing group coset pooling breaks hue invariance, and c) that increasing the number of hue rotations increases robustness to test-time hue shifts.”\n\n> 6. Can experiments in Section 4.1 include CNN+jitter as it seems to provide competitive performance in Section 4.2?\n\nWe included CNN+jitter - please find the results above. For long tailed MNIST, adding jitter makes solving the classification problem prohibitive, as color is required. For biased MNIST, performance decreases for small and improves for large  $\\sigma$, with CEConv still performing best."}, {"review_id": "F8B806CVUW", "reviewer": "Reviewer_9e3n", "summary": "The authors introduce a color equivariant convolutional neural network. To achieve this the authors represent the image in HSV format, and achieve hue equivariance using methods for rotational equivariance. This is possible since hue can be represented by an angle. The authors show that the proposed approach out performs standard CNNs and color invariant CNNs when there is a hue shift between the train and test set.", "questions": "* regarding equation 3. I believe the correlation should be between the feature maps and $C^{l+1}$ filters [7]. I think it could be clarifying to write that $c$ is the input channel and $i$ is output channel.\n* regarding equation 6: I'm not sure I understand how does H_n(k) acts on \\psi_c? In the case of group 2D rotation equivariance, the filters are transformed by in plane rotations. It is not clear to me how one can perform a 3D rotation of a filter with an arbitrary number of channels.\n* It seems to me that the network is designed for local hue equivariance, can the authors clarify the benefit of this over global hue equivariance (i.e, performing hue shift on the input image then processing all inputs with the same cnn, and combining representations at the final layer to get a hue-equivariant representation)?\n* Does computational expense improve if input images are converted to HSV before being passed into the network? In this case, I expect hue equivariance could be achieved by discretization of the 2D rotation group rather than a 3D transformation.\n* Have the authors experimented with finer/coarser discretizations of the hue/rotation group\n", "limitations": "* Limitations -- in particular the issue of computational cost -- are communicated", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "* Originality: The presented method of building a color equivariant CNN appears to be original. \n* Quality: The work appears to be of fairly good quality. \n* Clarity: The paper is well written. \n* Significance: The observation that color equivariance can be achieved by identifying hue with the rotation group is interesting. The results show the proposed approach leads to improved performance when there is a color based domain shift.", "weaknesses": "* Quality: I have some questions about the mathematical presentation, and experiment design (see questions).\n* Clarity: Some aspects were unclear to me, due to presentation or motivation (see questions)", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Great thanks for your helpful comments and appreciation for our work. Please find detailed answers to your questions and remarks below:\n\n---\n\n**Questions:**\n\n> 1. regarding equation 3. I believe the correlation should be between the feature maps and $C^{l+1}$ filters [7]. I think it could be clarifying to write that is the input channel and  is output channel.\n\nThank you, this is indeed a mistake from our side. We will correct / clarify in the paper.\n\n> 2. regarding equation 6: I'm not sure I understand how does H_n(k) acts on \\psi_c? In the case of group 2D rotation equivariance, the filters are transformed by in plane rotations. It is not clear to me how one can perform a 3D rotation of a filter with an arbitrary number of channels.\n\nExcellent question - as each \"pixel\" of a filter in a hidden layer lives in a high-dimensional space, applying 3D rotations is indeed not possible. 3D rotations are only applied at the input layer, whereas in hidden layers cyclic permutations are performed to retain equivariance throughout the network. We will better clarify this in the paper.\n\n> 3. It seems to me that the network is designed for local hue equivariance, can the authors clarify the benefit of this over global hue equivariance (i.e, performing hue shift on the input image then processing all inputs with the same cnn, and combining representations at the final layer to get a hue-equivariant representation)?\n\nInteresting alternative - we have briefly considered this, but initial experiments did not yield promising results and therefore did not further pursue this approach. The theoretical benefit of local over global hue invariance is that multiple objects in one image can be recognized invariantly in any combination of hues - this indeed appears to be a useful property. We will include a note on local vs. global equivariance in the Discussion.\n\n> 4. Does computational expense improve if input images are converted to HSV before being passed into the network? In this case, I expect hue equivariance could be achieved by discretization of the 2D rotation group rather than a 3D transformation.\n\nThanks, we have indeed considered this option. Converting an input to the HSV space introduces a discontinuity between 359-0 degrees, which can lead to suboptimal results when used as an input to a neural network. Alternatively, color spaces such as LAB could be used, where a hue shift can indeed be modeled as a 2D rotation. Eventually we did not see any benefit in 2D rotations, as the additional compute of 3D rotations is negligible on the scale of deep NNs, and this way the network could work directly on RGB images.\n\n> 5. Have the authors experimented with finer/coarser discretizations of the hue/rotation group?\n\nThis is indeed something we have looked at - in appendix D we provide an ablation study on the number of hue rotations, i.e. the discretization of the hue group. Figure 7 shows that increasing the number of rotations improves test-time hue shift performance. However, there is also a trade-off in model capacity as increasing the number of rotations increases the number of parameters in the model. This should be counteracted by reducing the width (number of channels) in the architecture to keep the number of parameters equal, thereby somewhat reducing model capacity."}, {"review_id": "BVTR5dgsYa", "reviewer": "Reviewer_w9HG", "summary": "This paper questions the importance of color variations for neural classifiers and proposes to use color-equivariant architectures in the case of unbalanced datasets.\nTo demonstrate the validity of the presented approach, the authors conduct experiments both on synthetic controlled datasets and on common object recognition benchmarks.\nAs the experiments show, the injection of color-equivariant layers leads to a slight improvement on almost all common benchmarks when the performance is measured on the original test set but the advantage of the presented method becomes more evident when the test data is corrupted with hue shifts.", "questions": "1. I ask the authors to discuss the weaknesses stated above. The main thing I am interested in is the usage of robust baseline models.\n\n1. While preserving the number of trainable parameters is valuable, I wonder if the network throughput remains the same for color-equivariant architectures in comparison with the baselines. If this is not the case, how significant is the decrease?", "limitations": "Limitations are addressed adequately.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "This paper studies an interesting and underinvestigated question of the importance of color representation for neural networks.\nThe submission is easy to read, and the motivation is well explained in the example of the Flowers dataset. \nThe authors have conducted a significant number of experiments to support their claims.\nAdditional strength is that the demonstrated performance improvement is achieved without increasing the number of trainable parameters (line 238).", "weaknesses": "1. While the idea of extending equivariance from geometric to photometric transformations is definitely interesting, the submitted manuscript, unfortunately, focuses on the only type of such transformations, i.e. hue shifts. Despite the case of the Flowers dataset is a perfect fit for this transformation, the authors do not discuss other use cases when this type of equivariance may be interesting in practice and just mention \"accidental recording conditions\" (line 3). For other datasets, hue shifts seem less meaningful, and the better robustness of the proposed CE-ResNets to such shifts at test time is explained by the fact the architecture was just intentionally designed for this scenario. Taking this into account, I find the scope of the paper a bit limited.\n\n1. In addition to being limited in the number of considered photometric transformations, the paper also considers a single task of object recognition. I would encourage the authors to consider other tasks as well, e.g. unsupervised domain adaptation.\n\n1. While the authors claim their approach makes networks more robust to test time corruptions (Tab. 1), they do not demonstrate other baselines aiming to provide robust outputs, e.g. adversarially robust models.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank reviewer w9HG for the helpful remarks and interest in our work. Please find a detailed response to your questions and suggestions below:\n\n---\n\n**Weaknesses:**\n\n> 1. While the idea of extending equivariance from geometric to photometric transformations is definitely interesting, the submitted manuscript, unfortunately, focuses on the only type of such transformations, i.e. hue shifts. [...] Taking this into account, I find the scope of the paper a bit limited.\n\nWe regret that we did not clarify this better. A hue-shift is truly fundamental: simply changing the light bulb or the time of day yields a different illumunant and thus a hue-shift. Being robust to hue-shifts is a still unsolved, important, and an active research direction (color constancy). See, eg, the following recent (top venue) publications below. \n\nWe will add these citations and motivation to related work. \n\n- Mahmoud and Brown. \"What else can fool deep learning? Addressing color constancy errors on deep neural network performance.\" ICCV. 2019.\n- Li, Bing, et al. \"Ranking-based color constancy with limited training samples.\" IEEE Transactions on Pattern Analysis and Machine Intelligence (2023).\n- Ono, Taishi, et al. \"Degree-of-linear-polarization-based color constancy.\" CVPR. 2022.\n\n\n\n\n> 2. In addition to being limited in the number of considered photometric transformations, the paper also considers a single task of object recognition. I would encourage the authors to consider other tasks as well, e.g. unsupervised domain adaptation.\n\nThis is true. Unfortunately, given the limited time of the rebuttal, we were not able to run additional tasks.\n\nPlease also see our answer to Reviewer gSRU:  We chose for  image classification because classification models, including the tested ResNets, are also commonly used as backbones for other applications. While we acknowledge that additional experiments on extra applications would strengthen the argument for generalization of our method, we instead opted, given limited compute resources, for a more thorough evaluation on multiple classification datasets. \n\n> 3. While the authors claim their approach makes networks more robust to test time corruptions (Tab. 1), they do not demonstrate other baselines aiming to provide robust outputs, e.g. adversarially robust models.\n\nAgreed. Additional comparisons with two baselines, CIConv and AugMix, can be found in the general author rebuttal, and will also be included in the paper.\n\n---\n\n**Questions:**\n\n> 1. I ask the authors to discuss the weaknesses stated above. The main thing I am interested in is the usage of robust baseline models.\n\nSee above.\n\n> 2. While preserving the number of trainable parameters is valuable, I wonder if the network throughput remains the same for color-equivariant architectures in comparison with the baselines. If this is not the case, how significant is the decrease?\n\nDefinitely. We do already discuss some compute efficiency in the Implementation section of the paper where the same number of channels, CEConv performs a factor $\\frac{|H_n|^2}{k^2} + |H_n|$ more MACs. However, the true MAC increase is lower since the number of channels in a CEConv network is reduced to maintain the same parameter count as the baseline.\n\n Unfortunately, given the limited time for the rebuttal, we could not yet provide these numbers in the rebuttal, but we will include these measurements in the final paper.\n\n\n"}]

Author rebuttal:
Thank you for the highly detailed and constructive feedback! In this rebuttal we address all points raised by the review team, leading to multiple improvements, including: evaluations against relevant baseline methods, clarifications, and insights. Below, we answer individual questions per reviewer in a point by point fashion. All answers will appear in the camera-ready version of the submission.

For the ColorMNIST experiments, we have performed additional experiments with color jitter augmentation. An updated version of Figure 2 is included in the rebuttal PDF. In both the "longtailed" setting (Figure 2a) and the low $\sigma$ settings of the "biased" setting (Figure 2b), color jitter hurts performance since the tasks require color as a discriminative feature. Only for the high $\sigma$ settings of the "biased" setting, color jitter improves performance since it induces color invariance. However, CECNN captures the best of both worlds: CECNN improves performance over the baseline Z2CNN in both tasks, regardless of $\sigma$.

For the image classification experiments, as multiple reviewers have suggested additional baselines for comparison, we will present the results here. We have included experiments with CIConv-W [1], as well as AugMix [2] - please find the results below. “Baseline” denotes a vanilla ResNet-18/44, [1] and [2] use the same ResNet with color invariant (CIconv) and equivariant (CEconv) convolutions, respectively. Despite discarding color information [1] indeed proves to be a robust baseline to hue shifts, though performing poorly on CIFAR due to the loss of detail by the edge detector. [2] slightly improves performance on both the original and hue-shifted test set, but its robustness lacks behind that of CEConv. Finally, for completeness we have trained CEConv with [2], yielding good performance on both test sets.

[1] Lengyel, Attila, et al. "Zero-shot day-night domain adaptation with a physics prior." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2021.

[2] Hendrycks, Dan et al. "AugMix: A Simple Data Processing Method to Improve Robustness and Uncertainty." Proceedings of the International Conference on Learning Representations (ICLR). 2020"

| Original test set | Caltech-101 | CIFAR-10 | CIFAR-100 | Flowers-102 | Oxford-IIIT Pet | Stanford Cars | STL-10 |
|:------------------|:-----------------|:-----------------|:------------------|:-----------------|:------------------|:-----------------|:-----------------|
| Baseline          | 71.61 $\pm$ 0.88 | 93.69 $\pm$ 0.16 | 71.28 $\pm$ 0.20 | 66.79 $\pm$ 0.89 | 69.87 $\pm$ 0.57  | 76.54 $\pm$ 0.10 | 83.80 $\pm$ 0.36 |
| Baseline + Jitter | 73.93 $\pm$ 0.73 | 93.03 $\pm$ 0.16 | 69.23 $\pm$ 0.44 | 68.75 $\pm$ 1.50 | 72.71 $\pm$ 0.68  | 80.59 $\pm$ 0.36 | 83.91 $\pm$ 0.38 |
| CEConv            | 70.16 $\pm$ 1.05 | 93.71 $\pm$ 0.26 | 71.37 $\pm$ 0.24 | 68.18 $\pm$ 0.45 | 70.24 $\pm$ 0.79  | 76.22 $\pm$ 0.19  | 84.24 $\pm$ 0.48 |
| CEConv + Jitter | 73.58 $\pm$ 0.68 | 93.51 $\pm$ 0.10 | 71.12 $\pm$ 0.57 | 74.17 $\pm$ 0.49 | 73.29 $\pm$ 0.63  | 79.79 $\pm$ 0.37  | 84.16 $\pm$ 0.10 |
| CIConv-W [1] | 72.85 $\pm$ 1.12 | 75.26 $\pm$ 0.57 | 38.81 $\pm$ 0.66 | 68.71 $\pm$ 0.29 | 61.53 $\pm$ 0.53  | 79.52 $\pm$ 0.42  | 80.71 $\pm$ 0.27 |
| CIConv-W [1] + Jitter | **74.38 $\pm$ 0.43** | 77.49 $\pm$ 0.53 | 42.27 $\pm$ 0.56 | 75.05 $\pm$ 0.39 | 64.23 $\pm$ 0.51  | *81.56 $\pm$ 0.32* | 81.88 $\pm$ 0.24 |
| Baseline + AugMix [2] | 71.92 $\pm$ 0.95 | 94.13 $\pm$ 0.22 | **72.64 $\pm$ 0.27** | 75.49 $\pm$ 0.24 | **76.02 $\pm$ 0.51**  | **82.32 $\pm$ 0.07**  | 84.99 $\pm$ 0.24 |
| CEConv + AugMix [2]   | 70.74 $\pm$ 1.12 | **94.22 $\pm$ 0.16** | 72.48 $\pm$ 0.18 | **78.10 $\pm$ 0.50** | 75.90 $\pm$ 0.22  | 80.81 $\pm$ 0.27  | **85.46 $\pm$ 0.30** |

| Hue-shifted test set | Caltech-101 | CIFAR-10 | CIFAR-100 | Flowers-102 | Oxford-IIIT Pet | Stanford Cars | STL-10 |
|:------------------|:-----------------|:-----------------|:-----------------|:-----------------|:------------------|:-----------------|:-----------------|
| Baseline | 51.14 $\pm$ 0.71 | 85.26 $\pm$ 0.56 | 47.01 $\pm$ 0.38 | 13.41 $\pm$ 0.35 | 37.56 $\pm$ 0.76  | 55.59 $\pm$ 0.74 | 67.60 $\pm$ 0.56 |
| Baseline + Jitter | 73.61 $\pm$ 0.60 | 92.91 $\pm$ 0.17 | 69.12 $\pm$ 0.47 | 68.44 $\pm$ 1.60 | 72.30 $\pm$ 0.49  | 80.65 $\pm$ 0.36 | 83.71 $\pm$ 0.35 |
| CEConv | 62.17 $\pm$ 1.01 | 90.90 $\pm$ 0.25 | 59.04 $\pm$ 0.45 | 33.33 $\pm$ 0.38 | 54.02 $\pm$ 1.34  | 67.16 $\pm$ 0.58 | 78.25 $\pm$ 0.51 |
| CEConv + Jitter | 73.57 $\pm$ 0.75 | **93.39 $\pm$ 0.08** | **71.06 $\pm$ 0.53** | 73.86 $\pm$ 0.39 | **72.94 $\pm$ 0.56**  | 79.79 $\pm$ 0.34 | **84.02 $\pm$ 0.14** |
| CIConv-W [1] | 71.92 $\pm$ 1.11 | 74.88 $\pm$ 0.54 | 37.09 $\pm$ 0.74 | 59.03 $\pm$ 0.62 | 60.54 $\pm$ 0.46  | 78.71 $\pm$ 0.33 | 79.92 $\pm$ 0.25 |
| CIConv-W [1] + Jitter | **74.40 $\pm$ 0.55** | 77.28 $\pm$ 0.54 | 42.30 $\pm$ 0.48 | **75.66 $\pm$ 0.27** | 63.93 $\pm$ 0.42  | **81.44 $\pm$ 0.26** | 81.54 $\pm$ 0.21 |
| Baseline + AugMix [2] | 51.82 $\pm$ 0.60 | 88.03 $\pm$ 0.26 | 51.39 $\pm$ 0.19 | 15.99 $\pm$ 0.28 | 48.04 $\pm$ 0.74  | 68.69 $\pm$ 0.73 | 72.19 $\pm$ 0.45 |
| CEConv + AugMix [2] | 62.29 $\pm$ 0.97 | 91.68 $\pm$ 0.21 | 60.75 $\pm$ 0.24 | 41.43 $\pm$ 0.97 | 62.27 $\pm$ 0.81  | 73.59 $\pm$ 0.30 | 80.17 $\pm$ 0.15 |

Meta-review: All reviewers find that the paper meets NeurIPS standards. Initially, there were concerns regarding unsupported claims and confusion in the definition. During the rebuttal, these concerns have been addressed and several reviewers increased their score.
Decision: Accept (poster)