Learning Color Equivariant Representations
Authors: Yulong Yang
Felix O'Mahony
Christine Allen-Blanchette
Venue: ICLR 2025 Poster (2025)
Keywords: Equivariant Neural Network
Geometric Deep Learning
Group Convolution
Primary area: interpretability and explainable AI
TL;DR: We achieve color equivariance in a convolutional neural network architecture by identifying hue and saturation transformations with the 2D rotation and 1D translation groups.

Abstract: In this paper, we introduce group convolutional neural networks (GCNNs) equivariant to color variation. GCNNs have been designed for a variety of geometric transformations from 2D and 3D rotation groups, to semi-groups such as scale. Despite the improved interpretability, accuracy and generalizability of these architectures, GCNNs have seen limited application in the context of perceptual quantities. Notably, the recent CEConv network uses a GCNN to achieve equivariance to hue transformations by convolving input images with a hue rotated RGB filter. However, this approach leads to invalid RGB values which break equivariance and degrade performance. We resolve these issues with a lifting layer that transforms the input image directly, thereby circumventing the issue of invalid RGB values and improving equivariance error by over three orders of magnitude. Moreover, we extend the notion of color equivariance to include equivariance to saturation and luminance shift. Our hue-, saturation-, luminance- and color-equivariant networks achieve strong generalization to out-of-distribution perceptual variations and improved sample efficiency over conventional architectures. We demonstrate the utility of our approach on synthetic and real world datasets where we consistently outperform competitive baselines.

Published as a conference paper at ICLR 2025
LEARNING COLOR EQUIVARIANT REPRESENTATIONS
Yulong Yang Felix O'Mahony 2,
Christine Allen-Blanchette '-t
Princeton University. Princeton, USA yulong. wwwnnggprnnnetoncc... cal 1118prinnettoo edu
2 University of Oxford, Oxford, UK
fel feli Felixomahonye@ge gmail. com,
ABSTRACT
In this paper, we introduce group convolutional neural networks (GCNNs) equiv- ariant to color variation. GCNNs have been designed for variety of geometric transformations from 2D and 3D rotation groups, to semi-groups such as scale. Despite the improved interpretability, accuracy and generalizability of these ar- chitectures, GCNNs have seen limited application in the context of perceptual quantities. Notably, the recent CEConv network uses a GCNN to achieve equivari- ance to hue transformations by convolving input images with a hue rotated RGB filter. However, this approach leads to invalid RGB values which break equivari- ance and degrade performance. We resolve these issues with a lifting layer that transforms the input image directly, thereby cireumventing the issue of invalid RGB values and improving equivariance error by over three orders of magnitude. Moreover, we extend the notion of color equivariance to include equivariance to saturation and luminance shift. Our hue-, saturation-, luminance- and color- equivariant networks achieve strong generalization to out-of-distribution perceptual variations and improved sample efficiency over conventional architectures. We demonstrate the utility of our approach on synthetic and real world datasets where we consistently outperform competitive baselines.
INTRODUCTION
The tremendous progress of image classification in the last decade be readily attributed to the development of deep convolutional neural networks (Krizhevsky et al.) 2012) Simonyan & Zisserman
2014 He et al., 20166). The highly nonlinear mapping and large parameter space does not lend itself to interpretation easily, but even in early networks, representations of geometry and color were observed and recognized for their importance (Krizhevsky et al. 2012 Lenc Vedaldi 2015). While a large body of literature has worked to improve the robustness of neural networks to geometric
transformations (Bruna & Mallat 2013) Cohen Welling 2016
Gilmer et al.) 2017 Qi al. 2017
Hinton et al.) |018] Esteves et al. 2018a] Greydanus et al. 2019 Zhong & Allen-Blanchedd Blanchette 2025
improving their robustness to perceptual variation has garnered considerably less attention.
A commonly used heuristic for improving network robustness to color variation is to perform mean subtraction and normalization on training set examples. This approach can work well when the training and testing datasets are drawn from the same distribution; however, for data that are collected at different points in time, or with different sensors, this is not likely to be the case. Consider, for example, the case of medical imaging where images of tissue samples collected from different labs (or from the same lab at different points in time) may have different characteristics due to variability in data collection protocols or imaging processes (Veta et al. 2016). This variability presents significant challenge for convolutional neural networks which have been found to be sensitive to color variations even the level of individual blocks (Engilberge et al.) 2017). Moreover, when presented with color perturbations, conventional networks exhibit a significant drop in classification performance (De & Pedersen 202II. One approach mitigate the effect of color variation is to ignore color entirely by enforcing color invariance. This can be done by converting input images to grayscale, or enforcing representation similarity across color as is done in Pakzad et al, (2022). This approach has its own challenges, however, since in many domains, color is an important cue
Equal contribution. Corresponding author.

---

Published as a conference paper at ICLR 2025
f(x)
p p(g.z)
Lo f(x))
Imput
Lifting
Saturation
Tuue
(a) Hue equivariance
(b) Lifting layer
Figure 1: Color-equivariant network. (a) The equivariance of our que-evuivar model is illustrated by the commutativity of the (hue) rotation and neural network mapping. hue rotation of 90 in the input image space (top-left to bottom-left), results in feature map rotation at each layer of the network (top-right to bottom-right). Corresponding feature maps are highlighted with blue border. (b) An input image (left) is lifted to the hue-saturation group (right) by shifting its hue and saturation values. For comparison, we illustrate the CEConv lifting layer in Appendix C.I
for classification. Another approach is use dataset augmentation, a technique that can improve robustness in the presence of known transformations (Benton et al.) (0220.. This approach, however, requires extended training times and does not provide improved interpretability.
In this work, we address the challenge of neural network sensitivity to color variation by leveraging the geometric structure of color group convolutional neural network. Geometric deep learning has gained strong interest in recent years due to its ability to capture information in interpretable and generalizable representations; by using a group convolutional neural network, we inherit these characteristics for the context of color variation. We represent color in the hue saturation luminance (HSL) color space, and leverage the insight that the huc, saturation, and luminance can be modeled with geometric group structure. Using this approach, network retains color information in structured representation throughout the processing pipeline, thereby allowing color information to be used and/or discarded intentionally.
Recently, Color Equivariant Convolutional Networks (Lengyel et al. 2024), proposed a group convolutional neural network architecture for hue-equivariant representation learning. Most notably, CECоny identifies hue transformations with the 2D rotation group. However, the proposed lifting layer introduces invalid RGB values which break equivariance, and degrade performance. To resolve this issue, our lifting layer operates on the input image instead of the network filters, which produces an equivariant descriptor without projection induced artifacts. This seemingly minor change improves equivariance error by more than three orders of magnitude, and stabilizes network performance across discretizations. We also expand the notion of color equivariance to capture variation saturation and luminance. Specifically, group. With this identification, we propose group-equivariant network where variations in hue, saturation and luminance represented geometric transformations.
identify variations in saturation and luminance with the ID translation
RELATED WORK
Group Convolutional Neural Networks. The generalization performance of convolutional neural networks on image processing tasks is attributed, in part, to the equivariance of planar convolution to 2D translations. This insight has garnered considerable attention and led strong interest in the development of convolutional neural networks equivariant to other symmetry groups (Kondor & Trivedi 2018 Cohen et al. (2019)). Previous works introduce group convolutional networks for finite and continuous groups. A framework for finite group convolution was developed in Cohen & WWelingg (2016). and demonstrated for the 2D rotation and reflection groups. The authors achieve equivariance specific symmetry groups by convolving input images and feature maps with the group orbit of learned filter bank. In Worrall et al. (2017), the authors design for equivariance to the continuous 2D rotation group by constraining filter representations with circular harmonic structure.
2

---

Published as a conference paper at ICLR 2025
Group convolutional networks have also been designed for use in the context of scale symmetry. In Esteves et al. (2018b), the authors represent input images in log-polar coordinates where 2D rotation and scale transformations present 2D translations. To navigate the semi group structure of scale transformations, the authors in Worrall & Welling (2019) approximate the scale space as finite and use dilated convolutions in a group equivariant architecture. The SREN network proposed in [Sun & Blu (2022) designs for equivariance to the continuous 2D rotation and isotropic scaling group by constraining filter representations be the linear combination of windowed Fourier basis.
Group convolutional networks have also been designed for groups acting in higher dimensions. For 3D rotational symmetry, Thomas et al. (2018) and Esteves et al. (2019) use spherical harmonics based representation with 3D point clouds and spherical image inputs respectively; and Batzner al. (2022) introduce an equívariant network for transformations of the Fuclidean group in 3D. In Finzi et al. (2020), the authors introduce a framework for general Lie groups whose exponential map is surjective, and [Chhen al. (2019a) introduces framework for more general manifolds. By designing for known symmetries in the task, these works provide improved interpretability, training efficiency and generalizability over conventional convolutional networks.
Other group equivariance methods encourage transformation equivariance using a soft penalty term on representation dissimilarity (Lene & Vedaldi, 2013; Gupta et al. 2023). These methods are separate from the group convolutional neural network literature, and different from what we propose. Our model leverages the geometric structure of hue, saturation and luminance to design color-equivariant networks, bringing the benefits of group-equivariant networks to perceptual transformations.
Color Invariance. Several prior works attempt to mitigate the effect of color variation in image processing tasks, for example, Chong al. (2008), the authors represent images in color space where pixel values are invariant to changes in luminance for luminance invariant image segmentation, and in Pakzad al. (2022). the authors penalize changes in their latent representations due to color variation for color invariant skin lesion identification. While color invariance resolves the effect of color variation, color has been identified as an important cue in representation learning (Engilberge et al. 2017 De & Pedersen 2021) which is discordant with the goal of color invariance. In contrast to these approaches, our work leverages the geometric structure of hue, saturation and luminance to construct convolutional neural network equivariant to variations in these quantities by design.
Most similar to ours is the model proposed in Lengyel et al. (2024), a hue-equivariant network for improved performance in the presence of color variation. Our color-equivariant networks differ from the work proposed in Lengyel et al. (2024) in two important ways. First, we lift the input image instead of the filters of the first layer which circumvents the issue of invalid hue rotations suffered by the network in Lengyel et al. (002)); and second, we expand the notion of color equivariance by introducing networks that are equivariant to hue, saturation, and luminance.
PRELIMINARIES
In this section we describe our notation, and review definitions of the group action, equivariance, and group convolution.
Notation. We use fl denote the 1-th feature map (f to denote input image), and f to denote the j-th channel of that feature map. We use vi to denote the i-th filter of the I-th layer, and vh., to denote the j-th channel of that filter.
Group action. Adapted from Gallier & Quaintance 2020). Given a set X and a group G, the action of the group G' on X is function yp G x X X satisfying the following:
1. For all g, h E and all x E X, p(g. (h,, x)) = x(gh,x)
2. For all x e X, (pl1 æ) where E G is the identity element of G.
The set X is called (left) G-set.
Equivariance. Adapted from Gallier & Quaintance 2020). Given two G-sets X and Y, and group actions yp : G x + X and o G x Y Y, a function f X + Y is said to be equivariant, and only if for all X, and g E G, f(p(g, x)) ) = Φ(g, f()). When f is invariant the action of
3

---

Published as a conference paper at ICLR 2025
G, the group action ø is the identity map and we write f(p(g, æ)) f(z). The equivariance of our network to hue shifts is illustrated in Figure la
Group convolution. In a conventional CNN, the input to convolution layer I, denoted f': + "R" is convolved with set of Kl+1 filters, denoted vif z² - RK where i ranges from to K*-1, The result of the convolution can be written:
K²
- = If vl((2)
v), E EZ
The convolution of conventional CNNs is equivariant to the action of the group (Z² +), that is, the group formed by summing over the integers. The more general group convolution can be written:
R²
If' * k)) = ΣΣ A(h)))
HEGR=1
and is equivariant to the action of the group G.
METHOD
In this section we present our color-equivariant network. We begin by presenting definitions for the hue and saturation groups and their group actions, then define the lifting layers and group convolution layers for our color-equivarian networks.
Hue group and group action. In the HSL color space, hue is represented by angular position, and can therefore be identified with the 2D rotation group. As in group convolutional network (Cohen & Welling 2016), we consider finite group representation. Specifically, we identify elements of the discretized hue group, HN. where the subscript N indicates the order of (i.e. the number of elements in) the group, with those of the cyclic group CN. We prove HN is a group in Appendix A.
We define the action of hue group on HSL images, x' X where x 2² - R3, and functions on the discrete hue group, y E Y, where Z x HN - R-K, An element of the hue group acts on an HSL. image by the group action yo HN x X - X, which shifts the hue channel of the image. Concretely, for an HSL image X defined as the concatenation of hue, saturation and luminance channels, i.c., x = (I'h, Ts, 21), the action of an element h, of the hue group HN is given by
pn x) = ((r), + 14) (mod c), Fe, #1);
where (-)mod(.) denotes the modulus operation and is applied to the pixel value which ranges from 0 to c.
An element of the hue group acts on function on the discrete hue group by the group action dn : H x Y + Y, which "rotates" the function on the group. Concretely, for function on the discrete hue group H defined as the concatenation of functions f element h4 in the huc group N is given by
JN), the action of an
@n, f) = (f(1++)(mod N); N).-.., (mood N))-
The action of S on an input image, and dy, on feature map are shown in Figure 6a We prove 4 and Φn are group actions Appendix B.I
We expand the notion of color equivariance proposed in Lengyel al. (2024) to include equivariance to saturation shifts. In the HSL color space, saturation can be represented by a real number in the interval [0, 1]: we introduce two approximations to give the saturation space group structure. First, we observe that there a bijection between the real numbers and the open interval (0, 1), so
Saturation group and group action.
use the structure of the group (R, +). Second, consider a finite subset of the group, a necessary and commonly used practice both conventional CNNs (recall that the translation group is infinite) and group-equivariant CNNs such Worrall & Welling (2019). With these approximations, the discretized saturation group SNy is isomorphic to the integers with addition, (Z, +). We prove Sy is group in Appendix A.2
We define the action of the saturation group on HSL images, X E X where I Z - R3, and functions on the discrete saturation group, y Y, where y : 722 x SN - R An element of the
4

---

Published as a conference paper at ICLR 2025
Sat-7* Festure Map
Shifted Orginnl
Figure 2: Saturation-cquivarian feature maps. We illustrate the equivariance of our saturation- equivariant model. saturation shift in the input image space (top-left bottom-left), results in feature map translation at each layer of the network (top-right to bottom-right). Corresponding feature maps are highlighted with blue border.
saturation group acts on an HSL image by the group action ys SN X + X, which shifts the saturation channel of the image. Concretely, for HSL image E X defined the concatenation of hue, saturation and luminance channels, i.c., x = (I), T., 21), the action of an element s; of the saturation group SN is given by
(x( = (Kh, min (x, + 8i, 21).
(5)
An element of the saturation group acts function the discrete saturation group by the group action 6, Sy x Y + Y, which "translates" the function on the group. Concretely, for function f on the discrete saturation group Sy defined the concatenation of functions f (fi..... fN). the action of an element 8; in the saturation group SN is given by
(s f) = (f1+s..... fN 0.
(6)
The action of input image, and d. feature map shown in Figure 2 ds are group actions Appendix B.2 We define the luminance group and group action similarly (see Appendix[F.I]
We prove ya, and
Hue-Saturation group action. We define the action the hue-saturation group as a composition of the hue and saturation group actions. An element of the hue-saturation group acts an HSL image by the group action Phs Hy x SM x X -> X, which shifts both the hue and saturation channels of the image. For HSL image x E X defined as the concatenation of hue, saturation and luminance channels, i.e., x = (In, F,, 22), the action of an element (h2 8,) of the hue-saturation group N x SM given by
Pho(((h sy), x) = (h,, ()).
element of the hue nue-saturation group acts on a function on the discrete saturation group by the group action dhs HN x Sas x Y Y, which "rotates" and "translates" the function the group. Concretely, for a function f on the discrete hue-saturation group HN x SM defined as the
concatenation of functions f = (ffi,...: fiM, (IM,Ja. Jai....,fNM! the action of an element (hi, 9;) in hue-saturation group Hy x Sy given by
@ne(( sy), f) on (hy (s),
(8)
We prove Phs and Φhe are group actions Appendix B
Lifting layer. The first layer of group convolutional neural network "lifts" the input image the group (see Figure [b). We can lift an input image to the product space of the image grid Z², and discretized hue shifts HN (ho, "i,....,NN] by convolving with hue shifted filters,
10 * (Paa)))))) = Σ Rlwhyyll
9E22
Here we denote an element of the product space Iz.i, where the subscript references element of z2 and j references the element h; in HN. The input image f and lifting filters tm.. are functions on z?? This is the approach taken CEonv (Lengyel et al. 2024). The authors shift the hue of a filter in the RGB space by rotating its values about axis passing through the point p (1, 1) shown
5

---

Published as conference paper at ICLR 2025
Rotated
Restored
Average Restored Image Emor
0.06
CEConV-N
0.02
Rotated
Restored
Hue Group Cardinality
Figure 3: Impact of order on hue rotation invertibility. Our lifting layer (blue) operates on HSL input images where each hue rotation invertible. The lifting layer proposed CEConv (orange) operates RGB filters and suffers from invalid hue rotations for all discretizations of the hue group except for N and N (i.c., symmetries of the axis-aligned RGB cube). (Left) We show the impact of invalid hue rotations a four pixel image. We rotate the image 60" using proposed lifting layer (top-left) and the CEConv lifting layer (botton-left). Subsequently applying -600 rotation yields an image that is indistinguishable from the original using approach (top-left), and one with visible artifacts using the CEConv approach (bottom-left) (Right) We show the average restored image error for both approaches. Our approach results in consistently negligible restored image error, however, the CEConv approach results in a restored image error exceeding 7% for all discretizations of the hue group except N 1 and =
in Figure 8blin in Appendix C.I As described in Lengyel et al. (2024). this approach results in invalid hue rotations for all discretizations of the hue group that not symmetries of the axis-aligned RGB cube (i.e., N = 1 and N = 3). To remedy this, the authors project invalid RGB values to the nearest point on the RGB cube. However, this projection breaks invertibility of the hue shift (see Figure (3). yielding descriptor that is only approximately hue equivariant.
Lifting to the group can alternatively be achieved by transforming the input image rather than the filters,
(gr5) = = fz)
(10)
We use this approach in our implementation as circumvents the issue of invalid hue rotations, producing reliably equivariant deseriptor. The lifting layers for the saturation group, luminance group, hue-saturation group, and hue-luminance group are constructed analogously.
Equivariance of the lifting layer. We show that lifting layer is equivariant hue shifts.
We let f (gx.))
(hy, (z) as in Equation (I0) and show that a hue shift of f results in a hue
shift of f In the first step we use the fact that ya is a group action (see Appendix B.1) and in the
second step use commutativity of the hue group:
4o (h (m f (z) -p (hyhm:: (z)
(11)
ph th (himiig (2)
(12)
Ph (hi: sph (hj, (0))))
(13)
(h f' (gx.))).
(14)
Equivariance of the saturation, luminance, hue-saturation, and luminnmne lifting layers can be shown analogously.
Group convolution layer. For all layers after the first layer, the feature maps f are on the product space Z2 x H.. Since hue shifts can only be performed three dimensional inputs, we leverage the identification of hue shifts with the discrete 2D rotation group, i.c., HN Cy and interpret the feature maps f' as functions on the product space G := 2 x Cx. With this interpretation perform group convolution on G as follows,
[f] * v() ΣΣ A(O)))
(15)

---

Published as a conference paper at ICLR 2025
Error Improvement
10
Hue-3*
Table 1: Generalization to global hue-shift. Classification error on the Hue-shift MNIST dataset is reported. Our hue-equivariant models (Hue-N) perform comparably to baselines on the in-distribution case (A/A), and outperform base- lines on the out-of-distribution case (A/B).
Hue-4*
5
10
10c
10
10a
Percentage Training Examples
Figure 4: Model sample efficiency. We show the error improvement (higher is better) the Z2CNN baseline as a function the per- centage of training examples used. The advan- tage of our Hue-N models increases as the per-
Network
A/A
A/B B
Params
Z2CNN
1.54 (0.10)
57.38 (22.06)
22,130
Hue-3*
1.79 (0.25)
1.81 (0.29)
22,658
Hue-4#
(0.25)
2.08 (0.15)
25,690
CHConv-3
1.79 (0.13)
1.83 (0.19)
28,739
centage of training examples used decreases.
CEConv-4
3.04 (0.27)
3.09 (0.25)
30,539
where vi is a function on G. Group convolution on the saturation, luminance, and hue-saturation- luminance groups performed analogously.
5 EXPERIMENTS
In this section, we highlight the sample efficiency of model in the context of hue generalization; the stability of our hue-cquivariant representations compared to CEConv (Lengyel et al.) 2024); the utility of notion of color equivariance that includes saturation and luminance; and the utility of our representations for color- based sorting. Our models achieve strong performance on extensive experiments, against competitive baselines. Additional experiments are presented in Appendix
5.1 HUE-SHIFT MNIST CLASSIFICATION
We demonstrate improved generalization to global hue-shifts and higher sample efficiency compared conventional CNN model with similar number of parameters on our Hue-shift MNIST dataset. Our dataset is a variation of MNIST (LeCun et al. 1998) with 60k training examples and 10k test examples classified into one of 10 categories. Additional details are provided Appendix D.I
Generalization to global hue-shift. The generalization performance of our hue-equivariant mod- els, Hue-N (N indicates the order of the discrete hue group), and a conventional CNN model, Z2CNN (Cohen & Welling 2016). are reported in Table I On the in-distribution test case (A/A), the performance of our model and the conventional CNN model are comparable. However, on the out-of-distribution test case (A/B), the performance of our model is preserved, while the performance of the conventional CNN model deteriorates significantly.
3
(a) Z2CNN
(b) Huc-4
Figure 5: Hue-shift MNIST feature map visualization. We compare the feature map trajectories of MNIST digits their hue is varied from 1 to 360". The color of the trajectory corresponds to the class label. (a) iSNE projection hue shifted feature map trajectories in the Z2CNN baseline model. As the hue of the input changes, the location of the digit in the feature space changes significantly. (b) iSNE projection hue shifted feature map trajectorics in our euvivaruiaii CNN. In contrast to the Z2CNN baseline, the location of the digit in the feature space changes minimally.
7

---

Published as a conference paper at ICLR 2025
Table Generalization to local hue-shift. Classification error on the 3D Shapes dataset is reported. Our model (Hue-N) and CEConv-N achieve improved generalization performance over the conven- tional CNN model (Z2CNN). Our approach is robust to the choice of hue group discretization, while the performance of CEConv deteriorates when N is not symmetry of the axis-aligned RGB cube.
Network
A/A
A/B
A/C
Params
Z2CNN
0.00 (0.00)
51.25 (9.59)
26.66 (19.60)
20,192
Hue-3
0.00 (0.00)
0.03 (0.04)
0.04 (0.03)
19,478
Hue-4*
0.00 (0.00)
0.00 (0.00)
0.00 (0.00)
21,832
CEConv-3
0.00 (0.00)
0.02 (0.03)
0.05 (0.04)
26,441
CEConv-4
0.00 (0.03)
0.60 (0.25)
0.53 (0.43)
30,792
The performance gap on the out-of-distribution test case can be attributed to the difference in internal representation structure. To understand this better, generate feature representation trajectories in the penultimate layer of our network and the baseline architecture by continuously varying the hue of an input image. We then visualize these trajectories using tSNE (van der Maaten & Hinton 2008) projection (see Figure 5). For the baseline architecture, the trajectories of different digits overlap for some hues (Figure Sap. but for our model, the trajectories of different digits are confined to separate clusters (Figure 5b
Model sample efficiency. The sample efficiency of our huc-equivariant models (Hue-N) and conventional CNN model (Z2CNN) are reported in Figure The advantage of our hue-equivariant models increases as the percentage of the training set used decreases, illustrating improved sample ef- ficiency (Elesedy 2022). The error improvement is defined as the difference between the performance of the proposed model and the performance of the baseline (Z2CNN).
5.2 HUE-SHIFT 3D SHAPES CLASSIFICATION
We demonstrate improved generalization to local hue-shifts and significantly reduced equivariance error compared to CEConv on 3D Shapes classification dataset (Burgess Kim 2018). The 3D Shapes dataset consists of RGB images of 3D shapes, where the color of the shape, the floor, and the walls vary across examples, as does the scale and orientation the shape. There are 48k examples in the training set, and 12k examples in each of the test sets. Additional details provided in Appendix D.2
Generalization to local hue-shift. The generalization performance of hue-equivariant mod- els, Hue-N (N indicates the cardinality of the discrete hue group), a conventional CNN model, Z2CNN (Cohen Welling, 2016), and the CEConv-N models are reported in Table 2 Our Hue-4 model outperforms all other models on the in-distribution (A/ A), global hue-shift out-o--distribution (A/B) and local hue-shift nut-of-distributte nificantly worse than all other hue- quevvquivaiaant models. We attribute the relatively poor generalization performance of the CEConv-4 model the feature map equivariance error.
(A/C) test sets. The performance of CEConv-4 is sig-
Festure Map
CECono-4 Feature Map
Hue Shitte Original
Huo
C6Comw4
shifeed feature maps
Equivariance error
Figure 6: Hue-equivariant feature maps. (a) We qualitatively compare hue-shifted feature maps produced using our lifting layer and the CEConv lifting layer. Highlighted feature maps obtained using our lifting layer (Hue-4) are qualitatively indistinguishable, while there are visible discrepancies the feature maps CEConv-4. (b) We quantitatively compare the equivariance-en shifted feature maps for each approach. The equivariance error resulting from our lifting layer (Hue-4) more than three orders magnitude lower than the error resulting from CEConv-4.
of the hue-
8

---

Published as a conference paper at ICLR 2025
Table 3: Generalization to saturation-shift. Classification error on the Camelyon17 dataset is reported. Our model (Sat-N and Hue-M-Sae M-Sat-N) achieves improved generalization performance over the conventional CNN model (ResNet50).
Networks
Error
Param
ResNet50
28.91 (7.58)
23.5M
Sat-3"
16.08 (2.68)
23.3M
Hue-4-Sat-3*
19.06 (4.92)
23.0M
CEConv-3
28.76 (9.93)
23.1M
Lifting layer equivariance error. The lifting layer equivariance error of our Hue-4 model and the CEConv-4 model reported in Figure 6b As discussed in Section 4 lifting layer operates HSL images and does not induce the invalid hue rotations observed in CEConv. We quantitatively assess how lifting layer invertibility impacts descriptor equivariance following Zhdanov et al. (2024).
We compute the equivariance eтror the relative equivariance,
[f(p x)) - on (h2, (x:)) [f (f,,,, x)) + on (h2, f(x))
Equivariance Error
(16)
and show that the equivariance error of CEConv-4 more than three orders of magnitude higher than Hue-4 model (see Figure 6b).
5.3 CAMELYON17 CLASSIFICATION SATURATION SHIFT THE WILD
We demonstrate improved generalization to saturation-shifts compared to ResNet50 (He et al. 2016 and CEConv-3 on the Camelyon17 classification dataset (Bandi et al. 2018 The Camelyon dataset consists of images of human tissue collected from five different hospitals. Variation in tissue images results from variation in the data collection and processing procedures. The dataset consists of 387,490 examples, 302,436 of which are used for training and 85,054 of which are used for testing. Hospitals 1, 2, and are represented the training set, and hospital is represented in the testing set. Additional details are provided in Appendix E.3
Generalization saturation-shift. We expand the notion color equivariance proposed in Lengyel et al. (2024) to capture variation in saturation. The generalization performance of our saturation- equivariant models, Resnet50 (He et al., (2066), and CEConv-3 are reported in Table Our saturation- and oolor-equivariant models outperform all other models, while CEConv performs comparably to the conventional CNN model.
5.4 COLOR SHIFT IN THE WILD
We demonstrate improved generalization to global color-shifts compared ResNet (He al. 2016) and CEConv on the Caltech-101 (Li et al. 2022), Oxford-IIT Pets (Parkhi et al.). Stanford
Table 4: Generalization to color-shift. Classification error on the Caltech-101, Oxford-IIT Pets, Stanford Cars, CIFAR-10, CIFAR-100, STL-10 datasets reported. Our models (Hue-N, Sat- N and Hue-M-Sat-N) outperform CEConv-N models on all datasets, and are competitive with ResNet. We ResNet-aug and Resnet-gray to denote ResNet architecture trained color-jitter augmented dataset and grayscale image dataset respectively.
Network
Caltech 101
CIFAR-10
CIFAR-100
Stanford Cars
Oxford Pets
STL-10
ResNet
32.68 (1,55)
7.86 (1.14)
32.00 (0.63)
25,41 (0.96)
31.52 (2.05)
18.59 (1.65)
ResNet-gray
33.79 (3.09)
8.45 (0.68)
32.04 (0.66)
24.71 (0.93)
30.38 (0.35)
18.71 (1.47)
ResNet-aug
32,90 (0.82)
(0.44)
32.27 (0.18)
22.38 (1.65)
30.06 (0.52)
17.89 (1.48)
Hue-3*
34.32 (0.34)
8,41 (0.39)
33.28 (0.63)
22.19 (2.20)
31.37 (3.50)
19.82 (0.98)
Hue-4*
32.23 (1.07)
8.83 (0.64)
34.70 (0.89)
20.38 (1.06)
27.39 (0.68)
20.73 (1.11)
Hue-4-Sat-3#
(1.07)
10.68 (0.78)
33.27 (0.31)
24.79 (3.87)
29.84 (1.34)
20.53 (0.73)
Sat-3#
41.64 (1.40)
9.24 (0.27)
39.33 (0.45)
31.90 (10.03)
36.87 (5.57)
20.71 (1.10)
CEConv-3
34.74 (0.83)
(0.33)
34.95 (0.44)
23,97 (1.56)
31.08 (2.54)
24.29 (1.31)
CFConv-4
33.52 (0.48)
9.28 (0.24)
35.46 (0.35)
24.08 (0.66)
33.70 (1,50)
21.90 (1.64)

---

Published as a conference paper at ICLR 2025
Hue-1
(b) ResNet44
Figure 7: Color sorting on CIFAR-10. We show images from CIFAR-10 automobile class ordered by pairwise distance using Hue-4 and ResNet44 feature maps. The structure of the Hue-4 feature maps naturally allow for color based sorting, whereas the ResNet44 feature maps do not. We include visualizations of the entire automobile class sorted using Hue-N and CEConv-N in Appendix
Cars (Krause et al., 2013). STL-10 (Coates et al. 201 CIFAR-10 10 and CIFAR-100 100 (Krizhevsky et al. 2009) datasets. report results on the ImageNet dataset in Appendix F.2 and provide additional details for all experiments in Appendix B
Generalization to color-shift. The generalization performance of our hue-equivariant models, Hue-N, saturation-cquivarii model, Sat-N, color-cquivariant models Hue-M-Sate ResNet44 model, and the CEConv-N models are reported in Table 4 Notably, our color-cquivarian network outperforms baseline models on all fine-grained classification tasks (i.e., Stanford Cars and Oxford- ITT Pets datasets). We attribute the performance gap between our Hue-4 model and the CEConv-4 model the significant difference their equivariance error.
Color sorting on CIFAR-10. While our hue-equivariant model performs on-par with the conventional CNN model this dataset, the structure of representation be leveraged for tasks that possible with conventional architectures alone. To illustrate this, we use our model to sort images in the automobile class by hue (see Figure 7ap. To produce hue ordering, we compute the hue-shifted pairwise Euclidean distance between hue group feature maps of instances in the automobile class. Images 21 and 22 are close the hue space if
arg ( (h,, (x1)), V (x22)1=
(17)
where (xx)) denotes the penultimate feature map for image j and h2 is the i-th element of the hue group. Sorting using the penultimate feature map of the ResNer44 model is shown in Figure 7b
6 CONCLUSION
In this paper we address the challenge of learning hue-, saturation- and luminance-cquivarian representations. Leveraging the observation that these perceptual transformations have geometric structure, we propose a group structure for each, and a group convolutional neural network that is equivariant to these transformations by design. By encoding perceptual variation in finite group structure, we achieve strong generalization performance and sample efficiency and can use learned representations for tasks such color-based sorting.
Limitations and future work. GCNNs are more computationally expensive than their conventional counterparts since they require computation of the filter orbit each layer. Concretely, the com- putational cost of GCNNs is approximately equal to that of conventional CNNs with filter bank size equal that of the augmented filter bank used in GCNNs (Cohen Welling 2016). GCNNs also require a finite filter orbit which can at best approximate a continuous group. Future work will investigate a continuous representation the group to address both these limitations.
ACKNOWLEDGMENNTS
We sincerely thank the ICLR 2025 reviewers and area chair for their thoughtful feedback; the discussions and additional experiments have meaningfully improved our paper. We thank Minseok Kim for their insights and discussion.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Abien Fred Agarap. Deep learning using rectified linear units. arXiv preprint arXiv: arXiv: 1803.08375, 2018.
Peter Bandi, Oscar Geessink, Quirine Manson, Marcory Van Dijk, Maschenka Balkenhol, Meyke Hermsen, Babak Ehteshami Bejnordi, Byungjae Lee, Kyunghyun Paeng, Aoxiao Zhong, et al. From detection of individual metastases to classification of lymph node status the patient level: the camelyon17 challenge. IEEE Transactions on Medical Imaging. 2018.
Simon Batzner, Albert Musaelian, Lixin Sun, Mario Geiger, Jonathan P Mailoa, Mordechai Kombluth, Nicola Molinari, Tess Smidt, and Boris Kozinsky. E equiivarat graph neural networks for data-efficient and accurate interatomic potentials. Nature communications, 13(1):24S5) 2022.
Gregory Benton, Mare Finzi, Pavel Izmailov, and Andrew G Wilson. Learning invariances in neural networks from training data, Advances neural information processing systems, 33:17605-17616, 2020.
Joan Bruna and Stéphane Mallat. Invariant scattering convolution networks. IEEE transactions on pattern analysis and machine intelligence, 35(8):1872-1888 2013.
Chris Burgess and Hyunjik Kim. 3d shapes dataset. http::///sthhbbooooooopop/i///ss 2018.
Hamilton Y Chong, Steven J Gortler, and Todd Zickler. A perception-basee color space for illumination-invaiian image processing. ACM Trunsactions on Graphics (TOG), 27(3):1-7, 2008.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. Proceedings of the fourteenth international conference on artificial intelligence and statistics, pp. 215-223. JMLR Workshop and Conference Proceedings, 2011.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International conference on machine learning, pp. 2990-2999, PMLR, 2016.
Taco Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling. Gauge equivariant convolutional networks and the icosahedral cnn. In International conference on Machine learning, pp. 1321-1332
PMLR, 2019a.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homoge- neous spaces. Advances in neural information processing systems, 32, 2019b.
Kanjar De and Marius Pedersen. Impact of colour on robustness of deep neural networks. In Proceedings of the IEEE/CVF international conference on computer vision, pp. 21-30, 2021.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference computer vision and pattern recognition, pp. 248-255. Icee, 2009.
Bryn Elesedy. Group symmetry in pac learning. In ICLR 2022 workshop on geometrical and topological representation learning, 2022.
Martin Engilberge, Edo Collins, and Sabine Süsstrunk. Color representation in deep neural networks. In 2017 IEEE International Conference on Image Processing (ICIP), pp. 2786-2790. IEEE, 2017.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical enns. In Proceedings of the European Conference Computer Vision (ECCV), pp. 52-68, 2018a.
Carlos Esteves, Christine Biien-Blanchetee, Xiaowei Zhou, and Kostas Daniilidis, Polar transformer networks. International Conference on Learning Representations, 2018b.
Carlos Esteves, Yinshuang Xu, Christine Allen-Blanchette, and Kostas Daniiliiis. Equivariant multi- view networks. Proceedings of the IEEE/CVF international conference on computer vision, pp. 1568-1577, 2019.
11

---

Published as a conference paper at ICLR 2025
Mare Finzi, Samuel Stanton, Pavel Izmailov, and Andrew Gordon Wilson. Generalizing convolutional neural networks for equivariance to lie groups on arbitrary continuous data. Internationaa Conference Machine Learning, pp. 3165-3176. PMLR, 2020.
Jean Gallier and Jocelyn Quaintance. Differential geometry and lie groups, volume 12. Springer, 2020.
Justin Gilmer, Samuel S Schoenholz, Patrick F Riley, Oriol Vinyals, and George E Dahl. Neural message passing for quantum chemistry. In International conference on machine learning, pp. 1263-1272. PMLR, 2017.
Samuel Greydanus, Misko Dzamba, and Jason Yosinski. Hamiltonian neural networks. Advances in neural information processing systems, 32, 2019.
Sharut Gupta, Joshua Robinson, Derek Lim, Soledad Villar, and Stefanie Jegelka. Structuring repre- sentation geometry with rotationally equivariant contrastive learning. In The Twelfth International Conference on Learning Representations, 2023.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Geoffrey E Hinton, Sara Sabour, and Nicholas Frosst. Matrix capsules with em routing. In Interna- tional conference on learning representations, 2018.
Sergey loffe. Batch normalization: Accelerating deep network training by reducing internal covariate shift. arXiv preprint arXiv:1502.03100 2015.
Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint 14X2:6:41..69880 2014.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks the action of compact groups. In International Conference on Machine Learning, pp.
2747-2755. PMLR, 2018
Jonathan Krause, Michael Stark, Jia Deng, and Fei-Fei. 3d object representations for fine-grained categorization. In 2013 IEEE International Conference on Computer Vision Workshops, pp. 554--561 2013. doi: 1..109//CCV..2013.77..
Alex Krizhevsky, Geoffrey Hinton, al. Learning multiple layers of features from tiny images. 2009.
Alex Krizhevsky, Ilya Sutskever, and Geoffrey E Hinton. Imagenet classification with deep convolu- tional neural networks. In Advances in Neural Information Processing Systems, 2012.
Ya Le and Xuan Yang. Tiny imagenet visual recognition challenge. CS 23/N, 7(7):3, 2015.
Yann LeCun, Léon Bottou, Yoshua Bengio, and Patrick Haffner. Gradient-based learning applied to document recognition. Proceedings of the IEEE, 86(11) 1):2278-22333 1998.
Yann LeCun, Fu Jie Huang, and Léon Bottou. Learning methods for generic object recognition with invariance to pose and lighting. Proceedings of the 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition, 2:11-104 Vol.2, 2004.
Karel Lenc and Andrea Vedaldi. Understanding image representations by measuring their equivariance and equivalence. Proceedings the IEEE conference on computer vision and pattern recognition, pp. 991-999, 2015.
Attila Lengyel, Ombretta Strafforello, Robert-Jan Bruintjes, Alexander Gielisse, and Jan van Gemert. Color equivariant convolutional networks. Advances in Neural Information Processing Systems, 36, 2024.
Fei-Fei Li, Marco Andreeto, Mare' Aurelio Ranzato, and Pietro Perona. Caltech 101, Apr 2022.
12

---

Published as a conference paper at ICLR 2025
Arezou Pakzad, Kumar Abhishek, and Ghassan Hamameh. Circle: Color invariant representation learning for unbiased classification of skin lesions. In European Conference on Computer Vision, pp. 203-219. Springer, 2022.
Omkar Parkhi, Andrea Vedaldi, Andrew Zisserman, and C. V. Jawahar. The oxford-iit pet dataset. URL https: rwwots ww..oo..accc..c...........oooo www.o/////....////////o/oooooo datp://s//aaappts/////
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference compuler vision and pattern recognition, pp. 652-660, 2017.
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. arXiv preprint arXiv: 1409. 1556, 2014.
Zikai Sun and Thierry Blu. Empowering networks with scale and rotation equivariance using similarity convolution. In The Eleventh International Conference on Learning Representations, 2022.
Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv: 1802.882/222 19, 2018.
Laurens van der Maaten and Geoffrey Hinton. Visualizing data using t-sne. Journal of Machine Learning Research, (86), 2008
Mitko Veta, Paul J Van Diest, Mehdi Jiwa, Shaimaa Al-Janabi, and Josien PW Pluim. Mitosis counting in breast cancer: Object-level interobserver agreement and comparison an automatic method. PloS one, 11(8):e0161286 2016.
Olivia Wiles, Sven Gowal, Florian Stimberg, Sylvestre Alvise-Rebuffi Ira Ktena, Krishnamurthy Dvijotham, and Taylan Cemgil. A fine-grained analysis on distribution shift. International Conference on Learning Representations, 2022.
Daniel Worrall and Max Welling.
Deep scale-spaces: Equivariance over scale. Advances in Neural Information Processing Systems, 32, 2019.
Daniel E Worrall, Stephan J Garbin, Daniyar Turmukhambetov, and Gabriel J Brostow. Harmonic networks: Deep translation and rotation equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5028-5037, 2017.
Maksim Zhdanov, David Ruhe, Maurice Weiler, Ana Lucie, Johannes Brandstetter, and Patrick Forré. Clifford-steerabble convolutional neural networks. International conference on Machine learning, 2024.
Tao Zhong and Christine Allen-Blanchette. Gagrasp: Geometric algebra diffusion for dexterous grasping. arXiv preprint arX1122503.001233 2025.
APPENDIX
A HUE AND SATURATION GROUP
In this section we prove the proposed discretized hue group and saturation group satisfy the axioms for a group. By definition, a group a non-empty set G along with binary operation on that satisfies:
1. Closure. For all o, beb G.
a a.bEC.
(18)
2. Associativity. For all a, b, G,
(a. b) (b.c).
(19)
13

---

1. Published as a conference paper at ICLR 2025
3. Identity element. There exists element 1 G so that for all G,
1 1
(20)
4. Inverse element. For each a & G, there exists a -1 E G so that,
a
(21)
A.1 HUE GROUP
We show the set HN,
HN = -0, /x,/N, 2x 2
(22)
where 0 E k < N, together with the binary operation, : HN x HN + HN, where
a a.br (a + b) (mod 2x))
(23)
for any a, b E HN. is a group. To do this, we prove the group axioms
18
Closure is satisfied if for any b E Hy, we have 2x and b = 2 for some integers ka and Ko, in the range 0 I ka, ko < N. The product of and can then be written
Hl.. Any a, E Hy can be expressed
2TT (kia + kb))
b
(mod 2T).
(24)
N
To prove closure, consider cases: kia + kb N, and ki kb N. In the first case, that is, ka + lib < N, we immediately have
2x (ka + kbb
a. b.b-
E HN.
(255)
N
since kn Ks is in the range ka + kb N. In the second case, that is, ka + ko 2 N, we
write ka + kb in quotient remainder form,
ki * N + k, kia + kb.
(26)
where km is the divisor, and k, is the remainder. Using this form we can write the product of elements a and b
2Tr (kim N + kir)
a b
(mod 2T))
(27)
N
20thy
2m +
(mod 2n)
(28)
N
2
(29)
N
Since 0 S kr N, we have that E HN, and closure is satisfied.
Associativity is satisfied if for any a, b, E HN. we have a (b-c) = (a b) cc. For any o, b, c E H write 2TR./N, /N, b = 2Tha/N, and c 2nkk/N for some integers his Kb and k in the range
0 < Ki, Kb, to < N. Then we have
((20) 2T (kia + kib)
2Tke
(a b) b)
(mod 2 + (mod 2nr).
(30)
N
N
Since modulo distributes over addition, the above can be written
2TTka
(2Tkk
2
(a b) c
(mod 2m)) +
(mod 2T))
(mod 2iT)
(31)
N
N
N
2
2
2TTh
(mod 2nr) (mod 2T)) +
(mod 2n)) (mod 2T) +
(mod 2T)
N
N
N
(32)
Using the identity property modulo (i.e., (a mod ()od a mod b), we can write
(2iThia
(2Thhi
22Thi
(a b) cc
(mod 2T) +
(mod 2T) (mod 2m) +
(mod 2T) (mod 2x))
N
N
N
(33)
14

---

Published as a conference paper at ICLR 2025
Using the distributive property of modulo, write
(( 2
2rrki
2TTic
(a. (a.b)- c==
+
(mod 2 +
(mod 2
(mod 2T)
(34)
N
N
N
2
2TT (Rib + kc)
+
(mod 2T)
(mod 2x)
(35)
N
N
= a (b-c) c),
(36)
and associativity is satisfied.
Identity element. An element e N is an identity element if for every a E HN. we have a a. Consider the element HN, then for every 2xka/N HN have
0 2TT (ka + 2) (mod 2Tr)
(37)
N
By commutativity of addition, we have 0 a and 0 is the identity element of HN.
Inverse element. There exists an inverse for every element of the group if for every a E Hy there e, where E H is the identity element.
exists an element a E HN so that a- = a a1 We can write any element a E HN. the form a 2 where is an integer in the range 0 can be written in the
O S ka < N. An inverse element a of a satisfying a a form a (x
(2T (Ka (N ka)) -22
( (mod 2T) 0.
a-a-l=
(mod 2T)
(38)
N
By commutativity addition we have a- -1. = 0. It remains to show that a 1 is in the set HN- Since k. is an integer in the range 0 S ka < N, then N - ka is in the range 0 < N - hi I N, and therefore, a- -N-NN)/ is in the set HN and existence of an inverse is satisfied.
Having shown that the set H together with the binary operation (a-b) b) + (a + b) (mod 2T) satisfies all four axioms of a group, we have that H fact a group.
A.2 SATURATION GROUP
We show that the set SN
SN = -2.., 1/N,9/N, N/N, 2
(39)
generated by S/N with & integer valued, together with the binary operation SN SN + SN. where
a a-b
(40)
for any a, b E SN is a group. To do this, we prove the group axioms 1
Closure is satisfied if for any a, Sy, we have b E SN. Any a, E Sy can be expressed m//N and 6 mysk/N for some integers m. and m&. The product of and then be
written
b ma * k N
my * (ma + mp) #
(41)
N
N
Since m = ma + my is an integer (the set of integers is closed under addition), we have that b = mok/N, is in the set Sy, and, closure is satisfied.
Associativity is satisfied if for any a, b, e E SN. we have a (b.cl c) = (a b) c. For any a, b, c E SN- can write a m. *K/N, , mork/N, and e m.rk/N for some integers mas my, and me. Then we can write,
(a. == (ma my k
Me *k N
(ma + my + mc) k
(42)
N
and similarly,
Ma k
my k
mc * *b)
(ma + my + me)
c- (b
+
(43)
N
N
N
N
15

---

Published as a conference paper at ICLR 2025
Therefore, (b- c) (a. c, and associativity is satisfied.
Identity element. An element e E SN is an identity element if for every a E SN. we have =a-e=0 Consider 0 Sv, then for every E SN where m.ck/N SN we have,
Ma k
0 k
(44)
N
By commutativity of addition, we have 0 a and 0 is the identity element of SN-
Inverse element. There exists inverse for every element of the group if for every a Sy there exists an element a E SN so that a a- = a We can write any element E SN. in the form a m. * where ma integer valued, An inverse element a -1 of satisfying
e, where Sx is the identity element.
0 can be written in the form a -ma*k/N
ma k
k = 0.
(45)
By commutativity of addition we have a O. It remains to show that a is in the set SN. Since -ma is an integer, a -m. is the set Sy and existence an inverse satisfied.
Having shown that the set Sy together with the binary operation (a b) + + b satisfies all four axioms of a group, we have that SN is in fact a group.
HUE AND SATURATION GROUP ACTION
In this section we prove the proposed hue group action and saturation group action satisfy the axioms for a group action. By definition of group action, şp is a group action if it satisfies the following properties:
1. For all g. h E G and all x E X
(lg. (h, ((gh, x)
(46)
2. For all x X
(11,x) = x
(47)
where E G is the identity element of G.
In the remainder of this section we refer to Equation (46 as 'property one' and Equation 47 as
'property two'.
B. HUE GROUP ACTION
Here we show that the proposed hue group action is indeed group action. The proposed hue group action on the input space is defined as
s (1e,, = ((rr, + hm)dmod (1),
(48)
in Equation g of the main text.
First we show the proposed hue group action чoд satisfies property one. For any hue shifts h, and hy, we have
sp (hi (h (h),
(hi, ((æn (y)d 2T), Ug
(49)
= (((xn + (yod 2TT) + fu,) (mod 2nT), xo x1))
(50)
(((x) (hy 14)) 2T) x7))
(51)
pPh((, ((((()+() + h4),
(52)
which shows that рh satisfies the property one. Now show (h satisfies property two. We have the identity element hue transformation hg = 0, and
Pn ( x) = ((x), + (mod 2 x,, 22))
(53)
(2,, m1) x,
(54)
which shows Ph satisfies property two. Having shown that 4n satisfies all properties of a group action, have that sou is, in fact, group action.
16

---

Published as a conference paper at ICLR 2025
B.2 SATURATION GROUP ACTION
Here we show the proposed saturation group action a group action. The proposed saturation group action is defined
Os(s; x)) = (æ%, min + 255), mt),
(55)
in Equation (5) of the main text. First we show the proposed saturation group action 9s satisfies property one. For any saturation shifts 8; and 8j, we have
(8(s (8.. x))
(s (In, min(x, + 8j, 255). .z1)
(56)
(In, min(min( + 85; 255) + 8i, 255), 21)
(57)
(In, min(x, + (8, + s.), a1) pa((s; + s)),
(58)
(59)
which shows &e satisfies property one. Now show yos satisfies property two. We have the identity element saturation transformation So = 0, and
(S0 x) (Ih, min ( 8o; 255), 21)
(60)
= (rh, Fay Tt) = vi
(61)
which shows y satisfies property two. Having shown that yps satisfies all properties of group action, we have that yo is, in fact, a group action.
C CECONV COMPARISON
In this section, we present qualitative comparisons of our Hue-N equivariant models and the CEConv- N equivariant models proposed in Lengyel et (2024)
C.1 LIFTING LAYER
The lifting layer proposed in CEConv transforms RGB filters in the first layer of the network (see Figure 8b). This approach only outputs valid RGB filters when the hue group is discretized to order N=1 (0 degree rotations) or N=3 (120 degree rotations). For other discretizations of the hue group, CEConv is only approximately equivariant. We detail the CEConv lifting layer in full in Section Lifting layer, and study the impact of invalid RGB rotations in Figure]
We propose lifting layer that transforms input images instead of network filters (see Figure Sa). This seemingly minor change improves equivariance error by more than three orders magnitude and stabilizes network performance across discretizations.
C.2 HUE-SHIFT MNIST FEATURE MAP VISUALIZATION
We present visualizations of feature maps produced by the Hue-3, Hue-4, CEConv-3 and CEConv 4 architectures, trained on the hue-shift MNIST dataset. Figure shows a comparison of the feature maps produced by our Hue-3 architecture and the CEConv-3 architecture, and Figure TO shows comparison of the feature maps produced by our Hue-4 architecture and the CEConv-4 architecture. The feature map visualizations in Figure are qualitatively similar, but in Figure I0 the 4 feature map visualization has noticeably more artifacts than the Hue-4 (ours) feature map visualization.
C.3 COLOR SORTING ON CIFAR-10 10
We present visualizations the sorted automobile class using Hue-N and CEConv-N architectures. We trained Hue-3, CEConV-3, Hue-4 and CEConv-4 on CIFAR-10 and used the resulting represen- tations to sort the automobile class by color. Figures II
12 13 and 14 show the automobile class
sorted by Hue-3 (ours), CEConv-3, Hue-4 (ours), and CEConv-4 respectively. Figures [1] and 12 are qualitatively similar; however, Figures [3 [4] are not. In particular, notice that there are blue images in the green band (around the center) of Figure 14 but not in Figure T3
17

---

Published as a conference paper at ICLR 2025
uitiigg
(a) Hue-N Lifting
(b) CEConv Lifting
Figure 8: Lifting layer. In Hue-N, input image (left) lifted the hue-saturation group (right) by shifting its hue and saturation values. (b) CEConv shifts the hue of filter in the RGB space by rotating its values about an axis passing through the point p (1, 1, 1). This approach results in invalid hue rotations for all discretizations of the hue group that are not symmetries of the axis-aligned RGB cube (i.e., N and N 3). (Top) RGB cube and yields valid rotations. (Bottom) N = 4. Rotations yield results where invalid RGB values that outside the RGB cube and needs to be projected.
= 3. Rotations symmetric to axis-aligned
(a) Hue-3
(b) CICCov-33
Figure 9: Hue shift MNIST feature map visualization. (SNE projection of huc shifted feature map trajectories in Hue-3 and CEConv-3.
(a) Hue-4
(b) CEConv-4
Figure 10: Hue shift MNIST feature map visualization. tSNE projection of hue shifted feature map trajectories in Hue-4 and CEConv-4. In contrast Hue-4, there is more ambiguous scatter in the feature maps of CEConv-4.
18

---

Published as a conference paper at ICLR 2025
Figure 11: Color sorting CIFAR-10 using Hue-3. We show images from CIFAR-10 automobile
class ordered by pairwise distance using Hue-3.
Figure 12: Color sorting on CIFAR-10 using CEConv-3. We show images from CIFAR-10 automobile class ordered by pairwise distance using CEConv-3.
Figure 13: Color sorting on CIFAR-10 using Hue-4, We show images from CIFAR-10 automobile class ordered by pairwise distance using Hue-4.
Figure 14: Color sorting on CIFAR-10 using CEConv-4. We show images from CIFAR-10 automobile class ordered by pairwise distance using CEConv-4.
19

---

Published as a conference paper at ICLR 2025
D DATASETS
D.I HUE-SHIFT MNIST
We introduce the Hue-shift MNIST dataset to evaluate the performance of our equivariant architectures the presence global hue shift. Examples 0' and 240 (see Figure 15a) and examples in the test set partitioned into an in-distribution set and an tittof-distributioo set. In-distribution examples are randomly assigned a hue from the same distribution as the training set, and out-of-distribution examples are randomly assigned hue outside of the training set distribution (i.e., between 240" and 360%) (see Figure 15b).
training set are randomly assigned a hue between
8
9
7
5
0
1
9
3
0
1
9
7
1
4
5
0
7
8
8
7
4
9
2
4
9
0
0
9
3
6
6
(a) Dataset examples
(b) Dataset examples
Figure 15: Hue-shift MNIST dataset. (a) Examples from the training dataset and in-distribution testing dataset are colored with a randomly selected hue between and 240". (b) Examples from dittrf-distributionn testing dataset B colored with randomly selected hue between 240 and 360
D.2 HUE-SHIFT 3D SHAPES
We introduce the Hue-shift 3D Shapes dataset to evaluate the performance of our equivariant archi- tectures the presence of local hue shift. Examples in the training set are randomly assigned a hue from the first half of the color space (colors 0-4 as defined in (Burgess & Kim 2018 ) and examples in the test set are are partitioned into an in-distribution set and two out-of-distribution test sets designed to measure robustness global hue-shift, and local hue-shift. In the out-of -ntto--distribttinn test set designed to measure robustness to global hue-shift, the color of the wall, the floor and the shape are randomly selected from the second half of the color space (colors 5-9 as defined in (Burgess & Kim 2018)). and in the out-of- df-iiiiribttiin test set designed to measure robustness to local hue-shift, the color of the wall and the floor are randomly selected from the first half of the color space, and the color of the shape is randomly selected from the second half of the color space.
sets. The
(a) Dataset examples
(b) Dataset B examples
(c) Dataset C examples
Figure 16: Hue-shift 3D Shapes dataset. (a) Examples from the training dataset and in-distribution testing dataset A. The color the wall, the floor and the shape are randomly selected from the first half of the color space (colors 0-4). (b) Examples from the out-of-distribution testing dataset B. The color the wall, the floor and the shape are randomly selected from the second half the color space (colors 5-9). (c) Examples from the out-of-distributio testing dataset C. The color of the wall and the floor are randomly selected from the first half of the color space, and the color the shape is randomly selected from the second half of the color space.
20

---

Published as a conference paper at ICLR 2025
D.3 SMALL NORB
We use the small NORB dataset to evaluate the performance of our equivariant architectures in the presence of luminance shifts. Examples in the training set are assigned with medium lighting conditions (2-3 as defined in (LeCun al. 2004)). ). and examples in the test set are are partitioned sets. Examples in testset A are in-distribution with lighting label 2-3 as defined in LeCun al. (2004); examples testset are out-of-distribution with lower lighting (lighting label (-1 as defined in LeCun et al. (2004)); and examples in testset C are -iittiiiuiittto with higher lighting (lighting label 4-5 as defined in LeCun et al. (2004). The out-of-distribution testsets are designed to measure robustness to luminance-shift.
into an in in-distribution set and two out-of-distribun
(a) Dataset A examples
(b) Dataset B examples
(c) Dataset C' examples
Figure 17: Small NORB dataset. (a) Examples from the training dataset and in-distribution testing dataset A. (b) Examples from the out-of-distribution testing dataset B with lower lighting conditions. (c) Examples from the -iittrriiiiittiionn testing dataset C' with higher lighting conditions.
D.4 TINY-IMAGENET
We use the Tiny-ImageNer dataset to evaluate the performance of our equivariant architectures in the presence of color shifts in large datasets. The Tiny-ImageNet dataset (Le & Yang/ 2015) is a downsized subset of ImageNet consisting of 100k 64x64 RGB images. Of the 1000 classes represented in ImageNet, 200 are represented in Tiny-ImageNet, each with 500 training examples, 50 validation examples, and 50 unlabeled test examples.
E IMPLEMENTATION AND TRAINING DETAILS
In this section we provide architectural details for our equivariant networks, and training details for each experiment. Our source code is publicly available online at htttts://ggthub com/ CAB-Lab Princeton/Learniiiee.. Color- Equivariann hrg.oo/oooooooooaaaaaaoaeeeeaaar
All experiments were performed over multiple random seeds to assess the robustness of the model to initialization. Performance statistics on the Hue-shift MNIST, Hue-shift 3D Shapes, and CIFAR- 10 datasets were computed over three random seeds (i.c., 1999, 2000, and 2001). Performance statistics the on Camelyon17 dataset were computed over five random seeds (i.e., 1997, 1998, 1999, 2000, and 2001). Performance statisties the ImageNet dataset were computed over random seeds (1999).
All models except ImageNet were trained on a shared research computing cluster. Each compute node allocates an Nvidia L40 GPU, 24 core partitions of an Intel Xeon Gold 5320 CPU, and 24GBs of DDR4 3200MHz RDIMMs. ImageNet was trained on compute nodes with 8 Nvidia L40 GPUs, two Intel Xeon Gold 5320 CPUs, and 512GBs of DDR4 3200MHz RDIMMs.
E. HUE-SHIFT MNIST
We compare the classification performance of our architecture and the Z2CNN architecture proposed in Cohen & Welling (0016). Our hue-equivariant architecture has the same number of layers as Z2CNN, but with reduced filter count to maintain similar number of parameters at each layer. In the final layer of our que-equivariant architecture, we perform hue group pooling to yield an hue- invariant representation.
We train our equivariant networks and the conventional architectures for 5 epochs with a batch size of 128. We optimize cross-entropy loss using the Adam optimizer (Kingma & Ba 2014) with
21

---

Published as a conference paper at ICLR 2025
31 = 0.9, and 32 0.999. We use initial learning rate of 10 for the Z² network, and 10-+ for the hue-equivariant network. We train CEConv architectures using the hyperparameters reported in (Lengyel et al. 2024).
E.2 HUE-SHIFT 3D SHAPES
We compare the classification performance of our architecture and the Z2CNN architecture proposed in Cohen & Welling (2016). Our hue-equivariant architectures are designed with the same network structure the baselines, but with a reduced filter count maintain similar number of parameters at each layer. In the final layer of our hue-equivariant architectures, we perform hue group pooling to yield an hue-invariant representation.
Following (Wiles et al. 2022), we train the ResNet architectures for 100k iterations with batch size of 128. We optimize over cross-entropy loss using SGD with learning rate of 10 Images are down-sampled by factor of 2 to train the CNN architectures. CNN architectures are trained as described in Section
E.3 CAMELYON17
We compare the classification performance of our architecture and the ResNet50 architecture (He et al. 2016). Our hue- and saturation-enn inturation-euuiariiant architectures have the same number of layers as ResNet50, but with reduced filter count to maintain a similar number of parameters each layer. In the final layer of our hue- and naturation-cquivariinn architectures, we perform group pooling to yield group invariant representation.
We train our equivariant networks and the conventional architectures for 10k iterations with batch size of 32. We optimize over cross-entropy loss using the Adam optimizer (Kingma & Ba 2014) with an initial learning rate of 10 31 = 0.9, and 8g 0.999. We train CEConv architectures using
the hyperparameters reported in Lengyel et al. 2024).
Node Train
Node
Tra
Node Train
Node Validatior
Node Test
0.25
3.06
recestage
2.00
0.35
0.04
0.30
0.00
0.00
0.00
0.00
0.00
200
130
150
100 Saturation
150
100
Figure 18: Camelyon-17 dataset saturation statistics. Example images and saturation statisties by node (hospital).
We present experimental results for four discretizations of the saturation space. Our choice of discretizations, i.e., d E (1/20, 1/10, 3/20, 1/2), are determined with consideration of the saturation range of the training and validation set (see Figure T8). For given choice of discretization d, an element 8; E Sy shifts the saturation of an input image by s; i d where is the maximum pixel value. We limit computational expense of lifting and convolution by performing these operations the truncated set (s-1- So, $1} for our Sat-3 model.
E.4 10
We evaluate the performance of model in the presence of natural hue shifts the CIFAR- 100 (Krizhevsky et al.) 2009) classification dataset. The dataset consists of 60k examples, 50k of which are used for training and 10k for testing.
22

---

Published as a conference paper at ICLR 2025
We compare the classification performance of our architecture and the ResNet44 architecture proposed in Cohen & Welling (2016). Our hue-, saturation- and color-equivariant architectures have the same number of layers as ResNet44, but with reduced filter count to maintain similar number of parameters at each layer. In the final layer of our networks, we perform group pooling to yield group invariant representation. We train equivariant networks and the conventional architectures for 300 epochs, and batch size of 128. We optimize over cross-entropy loss using SGD with an initial learning rate of 10 and cosine-annealing scheduler.
E.5 CIFAR-100
We evaluate the performance of our model in the presence of natural hue shifts on the CIFAR- 100 (Krizhevsky et al. 2009) classification dataset. The dataset consists of 60k examples, 40k of which are used for training, 10k for validation, and 10k for testing.
We compare the classification performance of our architecture and the ResNet44 architecture proposed in Cohen & Welling (2016). Our hue-, saturation- and color-equivariant architectures have the same number of layers as ResNet44, but with a reduced filter count to maintain a similar number of parameters each layer. In the final layer of our networks, we perform group pooling yield group invariant representation. We train our equivariant networks and the conventional architectures for 300 epochs, and batch size of 128. We optimize over cross-entropy loss using SGD with initial learning rate of 10-1 and cosine-annealing scheduler.
E.6 CALTECH-101
We optimize over a cross-entropy loss using Adam (Kingma Bal 2014) with an initial learning rate of 10-2 We evaluate the performance of our model in the presence of natural hue shifts on the Caltech- 101 (Li et al. 2022) classification dataset. The dataset consists of 9,146 examples, 2/3 of which are used for trainnn and 1/3 for testing.
We compare the classification performance of our architecture and the ResNet18 architecture proposed in He al. 2016). Our hue-, saturation- and color-equivariant architectures have the same number of layers as ResNet18, but with a reduced filter count to maintain a similar number of parameters each layer. In the final layer of our networks, we perform group pooling yield group invariant representation. We train our equivariant networks and the conventional architectures for 300 epochs, and batch size of 16.
E.7 STL-101
We evaluate the performance of our model in the presence of natural hue shifts on the STL-10 (Coates) jet al. 2011) T) classification dataset. The dataset consists of 5k training examples and 8k testing examples.
compare the classification performance of our architecture and the ResNet18 architecture proposed in He et al. (016). Our hue-, saturation- and color-equivariant architectures have the same number of layers as ResNet18, but with a reduced filter count to maintain a similar number of parameters at each layer. In the final layer of our networks, we perform group pooling to yield group invariant representation. We train our equivariant networks and the conventional architectures for 300 epochs, and batch size of 16. We optimize over cross-entropy loss using Adam (Kingma & Ba 2014) with an initial learning rate of 10-2
E.8 STANFORD CARS
We evaluate the performance of our model in the presence of natural hue shifts on the Stanford Cars (Krause et al.) 2013) classification dataset. As the original host for the Stanford cars dataset is no longer maintained, the dataset was constructed from the images] the devkif" and the annotation F B The dataset consists of 8, 8,144 training examples and 8,041 testing examples.
hactteeww/aaateeeeeaaaaaliiiaaan qi htt.h//www/ooooo///oo//////////a
23

---

Published as a conference paper at ICLR 2025
We compare the classification performance of our architecture and the ResNet18 architecture proposed in He et al. (2016). Our hue-, saturation- and color-equvvariant architectures have the same number of layers as R8sNet18, but with reduced filter count to maintain a similar number of parameters at each layer. In the final layer of our networks, we perform group pooling to yield group invariant representation. We train our equivariant networks and the conventional architectures for 300 epochs, and batch size of 16, We optimize over a cross-entropy loss using Adam (Kingma & Ba 2014) with an initial learning rate of 10-2
E.9 OXFORD- -IT PETS
We evaluate the performance of our model in the presence of natural hue shifts the Oxford-IIT Pets (Parkhi et a classification dataset. The dataset consists of 3,680 training examples and 3,669 testing examples.
We compare the classification performance of our architecture and the ResNet18 architecture proposed in He et al. 2016). Our hue-, saturation- and color-cquivarian architectures have the same number of layers as ResNet18, but with a reduced filter count to maintain a similar number of parameters at each layer. In the final layer of our networks, we perform group pooling yield group invariant representation. We train our equivariant networks and the conventional architectures for 300 epochs, and batch size of 16. We optimize over cross-entropy loss using Adam (Kingma & Ba 2014) with an initial learning rate of 10-2
E. SMALL NORB
We evaluate the performance of our model in the presence of natural luminance shifts small NORB (Le Cun et al.) 2004) classification dataset. The small NORB dataset consists of 48.6k 68.6k 96x96 grayscale images captured under 6 different lighting conditions, 9 clevations, and 18 azimuths.
We compare the classification performance of our architecture and the ResNet18 architecture proposed luminance-equivariance
in (He et al. 2016). Our luminance equivariance architectures have the same number of layers as ResNet18, but with reduced filter count to maintain similar number of parameters each layer. In the final layer of our networks, we perform group pooling to yield group invariant representation. We train our equivariant networks and the conventional architectures for 300 epochs, and batch size of 16. We optimize over a cross-entropy loss using Adam (Kingma & Ba 2014) with an initial learning rate of 10 2
E.11 IMAGENET
We evaluate the performance of our model in the presence of natural hue shifts the ImageNet (Deng) cet al.] 2009) classification dataset. The dataset consists 1000 categories with 14,197,122 annotated images.
We compare the classification performance of our architecture and the ResNet18 architecture proposed in (He et al. 2016). Our que-equivariant architectures have the same number of layers as ResNet18, but with a reduced filter count to maintain a similar number of parameters at each layer. In the final layer of our networks, we perform group pooling to yield group invariant representation. We train equivariant networks and the conventional architectures for 100 epochs, and a batch size of 64. We optimize over cross-entropy loss using Adam (Kingma & Ba 2014) with an initial learning rate of 10
ADDITIONAL RESULTS
F.1 LUMINANCE SHIFT SMALL NORB
We demonstrate improved generalization to luminance-shift compared to ResNet18 (He et al. 2016) on the small NORB classification dataset (LeCun et al. (2004). The small NORB dataset consist of categories of objects under different lighting conditions. Variation in images reflects variation in the location and strength of the lighting. Additional details are provided in Appendix D33
Luminance group and group action. We expand the notion of color equivariance proposed in (Lengyel al. 2024) to include equivariance luminance shifts. In the HSL color space,
24

---

Published as a conference paper at ICLR 2025
luminance be represented by real number in the interval [0, 1]. Using the approximations presented for construction of the discretized saturation group Sy. we similarly identify elements of the discretized luminance group LN, with those of the integers with addition, (Z, +).
We define the action of the luminance group on HSL images, 2 E where a 2² + R. and functions on the discrete luminance group. y E Y, where y = x Ln RRK An element of the NN x X - X, which shifts the
luminance group acts on an HSL image by the group action luminance channel of the image. Concretely, for HSL image X defined the concatenation of hue, saturation and luminance channels, i.e., I = (Th, La, Ti), the action of an element 4 of the luminance group LN given by
#)) (Ih, 2., min(m) + 4,,c);
622)
where e is the maximum luminance value. An element of the luminance group acts function on the discrete luminance group by the group action 0i LN x - Y. which "translate" the function the group. Concretely, for function on the discrete luminance group LN defined as the concatenation of functions f (J....,,x)) the action of an element 1 in the luminance group Ly is given by
si (4, f) (fi+t..... SN. .....,0
(63)
The action of fo: on an input image, and or on a feature map are shown in Figure
Generalization luminance-shift. We expand the notion of color equivariance proposed in (Lengyel et al., 2024) to capture variation in luminance. The generalization performance of our luminance-cquivvriann models and Resnet18 (He al. 2016) are reported in Table [ Our muminance-equivarii model significantly outperforms the ResNet model in all out-of-distributtoo cases.
Lum-7* Feature Map
Lum Shifted Original
Figure 19: Luminance equivariant feature maps. We illustrate the equivariance of our luminance- equivariant model. A luminance shift in the input image space (top-left to bottom-left), results in feature map translation each layer of the network (top-right to bottom-right). Corresponding feature maps are highlighted with a blue border.
Table 5: Generalization luminance-shift. Classification error on the small NORB dataset reported. Our model (Lum-3) achieves improved generalization performance over the conventional CNN model (ResNet18).
Networks
A/A
A/B
A/C
Params
ResNet18
8.32 (0.86)
37.70 (1.51)
33.88 (5.11)
11.2M
Lum-3*
7.04 (0.65)
18.45 (4.83)
25.14 (1.07)
11.1M
F.2 IMAGENET
We include classification results on the ImageNet dataset (Degg et al. 2009 provided in Appendix E
Additional details are
F.3 FOREGROUND- BACKGROUND SEGMENTATION
We demonstrate the utility of our method dataset (Li et al.) 2022). The dataset consists of 9, 146 examples with annotations containing bounding boxes and contours, which used to generate ground truth foreground masks. The training set
segmentation the Caltech- 101
25

---

Published as a conference paper at ICLR 2025
Table Generalization to color-shift. Classification error on the ImageNet dataset is reported. Our model (Hue-3) achieves generalization performance on-par with conventional CNN model (ResNet18) and baseline (CEConv-3).
Networks
Error
Params
ResNet18
30.66
11.7M
Hue-3#
30.28
11,4M
CEConv-3
30.71
11.4M
consists of 6941 examples, the validation set consists of 868 examples, and the test set consists of 868 examples. For uniformity of the dataset, we resize all input images and target masks to have height and width of 224x224.
We compare the quantitative and qualitative performance of a conventional foreground-backee backgroound segmentation architecture with our hue-equivariant variant. Each architecture is based on the bot- tleneck segmentation backbone described in Table 7 where all convolutional layers use residual connections, and each is followed by a batch normalization layer (loff 2015) and ReLU (Agarap) 2018) activation function. In the final layer, the conventional architecture uses max pooling layer followed by sigmoid activation function to produce the predicted gegground-backkrounn segmen- tation mask. Our hue-equivariant architecture is designed similarly, but uses group pooling layer instead of max pooling layer.
We report the quantitative performance of our hue-cquivariant model and the conventional model in segmentation
Table [8] and provide a qualitative comparison of the predicted masks Figure 20
Table 7: Foreground- segmentation network backbone.
Layer #
Layer type
Kernel size
Output channels
input
3
conv
7x7
32
2
conv
7x7
64
3
conv
7x7
128
cony
7x7
256
5
Ix1
256
conv
Ixl
256
7
1x1
256
deconv
7x7
128
deconv
7x7
64
10
deconv
7x7
32
11
deconv
7x7
3
12
pooling
1
Table 8: Coreground-backgroome segmentation. Segmentation error on the Caltech-101 dataset
reported. Our proposed solor-equivariant convolutions achieve improved performance over the conventional convolutions.
Convolution
Error
Params
Conv
11.25 (0.12)
1.1M
Hue-3*
9.67 (0.23)
1.2M
26

---

Published as a conference paper at ICLR 2025
Image
Mask
Hue-3*
Conv
Figure 20: Foreground-backee background segmentation. We show the foreground mask predicted by our approach and conventional architecture. (Left right) We show the input image, the ground truth foreground segmentation mask, the foreground segmentation mask predicted by our volor-equivariant network, and the foreground segmentation mask predicted by a conventional architecture
27


Reviews:
[{"review_id": "omODcYV7fl", "reviewer": "Reviewer_y47K", "summary": "This paper introduces group convolutional neural networks designed to be equivariant to variations in hue, saturation, and color in RGB images. It proposes a lifting layer that directly transforms the input image to produce a color-equivariant descriptor, effectively addressing the challenge of invalid RGB values found in previous works. Experiments on synthetic and real-world datasets highlight the robustness of the proposed method to color variations.", "questions": "Would the proposed method improve image classification performance on large-scale datasets like ImageNet or on tasks beyond image classification?", "limitations": "", "rating": 6, "confidence": 2, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "1.\tThe approach is technically compelling, as it enables convolutional neural networks to be equivariant to hue and saturation by design, leveraging the inherent geometric structure of these color components.\n\n2.\tResults across several benchmarks indicate that the proposed method offers stronger robustness to color variation compared to existing methods.\n\n3.\tThe paper is well-structured and easy to understand.", "weaknesses": "1.\tExperiments are limited to small toy datasets with pronounced hue or saturation shifts, leaving it unclear how the method would perform on large datasets like ImageNet.\n\n2.\tThe proposed method focuses on hue and saturation equivariance, but does not address other common variations in real-world data, such as 3D geometric transformations or lighting changes.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "WJwyBDH46R", "reviewer": "Reviewer_jSPt", "summary": "The paper extends the concept of color equivariance to include saturation shifts, in addition to hue transformations, then they design GCNNs that are equivariant to color variation by leveraging the geometric structure of color in the hue-saturation-luminance (HSL) color space, additionally a novel lifting layer is proposed to transform input images directly, avoiding invalid RGB values and significantly improving equivariance error.", "questions": "See weakness part.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "1. The paper is very well written and easy to read.\n\n2. The authors conducted extensive experiments using natural images (CIFAR10, CIFAR100, STL-10, etc.) as well as medical images (Camelyon-17) to validate the effectiveness of their algorithm. The experiments are thorough and detailed.\n\n3. Color equivariant representations are an important concept in computer vision, yet they are often overlooked in contemporary research. The authors effectively claim the significance of color in the era of deep learning in this work.", "weaknesses": "I believe this work has no significant drawbacks and meets the acceptance standards for ICLR. \n\nBeyond that, I am curious about two questions:\n\n1. This paper uses the HSL color space as the basis for designing experiments, specifically designing groups and group actions for hue and saturation, as well as a hue-saturation group action. I am interested to know whether this approach would also be applicable to other chromacity-luminance color spaces, such as CIELAB or YUV.  Additionally, the experiments are designed around hue shifts in the HSI series (i.e. H in HSV. HSL, HSB...). Could other chromacity shifts in different color spaces be designed to further demonstrate the robustness of the model?\n\n2. The experiments presented in this paper focus on classification tasks. I wonder if this design could be even more beneficial for higher-level tasks or those that rely more on semantic information, such as semantic segmentation. Do the authors have any plans to design similar experiments for such tasks?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "iOkEHcP6ME", "reviewer": "Reviewer_WTM5", "summary": "The paper presents a novel convolutional neural network (CNN) architecture tailored to handle color variations effectively. The authors introduce a color-equivariant group convolutional network (GCNN) that is robust to changes in hue and saturation, significantly improving performance in tasks involving perceptual color transformations. By employing a new lifting layer that operates directly on input images rather than filters, the model overcomes limitations of existing color-equivariant networks, achieving enhanced stability and reducing equivariance errors by orders of magnitude. This approach yields superior performance on both synthetic and real-world datasets, particularly excelling in generalizing to out-of-distribution color variations and enhancing sample efficiency.", "questions": "1, The additional computational load from group convolutions and lifting layers present a challenge for real-time applications. Therefore, it is essential to compare your method with common alternatives, such as color augmentation or robust contrastive learning approaches, which are known to enhance robustness to color and perceptual variations. These comparisons would be especially valuable for demonstrating the unique advantages of color equivariance over standard augmentation techniques.", "limitations": "", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1, The paper presents a novel approach to color-equivariant convolutional neural networks (CNNs) by extending GCNNs to handle hue and saturation changes in the HSL color space. This is achieved with an inventive lifting layer that acts directly on input images, improving stability and reducing errors by avoiding invalid RGB values—a common issue in previous architectures.\n\n2, The authors demonstrate a thorough experimental design using various synthetic and real-world datasets (e.g., Hue-shift MNIST, Camelyon17) known for color variations, showing the model’s robustness. Both quantitative and qualitative analyses, including sample efficiency and feature map visualizations, offer a detailed evaluation. Comparisons with competitive baselines like CEConv and ResNet confirm the model’s generalizability and efficiency.\n\n3, The paper is well-structured, with a logical flow from problem statement to methodology and results. Clear explanations of the hue- and saturation-equivariant group actions, along with detailed figures, aid in understanding the technical concepts. Comprehensive appendices further enhance reproducibility and transparency by providing proofs and additional experiment details.", "weaknesses": "1, The paper extends GCNNs to be equivariant to hue and saturation but does not address luminance variations, which are also important for color representation. Although the authors suggest that luminance invariance may be approximately preserved, this may not hold in real-world applications, such as medical imaging or scenes with varying lighting. Future work could integrate luminance into equivariant transformations or evaluate its impact on specific tasks.\n\n2, The model is compared to standard CNNs (e.g., ResNet) and color-equivariant networks (e.g., CEConv), but not to other common approaches for handling color variations, such as color augmentation techniques or robust contrastive learning models. Adding these comparisons would strengthen the case for this method’s effectiveness in handling color shifts.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "B4m5Q8iyWT", "reviewer": "Reviewer_7Wsa", "summary": "This paper builds on the approach “Color Equivariant Convolutional Networks” [16, Neurips 2023], modifying their lifting layer by operating on the input image rather than on the filters. Additionally, it generalizes the equivariance, previously only defined on the hue channel to the  saturation channel. Experiments on synthetic datasets (hue shift MNIST and hue shift 3D shape) and on CIFAR, Caltech, Camelyon17, Caltech, STL-101, Stanford Cars, and Oxford pets dataset show improved classification performance compared to the CE-conv3 approach [16] and to Resnet baselines with same number of parameters.", "questions": "1/ Why not use the experimental protocol of [16] that seems well detailed in their paper for Table 4?\n\n2/ What would be the pros and cons of using the proposed approach vs the SOTA (or a strong baseline from the leaderboard such as Group DRO) on Camelyon17 ?\n\n3/ Could you explain how the different cardinalities have been chosen for the different experiments?\n\n4/ In the limitation section, the computation overhead of the approach is mentioned, without mentioning explicit numbers compared to the baseline’s training time, could this information be mentioned?", "limitations": "", "rating": 5, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "* The paper is well written and illustrated.\n* The approach is well motivated and addresses concrete tasks, for instance improving classification robustness in case of different imaging processes coming from different labs in the case of medical imagery. \n* The claims are supported by experiments. \n* Some limitations of the approach are presented.  \n* The code will be open source.\n* The appendix provides necessary information to reproduce results.\n* The approach designs color equivariance in group CCNN models, which is an interesting property.", "weaknesses": "* The claim of improved equivariance by order of magnitude over [16] on the 3D shape dataset is true but the error of [16] is already very low on this dataset, 0.05 at most for CEconv-3. As the error is very small, this difference in magnitude will not translate into drastic improvements in practice.\n* [Comparison to stronger baselines] The comparison to [16] on non-synthetic datasets displays larger errors than in [16]. Why not use their experimental protocol that seems well detailed in their paper? \n[Incomplete comparison] Only results with CE-conv3 and CE-conv4 appear in the current paper, while [16] often shows better results with CE-conv2.\n* [No comparison to SOTA] Further discussion on the Camelyon-17 results would be interesting: Looking at the leaderboard on this dataset, https://wilds.stanford.edu/leaderboard/ , we note a number of approaches are performing better than the proposed approach. What would be the pros and cons of using the proposed approach vs the SOTA ?       \n* The cardinality N of the group used in the experiments is not discussed. In addition, it is only introduced on page 7, it would help the reader to define the notation before as all figures are using it.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8LSoVmB9Th", "reviewer": "Reviewer_r9ZC", "summary": "The authors propose GCNNs which are equivariant to color variation. GCNNs have been widely used for geometric transformations while only a few research has attempted to apply GCNNs to color-equivariant networks to achieve generalization to out-of-distribution samples. Throughout the paper, the authors reasonably justified the reason for designing the color-equivariant networks and proposed a proper method for dealing with such situations. Both quantitative and qualitative evaluations are conducted in the paper.", "questions": "1. Fig. 1: Baseline comparison with CEConvs\n\n2. Fig. 4\n    - What is the “Model sample efficiency”? \n    - How can the error improvement be computed?\n    - Fig. 4 shows that there is no improvement over Z2CNN if 10% or 100% of training example is used. Why does it happen?\n    - On the other hand, the advantage of the proposed methods are shown effective when only a little portion of the training example is used. What does this mean?\n\n3. Table 1: Baseline comparison with CEConvs\n\n4. Fig 5: Baseline comparison with CEConvs\n\n5. Fig 7: Baseline comparison with CEConvs\n\n6. What is the difference between the global hue-shift out-of-distribution (A/B) and the local hue-shift out-of-distribution (A/C)?", "limitations": "", "rating": 3, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- Interpretable visual illustrations of the results\n- Interesting motivation\n- Decent paper writing", "weaknesses": "Overall, it is clear to me that the proposed method is better than the baselines (ResNet50, Z2CNN) under the generalization testing setup. However, the improvements over CEConvs look marginal (performance-wise and method-wise). Other weaknesses I have found are:\n\n1. Related Works: \n    - What is the connection/difference of the proposed setting with Domain Generalization?\n2. Missing baselines to support the argument in L44-50.\n    - converting input image to grayscale \n    - data augmentation", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)