SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Color variations are simulated by biased ColorMNIST, a 10-class classification problem where each class  $c$  has its own characteristic hue  $\theta_c$  defined in degrees, distributed uniformly on the hue circle. The exact color of each digit  $x$  is sampled according to  $\theta_x \sim \mathcal{N}(\theta_c, \sigma)$ . We generate multiple datasets by varying  $\sigma$  between 0 and  $10^6$ , where  $\sigma = 0$  results in a completely deterministic color for each class and  $\sigma = 10^6$  in an approximately uniform distribution for  $\theta_x$ . For small  $\sigma$ , color is thus highly informative of the class, whereas for large  $\sigma$  the classification needs to be performed based on shape. The dataset is visualized in Appendix B.1. We train all models on the train set of 1.000 samples for 1500 epochs and evaluate on the test set of 10.000 samples. The test accuracies for different  $\sigma$  are shown in Fig. 2b. CECNN outperforms Z2CNN across all standard deviations, indicating CEConvs allow for a more efficient internal color representation. The color invariant CECNN network outperforms the equivariant CECNN model from  $\sigma \geq 48$ . Above this value color is no longer informative for the classification task and merely acts as noise unnecessarily consuming model capacity, which is effectively filtered out by the color invariant networks. The results of the grayscale Z2CNN are omitted as they are significantly worse, ranging between  $89.89\% (\sigma = 0)$  and  $79.94 (\sigma = 10^6)$ . Interestingly, CECNN with coset pooling outperforms the grayscale Z2CNN. This is due to the fact that a CECNN with coset pooling is still able to distinguish between small color changes and therefore can partially exploit color information. Networks trained with color jitter are unable to exploit color information for low  $\sigma$ ; see Appendix B.2 for detailed results.
4.2 Image classification
Setup We evaluate our method for robustness to color variations on several natural image classification datasets, including CIFAR-10 and CIFAR-100 [27], Flowers-102 [35], STL-10 [6], Oxford-IIIT Pet [40], Caltech-101 [31], Stanford Cars [26] and ImageNet [10]. We train a baseline and color equivariant (CE-)ResNet [20] with 3 rotations and evaluate on a range of test sets where we gradually apply a hue shift between  $-180^{\circ}$  and  $180^{\circ}$ . For high-resolution datasets (all except CIFAR) we train a ResNet-18 architecture and use default ImageNet data augmentations: we scale to 256 pixels, random crop to 224 pixels and apply random horizontal flips. For the CIFAR datasets we use the ResNet-44 architecture and augmentations from [7], including random horizontal flips and translations of up to 4 pixels. We train models both with and without color jitter augmentation to separately evaluate the effect of equivariance and augmentation. The CE-ResNets are downscaled in width to match the parameter count of the baseline ResNets. We have also included AugMix [21] and CICnv [29] as baselines for comparison. Training is performed for 200 epochs using the Adam [25] optimizer with a learning rate of 0.001 and the OneCycle learning rate scheduler. All our experiments use PyTorch and run on a single NVIDIA A40 GPU.
Hybrid networks In our toy experiments we enforce color equivariance throughout the network. For real world datasets however, we anticipate that the later layers of a CNN may not benefit from enforcing parameter sharing between colors, if the classes of the dataset are determined by color
specific features. We therefore evaluate hybrid versions of our color equivariance networks, denoted by an integer suffix for the number of ResNet stages, out of a possible four, that use CEConvs.
Original test set	Caltech	C-10	C-100	Flowers	Ox-Pet	Cars	STL10	ImageNet
Baseline	71.61	93.69	71.28	66.79	69.87	76.54	83.80	69.71
CICConv-W	72.85	75.26	38.81	68.71	61.53	79.52	80.71	65.81
CEConv	70.16	93.71	71.37	68.18	70.24	76.22	84.24	66.85
CEConv-2	71.50	93.94	72.20	68.38	70.34	77.06	84.50	70.02
Baseline + jitter	73.93	93.03	69.23	68.75	72.71	80.59	83.91	69.37
CICConv-W + jitter	74.38	77.49	42.27	75.05	64.23	81.56	81.88	65.95
CEConv + jitter	73.58	93.51	71.12	74.17	73.29	79.79	84.16	65.57
CEConv-2 + jitter	72.61	93.86	71.35	71.72	72.80	80.32	84.46	69.42
Baseline + AugMix	71.92	94.13	72.64	75.49	76.02	82.32	84.99	-
CEConv + AugMix	70.74	94.22	72.48	78.10	75.90	80.81	85.46	-
Hue-shifted test set
Baseline	51.14	85.26	47.01	13.41	37.56	55.59	67.60	54.72
CICConv-W	71.92	74.88	37.09	59.03	60.54	78.71	79.92	64.62
CEConv	62.17	90.90	59.04	33.33	54.02	67.16	78.25	56.90
CEConv-2	64.51	91.43	62.11	33.32	51.14	68.17	77.80	62.26
Baseline + jitter	73.61	92.91	69.12	68.44	72.30	80.65	83.71	67.10
CICConv-W + jitter	74.40	77.28	42.30	75.66	63.93	81.44	81.54	65.03
CEConv + jitter	73.57	93.39	71.06	73.86	72.94	79.79	84.02	64.52
CEConv-2 + jitter	73.03	93.80	71.33	71.44	72.58	80.28	84.31	68.74
Baseline + AugMix	51.82	88.03	51.39	15.99	48.04	68.69	72.19	-
CEConv + AugMix	62.29	91.68	60.75	41.43	62.27	73.59	80.17	-
Table 1: Classification accuracy in % of vanilla vs. color equivariant (CE-)ResNets, evaluated both on the original and hue-shifted test sets. Color equivariant CNNs perform on par with vanilla CNNs on the original test sets, but are significantly more robust to test-time hue shifts.
Results We report both the performance on the original test set, as well as the average accuracy over all hue shifts in Table 1. For brevity we only show the fully equivariant and hybrid-2 networks, a complete overview of the performances of all hybrid network configurations and error standard deviations can be found in Appendix C.1. Between the full color equivariant and hybrid versions of our CE-ResNets, at least one variant outperforms vanilla ResNets on most datasets on the original test set. On most datasets the one- or two-stage hybrid versions are the optimal CE-ResNets, providing a good trade-off between color equivariance and leaving the network free to learn color specific features in later layers. CE-ResNets are also significantly more robust to test-time hue shifts, especially when trained without color jitter augmentation. Training the CE-ResNets with color jitter further improves robustness, indicating that train-time augmentations complement the already hard-coded inductive biases in the network. We show the detailed performance on Flowers-102 for all test-time hue shifts in Fig. 1b. The accuracy of the vanilla CNN quickly drops as a hue shift is applied, whereas the CE-CNN performance peaks at  $-120^{\circ}$ ,  $0^{\circ}$  and  $120^{\circ}$ . Applying train-time color jitter improves the CNN's robustness to the level of a CNN with grayscale inputs. The CE-CNN with color jitter outperforms all models for all hue shifts. Plots for other datasets are provided in Appendix C.2.
Color selectivity To explore what affects the success of color equivariance, we investigate the color selectivity of a subset of the studied datasets. We use the color selectivity measure from [42] and average across all neurons in the baseline model trained on each dataset. Fig. 3 shows that color selective datasets benefit from using color equivariance up to late stages, whereas less color selective datasets do not.
Feature representations of color equivariant CNNs We use the Neuron Feature [42] (NF) visualization method to investigate the internal feature representation of the CE-ResNet. NF computes a weighted average of the  $N$  highest activation input patches for each filter at a certain layer, as such representing the input patch that a specific neuron fires on. Fig. 4 shows the NF  $(N = 50)$  and top-3 input patches for filters at the final layers of stages 1-4 of a CE-ResNet18 trained on Flowers-102.
Color Equivariant stages vs. accuracy improvement
Figure 3: Color selective datasets benefit from using color equivariance up to late stages, whereas less color selective datasets do not. We compute average color selectivity [42] of all neurons in the baseline CNN trained on each dataset, and plot the accuracy improvement of using color equivariance in hybrid and full models, coloring each graphed dataset for color selectivity.
Figure 4: Neuron Feature [42] (NF) visualization with top-3 patches at different stages of a CE-ResNet18 trained on Flowers-102. Rows represent different rotations of the same filter. As expected, each row of a NF activates on the same shape in a different color.
Different rows represent different rotations of the same filter. As expected, each row of a NF activates on the same shape in a different color, demonstrating the color sharing capabilities of CEConvs. More detailed NF visualization are provided in Appendix C.4.
Ablation studies We perform ablations to investigate the effect of the number of rotations, the use of group coset pooling, and the strength of train-time color jitter augmentations. In short, we find that a) increasing the number of hue rotations increases robustness to test-time hue shifts at the cost of a slight reduction in network capacity, b) removing group coset pooling breaks hue invariance, and c) hue equivariant networks require lower intensity color jitter augmentations to achieve the same test-time hue shift robustness and accuracy. The full results can be found in Appendix D.
In this work, we propose Color Equivariant Convolutions (CEConvs) which enable feature sharing across colors in the data, while retaining discriminative power. Our toy experiments demonstrate benefits for datasets where the color distribution is long-tailed or biased. Our proposed fully equivariant CECNNs improve performance on datasets where features are color selective, while hybrid versions that selectively apply CEConvs only in early stages of a CNN benefit various classification tasks.
Limitations CEConvs are computationally more expensive than regular convolutions. For fair comparison, we have equalized the parameter cost of all models compared, at the cost of reducing the number of channels of CECNNs. In cases where color equivariance is not a useful prior, the reduced capacity hurts model performance, as reflected in our experimental results.
Pixel values near the borders of the RGB cube can fall outside the cube after rotation, and subsequently need to be reprojected. Due to this clipping effect the hue equivariance in Eq. (9) only holds approximately. As demonstrated empirically, this has only limited practical consequences, yet future work should investigate how this shortcoming could be mitigated.
Local vs. global equivariance The proposed CEConv implements local hue equivariance, i.e. it allows to model local color changes in different regions of an image separately. In contrast, global equivariance, e.g. by performing hue shifts on the full input image, then processing all inputs with the same CNN and combining representations at the final layer to get a hue-equivariant representation, encodes global equivariance to the entire image. While we have also considered such setup, initial experiments did not yield promising results. The theoretical benefit of local over global hue equivariance is that multiple objects in one image can be recognized equivariantly in any combination of hues - empirically this indeed proves to be a useful property.
Future work The group of hue shifts is but one of many possible transformations groups on images. CNNs naturally learn features that vary in both photometric and geometric transformations [5, 37]. Future work could combine hue shifts with geometric transformations such as roto-translation [7] and scaling [49]. Also, other photometric properties could be explored in an equivariance setting, such as saturation and brightness.
Our proposed method rotates the hue of the inputs by a predetermined angle as encoded in a rotation matrix. Making this rotation matrix learnable could yield an inexact but more flexible type of color equivariance, in line with recent works on learnable equivariance [34, 46]. An additional line of interesting future work is to incorporate more fine-grained equivariance to continuous hue shifts, which is currently intractable within the GConv-inspired framework as the number multiply-accumulate operations grow quadratically with the number of hue rotations.
Broader impact Improving performance on tasks where color is a discriminative feature could affect humans that are the target of discrimination based on the color of their skin. CEConvs ideally benefit datasets with long-tailed color distributions by increasing robustness to color changes, in theory reducing a CNN's reliance on skin tone as a discriminating factor. However, careful and rigorous evaluation is needed before such properties can be attributed to CECNNs with certainty.
This project is supported in part by NWO (project VI.Vidi.192.100).
[1] Naif Alshammari, Samet Akcay, and Toby P. Breckon. On the impact of illumination-invariant image pre-transformation for contemporary automotive semantic scene understanding. In 2018 IEEE Intelligent Vehicles Symposium (IV), pages 1027-1032, 2018.
[2] Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen A. J. Eppenhof, Josien P. W. Pluim, and Remco Duits. Roto-translation covariant convolutional networks for medical image analysis. In Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, pages 440–448. Springer International Publishing, 2018.
[3] Inês Bramão, Luís Fáisca, Karl Magnus Petersson, and Alexandra Reis. The contribution of color to object recognition. In Ioannis Kypraios, editor, Advances in Object Recognition Systems, chapter 4. IntechOpen, Rijeka, 2012.
[4] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Velickovic. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges. CoRR, abs/2104.13478, 2021.
[5] Robert-Jan Bruintjes, Tomasz Motyka, and Jan van Gemert. What affects learned equivariance in deep image recognition models? arXiv preprint arXiv:2304.02628, 2023.
[6] Adam Coates, Andrew Ng, and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelligence and statistics, pages 215-223. JMLR Workshop and Conference Proceedings, 2011.
[7] Taco S. Cohen and Max Welling. Group equivariant convolutional networks. In Proceedings of the 33rd International Conference on International Conference on Machine Learning - Volume 48, ICML'16, page 2990-2999. JMLR.org, 2016.
[8] Ian R. Cole. Modelling CPV. 6 2015.
[9] Marco Cotogni and Claudio Cusano. Offset equivariant networks and their applications. Neurocomputing, 502:110-119, 2022.
[10] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition, pages 248–255, 2009.
[11] Li Deng. The mnist database of handwritten digit images for machine learning research. IEEE Signal Processing Magazine, 29(6):141-142, 2012.
[41] Ivet Rafegas and Maria Vanrell. Color representation in cnns: Parallelisms with biological vision. In 2017 IEEE International Conference on Computer Vision Workshops (ICCVW), pages 2697-2705, 2017.
[42] Ivet Rafegas and Maria Vanrell. Color encoding in biologically-inspired convolutional neural networks. Vision Research, 151:7-17, 2018. Color: cone opponency and beyond.
[43] Ivet Rafegas, Maria Vanrell, Luis A. Alexandre, and Guillem Arias. Understanding trained cnns by indexing neuron selectivity. Pattern Recognition Letters, 136:318-325, 2020.
[44] M. Rath and A. Condurache. Boosting deep neural networks with geometrical prior knowledge: A survey. ArXiv, abs/2006.16867, 2020.
[45] B Regan, C Julliot, Bruno Simmen, Françoise Vienot, P.C. Charles-Dominique, and John Mollon. Fruits, foliage and the evolution of primate colour vision. Philosophical transactions of the Royal Society of London. Series B, Biological sciences, 356:229-83, 03 2001.
[46] David W. Romero and Suhas Lohit. Learning partial equivariances from data. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems, volume 35, pages 36466-36478. Curran Associates, Inc., 2022.
[47] Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In I. Guyon, U. Von Luxburg, S. Bengio, H. Wallach, R. Fergus, S. Vishwanathan, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
[48] Christos Sakaridis, Dengxin Dai, and Luc Van Gool. Guided curriculum model adaptation and uncertainty-aware evaluation for semantic nighttime image segmentation. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV), October 2019.
[49] Ivan Sosnovik, Artem Moskalev, and Arnold Smeulders. Disco: accurate discrete scale convolutions. In Proceedings of the 32nd British Machine Vision Conference (BMVC), 2021.
[50] Ivan Sosnovik, Michal Szmaja, and Arnold Smeulders. Scale-equivariant steerable networks. In International Conference on Learning Representations, 2020.
[51] Jan C Van Gemert. Exploiting photographic style for category-level image classification by generalizing the spatial pyramid. In Proceedings of the 1st ACM International Conference on Multimedia Retrieval, pages 1-8, 2011.
[52] Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
[53] Daniel Worrall and Max Welling. Deep scale-spaces: Equivariance over scale. In Advances in Neural Information Processing Systems, volume 32, pages 7366-7378. Curran Associates, Inc., 2019.
[54] Xuanyu Zhu, Yi Xu, Hongteng Xu, and Changjian Chen. Quaternion convolutional neural networks. In Proceedings of the European Conference on Computer Vision (ECCV), September 2018.
A Derivation of  $H_{n}$
Rotation around an arbitrary unit vector  $\mathbf{u}$  by angle  $\theta$  can be decomposed into five simple steps [8]:
1. rotating the vector such that it lies in one of the coordinate planes, e.g.  $xz$  using  $M_{xz}$
2. rotating the vector such that it lies on one of the coordinate axes, e.g.  $x$  using  $M_{x}$
3. rotating the point around vector  $\mathbf{u}$  on axis  $x$  using  $R_{x}$ ;
4. reversing the rotation in step 2. using  $M_x^{-1} = M_x^T$ ;
5. reversing the rotation in step 1. using  $M_{xz}^{-1} = M_{xz}^{T}$
These operations can be combined into a single matrix:
$$
\begin{array}{l} R _ {\mathbf {u}, \theta} = M _ {x z} ^ {T} \left(M _ {x} ^ {T} \left(R _ {x, \theta} \left(M _ {x z} \left(M _ {x z}\right)\right)\right)\right) (16) \\ = M _ {x z} ^ {T} M _ {x} ^ {T} R _ {x, \theta} M _ {x z} M _ {x z} (17) \\ = \left[ \begin{array}{c c c} \cos \theta + u _ {x} ^ {2} (1 - \cos \theta) & u _ {x} u _ {y} (1 - \cos \theta) - u _ {z} \sin \theta & u _ {x} u _ {z} (1 - \cos \theta) + u _ {y} \sin \theta \\ u _ {y} u _ {x} (1 - \cos \theta) + u _ {z} \sin \theta & \cos \theta + u _ {y} ^ {2} (1 - \cos \theta) & u _ {y} u _ {z} (1 - \cos \theta) - u _ {x} \sin \theta \\ u _ {z} u _ {x} (1 - \cos \theta) - u _ {y} \sin \theta & u _ {z} u _ {y} (1 - \cos \theta) + u _ {x} \sin \theta & \cos \theta + u _ {z} ^ {2} (1 - \cos \theta) \end{array} \right]. (18) \\ \end{array}
$$
Substituting  $\mathbf{u} = \left[\frac{1}{\sqrt{3}}, \frac{1}{\sqrt{3}}, \frac{1}{\sqrt{3}}\right]$  yields
$$
R _ {\mathbf {u}, \theta} = \left[ \begin{array}{c c c} \cos \theta + \frac {1}{3} (1 - \cos \theta) & \frac {1}{3} (1 - \cos \theta) - \frac {1}{\sqrt {3}} \sin \theta & \frac {1}{3} (1 - \cos \theta) + \frac {1}{\sqrt {3}} \sin \theta \\ \frac {1}{3} (1 - \cos \theta) + \frac {1}{\sqrt {3}} \sin \theta & \cos \theta + \frac {1}{3} (1 - \cos \theta) & \frac {1}{3} (1 - \cos \theta) - \frac {1}{\sqrt {3}} \sin \theta \\ \frac {1}{3} (1 - \cos \theta) - \frac {1}{\sqrt {3}} \sin \theta & \frac {1}{3} (1 - \cos \theta) + \frac {1}{\sqrt {3}} \sin \theta & \cos \theta + \frac {1}{3} (1 - \cos \theta) \end{array} \right], \tag {19}
$$
and lastly, rearranging and substituting  $\theta = \frac{2k\pi}{n}$  results in
$$
H _ {n} (k) = \left[ \begin{array}{c c c} \cos \left(\frac {2 k \pi}{n}\right) + a & a - b & a + b \\ a + b & \cos \left(\frac {2 k \pi}{n}\right) + a & a - b \\ a - b & a + b & \cos \left(\frac {2 k \pi}{n}\right) + a \end{array} \right]. \tag {20}
$$
with  $n$  the total number of discrete rotations in the group,  $k$  the rotation,  $a = \frac{1}{3} - \frac{1}{3}\cos\left(\frac{2k\pi}{n}\right)$  and  $b = \sqrt{\frac{1}{3}}*\sin\left(\frac{2k\pi}{n}\right)$ .
B ColorMNIST
B.1 Dataset visualization
Long-tailed ColorMNIST dataset The training samples of the Longtailed ColorMNIST dataset are depicted in Fig. 5, clearly indicating a class imbalance.
Figure 5: Long-tailed ColorMNIST. Note the strong class imbalance in the dataset. Best viewed in color.
Biased ColorMNIST dataset A small subset of the samples of Biased ColorMNIST is shown in Fig. 6 for  $\sigma = 0$  (a) and  $\sigma = 36$  (b), respectively. Note that the samples in (a) have a deterministic color, whereas in (b) exhibit some variation in hue.
(b)
Figure 6: Samples from Biased ColorMNIST for  $\sigma = 0$  (a) and  $\sigma = 36$  (b), respectively. Best viewed in color.
B.2 Additional experiments
Results with color jitter augmentation We performed both ColorMNIST experiments with color jitter augmentations. The results are shown in Fig. 7. (a) For long-tailed ColorMNIST, adding jitter makes solving the classification problem prohibitive, as color is required. Z2CNN and CECNN with jitter therefore perform no better than as the CECNN model with coset pooling. (b) For biased MNIST, performance decreases for small and improves for large  $\sigma$ , with CEConv still performing best.
(b)
Figure 7: Color equivariant convolutions efficiently share shape information across different colors. CECNN outperforms a vanilla network in both a long-tailed class imbalance setting (a), where MNIST digits are to be classified based on both shape and color, and a color biased setting (b), where the color of each class  $c$  is sampled according to  $\theta_d \sim \mathcal{N}(\theta_c, \sigma)$ .
Long-tailed ColorMNIST with weighted loss We performed the longtailed ColorMNIST experiment both with a uniformly weighted loss and a loss where classes are weighted inversely to their frequency according to  $w_{i} = \frac{N}{c*n_{i}}$ , where  $w_{i}$  denotes the weight for class  $i$ ,  $N$  the number of samples in the training set,  $c$  the number of classes, and  $n_{i}$  the number of samples for class  $i$ . The results are shown in Fig. 8. We observed no significant difference between the two setups, with the CECNN without coset pooling outperforming the other models by a large margin in both.
Figure 8: Per-class accuracy of various models trained with a loss function weighted by inverse class frequency. CECNN without coset pooling outperforms all other models, with no significant differences compared to an uniformly weighted loss function.
C Classification experiments
C.1 Overview of all CE-ResNet configurations
Table 2 shows an overview of the classification accuracies of all baselines and equivariant architectures. CEConv-x denotes the number of ResNet stages with CE convolutions with CEConv-4 (3 for CIFAR) being a fully equivariant ResNet. In nearly all cases, early equivariance is beneficial for improving classification accuracy on both the original as well as the hue shifted test sets. In case of the Flowers-
102 dataset late equivariance seems to have a significant advantage, whereas for Caltech-101 and Stanford Cars the color equivariance bias does not seem to have much added value.
Table 2: Classification accuracy on various datasets. CEConv- $s$  denotes a ResNet with  $s$  color equivariant stages. We report results for models trained with and without color jitter augmentation. (Hybrid) color equivariant networks improve performance over the baseline model on both the original as well as the hue-shifted test set.
C.2 Test-time hue shift plots