SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Attila Lengyel Ombretta Strafforello Robert-Jan Bruintjes Alexander Gielisse Jan van Gemert
Computer Vision Lab  
Delft University of Technology  
Delft, The Netherlands
Color is a crucial visual cue readily exploited by Convolutional Neural Networks (CNNs) for object recognition. However, CNNs struggle if there is data imbalance between color variations introduced by accidental recording conditions. Color invariance addresses this issue but does so at the cost of removing all color information, which sacrifices discriminative power. In this paper, we propose Color Equivariant Convolutions (CEConvs), a novel deep learning building block that enables shape feature sharing across the color spectrum while retaining important color information. We extend the notion of equivariance from geometric to photometric transformations by incorporating parameter sharing over hue-shifts in a neural network. We demonstrate the benefits of CEConvs in terms of downstream performance to various tasks and improved robustness to color changes, including train-test distribution shifts. Our approach can be seamlessly integrated into existing architectures, such as ResNets, and offers a promising solution for addressing color-based domain shifts in CNNs.
Color is a powerful cue for visual object recognition. Trichromatic color vision in primates may have developed to aid the detection of ripe fruits against a background of green foliage [38, 45]. The benefit of color vision here is two-fold: not only does color information improve foreground-background segmentation by rendering foreground objects more salient, color also allows diagnostics, e.g. identifying the type (orange) and ripeness (green) where color is an intrinsic property facilitating recognition [3], as illustrated in Fig. 1a. Convolutional neural networks (CNNs) too exploit color information by learning color selective features that respond differently based on the presence or absence of a particular color in the input [42].
Unwanted color variations, however, can be introduced by accidental scene recording conditions such as illumination changes [29, 48], or by low color-diagnostic objects occurring in a variety of colors, making color no longer a discriminative feature but rather an undesired source of variation in the data. Given a sufficiently large training set that encompasses all possible color variations, a CNN learns to become robust by learning color invariant and equivariant features from the available data [36, 37]. However, due to the long tail of the real world it is almost impossible to collect balanced training data for all scenarios. This naturally leads to color distribution shifts between training and test time, and an imbalance in the training data where less frequently occurring colors are underrepresented. As CNNs often fail to generalize to out-of-distribution test samples, this can have significant impact on many real-world applications, e.g. a model trained mostly on red cars may struggle to recognize the exact same car in blue.
Color invariance addresses this issue through features that are by design invariant to color changes and therefore generalize better under appearance variations [14, 17]. However, color invariance comes at the loss of discriminative power as valuable color information is removed from the model's
Figure 1: Color plays a significant role in object recognition. (a) The absence of color makes flowers less distinct from their background and thus harder to classify. The characteristic purple-blue color of the Monkshood (Class A) enables a clear distinction from the Snapdragon (Class B) [35]. On the other hand, relying too much on colors might negatively impact recognition to color variations within the same flower class. (b) Image classification performance on the Flower-102 dataset [35] under a gradual variation of the image hue. Test-time hue shifts degrade the performance of CNNs (ResNet-18) drastically. Grayscale images and color augmentations result in invariance to hue variations, but fail to capture all the characteristic color features of flowers. Our color equivariant network (CE-ResNet-18-1) enables feature sharing across the color spectrum, which helps generalise to underrepresented colors in the dataset, while preserving discriminative color information, improving classification for unbalanced color variations.
internal feature representation [18]. We therefore propose to equip models with the less restrictive color equivariance property, where features are explicitly shared across different colors through a hue transformation on the learned filters. This allows the model to generalize across different colors, while at the same time also retaining important color information in the feature representation.
An RGB pixel can be decomposed into an orthogonal representation by the well-known hue-saturation-value (HSV) model, where hue represents the chromaticity of a color. In this work we extend the notion of equivariance from geometric to photometric transformations by hard-wiring parameter sharing over hue-shifts in a neural network. More specifically, we build upon the seminal work of Group Equivariant Convolutions [7] (GConvs), which implements equivariance to translations, flips and rotations of multiples of 90 degrees, and formulates equivariance using the mathematical framework of symmetry groups. We introduce Color Equivariant Convolutions (CEConvs) as a novel deep learning building block, which implements equivariance to the  $H_{n}$  symmetry group of discrete hue rotations. CEConvs share parameters across hue-transformed filters in the input layer and store color information in hue-equivariant feature maps.
CEConv feature maps contain an additional dimension compared to regular CNNs, and as a result, require larger filters and thus more parameters for the same number of channels. To evaluate equivariant architectures, it is common practice to reduce the width of the network to match the parameter count of the baseline model. However, this approach introduces a trade-off between equivariance and model capacity, where particularly in deeper layers the quadratic increase in parameter count of CEConv layers makes equivariance computationally expensive. We therefore investigate hybrid architectures, where early color invariance is introduced by pooling over the color dimension of the feature maps. Note that early color invariance is maintained throughout the rest of the network, despite the use of regular convolutional layers after the pooling operation. Limiting color equivariant filters to the early layers is in line with the findings that early layers tend to benefit the most from equivariance [5] and learn more color selective filters [37, 42].
We rigorously validate the properties of CEConvs empirically through precisely controlled synthetic experiments, and evaluate the performance of color invariant and equivariant ResNets on various more realistic classification benchmarks. Moreover, we investigate the combined effects of color equivariance and color augmentations. Our experiments show that CEConvs perform on par or better
than regular convolutions, while at the same time significantly improving the robustness to test-time color shifts, and is complementary to color augmentations.
- We show that convolutional neural networks benefit from using color information, and at the same time are not robust to color-based domain shifts.
- We introduce Color Equivariant Convolutions (CEConvs), a novel deep learning building block that allows feature sharing between colors and can be readily integrated into existing architectures such as ResNets.
- We demonstrate that CEConvs improve robustness to train-test color shifts in the input.
All code and experiments are made publicly available on https://github.com/Attila94/CEConv.
**Equivariant architectures** Translation equivariance is a key property of convolutional neural networks (CNNs) [23, 28]: shifting the input to a convolution layer results in an equally shifted output feature map. This allows CNNs to share filter parameters over spatial locations, which improves both parameter and data efficiency as the model can generalize to new locations not covered by the training set. A variety of methods have extended equivariance in CNNs to other geometric transformations [44], including the seminal Group Equivariant Convolutions [7] for rotations and flips, and other works concerning rotations [2, 30, 52], scaling [50, 53] and arbitrary Lie groups [32]. Yet to date, equivariance to photometric transformations has remained largely unexplored. Offset equivariant networks [9] constrain the trainable parameters such that an additive bias to the RGB input channels results in an equal bias in the output logits. By applying a log transformation to the input the network becomes equivariant to global illumination changes according to the Von Kries model [13]. In this work we explore an alternative approach to photometric equivariance inspired by the seminal Group Equivariant Convolution [7] framework.
Color in CNNs Recent research has investigated the internal representation of color in Convolutional Neural Networks (CNNs), challenging the traditional view of CNNs as black boxes. For example, [41, 42] introduces the Neuron Feature visualization technique and characterizes neurons in trained CNNs based on their color selectivity, assessing whether a neuron activates in response to the presence of color in the input. The findings indicate that networks learn highly color-selective neurons across all layers, emphasizing the significance of color as a crucial visual cue. Additionally, [43] classifies neurons based on their class selectivity and observes that early layers contain more class-agnostic neurons, while later layers exhibited high class selectivity. A similar study has been performed in [12], further supporting these findings. [36, 37] investigate learned symmetries in an InceptionV1 model trained on ImageNet [10] and discover filters that demonstrated equivariance to rotations, scale, hue shifts, and combinations thereof. These results motivate color equivariance as a prior for CNNs, especially in the first layers. Moreover, in this study, we will employ the metrics introduced by [42] to provide an explanation for several of our own findings.
Color priors in deep learning Color is an important visual discriminator [15, 19, 51]. In classical computer vision, color invariants are used to extract features from an RGB image that are more consistent under illumination changes [14, 17, 18]. Recent studies have explored using color invariants as a preprocessing step to deep neural networks [1, 33] or incorporating them directly into the architecture itself [29], leading to improved robustness against time-of-day domain shifts and other illumination-based variations in the input. Capsule networks [22, 47], which use groups of neurons to represent object properties such as pose and appearance, have shown encouraging results in image colorization tasks [39]. Quaternion networks [16, 54] represent RGB color values using quaternion notation, and employ quaternion convolutional layers resulting in moderate improvements in image classification and inpainting tasks. Building upon these advancements, we contribute to the ongoing research on integrating color priors within deep neural architectures.
3 Color equivariant convolutions
3.1 Group Equivariant Convolutions
A CNN layer  $\Phi$  is equivariant to a symmetry group  $G$  if for all transformations  $g\in G$  on the input  $x$  the resulting feature mapping  $\Phi (x)$  transforms similarly, i.e., first doing a transformation and then the mapping is similar to first doing the mapping and then the transformation. Formally, equivariance is defined as
$$
\Phi \left(T _ {g} x\right) = T _ {g} ^ {\prime} \Phi (x), \quad \forall g \in G, \tag {1}
$$
where  $T_{g}$  and  $T_{g}^{\prime}$  are the transformation operators of group action  $g$  on the input and feature space, respectively. Note that  $T_{g}$  and  $T_{g}^{\prime}$  can be identical, as is the case for translation equivariance where shifting the input results in an equally shifted feature map, but do not necessarily need to be. A special case of equivariance is invariance, where  $T_{g}^{\prime}$  is the identity mapping and the input transformation leaves the feature map unchanged:
$$
\Phi (T _ {g} x) = \Phi (x), \quad \forall g \in G. \tag {2}
$$
We use the definition from [7] to denote the  $i$ -th output channel of a standard convolutional layer  $l$  in terms of the correlation operation  $(\star)$  between a set of feature maps  $f$  and  $C^{l+1}$  filters  $\psi$ :
$$
[ f \star \psi^ {i} ] (x) = \sum_ {y \in \mathbb {Z} ^ {2}} \sum_ {c = 1} ^ {C ^ {l}} f _ {c} (y) \psi_ {c} ^ {i} (y - x). \tag {3}
$$
Here  $f:\mathbb{Z}^2\to \mathbb{R}^{C^l}$  and  $\psi^i:\mathbb{Z}^2\to \mathbb{R}^{C^l}$  are functions that map pixel locations  $x$  to a  $C^l$ -dimensional vector. This definition can be extended to groups by replacing the translation  $x$  by a group action  $g$ :
$$
[ f \star \psi^ {i} ] (g) = \sum_ {y \in \mathbb {Z} ^ {2}} \sum_ {c} ^ {C ^ {l}} f _ {c} (y) \psi_ {c} ^ {i} \left(g ^ {- 1} y\right) \tag {4}
$$
As the resulting feature map  $f \star \psi^i$  is a function on  $G$  rather than  $\mathbb{Z}^2$ , the inputs and filters of all hidden layers should also be defined on  $G$ :
$$
[ f \star \psi^ {i} ] (g) = \sum_ {h \in G} \sum_ {c} ^ {C ^ {l}} f _ {c} (h) \psi_ {c} ^ {i} \left(g ^ {- 1} h\right) \tag {5}
$$
Invariance to a subgroup can be achieved by applying a pooling operation over the corresponding cosets. For a more detailed introduction to group equivariant convolutions, please refer to [4, 7].
3.2 Color Equivariance
We define color equivariance as equivariance to hue shifts. The HSV color space encodes hue by an angular scalar value, and a hue shift is performed as a simple additive offset followed by a modulo operator. When projecting the HSV representation into three-dimensional RGB space, the same hue shift becomes a rotation along the  $[1, 1, 1]$  diagonal vector.
We formulate hue equivariance in the framework of group theory by defining the group  $H_{n}$  of multiples of  $360 / n$ -degree rotations about the  $[1,1,1]$  diagonal vector in  $\mathbb{R}^3$  space.  $H_{n}$  is a subgroup of the  $SO(3)$  group of all rotations about the origin of three-dimensional Euclidean space. We can parameterize  $H$  in terms of integers  $k,n$  as
$$
H _ {n} (k) = \left[ \begin{array}{c c c} \cos \left(\frac {2 k \pi}{n}\right) + a & a - b & a + b \\ a + b & \cos \left(\frac {2 k \pi}{n}\right) + a & a - b \\ a - b & a + b & \cos \left(\frac {2 k \pi}{n}\right) + a \end{array} \right] \tag {6}
$$
with  $n$  the total number of discrete rotations in the group,  $k$  the rotation,  $a = \frac{1}{3} -\frac{1}{3}\cos \left(\frac{2k\pi}{n}\right)$  and  $b = \sqrt{\frac{1}{3}} *\sin \left(\frac{2k\pi}{n}\right)$ . The group operation is matrix multiplication which acts on the continuous  $\mathbb{R}^3$  space of RGB pixel values. The derivation of  $H_{n}$  is provided in Appendix A.
Color Equivariant Convolution (CEConv) Let us define the group  $G = \mathbb{Z}^2 \times H_n$ , which is a direct product of the  $\mathbb{Z}^2$  group of discrete 2D translations and the  $H_n$  group of discrete hue shifts. We can then define the Color Equivariant Convolution (CEConv) in the input layer as:
$$
[ f \star \psi^ {i} ] (x, k) = \sum_ {y \in \mathbb {Z} ^ {2}} \sum_ {c = 1} ^ {C ^ {l}} f _ {c} (y) \cdot H _ {n} (k) \psi_ {c} ^ {i} (y - x). \tag {7}
$$
We furthermore introduce the operator  $\mathcal{L}_g = \mathcal{L}_{(t,m)}$  including translation  $t$  and hue shift  $m$  acting on input  $f$  defined on the plane  $\mathbb{Z}^2$ :
$$
[ \mathcal {L} _ {g} f ] (x) = [ \mathcal {L} _ {(t, m)} f ] (x) = H _ {n} (m) f (x - t) \tag {8}
$$
Since  $H_{n}$  is an orthogonal matrix, the dot product between a hue shifted input  $H_{n}f$  and a filter  $\psi$  is equal to the dot product between the original input  $f$  and the inverse hue shifted filter  $H_{n}^{-1}\psi$ :
$$
H _ {n} f \cdot \psi = \left(H _ {n} f\right) ^ {T} \psi = f ^ {T} H _ {n} ^ {T} \psi = f \cdot H _ {n} ^ {T} \psi = f \cdot H _ {n} ^ {- 1} \psi . \tag {9}
$$
Then the equivariance of the CEConv layer can be derived as follows (using  $C^l = 1$  for brevity):
$$
\begin{array}{l} [ [ \mathcal {L} _ {(t, m)} f ] \star \psi^ {i} ] (x, k) = \sum_ {y \in \mathbb {Z} ^ {2}} H _ {n} (m) f (y - t) \cdot H _ {n} (k) \psi^ {i} (y - x) \\ = \sum_ {y \in \mathbb {Z} ^ {2}} f (y) \cdot H _ {n} (m) ^ {- 1} H _ {n} (k) \psi^ {i} (y - (x - t)) \\ = \sum_ {y \in \mathbb {Z} ^ {2}} f (y) \cdot H _ {n} (k - m) \psi^ {i} (y - (x - t)) \tag {10} \\ = [ f \star \psi^ {i} ] (x - t, k - m) \\ = \left[ \mathcal {L} _ {(t, m)} ^ {\prime} [ f \star \psi^ {i} ] \right] (x, k) \\ \end{array}
$$
Since input  $f$  and feature map  $[f \star \psi]$  are functions on  $\mathbb{Z}^2$  and  $G$ , respectively,  $\mathcal{L}_{(t,k)}$  and  $\mathcal{L}'_{(t,k)}$  represent two equivalent operators acting on their respective groups. For all subsequent hidden layers the input  $f$  and filters  $\psi^i$  are functions on  $G$  parameterized by  $x, k$ , and the hidden layer for CEConv is defined as:
$$
\left[ f \star \psi^ {i} \right] (x, k) = \sum_ {y \in \mathbb {Z} ^ {2}} \sum_ {r = 1} ^ {n} \sum_ {c = 1} ^ {C ^ {l}} f _ {c} (y, r) \cdot \psi_ {c} ^ {i} (y - x, (r - k) \% n), \tag{11}
$$
where  $n$  is the number of discrete rotations in the group and  $\%$  is the modulo operator. In practice, applying a rotation to RGB pixels will cause some pixel values to fall outside of the RGB cube, which will then have to be reprojected within the cube. Due to this discrepancy, Eq. (9) only holds approximately, though in practice this has only limited consequences, as we empirical show in Appendix D.
Tensor operations We implement CEConv similarly to GConv [7]. GConv represents the pose associated with the added spatial rotation group by extending the feature map tensor  $X$  with an extra dimension  $G^{l}$  to size  $[C^l, G^l, H, W]$ , denoting the number of channels, transformations that leave the origin invariant, and height and width of the feature map at layer  $l$ , respectively (batch dimension omitted). Similarly, a GConv filter  $\tilde{F}$  with spatial extent  $k$  is of size  $[C^{l + 1}, G^{l + 1}, C^l, G^l, k, k]$ . The GConv is then defined in terms of tensor multiplication operations as:
$$
X _ {c ^ {\prime}, g ^ {\prime}, \therefore ;} ^ {l + 1} = \sum_ {c} ^ {C ^ {l}} \sum_ {g} ^ {G ^ {l}} \tilde {F} _ {c ^ {\prime}, g ^ {\prime}, c, g, \therefore ;} ^ {l} \star X _ {c, g, \therefore ;} ^ {l}, \tag {12}
$$
where  $(\cdot)$  denotes tensor slices. Note that in the implementation, a GConv filter  $F$  only contains  $[C^{l + 1}, C^l, G^l, k, k]$  unique parameters - the extra  $G^{l + 1}$  dimension is made up of transformed copies of  $F$ .
As the RGB input to the network is defined on  $\mathbb{Z}^2$ , we have  $G^{1} = 1$  and  $\tilde{F}$  has size  $[C^{l + 1}, G^{l + 1}, 3, 1, k, k]$ . The transformed copies in  $G^{l + 1}$  are computed by applying the rotation matrix from Eq. (6):
$$
\tilde {F} _ {c ^ {\prime}, g ^ {\prime},:, 1, u, v} ^ {1} = H _ {n} \left(g ^ {\prime}\right) F _ {c ^ {\prime},:, 1, u, v} ^ {1}. \tag {13}
$$
In the hidden layers  $\tilde{F}$  contains cyclically permuted copies of  $F$ :
$$
\tilde {F} _ {c ^ {\prime}, g ^ {\prime}, c, g, u, v} ^ {l} = F _ {c ^ {\prime}, c, (g + g ^ {\prime}) \% n, u, v} ^ {l}. \tag{14}
$$
Furthermore, to explicitly share the channel-wise spatial kernel over  $G^l$  [30], filter  $F$  is decomposed into a spatial component  $S$  and a pointwise component  $P$  as follows:
$$
F _ {c ^ {\prime}, c, g, u, v} ^ {l} = S _ {c ^ {\prime}, c, 1, u, v} \cdot P _ {c ^ {\prime}, g ^ {\prime}, c, g, 1, 1} \tag {15}
$$
$F$  is precomputed in each forward step prior to the convolution operation in Eq. (12).
Input normalization is performed using a single value for the mean and standard deviations rather than per channel, as is commonly done for standard CNNs. Channel-wise means and standard deviations break the equivariance property of CECNN as a hue shift could no longer be defined as a rotation around the  $[1, 1, 1]$  diagonal. Experiments have shown that using a single value for all channels instead of channel-wise normalization has no effect on the performance.
Compute efficiency CEConvs create a factor  $|H_{n}|$  more feature maps in each layer. Due to the decomposition in Eq. (15), the number of multiply-accumulate (MAC) operations increase by only a factor  $\frac{|H_n|^2}{k^2} + |H_n|$ , and the number of parameters by a factor  $\frac{|H_n|}{k^2} + 1$ . See Appendix C.3 for an overview of parameter counts and MAC operations.
4.1 When is color equivariance useful?
Color equivariant convolutions share shape information across different colors while preserving color information in the group dimension. To demonstrate when this property is useful we perform two controlled toy experiments on variations of the MNIST [11] dataset. We use the Z2CNN architecture from [7], and create a color equivariant version of the network called CECNN by replacing all convolutional layers by CEConv with three rotations of  $120^{\circ}$ . The number of channels in CECNN is scaled such as to keep the number of parameters approximately equal to the Z2CNN. We also create a color invariant CECNN by applying coset max-pooling after the final CEConv layer, and a color invariant Z2CNN by converting the inputs to grayscale. All experiments are performed using the Adam [24] optimizer with a learning rate of 0.001 and the OneCycle learning rate scheduler. No data augmentations are used. We report the average performance over ten runs with different random initializations.
Color imbalance is simulated by long-tailed ColorMNIST, a 30-class classification problem where digits occur in three colors on a gray background, and need to be classified by both number (0-9) and color (red, green, blue). The number of samples per class is drawn from a power law distribution resulting in a long-tailed class imbalance. Sharing shape information across colors is beneficial as a certain digit may occur more frequently in one color than in another. The train set contains a total of 1,514 training samples and the test set is uniformly distributed with 250 samples per class. The training set is visualized in Appendix B.1. We train all four architectures on the dataset for 1000 epochs using the standard cross-entropy loss. The train set distribution and per-class test accuracies for all models are shown in Fig. 2a. With an average accuracy of  $91.35 \pm 0.40\%$  the CECNN performs significantly better than the CNN with  $71.59 \pm 0.61\%$ . The performance increase is most significant for the classes with a low sample size, indicating that CEConvs are indeed more efficient in sharing shape information across different colors. The color invariant Z2CNN and CECNN networks, with an average accuracy of  $24.19 \pm 0.53\%$  and  $29.43 \pm 0.46\%$ , respectively, are unable to discriminate between colors. CECNN with coset pooling is better able to discriminate between foreground and background and therefore performs slightly better. We repeated the experiment with a weighted loss and observed no significantly different results. We have also experimented with adding color jitter augmentations, which makes solving the classification problem prohibitive, as color is required. See Appendix B.2 for both detailed results on both experiments.
(b)
Figure 2: Color equivariant convolutions efficiently share shape information across different colors. CECNN outperforms a vanilla network in both a long-tailed class imbalance setting (a), where MNIST digits are to be classified based on both shape and color, and a color biased setting (b), where the color of each class  $c$  is sampled according to  $\theta_d \sim \mathcal{N}(\theta_c, \sigma)$ .