SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fig. 9 shows the test accuracies under a test time hue shift on all datasets in the paper. Each figure includes a regular ResNet, a color equivariant ResNet-x (CE-ResNet-x) and a ResNet-x with color equivariant convolutions in the first ResNet stage (CE-ResNet-x-1), trained with and without color jitter augmentation. Finally, the plot shows the accuracy of a ResNet-x trained on grayscale inputs. CEConv improves robustness to test-time hue shifts on all datasets.
Figure 9: Test accuracy on various classification datasets under a test time hue shift.
C.3 CE-ResNet configurations
The configurations of the color equivariant ResNet with three hue rotations, as used in the classification experiment in Section 4.2, are shown in Table 3. CE stages 0 denotes a regular ResNet.
Model	CE stages	Width	Parameters (M)	MACs (G)
ResNet-18	0	64	11.69	3.59
1	63	11.38	5.66
2	63	11.57	7.37
3	61	11.54	8.80
4	55	11.79	10.32
ResNet-44	0	32	2.64	0.78
1	31	2.51	1.23
2	30	2.50	1.63
3	27	2.60	1.83
Table 3: Color equivariant ResNet configurations.
C.4 Neuron Feature visualizations
Fig. 10 shows the Neuron Feature [42] (NF) visualization with top-3 patches of two neurons at different stages in a CE-ResNet18 trained on Stanford Cars. As expected, each row of a NF activates on the same shape in a different color. We show neurons that are insensitive to color (top row) and neurons that are sensitive to color (bottom row).
Figure 10: Neuron Feature [42] (NF) visualization with top-3 patches of two neurons at different stages in a CE-ResNet18 trained on Stanford Cars. Rows represent different rotations of the same filter.
D Ablation studies
Strength of color jitter augmentation Fig. 11 shows the effect of hue jitter augmentation during training on both a color equivariant ResNet-18 with 3 rotations (a) and a regular ResNet-18 (b) trained on Flowers-102. All runs have been repeated 3 times and the mean performance is reported. As expected, the color equivariant network (a) without jitter augmentation is equivariant to rotations of multiples of 120 degrees, but performance quickly degrades. Applying slight (0.1) hue jitter during training both helps in an absolute sense, increasing performance over all rotations, and makes the network more robust to hue changes as shown by the increasing width of the peaks. Further increasing the strength of the augmentation results in a uniform performance over all hue shifts, indicated by the flat lines. There appears to be no significant difference for jitter strength  $>0.2$ . In comparison, the
regular ResNet (b) trained without hue augmentation shows a single peak around 0 degrees, which increases in width when applying more severe augmentation. Note that the increase in absolute performance is smaller compared to the color equivariant network. The reason for this is that the equivariant architecture only requires augmentation "between" the discrete rotations to which it is already robust, as opposed to the full scale of hue shifts for the baseline architecture. Augmentation and equivariance thus exhibit a remarkable synergistic interaction.
(b)
Figure 11: Effect of hue jitter augmentation on a color equivariant (a) and a regular (b) ResNet-18.
Group coset pooling We have removed the group coset pooling operation by flattening the feature map group dimension into the channel dimension in the penultimate layer, before applying the final classification layer. As shown in Fig. 12, the model without pooling layer is no longer invariant to hue shifts and behaves identically to the baseline model.
Figure 12: CE-ResNet without group coset pooling behaves similarly to a regular ResNet (average over 5 runs).
Number of color rotations We investigate the effect of the number of hue rotations in color equivariant convolutions by training CE-ResNets with 2-10 rotations on Flowers-102. Fig. 13 shows the test accuracies for rotations 1-5 (a) and 6-10 (b), respectively. Note that, for this particular dataset, more hue rotations not only lead to better robustness to test-time hue shifts, but also to better absolute performance. However, there is a trade-off between number of rotations and model capacity, as increasing the number of rotations increases the number of parameters in the model, and the model width needs to be scaled down to keep the number of parameters equal. Both the optimal number of color rotations and network width therefore depend on the amount of color vs. the complexity of the data, and therefore both need to be carefully calibrated per dataset.
As expected, the number of peaks increases with the number of hue rotations, though interestingly, the peaks do vary in height. This is an artifact due the way test-time hue shifts are applied to the input images. When RGB pixels are rotated about the [1,1,1] diagonal, values near the borders of the RGB cube tend to fall outside the cube and subsequently need to be reprojected. This reprojection is not modeled by the filter transformations in the CEConv layers, and subsequently
causes a discrepancy between the filter and the image transformations. Indeed, when the test-time hue shift is instead implemented through a rotation in RGB space without reprojecting into the cube, this artifact disappears and all peaks are of equal height, as shown in Fig. 13 (c-d). Note that rotations of multiples of 120 degrees always end up within the RGB cube, which is why this artifact does never occur at -120, 0 and 120 degrees. Future work should further investigate the extent to which this discrepancy is problematic in practice, and look into alternative solutions.
(a) Test-time hue shift with reprojection.
(c) Test-time hue shift without reprojection.
Figure 13: The effect of the number of hue rotations in color equivariant convolutions on downstream performance. More rotations increase robustness to test-time hue shifts. Note that in (a-b) the peaks are not of equal height due to clipping effects near the boundaries of the RGB cube. This artifact disappears when the test-time hue shift is also applied without reprojection, resulting in peaks of equal height (c-d).
(b) Test-time hue shift with reprojection.
(d) Test-time hue shift without reprojection.