SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Neural networks contain permutation symmetries in their weights, meaning that given, e.g., a neural network of the form (1), with pointwise applied nonlinearity  $\sigma$ , we can arbitrarily permute the inputs and outputs of each layer
$$
W _ {1} \mapsto P _ {1} W _ {1}, \quad W _ {j} \mapsto P _ {j} W _ {j} P _ {j - 1} ^ {T}, \quad W _ {L} \mapsto W _ {L} P _ {L - 1} ^ {T},
$$
and obtain a functionally equivalent net. The reader should note the similarity to the kernel constraint (3). It was conjectured by [13] that given two nets of the same type trained on the same data, it should be possible to permute the weights of one net to put it in the same loss-basin as the other net. Recently, this conjecture has gained quite strong empirical support [1, 22].
When two nets are in the same loss-basin, they exhibit close to linear mode connectivity, meaning that the loss/accuracy barrier on the linear interpolation between the weights of the two nets will be close to zero. Let the weights of two nets be given by  $\theta_{1}$  and  $\theta_{2}$  respectively. In this paper we define the barrier of a performance metric  $\zeta$  on the linear interpolation between the two nets by
$$
\frac {\frac {1}{2} \left(\zeta \left(\theta_ {1}\right) + \zeta \left(\theta_ {2}\right)\right) - \zeta \left(\frac {1}{2} \left(\theta_ {1} + \theta_ {2}\right)\right)}{\frac {1}{2} \left(\zeta \left(\theta_ {1}\right) + \zeta \left(\theta_ {2}\right)\right)}. \tag {4}
$$
Here  $\zeta$  will most commonly be the test accuracy. Previous works [15, 13, 1, 22] have defined the barrier in slightly different ways. In particular they have not included the denominator which makes it difficult to compare scores for models with varying performance. We will refer to (4) without the denominator as the absolute barrier. Furthermore we evaluate the barrier only at a single interpolation point—halfway between  $\theta_{1}$  and  $\theta_{2}$ —as compared to earlier work taking the maximum of barrier values when interpolating between  $\theta_{1}$  and  $\theta_{2}$ . We justify this by the fact that the largest barrier value
Figure 3: For every CNN that is trained with horizontal flipping data augmentation, there is a corresponding equally likely CNN that was initialized with filters horizontally flipped and trained on horizontally flipped images. This CNN is the same as the original but with flipped filters, also after training. According to the permutation conjecture [13], there should be a permutation of the channels of the flipped CNN that aligns it close to the original CNN. This implies that the CNN is close to a GCNN. Lighter blue means a spot in the parameter landscape with higher accuracy.
is practically almost always halfway between  $\theta_{1}$  and  $\theta_{2}$  (in fact [13] also only evaluate the halfway point in their experiments). The permutation conjecture can now be informally stated as follows.
Conjecture 3.1 (Entezari et al. [13, Sec. 3.2]). Most SGD solutions belong to a set  $S$  whose elements can be permuted in such a way that there is no barrier on the linear interpolation between any two permuted elements in  $S$ .
Importantly, we note that when applied to CNNs, the conjecture should be interpreted in the sense that we only permute the channel dimensions of the convolution weights, not the spatial dimensions. We will now explain why applying Conjecture 3.1 to CNNs that are trained on invariant data distributions leads to a conjecture stating that most SGD solutions are close to being GCNNs. For simplicity, we discuss the case where  $G =$  horizontal flips of the input image, but the argument works equally well for vertical flips or 90 degree rotations. The argument is summarised in Figure 3.
We will consider an image classification task and the common scenario where images do not change class when they are horizontally flipped, and where a flipped version of an image is equally likely as the original (in practice this is very frequently enforced using data augmentation). Let  $\tau$  be the horizontal flipping function on images. We can also apply  $\tau$  to feature maps in the CNN, whereby we mean flipping the spatial horizontal dimension of the feature maps.
Initialize one CNN  $\Phi$  and copy the initialization to a second CNN  $\hat{\Phi}$ , but flip all filters of  $\hat{\Phi}$  horizontally. If we let  $x_0, x_1, x_2, \ldots$  be the samples drawn during SGD training of  $\Phi$ , then an equally likely drawing of samples for SGD training of  $\hat{\Phi}$  is  $\tau(x_0), \tau(x_1), \tau(x_2), \ldots$ . After training  $\Phi$  using SGD and  $\hat{\Phi}$  using the horizontally flipped version of the same SGD path,  $\hat{\Phi}$  will still be a copy of  $\Phi$  where all filters are flipped horizontally. This means according to Conjecture 3.1 that  $\Phi$  can likely be converted close to  $\hat{\Phi}$  by only permuting the channels in the network.
Let's consider the  $j$ 'th convolution kernel  $\psi \in \mathbb{R}^{c_j\times c_{j - 1}\times k\times k}$  of  $\Phi$ . There should exist permutations  $P_{j}$  and  $P_{j - 1}$  such that  $\tau (\psi)\approx P_j\psi P_{j - 1}^T$ , where the permutations act on the channels of  $\psi$  and  $\tau$  on the spatial  $k\times k$  part. But if we assume that the  $P_{j}$ 's are of order 2 so that they define representations of the horizontal flipping group, this is precisely the kernel constraint (3), which makes the CNN close to a GCNN! It seems intuitive that the  $P_{j}$ 's are of order 2, i.e. that if a certain channel should be permuted to another to make the filters as close to each other as possible then it should hold the other way around as well. However, degeneracies such as multiple filters being close to equal can in practice hinder this intuition. Nevertheless, we conclude with the following conjecture which in some sense is a weaker version of Conjecture 3.1, as we deduced it from that one.
Conjecture 3.2. Most SGD CNN-solutions on image data with a distribution that is invariant to horizontal flips of the images will be close to GCNNs.
2This is not strictly true of networks that contain stride-2 convolutions or pooling layers with padding, as they typically start at the left edge of the image and if the image has an even width won't reach to the right edge of the image, meaning that the operation is nonequivalent to performing it on a horizontally flipped input. Still it will be approximately equivalent. We will discuss the implications of this defect in Section A.2.1.
Table 1: Types of VGG11 nets considered. All except "w/o aug" are trained with horizontal flipping augmentation.
Name	Description
CNN	Ordinary VGG11 trained using cross-entropy loss (C.-E.). 9.23M parameters.
CNN w/o aug	Ordinary VGG11 trained without horizontal flipping augmentation.
CNN + inv-loss	Ordinary VGG11 trained using C.-E. and invariance loss (inv-loss).
CNN + late inv-loss	Ordinary VGG11 trained using C.-E. and inv-loss after 20% of the epochs.
GCNN	A regular horizontal flipping GCNN trained using C.-E. 4.61M parameters.
PGCNN	A partial horizontal flipping GCNN trained using C.-E. The first two conv-layers are G-conv-layers and the rest are ordinary conv-layers. 9.19M parameters.
PGCNN + late inv-loss	The PGCNN trained using C.-E. and inv-loss after 20% of the epochs.
A measure for closeness to being a GCNN. To measure how close a CNN  $\Phi$  is to being a GCNN we can calculate the barrier, as defined in (4), of the linear interpolation between  $\Phi$  and a permutation of the flipped version of  $\Phi$ . We call this barrier for the test accuracy the GCNN barrier.
The aim of the experimental section is to investigate two related questions.
(Q1) If a CNN is trained to be invariant to horizontal flips of input images, is it a GCNN? This question is related to the theoretical investigation of layerwise equivariance in Section 2.
(Q2) If a CNN is trained on a horizontal flipping-invariant data distribution, will it be close to a GCNN? This is Conjecture 3.2.
To answer these two questions we evaluate the GCNN barrier for CNNs trained on CIFAR10 and ImageNet. We look at CNNs trained with horizontal flipping data augmentation to answer (Q2) and CNNs trained with an invariance loss on the logits to answer (Q1). In fact, for all training recipes horizontal flipping data augmentation is used. The invariance loss applied during training of a CNN  $\Phi$  is given by  $\| \Phi (x) - \Phi (\tau (x))\|$ , where  $\tau$  horizontally flips  $x$ . It is added to the standard cross-entropy classification loss. To evaluate the invariance of a CNN  $\Phi$  we will calculate the relative invariance error  $\| \Phi (x) - \Phi (\tau (x))\| /(0.5\| \Phi (x)\| +0.5\| \Phi (\tau (x))\|)$ , averaging over a subset of the training data. Another way to obtain invariance to horizontal flips is to use some sort of self-supervised learning approach. We will investigate self-supervised learning of ResNets in Section 4.2.
To align networks we will use activation matching [25, 1]. In activation matching, the similarity between channels in feature maps of the same layer in two different networks is measured over the training data and a permutation that aligns the channels as well as possible between the networks according to this similarity is found. Furthermore, we will use the REPAIR method by [22], which consists of—after averaging the weights of two networks—reweighting each channel in the obtained network to have the average batch statistics of the original networks. This reweighting can be merged into the weights of the network so that the original network structure is preserved. REPAIR is a method to compensate for the fact that when two filters are not perfect copies of each other, the variance of the output of their average will be lower than the variance of the output of the original filters. We will also report results without REPAIR.
Experimental details can be found in Appendix A. We provide code for merging networks with their flipped selfs at https://github.com/georg-bn/layerwise-equivariance.
4.1 VGG11 on CIFAR10
We train VGG11 nets [33] on CIFAR10 [23]. We will consider a couple of different versions of trained VGG11 nets, they are listed in Table 1.
We train 24 VGG11 nets for each model type and discard crashed runs and degenerate  $\mathrm{run}^3$  to obtain 18-24 good quality nets of each model type. In Figure 1 we visualize the filters of the first layer in two
<sup>3</sup>By degenerate run we mean a net with accuracy  $10\%$ . There was one GCNN and one CNN + late inv-loss for which this happened.
Table 2: Statistics for VGG11 nets trained on CIFAR10.
Name	Accuracy	Invariance Error	GCNN Barrier
CNN	0.901 ± 2.1 · 10-3	0.282 ± 1.8 · 10-2	4.00 · 10-2 ± 4.9 · 10-3
CNN w/o aug	0.879 ± 1.8 · 10-3	0.410 ± 4.0 · 10-2	4.98 · 10-2 ± 6.1 · 10-3
CNN + inv-loss	0.892 ± 2.3 · 10-3	0.0628 ± 6.7 · 10-3	1.90 · 10-3 ± 8.9 · 10-4
CNN + late inv-loss	0.902 ± 2.8 · 10-3	0.126 ± 1.6 · 10-2	1.33 · 10-2 ± 2.8 · 10-3
GCNN	0.899 ± 2.5 · 10-3	1.34 · 10-6 ± 1.5 · 10-7	8.92 · 10-4 ± 1.6 · 10-3
PGCNN	0.902 ± 3.7 · 10-3	0.274 ± 2.0 · 10-2	3.09 · 10-2 ± 7.8 · 10-3
PGCNN + late inv-loss	0.915 ± 2.3 · 10-3	0.124 ± 1.5 · 10-2	7.56 · 10-3 ± 1.8 · 10-3
Table 3: Results for ResNet50's trained using various methods on ImageNet. The model with an asterisk is trained by us. The last five are self-supervised methods. Flip-Accuracy is the accuracy of a net with all conv-filters horizontally flipped. Halfway Accuracy is the accuracy of a net that has merged the weights of the original net and the net of the net with flipped filters—after permuting the second net to align it to the first. For the accuracy and barrier values w/o REPAIR we still reset batch norm statistics after merging by running over a subset of the training data.
Training Method	Invariance	Flip-Accuracy	HalfwayAccuracy, no REPAIR	HalfwayAccuracy	GCNNBarrier,no REPAIR	GCNNBarrier
Error	Accuracy
Torchvision new	0.284	0.803	0.803	0.731	0.759	0.0893	0.0545
Torchvision old	0.228	0.761	0.746	0.718	0.725	0.0467	0.0384
*Torchvision old+ inv-loss	0.0695	0.754	0.745	0.745	0.745	0.00636	0.0054
BYOL	0.292	0.704	0.712	0.573	0.624	0.19	0.118
DINO	0.169	0.753	0.744	0.611	0.624	0.184	0.166
Moco-v3	0.16	0.746	0.735	0.64	0.681	0.136	0.0805
Simsiam	0.174	0.683	0.667	0.505	0.593	0.252	0.121
VGG11 nets. More such visualizations are presented in Appendix A. We summarise the statistics of the trained nets in Table 2. The most interesting findings are that the models with low invariance error also have low GCNN barrier, indicating that invariant models are layerwise equivariant. Note that the invariance error and GCNN barrier should both be zero for a GCNN. In order to have something to compare the numbers in Table 2 to, we provide barrier levels for merging two different nets in Table 4 in the appendix. Of note is that merging a model with a flipped version of itself is consistently easier than merging two separate models. In Figure 4 in the appendix, we show the distribution of permutation order for channels in different layers. The matching method sometimes fails to correctly match GCNN channels (for which we know ground truth matches), but overall it does a good job. In general, the unconstrained nets seem to learn a mixture between invariant and regular GCNN channels. Preliminary attempts with training GCNN-VGG11s with such a mixture of channels did however not outperform regular GCNNs.
4.2 ResNet50 on ImageNet
Next we look at the GCNN barrier for ResNet50 [21] trained on ImageNet [11]. We consider a couple of different training recipes. First of all two supervised methods—the latest Torchvision recipe [35] and the old Torchvision recipe [36] which is computationally cheaper. Second, four self-supervised methods: BYOL [19], DINO [7], Moco-v3 [9] and Simsiam [8]. The results are summarised in Table 3. There are a couple of very interesting takeaways. First, the GCNN barrier for the supervised methods is unexpectedly low. When merging two separate ResNet50's trained on ImageNet, [22] report an absolute barrier of 20 percentage points, whereas we here see barriers of 4.4 percentage points for the new recipe and 2.9 percentage points for the old recipe. This indicates that it is easier to merge a net with a flipped version of itself than with a different net. However, we also observe that for the self-supervised methods the barrier is quite high (although still lower than 20 percentage points). This is curious since they are in some sense trained to be invariant to aggressive data augmentation—including horizontal flips. Finally we note that when training with an invariance loss, the GCNN barrier vanishes, meaning that the obtained invariant net is close to being a GCNN.
The reader may have noticed that the "Flip-Accuracy", i.e., accuracy of nets with flipped filters, is markedly different than the accuracy for most nets in Table 3. This is a defect stemming from using image size 224. We explain this and present a few results with image size 225 in Appendix A.2.1.
We studied layerwise equivariance of ReLU-networks theoretically and experimentally. Theoretically we found both positive and negative results in Section 2, showing that layerwise equivariance is in some cases guaranteed given equivariance but in general not. In Section 3, we explained how Entezari et al.'s permutation conjecture 3.1 can be applied to a single CNN and the same CNN with horizontally flipped filters. From this we extrapolated Conjecture 3.2 stating that SGD CNN solutions are likely to be close to being GCNNs, also leading us to propose a new measure for how close a CNN is to being a GCNN—the GCNN barrier. In Section 4 we saw quite strong empirical evidence for the fact that a ReLU-network that has been trained to be equivariant will be layerwise equivariant. We also found that it is easier to merge a ResNet with a flipped version of itself, compared to merging it with another net. Thus, Conjecture 3.2 might be a worthwhile stepping stone for the community investigating Conjecture 3.1, in addition to being interesting in and of itself. If a negative GCNN barrier is achievable, it would imply that we can do "weight space test time data augmentation" analogously to how merging two separate nets can enable weight space ensembling.
This work was partially supported by the Wallenberg AI, Autonomous Systems and Software Program (WASP) funded by the Knut and Alice Wallenberg Foundation. The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at the Chalmers Centre for Computational Science and Engineering (C3SE) partially funded by the Swedish Research Council through grant agreement no. 2022-06725 and by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
[1] Samuel Ainsworth, Jonathan Hayase, and Siddhartha Srinivasa. Git re-basin: Merging models modulo permutation symmetries. In The Eleventh International Conference on Learning Representations, 2023. URL https://openreview.net/forum?id=CQsmMYmlP5T.
[2] Erik J. Bekkers, Maxime W. Lafarge, Mitko Veta, Koen A. J. Eppenhof, Josien P. W. Pluim, and Remco Duits. Roto-Translation Covariant Convolutional Networks for Medical Image Analysis. In Alejandro F. Frangi, Julia A. Schnabel, Christos Davatzikos, Carlos Alberola-López, and Gabor Fichtinger, editors, Medical Image Computing and Computer Assisted Intervention – MICCAI 2018, Lecture Notes in Computer Science, pages 440–448, Cham, 2018. Springer International Publishing. ISBN 978-3-030-00928-1. doi: 10.1007/978-3-030-00928-1_50.
[3] Georg Bokman and Fredrik Kahl. A case for using rotation invariant features in state of the art feature matchers. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops, 2022.
[4] Georg Bokman, Fredrik Kahl, and Axel Flinth. Zz-net: A universal rotation equivariant architecture for 2d point clouds. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 10976-10985, 2022.
[5] Robert-Jan Bruintjes, Tomasz Motyka, and Jan van Gemert. What affects learned equivariance in deep image recognition models? In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) Workshops, pages 4838-4846, June 2023.
[6] Lucas Brynte, Georg Bokman, Axel Flinth, and Fredrik Kahl. Rigidity preserving image transformations and equivariance in perspective. In Scandinavian Conference on Image Analysis, 2023.
[7] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In Proceedings of the International Conference on Computer Vision (ICCV), 2021.
[37] Soledad Villar, David W Hogg, Kate Storey-Fisher, Weichi Yao, and Ben Blum-Smith. Scalars are universal: Equivariant machine learning, structured like classical physics. In Thirty-Fifth Conference on Neural Information Processing Systems, 2021. URL https://openreview.net/forum?id=ba27-RzNaIv.
[38] Maurice Weiler and Gabriele Cesa. General e(2)-equivariant steerable cnns. In H. Wallach, H. Larochelle, A. Beygelzimer, F. d'Alché-Buc, E. Fox, and R. Garnett, editors, Advances in Neural Information Processing Systems, volume 32. Curran Associates, Inc., 2019. URL https://proceedings.neurips.cc/paper/2019/file/45d6637b718d0f24a237069fe41b0db4-Paper.pdf.
[39] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3D Steerable CNNs: Learning Rotationally Equivariant Features in Volumetric Data. In Advances in Neural Information Processing Systems, volume 31. Curran Associates, Inc., 2018.
[40] Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning Steerable Filters for Rotation Equivariant CNNs. In 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 849-858, Salt Lake City, UT, June 2018. IEEE. ISBN 978-1-5386-6420-9. doi: 10.1109/CVPR.2018.00095.
[41] Jeffrey Wood and John Shawe-Taylor. Representation theory and invariant neural networks. Discrete Applied Mathematics, 69(1-2):33-60, August 1996. ISSN 0166218X. doi: 10/c3qmr6. URL https://linkinghub.elsevier.com/retrieve/pii/0166218X95000753.
[42] Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic Networks: Deep Translation and Rotation Equivariance. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pages 5028-5037, 2017.
[43] Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnabas Poczos, Russ R Salakhutdinov, and Alexander J Smola. Deep Sets. In Advances in Neural Information Processing Systems, volume 30. Curran Associates, Inc., 2017.
Figure 4: Number of channels of different permutation order that are found by activation matching in different VGG11 versions. In the ideal case, for GCNNs all channels should have order  $\leq 2$ . The matching method fails this, but not extremely badly. The main reason for failure that we have observed is that layers contain many zero-filters.
A Experimental details and more results
A.1 VGG11 on CIFAR10
We present the distribution of various channels types in Figure 4. We present statistics for merging two different nets in Table 4.
In Figures 5, 6 we show the convolution kernels of the second convolution layers of the same nets as in Figure 1. Figures 7, 8 show the filters of a GCNN-VGG11, which has hardcoded equivariance in each layer.
There are some interesting takeaways from these filter visualizations. First of all, the number of zero-filters in Figure 6 – second layer of an ordinary VGG11 – and Figure 8 – second layer of a GCNN-VGG11 – is quite high and it is lower for the VGG11 trained using invariance loss – Figure 5. Also, the GCNN has more zero-filters than the ordinary net. It would be valuable to investigate why these zero-filters appear and if there is some training method that doesn’t lead to such apparent under-utilization of the network parameters.
Second, naturally when enforcing maximally equivariant layers using a lifting convolution in the first layer (recall Section 2.2), the network is forced to learn duplicated horizontal filters – see Figure 7 e.g. the all green filter. This could explain the worse performance of GCNN:s relative to ordinary VGG11:s. The choice of representations to use in each layer of a GCNN remains a difficult problem, and as we mentioned in the main text, in our initial experiments it did not seem to work to set the representations similar to those observed in Figure 4. Further research on the optimal choice of representations would be very valuable. However, as discussed in Section C.1, it may be the case that no layerwise equivariant net outperforms nets which are not restricted to be equivariant in late layers.
A.1.1 Training details
The nets were trained on NVIDIA T4 GPUs on a computing cluster. We train 4 nets in parallel on each GPU, each run taking between 1 and 2 hours, depending on net type. This yields less than  $(24/4) \cdot 6 \cdot 2 = 72$  compute hours in total.
Figure 5: Illustration of how a VGG11 encodes the horizontal flipping symmetry. The 128 filters in the second convolutional layer of a VGG11-net trained on CIFAR10 are shown, where each filter is next to a filter in a permuted version of the horizontally flipped convolution kernel. Both the input and output channels of the flipped convolution kernel are permuted, as described in Section 3. The 64 input channels are projected by a random projection matrix to 3 dimensions for visualization as RGB. The net is trained with an invariance loss to output the same logits for horizontally flipped images. It has learnt very close to a GCNN structure where the horizontally flipped filters can be permuted close to the original.