SlowGuess/NeurIPS_2023_OCR huggingface dataset:
For all experiments, we use near-identical, parameter-matched architectures in which only the type of invariant map differs. To isolate the effects of the invariant map, all models are comprised of a single  $G$ -Conv block followed by either Max  $G$ -Pooling or the  $G$ -TC layer, and an MLP Classifier. Here, we perform only pure  $G$ -Conv, without translation. Thus, we use filters the same size as the input in all models. The  $G$ -Conv block is comprised of a  $G$ -Conv layer, a batch norm layer, and an optional nonlinearity. For the Max  $G$ -Pool model, ReLU is used as the nonlinearity. Given the third-order nonlinearity of the  $G$ -TC, we omit the nonlinearity in the  $G$ -Conv block in the  $G$ -TC Model. The  $G$ -TC layer increases the dimensionality of the output of the  $G$ -Conv block; consequently the input dimension of the first layer of the MLP is larger and the weight matrix contains more parameters than for the Max  $G$ -Pool model. To compensate for this, we increase the dimension of the output of the first MLP layer in the Max Model, to match the overall number of parameters.
All models are trained with a cross-entropy loss, using the Adam optimizer, a learning rate of 0.00005, weight decay of 0.00001, betas of [0.9, 0.999], epsilon of  $10^{-8}$ , a reduce-on-plateau learning rate scheduler with a factor of 0.5, patience of 2 epochs, and a minimum learning rate of 0.0.0001. Each model is trained with four random seeds [0, 1, 2, 3], and results are averaged across seeds.
G.1 MNIST Experiments:  $SO(2)$  and  $O(2)$  on  $\mathbb{R}^2$
G.1.1 Datasets
The  $SO(2)$ -MNIST dataset is generated by applying a random 2D planar rotation to each digit in the MNIST [36] training and test datasets. This results in training and test sets with the standard sizes of 60,000 and 10,000. For the  $O(2)$ -MNIST, each image is randomly flipped and rotated—i.e. transformed by a random element of the group  $O(2)$ . A random  $20\%$  of the training dataset is set aside for model validation and is used to tune hyperparameters. The remaining  $80\%$  is used for training. Images are additionally downsized with interpolation to  $16 \times 16$  pixels.
G.1.2 Models and Training
C8-CNN
TC. The  $G$ -Conv block consists of an  $C8$ -Conv block using 24 filters followed by Batch Norm. Next, the  $G$ -TC Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The  $G$ -Conv block consists of an  $C8$ -Conv block using 24 filters followed by Batch Norm and a ReLU nonlinearity. Next, a Max  $G$ -Pooling Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer. The first layer has output dimension 275, to compensate for the difference in output size of the  $G$ -TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
D16-CNN
TC. The  $G$ -Conv block consists of an  $D16$ -Conv block using 24 filters followed by Batch Norm. Next, the  $G$ -TC Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The  $G$ -Conv block consists of an  $D16$ -Conv block using 24 filters followed by Batch Norm and a ReLU nonlinearity. Next, a Max  $G$ -Pooling Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer. The first layer has output dimension 2, 380, to compensate for the difference in output size of the  $G$ -TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
G.2 ModelNet10 Experiments:  $O$  and  $O_h$  acting on  $\mathbb{R}^3$
G.2.1 Datasets
The ModelNet10 dataset is downsampled and voxelized to a  $10 \times 10 \times 10$  grid. for the  $O$ -ModelNet10 Dataset, each datapoint is transformed by a random cubic rotation. For the  $O_h$ -ModelNet10 Dataset, each datapoint is transformed by a random cubic rotation and flip. The standard training and testing sets are used. A random  $20\%$  of the training dataset is set aside for model validation and is used to tune hyperparameters. The remaining  $80\%$  is used for training.
G.2.2 Models and Training
O-CNN
TC. The  $G$ -Conv block consists of an  $O$ -Conv block using 24 filters followed by a IID 3D Batch Norm Layer. Next, the  $G$ -TC Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear layer, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The  $G$ -Conv block consists of an  $O$ -Conv block using 24 filters followed by a IID 3D Batch Norm Layer and a ReLU nonlinearity. Next, a Max  $G$ -Pooling Layer is applied. The output is raveled and passed to a three-layer MLP using 1d Batch Norm and ELU nonlinearity and after each linear
layer. The first layer has output dimension 5, 420, to compensate for the difference in output size of the  $G$ -TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
$O_{h}$  -CNN
TC. The  $G$ -Conv block consists of an  $O_h$ -Conv block using 24 filters followed by a IID 3D Batch Norm Layer. Next, the  $G$ -TC Layer is applied. The output is raveled and passed to a three-layer MLP, with all three layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.
Max. The  $G$ -Conv block consists of an  $O_h$ -Conv block using 24 filters followed by a IID 3D Batch Norm Layer and a ReLU nonlinearity. Next, a Max  $G$ -Pooling Layer is applied. The output is raveled and passed to a three-layer MLP. The first layer has output dimension 20,000, to compensate for the difference in output size of the  $G$ -TC Layer. The remaining two layers having output dimension 64. Finally a linear layer is applied, with output dimension 10, for the 10 object categories.