SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Suppose that  $\Psi$  is an intertwiner between  $(\rho_i, \mathcal{F}_i^H)$  and  $(\sigma_1, \mathcal{F}_1^G)$ . Using the factorization property of induced representations [5.1] there is a canonical basis of the space  $\mathrm{Hom}_H[(\rho_i, \mathcal{F}_i^H), \mathrm{Res}_H^G[(\sigma_1, \mathcal{F}_1^G)]) \cong \mathrm{Hom}_G[\mathrm{Ind}_H^G[(\rho_i, \mathcal{F}_i^H)], (\sigma_1, \mathcal{F}_1^G)]$  and we may write  $\Psi$  uniquely as  $\Psi = \Psi^\dagger \circ \Phi_\rho$  where  $\Phi_\rho$  is an  $H$ -equivariant map and  $\Psi^\dagger$  is a  $G$ -equivariant map. Thus, any boundary between  $H$  and  $G$  layers can be written as an  $H$ -equivariant layer between  $(\rho_i, \mathcal{F}_i^H)$  and  $\mathrm{Ind}_H^G[(\rho_i, \mathcal{F}_i^H)]$  followed by a  $G$ -equivariant layer between  $\mathrm{Ind}_H^G[(\rho_i, \mathcal{F}_i^H)]$  and  $(\sigma_1, \mathcal{F}_1^H)$ . In this way, induction is all you need and all possible latent  $G$ -equivariant architectures can be written in terms of the induction representation.
6.1 Datasets & Evaluation Metrics
We evaluate the performance of our method on three single-object pose estimation datasets. These datasets require making predictions in SO(3) from single 2D images. SYMSOL [14] consists of a set of images of marked and unmarked geometric solids, taken from different vantage points. Training data is annotated with viewing direction. Some objects have symmetries so that there are multiple equivalent viewing directions, which requires learning distributions over poses. PASCAL3D+ [13] is a popular benchmark for object pose estimation composed of real images of objects from twelve categories. This dataset is challenging due to the large variation in object appearances and the presence of novel object instances in the test set. To be consistent with the baselines, we augment the training data with synthetic renderings [45] and evaluate performance on the PASCALVOC_val set. For more details on the benchmark datasets and additional numerical experiments, see B.
Figure 6: Diagram of an Equivariant Image to Sphere Convolution. At each unit vector  $\hat{n} \in S^2$  the kernel  $\kappa(\hat{n}:p)$  is dependent on the image point  $p = (x,y) \in \mathbb{R}^2$ . Equivariance constraints put restrictions on the allowed form of  $\kappa(\hat{n}:p)$ . Similar to a standard convolution, the kernel  $\kappa$  has a user defined receptive field.
When a single ground truth rotation label is provided, we evaluate the method using the geodesic distance between the predicted and ground truth rotation matrices, reported as either median rotation error or accuracy at a given rotation error threshold. For SYMSOL, which provides the full set of equivalent rotations associated with an image, we measure the accuracy of the learned pose distribution using average log-likelihood. This is also the accuracy metric used in [12].
6.2 Implementation & Training Details
For the results presented in, we use a ResNet encoder with weights pre-trained on ImageNet. With 224x224 images as input, this generates a 7x7 feature map with 2048 channels.
The filters in the induction layer are instantiated using the e2nn [38] package. The maximum
frequency is set at  $\ell = 6$ . The output of the induction layer is a 64-channeled  $S^2$  signal with fibers transforming in the trivial representation of SO(3). After the induction layer, a spherical convolution operation is performed using a filter that is parameterized in the Fourier domain, which generates an 8-channel signal over SO(3). A spherical non-linearity is applied by mapping the signal to the spatial domain, applying a ReLU, then mapping back to the Fourier domain. One final spherical convolution with a locally supported filter is performed to generate a one-dimensional signal on SO(3). The output signal is queried using an SO(3) HEALPix grid (recursion level 3 during training, 5 during evaluation) and then normalized using a softmax following [14].  $S^2$  and SO(3) convolutions were performed using the e3nn [43] package. The network was initialized and trained using PyTorch [46].
In order to create a fair comparison to existing baselines, batch size (64), number of epochs (40), optimizer (SGD) and learning rate schedule (StepLR) were chosen to be the same as that of [12]. Numerical experiments were implemented on NVIDIA P-100 GPUs.
6.3 Comparison to Baselines
We compare our method's performance to competitive pose estimation baselines. We include regression methods, [29, 30, 33], that perform well on datasets where objects have a single valid pose (e.g. are non-symmetric or symmetry is disambiguated in labels). We also baseline against methods that model pose with parametric families of distributions, [35, 47, 34, 36], an implicit model [14], and the Fourier basis of
$SO(3)$  [12]. To make the comparison fair, all methods use the same-sized ResNet backbone for each experiment, and we report results as stated in the original papers where possible.
SYMSOL Results Performance on the SYMSOL dataset is reported in Table I. Our method achieves the highest average log-likelihood on SYMSOL I. Importantly, we observe a significant improvement over Klee et al. [12] on all objects, which indicates that our induction layer is more effective than its hand-designed orthographic projection. On SYMSOL II, our method slightly underperforms Murphy et al. [14], which has much higher expressivity on the output since it is an implicit model. However, we demonstrate that our approach, which preserves the symmetry present in the images, is better with less data, as shown in Table 2.
Table 1: Average log likelihood (the higher the better ↑) on SYMSOL I & II. Per [14], a single model is trained on all classes in SYMSOL I and a separate model is trained on each class in SYMSOL II.
avg	cone	SYMSOL I (↑)	io	avg	SYMSOL II (↑)
cyl	tet	cube	ico	sphX	cylo	tetX
Deng et al. [34]	-1.48	0.16	-0.95	0.27	-4.44	-2.45	2.57	1.12	2.99	3.61
Prokudin et al. [35]	-1.87	-3.34	-1.28	-1.86	-0.50	-2.39	0.48	-4.19	4.16	1.48
Gilitschenski et al. [48]	-0.43	3.84	0.88	-2.29	-2.29	-2.29	3.70	3.32	4.88	2.90
Murphy et al. [14]	4.10	4.45	4.26	5.70	4.81	1.28	7.57	7.30	6.91	8.49
Klee et al. [12]	3.41	3.75	3.10	4.78	3.27	2.15	4.84	3.74	5.18	5.61
Ours	5.11	4.91	4.22	6.10	5.73	4.69	6.20	7.10	6.01	5.62
Table 2: Average log likelihood on SYMSOL I & II with  ${10}\%$  of training data.
avg	cone	10% SYMSOL I (↑)	10% SYMSOL II (↑)
cyl	tet	cube	ico	avg	sphX	cylo	tetX
Murphy et al. [14]	-7.94	-1.51	-2.92	-6.90	-10.04	-18.34	-0.73	-2.51	2.02	-1.70
Klee et al. [12]	2.98	3.51	2.88	3.62	2.94	1.94	3.61	3.12	3.87	3.84
Ours	3.01	3.63	3.01	3.53	3.02	1.91	3.54	2.88	3.71	4.04
PASCAL3D+ Results Our method achieves state-of-the-art performance on PASCAL3D+ with an average median rotation error of 9.2 degrees, as reported in Table 3. Even though object symmetries are consistently disambiguated in the labels, modeling pose as a distribution is beneficial for noisy images where there is insufficient information to resolve the pose exactly. Because our induction layer produces representations on the Fourier basis of SO(3), it naturally allows for capturing this uncertainty as a distribution over SO(3). While both our method and [12] leverage SO(3) equivariant
layers to improve generalization, we find our method achieves higher performance. We believe our induction layer is more robust to variations in how the images are rendered or captured, which is important for PASCAL3D+, since the data is aggregated from many sources. Moreover, our method does not restrict features to the hemisphere, which could be beneficial for objects, like bikes and chairs, that do not fully self-occlude their backsides.
Table 3: Rotation prediction on PASCAL3D+. First column is the average over all categories.
Median rotation error in degrees (↓)
avg	plane	bike	boat	bottle	bus	car	chair	table	mbike	sofa	train	tv
Mohlin et al. 47	11.5	10.1	15.6	24.3	7.8	3.3	5.3	13.5	12.5	12.9	13.8	7.4	11.7
Prokudin et al. 35	12.2	9.7	15.5	45.6	5.4	2.9	4.5	13.1	12.6	11.8	9.1	4.3	12.0
Tulsiani and Malik 29	13.6	13.8	17.7	21.3	12.9	5.8	9.1	14.8	15.2	14.7	13.7	8.7	15.4
Mahendran et al. 30	10.1	8.5	14.8	20.5	7.0	3.1	5.1	9.5	11.3	14.2	10.2	5.6	11.7
Liao et al. 33	13.0	13.0	16.4	29.1	10.3	4.8	6.8	11.6	12.0	17.1	12.3	8.6	14.3
Murphy et al. 14	10.3	10.8	12.9	23.4	8.8	3.4	5.3	10.0	7.3	13.6	9.5	6.4	12.3
Klee et al. 12	9.8	9.2	12.7	21.7	7.4	3.3	4.9	9.5	9.3	11.5	10.5	7.2	10.6
Yin et al. 36	9.4	8.6	11.7	21.8	6.9	2.8	4.8	7.9	9.1	12.2	8.1	6.9	11.6
Ours (ResNet-50)	10.2	9.2	13.1	30.6	6.7	3.1	4.8	8.7	5.4	11.6	11.0	5.8	10.6
Ours	9.2	9.3	12.6	17.0	8.0	3.0	4.5	9.4	6.7	11.9	12.1	6.9	9.9
In conclusion, we have argued that any network that learns a three-dimensional model of the world from two-dimensional images must satisfy certain consistency properties. We have shown how these consistency properties translate into an SO(2)-equivariance constraint. Using the induced representation we have derived an explicit form for any neural networks that satisfies said consistency constraint. We have proposed an induction/restiction layer, which is a learnable network layer that satisfies the derived consistency equation. We have shown that the induction layer satisfies both a completeness property and universal property and, up to isomorphism, is unique. Furthermore, we have shown that the methods of [12, 10, 11] can be realized as specific instances of the induction layer.
The framework that we have developed is general and can be applied to other computer vision problems with different symmetries. For example, as was noted in [49], the cryogenic electronic microscopy orientation estimation problem has a latent SO(3) symmetry but a manifest SO(2)  $\times \mathbb{Z}_2 \cong O(2)$  (as opposed to an SO(2)) symmetry. With a slight modification H the results presented in the main text allow for the construction of an induction layer that leverages this observation.
Future Work In many structure-from-motion tasks, one has access to multiple images of the same object, taken at either known or unknown vantage points. Our work considers only single view pose estimation. A natural generalization of our work is to include stereo measurements into the induced/restricted representation framework. [50, 51] use transformer architectures to learn models of three-dimensional objects from two-dimensional images. Furthermore, in this work we have only considered supervised learning, but our framework naturally generalizes to unsupervised settings like [23]. Another natural extension of our work would be to include transformers into the framework presented here, which only applies to convolutional networks.
In deep learning, we often wish to construct a neural network that respects a latent symmetry  $G$  that does not have an explicit action on the input data space. We have show how the induced representation can be used to construct latent  $G$ -equivariant neural networks. Our work provides a systematic way to construct neural architectures that accept any format of inputs and respect the latent symmetries of the problem.
Owen Howell thanks the National Science Foundation Graduate Research Fellowship Program (NSF-GRFP) for financial support. Ondrej Biza, David Klee and Robin Walters were supported by National Science Foundation (NSF) grants 2107256 and 2134178. Linfeng Zhao was additionally supported by NSF grant 2107256. Ondrej Biza also acknowledges support from NSF grants 1750649
and 1763878. Owen Howell thanks Dr. Thomas Sayre-Maccord for invaluable advice and logistics help. Owen Howell further thanks Liam Pavlovic and Dr. David Rosen for useful discussions.