Title: Equivariant Single View Pose Prediction Via Induced and Restriction Representations
Venue: NeurIPS 2023
Accepted: yes
Keywords: Equivarient Machine Learning

Pose Prediction

Computer Vision
Abstract: Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three dimensions to make predictions about novel images. However, imposing $SO(3)$-equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of $SO(3)$ will rotate an image out of plane. We show that an algorithm that learns a three-dimensional representation of the world from two dimensional images must satisfy certain consistency properties which we formulate as $SO(2)$-equivariance constraints. We use the induced representation of $SO(2)$ on $SO(3)$ to construct and classify architectures that have two-dimensional inputs and 
which satisfy these consistency constraints. We prove that any architecture which respects said consistency constraints can be realized as an instance of our construction. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new algorithm that is a learnable generalization of previously considered methods. We test our architecture on three pose predictions task and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.

Equivariant Single View Pose Prediction Via Induced and Restricted Representations

Abstract

Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three-dimensions to make predictions about novel images. However, imposing SO(3) -equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of SO(3) will rotate an image out of the plane. We show that an algorithm that learns a three-dimensional representation of the world from two-dimensional images must satisfy certain consistency properties which we formulate as SO(2) -equivariance constraints. We use the induced and restricted representations of SO(2) on SO(3) to construct and classify architectures that satisfy these consistency constraints. We prove our construction realizes all possible architectures that respect these constraints. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new model that generalizes previously considered methods and contains additional trainable parameters. We test our architecture on three pose prediction tasks and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.

1 Introduction

One of the fundamental problems in computer vision is learning representations of 3D objects from 2D images [1-3]. By understanding how image features correspond to a physical object, a model can generalize better to novel views of the object, for instance, when estimating the pose of an object. In general, neural networks that respect the symmetries of a problem are more noise robust and data efficient, while also less prone to over-fitting [4]. Three-dimensional space has a natural symmetry group of three-dimensional rotations and three-dimensional translations, SE(3) . While we would like to leverage this symmetry to design improved neural architectures, serious challenges exist in incorporating 3D symmetry when applied to image data. Specifically, a projection of a threedimensional scene into a two-dimensional plane does not transform equivariantly under all elements of SE(3) . This is because there is no a priori model for how two-dimensional images transform under out-of-plane object rotations. Cohen and Welling [5] showed how to design neural networks that are explicitly SO(2) -equivariant and accept images as inputs. However, SO(2) -equivariant methods ignore the fact that the group SO(3) acts on the space of pose configurations.

Equivariant neural networks are much more constrained than general multi-layer perceptrons. The requirement of equivariance to a group G places strict restrictions on the allowed linear maps and the

allowed non-linear functions in each network layer [5, 6]. Because of this, the possible structures of G -equivariant neural networks can be completely classified based on the representation theory of the group G [4, 7, 8]. For compact groups, it is possible to characterize the structure of all possible kernels of G -equivariant networks [8].

(f) ∈ F↑

Φ(h· f) ∈ F↑

h E SO(2)

Φ

hE SO(2)

fEF

h·fEF

We argue that any equivariant machine learning algorithm that builds a three-dimensional model of the world from two-dimensional images must satisfy a natural geometric consistency property. This consistency property can be stated as SO(2) -equivariance with respect to only the SO(2) subgroup acting along the camera viewing orientation (see Figure 1). We give a complete characterization of maps that satisfy this property. This derivation uses the socalled restricted representation of SO(2) since the group action is restricted from the full SO(3) . Using the Frobenius Reciprocity theorem, we show that this geometric constraint can also be derived using induced representations, a type of representation of SO(3) constructed from representations of SO(2) . The classification theorems derived in [5, 8, 4] assume that both the input and output layers are G -equivariant. The construction presented in 4 is different; we map H -equivariant functions to G -equivariant functions. Our arguments using induced and restricted representations give a natural generalization of equivariant maps between different groups. We derive analogies of the theorems presented in [9, 8] for the induced and restricted representations.

1.1 Importance and Contribution

In this work, we will show how induced and restricted representations can be used to construct neural architectures that accept image data and leverage SO(3) -equivariant methods to avoid learning nuisance transformations in three-dimensional space.

We show that our construction satisfies two desirable theoretical properties, completeness and universality . Let H ⊆ G . We focus on the case G = SO (3) and let H = SO (2) but we give a theoretical analysis for general groups. Specifically, the induced representation construction is complete in that all group-valued functions on G can be induced from a set of group-valued functions on H . The construction is universal in that all multi-linear maps that map H -equivariant functions to G -equivariant functions are specific cases of the induced representation, modulo isomorphism. Furthermore, we show that the architectures proposed in [10, 11] are special cases of our construction for the icosahedral group G = A 5 and the construction proposed in [12] is a special case of our construction for the three-dimensional rotation group G = SO(3) . Our method achieves state of the art performance for orientation prediction on PASCAL3D+ [13] and SYMSOL [14] datasets.

Contributions:

We propose a unified theory for learning three-dimensional representations from twodimensional images. We show that algorithms that learn three-dimensional representations from two-dimensional images must satisfy certain consistency properties, which are equivalent to SO(2) -steerability constraints.

We introduce a fully differentiable layer called an induction/restriction layer that maps signals on the plane into signals on the sphere. We show that the induction/restriction layer satisfies a natural consistency constraint and prove both a completeness and universal property for our construction.

Our method achieves SOTA performance for orientation prediction on PASCAL3D+ and SYMSOL datasets.

2 Related Work

Equivariant Learning Incorporating task symmetry into the design of neural networks has been effective in domains such as computer vision [15, 16], point cloud processing [17, 18], and robotics [19]. Cohen and Welling [9] introduced steerable kernels which are a trainable layer that can be used to build networks that are equivariant to 2D [5, 20] and 3D transformations [21, 22]. The majority of past works have studied end-to-end equivariant models, where the input can be transformed by the action of the group and all layers are equivariant, in this work, we explore how to 'fuse' SO (2) and SO (3) equivariant layers.

There has been growing interest in leveraging 3D symmetry from 2D inputs. [23, 24] learned a 3D transformable latent space from images of a single object. [25] trained a convolutional network to predict pre-trained SO(3) equivariant embeddings, while [11, 10, 12] mapped image features onto elements of the discrete group of SO(3) , using structured viewpoints or a hand-coded projections, respectively. In contrast to prior work, we provide a theoretical foundation for learned equivariant mappings from 2D to 3D, which additionally guides us to introduce a more general and effective trainable operation.

Object Pose Estimation Predicting the 3D orientation of objects is an important problem in fields like autonomous driving [26], robotics [27] and cryogenic electron microscopy [28]. Many works [29, 30] have used a regression approach, and others [31-33] have identified ways to mitigate the discontinuities along the SO(3) manifold. More recent works have explored ways to model orientation as a distribution over 3D rotations, which handles object symmetries and captures uncertainty. [34], [35] and [36] predict parameters for Bingham, von Mises and Laplace distributions, respectively. These families of distributions can have limited expressivity, so other work explored using implicit networks [14] or the Fourier basis [12] to model more complex pose distributions. Inspired by [12, 23], we parameterize the latent object pose as a distribution on SO (3) and then ask what constraints need to be imposed to enforce SO(2) ⊆ SO(3) -equivariance A.0.4, which is a generalization of SO(2) -equivariance.

3 Background

We introduce the induced and restricted representations. For a more extensive review of representation theory, see A.

Let G be a group and V be a vector space over C . A representation ( ρ , V ) of G is a map ρ : G → Hom[ V, V ] such that

Concisely, a group representation is an embedding of a group into a set of matrices. The matrix embedding must obey the multiplication rule of the group. We now introduce the restricted representation and induced representation .

Restricted Representation Let H ⊆ G be a subgroup. Let ( ρ , V ) be a representation of G . The restricted representation of ( ρ , V ) from G to H is denoted as Res G H [( ρ , V )] . Intuitively, Res G H [( ρ , V )] can be viewed as ( ρ , V ) evaluated on the subgroup H of G . Specifically,

For a more in depth discussion of the restricted representation, please see A.

Induced Representation The induced representation is a way to construct representations of a larger group G out of representations of a subgroup H ⊆ G . Let ( ρ , V ) be a representation of H . The induced representation of ( ρ , V ) from H to G is denoted as Ind G H [( ρ , V )] . Define the space of functions

Then the induced representation is defined as ( π , F ) = Ind G H [( ρ , V )] where the induced action π acts on the function space F via

Please see A for an in depth discussion of the induced representation. The induced and restricted representations are adjoint functors [37].

4 Method

Convolutional networks or vision transformers are typically used to extract spatial feature maps from 2D images. For convenience we ignore discretization and treat the feature maps as having continuous inputs f : R 2 → R d . To leverage spatial symmetries in 3D, we would like to map our features f from a plane onto a sphere: g : S 2 → R D . Klee et al. [12] proposed one such mapping, where the planar feature map is stretched over a hemisphere, but other possible mappings exist.

We formalize the equivariance property that every projection should have through the theory of induced and restricted representations. The constraints that we impose have an intuitive geometric interpretation. We give a complete characterization of all possible linear and equivariant projections Φ from planar features to a spherical representation. Our general formulation includes [12] as a special case, and we show that a learnable equivariant projection leads to better predictive models.

4.1 Equivariant 2D to 3D Projection by Induced and Restricted Representations

We first derive the SO(2) -equivariance constraint for the most general linear mapping from images to spherical signals.

Image inputs We first describe F the space of image input signals. Let V and V ↑ be vector spaces. Let F be the vector space of all V -valued signals defined on the plane

Elements of F are sometimes called SE(2) -steerable feature fields [20]. The group SE(2) = R 2 /multicloseright SO(2) of 2D translations and rotations acts on F via representation π . Each h ∈ SE(2) has a unique factorization h = ¯ hh c where ¯ h ∈ R 2 is a translation and h c ∈ SO(2) is a rotation. Let ( ρ , V ) be an SO(2) -representation describing the transformation of the fibers of the features f . Then the action π is defined

so that ( π , F ) = Ind SE(2) SO(2) [( ρ , V )] and ( π , F ) gives a representation of the group SE(2) [9].

Spherical outputs We would like to map signals in F to functions from S 2 into the vector space V ↑ . Let F ↑ be the vector space of all such outputs defined as

The group SO(3) acts on the vector space F ↑ via

where ρ ↑ ( g ) describes the SO(3) fiber representation.

SO(2) -equivariant image to sphere Let H = SO(2) be the SO(2) subgroup of SO(3) that corresponds to in-plane rotations of the image. Our goal is to classify H -equivariant linear maps Φ : F → F ↑ . This is equivalent to the constraint that

The constraint enforces equivariance with respect to SO(2) transformations. By definition, the evaluation of π ↑ ( h ) at h ∈ SO(2) is the restricted representation π ↑ ( h ) = Res SO(3) SO(2) [ π ↑ ]( h ) .

SO(2)

SO(3)

SO(3)

SO(2)

SO(3)

p3,

V3

(D3

(&M

(p3, V3)

(D3

W3)

V2)

P2,

D²,

(p2, V2)

D²,

W2)

(p1, V1)

(D1,

, W1)

(p1, Vi)

(D1

W1)

Po,

%

(D°,

(M

(po, Vo)

(D, Wo)

4.2 Solving the Kernel Constraint

We use tools from [38, 8] to solve for the space of all possible maps satisfying the constraint 1, giving the trainable space for the image to sphere layer.

Our conclusion is that instead of mapping arbitrary SO(2) -input representations to arbitrary SO(2) -output representations, the allowed input and output representations ( ρ , V ) and ( ρ ↑ , V ↑ ) must satisfy additional constraints. Specifically, not every representation can be realized as the restriction of an SO(3) to SO(2) representation 2. Although in this paper we focus on orientation estimation, the equivariant framework in Section C.0.1 is more general. In the Appendix D, we formulate and solve analogous equivariance constraints for both 6DoF-pose estimation and monocular volume reconstruction.

Theorem 1. The constraint in Equation 1 can be solved exactly using the results of [38, 8]. The most linear general map Φ : F → F ↑ can be expanded as

where κ : R 2 × S 2 → Hom[ V, V ↑ ] . Then, the exact form of κ can be written as

where Y /lscript (ˆ n ) is the vectorization of the /lscript -type spherical harmonics and each F /lscript ( r ) is a standard SO(2) -steerable kernel [9, 38] that has input SO(2) -representation ( ρ , V ) and output SO(2) -representation ( ρ /lscript , V /lscript ) = ( ρ , V ) ⊗ Res SO(3) SO(2) [( D /lscript , V /lscript )] .

The proof of this statement is given in Appendix F. Note that similar to [18, 6] the tensor product structure of the SO(2) and SO(3) irreducible representations determine the allowed input and output representations of the matrix valued harmonic coefficients F /lscript ( r ) .

4.3 Including Non-Linearities

In Section 4.2, we considered the most general linear maps that satisfied the generalized equivariance constraint. Adding non-linearities should allow for more expressiveness. Understanding nonlinearities between equivariant layers is still an active area of research [39-42].

One way to include non-linearity is to apply standard SO(3) non-linearities after the linear induction layer. After applying the linear mapping described in C, we apply an additional spherical non-linearity [43] to the signal on S 2 . This is the method we employ for the results presented in 6.2. As shown in G it is also possible to include tensor-product based non-linearity analogous to the results of [18, 6].

5 Theory

5.1 Universal Property

In section 4 we showed how the restriction representation arises naturally when constructing SO(3) -equivariant architectures for image data. However, there is no a priori choice of the hidden SO(3) representation. We show that with this choice, our construction satisfies a universal property and is unique up to isomorphism [44].

A standard result in group theory establishes the following universal property of induced representations, as stated in [37]:

Theorem 2. Let H ⊆ G . Let ( ρ , V ) be any H -representation. Let Ind G H ( ρ , V ) be the induced representation of ( ρ , V ) from H to G . Then, there exists a unique H -equivariant linear map Φ ρ : V → Ind G H V such that for any G -representation ( σ , W ) and any H -equivariant linear map Ψ : V → W , there is a unique G -equivariant map Ψ ↑ : Ind G H V → W such that the diagram 3 is commutative.

This theorem can be applied to the construction proposed in 1 to prove a universality property, similar to the results of [5] for G -equivariant neural networks.

Hom G [ Ind G H ( ρ , V ) , ( σ , W )] and Ψ ρ : V → Ind G H V is ( σ , W ) independent.

Let ( ρ , V ) be a H -representation and let ( σ , W ) be a G -representation. Let Ψ : V → W where Ψ is an intertwiner of a the H -representation and the restriction of the G -representation to an H -representation so that

so that Ψ ∈ Hom H [( ρ , V ) , Res G H ( σ , W )] . The universal property of the induced representation allows us to write any such Ψ in a canonical form. Specifically, as illustrated in Figure 5.1, we can always uniquely decompose Ψ = Ψ ↑ ◦ Φ ρ where Ψ ↑ ∈

(

ρ

, V

)

(

σ

, W

)

(

ρ

, V

)

(

σ

, W

)

Ψ

ρ

(

h

)

σ

(

g

)

σ

(

h

)

Ψ

∼

=

(

ρ

, V

)

(

σ

, W

)

Ind

G

H

(

ρ

, V

)

Ind

G

H

(

ρ

, V

)

(

ρ

, V

)

(

σ

, W

)

Φ

ρ

Ψ

ρ

(

h

)

σ

(

g

)

σ

(

h

)

Ψ

↑

[Ind

G

H

ρ

](

h

)

[Ind

G

H

σ

](

g

)

Ψ

↑

Φ

ρ

Ψ

Factorization Property of H ⊆ G Neural Networks We use the universal property of induced representations to show that all possible latent G -equivariant architectures can be written in terms of the induced representation. At each layer of a equivariant neural network, we have a set of functions from a homogeneous space of a group into some vector space [6]. Let X H i be a set of homogeneous spaces of the group H and let X G j be a set of homogeneous spaces of the group G . Let V i H and W G j be a set of vector spaces. Then, consider the function spaces

The group H acts on the homogeneous spaces X H i and the group G acts on the homogeneous spaces X G j so that the function spaces F i H and F G j form representations of H and G , respectively

Suppose we wish to design a downstream G -equivariant neural network that accepts as signals functions that live in the vector space F H 0 and transform in the ρ 0 representation of H . Thus,

( ρ 0 , F H 0 ) is a H -representation, but not necessarily a G -representation. At some point, in the architecture, a layer F i H must be H equivariant on the left and both H and G -equivariant on the right. Let us call the layer that is both H and G -equivariant F G 1 .

Suppose that Ψ is an intertwiner between ( ρ i , F i H ) and ( σ 1 , F G 1 ) . Using the factorization property of induced representations 5.1, there is a canonical basis of the space Hom H [( ρ i , F i H ) , Res G H [( σ 1 , F G 1 )]] ∼ = Hom G [Ind G H [( ρ i , F i H )] , ( σ 1 , F G 1 )] and we may write Ψ uniquely as Ψ = Ψ ↑ ◦ Φ ρ where Φ ρ is an H -equivariant map and Ψ ↑ is a G -equivariant map. Thus, any boundary between H and G layers can be written as an H -equivariant layer between ( ρ i , F i H ) and Ind G H [( ρ i , F i H )] followed by a G -equivariant layer between Ind G H [( ρ i , F i H )] and ( σ 1 , F H 1 ) . In this way, induction is all you need and all possible latent G -equivariant architectures can be written in terms of the induction representation.

6 Experiments

6.1 Datasets & Evaluation Metrics

We evaluate the performance of our method on three single-object pose estimation datasets. These datasets require making predictions in SO(3) from single 2D images. SYMSOL [14] consists of a set of images of marked and unmarked geometric solids, taken from different vantage points. Training data is annotated with viewing direction. Some objects have symmetries so that there are multiple equivalent viewing directions, which requires learning distributions over poses. PASCAL3D+ [13] is a popular benchmark for object pose estimation composed of real images of objects from twelve categories. This dataset is challenging due to the large variation in object appearances and the presence of novel object instances in the test set. To be consistent with the baselines, we augment the training data with synthetic renderings[45] and evaluate performance on the PASCALVOC_val set. For more details on the benchmark datasets and additional numerical experiments, see B.

When a single ground truth rotation label is provided, we evaluate the method using the geodesic distance between the predicted and ground truth rotation matrices, reported as either median rotation error or accuracy at a given rotation error threshold. For SYMSOL, which provides the full set of equivalent rotations associated with an image, we measure the accuracy of the learned pose distribution using average log-likelihood. This is also the accuracy metric used in [12].

6.2 Implementation & Training Details

For the results presented in 6, we use a ResNet encoder with weights pre-trained on ImageNet. With 224x224 images as input, this generates a 7x7 feature map with 2048 channels.

The filters in the induction layer are instantiated using the e2nn [38] package. The maximum

frequency is set at /lscript = 6 . The output of the induction layer is a 64 -channeled S 2 signal with fibers transforming in the trivial representation of SO(3) . After the induction layer, a spherical convolution operation is performed using a filter that is parameterized in the Fourier domain, which generates an 8-channel signal over SO(3) . A spherical non-linearity is applied by mapping the signal to the spatial domain, applying a ReLU, then mapping back to the Fourier domain. One final spherical convolution with a locally supported filter is performed to generate a one-dimensional signal on SO(3) . The output signal is queried using an SO(3) HEALPix grid (recursion level 3 during training, 5 during evaluation) and then normalized using a softmax following [14]. S 2 and SO(3) convolutions were performed using the e3nn [43] package. The network was initialized and trained using PyTorch [46].

In order to create a fair comparison to existing baselines, batch size (64), number of epochs (40), optimizer (SGD) and learning rate schedule (StepLR) were chosen to be the same as that of [12]. Numerical experiments were implemented on NVIDIA P-100 GPUs.

6.3 Comparison to Baselines

We compare our method's performance to competitive pose estimation baselines. We include regression methods, [29, 30, 33], that perform well on datasets where objects have a single valid pose (e.g. are non-symmetric or symmetry is disambiguated in labels). We also baseline against methods that model pose with parametric families of distributions, [35, 47, 34, 36], an implicit model [14], and the Fourier basis of

SO (3) [12]. To make the comparison fair, all methods use the same-sized ResNet backbone for each experiment, and we report results as stated in the original papers where possible.

SYMSOL Results Performance on the SYMSOL dataset is reported in Table 1. Our method achieves the highest average log-likelihood on SYMSOL I. Importantly, we observe a significant improvement over Klee et al. [12] on all objects, which indicates that our induction layer is more effective than its hand-designed orthographic projection. On SYMSOL II, our method slightly underperforms Murphy et al. [14], which has much higher expressivity on the output since it is an implicit model. However, we demonstrate that our approach, which preserves the symmetry present in the images, is better with less data, as shown in Table 2.

PASCAL3D+ Results Our method achieves state-of-the-art performance on PASCAL3D+ with an average median rotation error of 9.2 degrees, as reported in Table 3. Even though object symmetries are consistently disambiguated in the labels, modeling pose as a distribution is beneficial for noisy images where there is insufficient information to resolve the pose exactly. Because our induction layer produces representations on the Fourier basis of SO(3) , it naturally allows for capturing this uncertainty as a distribution over SO(3) . While both our method and [12] leverage SO(3) equivariant

layers to improve generalization, we find our method achieves higher performance. We believe our induction layer is more robust to variations in how the images are rendered or captured, which is important for PASCAL3D+, since the data is aggregated from many sources. Moreover, our method does not restrict features to the hemisphere, which could be beneficial for objects, like bikes and chairs, that do not fully self-occlude their backsides.

7 Conclusion

In conclusion, we have argued that any network that learns a three-dimensional model of the world from two-dimensional images must satisfy certain consistency properties. We have shown how these consistency properties translate into an SO(2) -equivariance constraint. Using the induced representation we have derived an explicit form for any neural networks that satisfies said consistency constraint. We have proposed an induction/restriction layer , which is a learnable network layer that satisfies the derived consistency equation. We have shown that the induction layer satisfies both a completeness property and universal property and, up to isomorphism, is unique. Furthermore, we have shown that the methods of [12, 10, 11] can be realized as specific instances of the induction layer.

The framework that we have developed is general and can be applied to other computer vision problems with different symmetries. For example, as was noted in [49], the cryogenic electronic microscopy orientation estimation problem has a latent SO(3) symmetry but a manifest SO(2) × Z 2 ∼ = O (2) (as opposed to an SO(2) ) symmetry. With a slight modification H, the results presented in the main text allow for the construction of an induction layer that leverages this observation.

Future Work In many structure-from-motion tasks, one has access to multiple images of the same object, taken at either known or unknown vantage points. Our work considers only single view pose estimation. A natural generalization of our work is to include stereo measurements into the induced/restricted representation framework. [50, 51] use transformer architectures to learn models of three-dimensional objects from two-dimensional images. Furthermore, in this work we have only considered supervised learning, but our framework naturally generalizes to unsupervised settings like [23]. Another natural extension of our work would be to include transformers into the framework presented here, which only applies to convolutional networks.

In deep learning, we often wish to construct a neural network that respects a latent symmetry G that does not have an explicit action on the input data space. We have show how the induced representation can be used to construct latent G -equivariant neural networks. Our work provides a systematic way to construct neural architectures that accept any format of inputs and respect the latent symmetries of the problem.

Acknowledgments and Disclosure of Funding

Owen Howell thanks the National Science Foundation Graduate Research Fellowship Program (NSF-GRFP) for financial support. Ondrej Biza, David Klee and Robin Walters were supported by National Science Foundation (NSF) grants 2107256 and 2134178. Linfeng Zhao was additionally supported by NSF grant 2107256. Ondrej Biza also acknowledges support from NSF grants 1750649

and 1763878. Owen Howell thanks Dr. Thomas Sayre-Maccord for invaluable advice and logistics help. Owen Howell further thanks Liam Pavlovic and Dr. David Rosen for useful discussions.

References

David Marr. Vision: A computational investigation into the human representation and processing of visual information . MIT press, 2010.

Richard Hartley and Andrew Zisserman. Multiple View Geometry in Computer Vision . Cambridge University Press, 2 edition, 2004. doi: 10.1017/CBO9780511811685.

Onur Ozyesil, Vladislav Voroninski, Ronen Basri, and Amit Singer. A survey of structure from motion, 2017. URL https://arxiv.org/abs/1701.08493 .

Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veliˇ ckovi´ c. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021. URL https://arxiv.org/abs/2104.13478 .

Taco S. Cohen and Max Welling. Steerable cnns. axriv , 2016. doi: 10.48550/ARXIV.1612.08498. URL https://arxiv.org/abs/1612.08498 .

Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups, 2018. URL https://arxiv.org/abs/1802.03690 .

S. Cohen, Mario Geiger, and Maurice Weiler. Intertwiners between induced representations (with applications to the theory of equivariant neural networks), 2018. URL https://arxiv.org/abs/1803.10743 .

Leon Lang and Maurice Weiler. A wigner-eckart theorem for group equivariant convolution kernels, 2020. URL https://arxiv.org/abs/2010.10952 .

Taco S. Cohen and Max Welling. Group equivariant convolutional networks. axriv , 2016. doi: 10.48550/ ARXIV.1602.07576. URL https://arxiv.org/abs/1602.07576 .

David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to icosahedral projection for SO(3) object reasoning from single-view images, 2022. URL https://arxiv.org/abs/2207.08925 .

Carlos Esteves, Yinshuang Xu, Christine Allen-Blanchette, and Kostas Daniilidis. Equivariant multi-view networks, 2019. URL https://arxiv.org/abs/1904.00993 .

David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning equivariant features for efficient pose prediction. In International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=_2bDpAtr7PI .

Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision , pages 75-82, 2014. doi: 10.1109/WACV.2014.6836101.

Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions on the rotation manifold, 2022.

Yann LeCun, Yoshua Bengio, et al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks , 3361(10):1995, 1995.

Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint arXiv:1803.02155 , 2018.

Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 652-660, 2017.

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation-equivariant neural networks for 3d point clouds, 2018.

Dian Wang, Jung Yeon Park, Neel Sortur, Lawson L. S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry, 2022. URL https: //arxiv.org/abs/2211.09231 .

Maurice Weiler and Gabriele Cesa. General e (2) -equivariant steerable cnns, 2021.

Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, 2018.

Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling. Spherical cnns, 2018.

Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S. Cohen. Explorations in homeomorphic variational auto-encoding, 2018.

Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204.11371 , 2022.

Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, and Ameesh Makadia. Cross-domain 3d equivariant image embeddings. In International Conference on Machine Learning , pages 1812-1822. PMLR, 2019.

Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research , 32(11):1231-1237, 2013.

Yu Xiang, Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint arXiv:1711.00199 , 2017.

Ellen D. Zhong, Tristan Bepler, Joseph H. Davis, and Bonnie Berger. Reconstructing continuous distributions of 3d protein structure from cryo-em images, 2020.

Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints, 2015.

Siddharth Mahendran, Haider Ali, and Rene Vidal. A mixed classification-regression framework for 3d pose estimation from 2d images, 2018.

Yi Zhou, Connelly Barnes, Jingwan Lu, Jimei Yang, and Hao Li. On the continuity of rotation representations in neural networks, 2020.

Romain Brégier. Deep regression on manifolds: A 3d rotation case study, 2021.

Shuai Liao, Efstratios Gavves, and Cees G. M. Snoek. Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres, 2019.

Haowen Deng, Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilic, and Tolga Birdal. Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation, 2020.

Sergey Prokudin, Peter Gehler, and Sebastian Nowozin. Deep directional statistics: Pose estimation with uncertainty quantification, 2018.

Yingda Yin, Yang Wang, He Wang, and Baoquan Chen. A laplace-inspired distribution on SO(3) for probabilistic rotation estimation. In The Eleventh International Conference on Learning Representations , 2023. URL https://openreview.net/forum?id=Mvetq8DO05O .

Tullio Ceccherini-Silberstein, Fabio Scarabotti, and Filippo Tolli. Harmonic Analysis on Finite Groups: Representation Theory, Gelfand Pairs and Markov Chains . Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2008. doi: 10.1017/CBO9780511619823.

Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns, 2018.

Daniel Franzen and Michael Wand. Nonlinearities in steerable so(2)-equivariant cnns, 2021.

Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs, 2021.

Adrien Poulenard and Leonidas J. Guibas. A functional approach to rotation equivariant non-linearities for tensor field networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pages 13169-13178, 2021. doi: 10.1109/CVPR46437.2021.01297.

Yinshuang Xu, Jiahui Lei, Edgar Dobriban, and Kostas Daniilidis. Unified fourier-based kernel and nonlinearity design for equivariant networks on homogeneous spaces, 2022.

Mario Geiger and Tess Smidt. e3nn: Euclidean neural networks, 2022.

Tom Leinster. Basic category theory, 2016.

Hao Su, Charles R. Qi, Yangyan Li, and Leonidas Guibas. Render for cnn: Viewpoint estimation in images using cnns trained with rendered 3d model views, 2015.

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Yang, Zach DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, high-performance deep learning library, 2019.

David Mohlin, Gerald Bianchi, and Josephine Sullivan. Probabilistic regression with huber distributions, 2021.

Igor Gilitschenski, Roshni Sahoo, Wilko Schwarting, Alexander Amini, Sertac Karaman, and Daniela Rus. Deep orientation uncertainty learning based on a bingham loss. In International Conference on Learning Representations , 2020. URL https://openreview.net/forum?id=ryloogSKDS .

Gabriele Cesa, Arash Behboodi, Taco Cohen, and Max Welling. On the symmetries of the synchronization problem in cryo-EM: Multi-frequency vector diffusion maps on the projective plane. In Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems , 2022. URL https://openreview.net/forum?id=owDcdLGgEm .

Ondrej Biza, Sjoerd van Steenkiste, Mehdi S. M. Sajjadi, Gamaleldin F. Elsayed, Aravindh Mahendran, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames, 2023.

Mehdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Paveti´ c, Mario Luˇ ci´ c, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer, 2022.

A. Zee. Group Theory in a Nutshell for Physicists . In a Nutshell. Princeton University Press, 2016. ISBN 9780691162690. URL https://books.google.com/books?id=FWkujgEACAAJ .

J. P. Serre. Groupes finis, 2005. URL https://arxiv.org/abs/math/0503154 .

Tullio Ceccherini-Silberstein, Fabio Scarabotti, and Filippo Tolli. Induced representations and Mackey theory , page 399-425. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2018. doi: 10.1017/9781316856383.012.

Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes, 2015.

Jiehong Lin, Hongyang Li, Ke Chen, Jiangbo Lu, and Kui Jia. Sparse steerable convolutions: An efficient learning of SE(3)-equivariant features for estimation and tracking of object poses in 3d space. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems , 2021. URL https://openreview.net/forum?id=Fa-w-10s7YQ .

Erik Jenner and Maurice Weiler. Steerable partial differential operators for equivariant neural networks, 2022.

Jaime Spencer, Chris Russell, Simon Hadfield, and Richard Bowden. Deconstructing self-supervised monocular reconstruction: The design decisions that matter, 2022.

Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J. Fleet. Monocular depth estimation using diffusion models, 2023.

Xingtong Liu, Ayushi Sinha, Masaru Ishii, Gregory D. Hager, Austin Reiter, Russell H. Taylor, and Mathias Unberath. Dense depth estimation in monocular endoscopy with self-supervised learning methods, 2019.

Victor M. Batlle, J. M. M. Montiel, and Juan D. Tardos. Photometric single-view dense 3d reconstruction in endoscopy, 2022.

Michaël Fonder, Damien Ernst, and Marc Van Droogenbroeck. M4depth: Monocular depth estimation for autonomous vehicles in unseen environments, 2022.

Saro Passaro and C. Lawrence Zitnick. Reducing so(3) convolutions to so(2) for efficient equivariant gnns, 2023.

Kurt Hornik, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are universal approximators. Neural Networks , 2(5):359-366, 1989. ISSN 0893-6080. doi: https://doi.org/ 10.1016/0893-6080(89)90020-8. URL https://www.sciencedirect.com/science/article/pii/ 0893608089900208 .

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper shows that algorithms which learn 3D representations from 2D images must satisfy certain consistency properties, which are equivalent to SO(2)-steerable constraints. To this end, a differentiable induction layer is proposed to map signals on the plane into signals on the sphere. The method is evaluated on PASCAL3D+ and SYMSOL datasets to verify the validity of performing orientation prediction.\n\nSTRENGTHS:\n1) A theoretical foundation for learned equivariant mappings from 2D to 3D is presented.\n\n2) The theoretical background and analyses presented in this paper is thorough.\n\nWEAKNESSES:\nMajor:\n1) The implementation part is too short. It is unclear how the implementation details reflect the theoretical parts. Could the authors elaborate the connection between the theory and the implementation with more details? \n\n2) There is no experimental analysis of how the learning-based framework satisfies the mentioned consistency and the properties. There could be some visualizations and ablation studies to show the effectiveness of the proposed method.\n\n3) The paper mentioned that some previous methods are the special cases of the proposed learning-based method. However, from the experimental results, we can see that the proposed method does not always outperform the compared methods. In some cases, the learned layer's performance is even much worse than that of other methods. Moreover, it seems not convincing that the learned layer can avoid learning nuisance transformations. The analyses of failure cases might be helpful (for example, to explain why on some objects, the proposed method performs much worse than other methods).\n\n\nMinor:\n1) The size of figure 3 is too strange.\n\n2) Many references are not complete.\n\n3) Some references should be their conference/journal version instead of the arXiv version.\n\n4) Some typos:\n\n-- L35: \"then\"\n\n-- L64: \"a SO(2)-steerable constraints\"\n\n-- L158: \"an standard\"\n\n-- L195: \"a the\"\n\n-- L206: \"spaces .Then\"\n\n-- L209: \"respectively\"\n\n-- L230: \"directions. which\"\n\n-- L300: \"stero\"\n\n-- L306: \"have show\"\n\n-- Figure 7: \"a image\"\n\n-- Figure 7: \"a induced\"\n\n-- Figure 7: \"with and SO(3)\"\n\nQUESTIONS:\n1) L33: what does \"group truth\" mean?\n\n2) What is the number of layers of the default ResNet in the experiments? Why does the ResNet-50 version have very poor performance for the boat category?\n\nLIMITATIONS:\nFrom the experimental results, the learned layer can be much worse than some hand-coded methods. However, this is not well addressed in the paper. This could be mentioned as a limitation.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-18"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis is a theory paper. The paper presents the general form of H equivariant linear maps that lift the signals defined on R2 to signals defined on S2, which is defined by convolution with kernel constraints. The main paper focuses on the case where H=SO(2) and G=SO(3). Under such a setting, the authors evaluate an SO(3) distribution regression task and show better performance there.\n\nSTRENGTHS:\n- The paper proposes a theoretical framework to study the group equivariance under projection, which is important when studying 3D reconstruction-related tasks\n- The formulation for the SO(2)-SO(3) setting is clean and easy to implement with existing equivariant network libraries.\n- The theory can be found somehow useful for the SO(3) distribution regression task.\n\nWEAKNESSES:\n- One main limitation of the proposed theory is that the authors only extensively demo the case of SO(2)—SO(3), but lack more general application settings on other groups, for example, at least SE(2) and SE(3), which has wider application settings for vision. Line 134 has an equation for the semi-direct product SE(2) but is not further illustrated as far as the reviewer understands.\n- Another inevitable drawback of studying the single view SO(2)—SO(3) case is that only rotation along the camera optical axis strictly obeys the equivariance since when the object is rotated in 3D or the camera is rotated towards another direction (viewpoint) the content of the image will change due to the nature of the projection, under such case, the single view has no way to guarantee any equivariance. The reviewer thinks studying the multi-view case is more reasonable to address the view content change due to the 3D rotation. This limitation is mentioned by the author and the reviewer thinks it’s relatively fair.\n- For equivariance theory paper, making it more rigorous, people always try to include the case of higher-order tensor features not just with trivial (zero-order) features. In Line 246, as far as the reviewer understands, the paper implementation only tests with zero order features, the reviewer is curious about the performance of including higher-order features.\n\nQUESTIONS:\nsee weakness\n\nLIMITATIONS:\nThe reviewer hasn't found explicit social impact claim.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-06"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis work tackles the challenging problem of achieving SO(3) equivariance over 2D projected images for pose estimation tasks. They propose a unified 2-step framework by adding an induced layer to turn SO(2)-equivariant representations into spherical signal, then using SO(3)-equivariant convolution to get the final pose distribution prediction. They have laid a general mathematical derivation of SO(3) equivariance with restriction and induction representation that satisfies the universal property, which can well summarize existing works as special cases. Comparable results have been achieved in 3 (one from supp.) well-adopted public benchmark for pose estimation.\n\nSTRENGTHS:\n- A novel and universal theory has been proposed to achieve SO(3) equivariance over the challenging 2D image-based pose estimation, which covers several previous works as special cases\n- The proposed framework is able to extend to 6D pose estimation and monocular density reconstruction problem, which impact is potentially to be significant for a lot of other 2D-to-3D learning tasks\n- Compared to the main paper, the additional full derivation in the supp. is clear and detailed\n\nWEAKNESSES:\n- My major concern is on the contribution significance, and the author might want to give a better clarification on the original contribution of the paper.\n  - The claimed major contribution of induced layer seems to mainly come from this [Image to Sphere(Klee et al.)](https://arxiv.org/abs/2302.13926) paper. \n  - Given we already have a lot of works tackling SO(2)-equivariant convolutions on image space, and SE(3)-equivariant operation on 3D space, it seems a trivial contribution to get a SO(3)-equivariant framework for 2D images by simply connecting existing tools from e2nn and e3nn package. Like one possible simple solution might be first lifting the 2D image using a pre-trained monocular depth, and then use SE(3) equivariant convolutions to do 6D pose estimation on the 3D space, thus we do not need to do this complex spherical signal lifting and follow-up neural operations. \n  - The theory on equivariant 6D pose estimation and monocular density volume seems interesting, but it is put in the supp. without any experimental support. \n\n- Though we have this nice equivariant property, the results are just comparable to previous SOTA methods, and on ModelNet40 it is obviously worse than the Klee et al. results, which doesn't seem to match with the conclusion drawn from the main paper.\n\n- Sec 4.1 seems to be poorly written, and the logic flow is very confusing to understand the connections between the 3 paragraphs, like the part mapping F to S2 is missing, and I couldn't find which one is the exact induced representation. \n\n- The theory on universal property in Sec. 5 should be put after Sec. 3, so that we can better understand the unique purpose of having both H-equivariant restriction representation and H-to-G induction representation. Also all designs in Sec. 4 can be better supported in a theoretical way before listing them out. Also the completeness property should be put in the main paper instead of only in the supp\n\n- Line 114 should be $R^D \\to S^2$, since we are mapping feature from a plane onto a sphere\n\nQUESTIONS:\n1. It seems that S2-conv and SO(3) are both spherical convolutions according to Sec. 6.2, I am wondering what is the major difference between these two operations.\n\n2. The author mentions that the implicit function seems to have unique advantage in pose distribution estimation, can the propose framework be potentially extended to support implicit function prediction in the near future?\n\nLIMITATIONS:\nNA", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

comments: SUMMARY:
The paper studies learning equivariant SO3 representations from 2D images. The authors first propose a generalized theory that shows SO2 equivariance constraint for image to spherical signals. Then they propose a construction algorithm called induction layer to implement equivariance. The proposed method outperforms relevant baselines on pose estimation on SYMBOL and PASCAL 3D+.

STRENGTHS:
The paper studies an important problem in 3D vision  (equivariant-SO3 from 2D input) and shows its empirical advantage in 3D pose estimation. This has a potential impact on lots of downstream tasks. 
They provide a unified theory that shows previous works are their special cases. 
The empirical results are convincing, especially results with 10% training data.

WEAKNESSES:
1. It is not clear how to construct the induction layer from the paper. More details may be provided on how to implement the layer, e.g. how to solve equation 2.
2. The experiments are object-centric 6D pose estimation. I wonder if the method can generalizes to cluttered scenes like those in BOP challenges (https://bop.felk.cvut.cz/challenges/)

QUESTIONS:
Please see weakness section

LIMITATIONS:
The limitation is discussed.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-08

comments: SUMMARY:
The paper focuses on the task of 3d pose prediction. The paper proposes a set of consistency constraints for learning 3d representation and shows that few of the previously proposed neural architectures follow these constraints. Further using their constraints they propose a new architecture with an induction layer that maps feature maps to a feature sphere. Using this new architecture they show that their method can achieve state-of-the-art performance on PASCAL3D+ and Symsol 2 dataset.

STRENGTHS:
-> the paper proposes a unifying theory that encompases previously proposed architecture

-> the paper achieves strong performance on PASCAL3D+ and SYMSOL II

-> the paper is well written and described

WEAKNESSES:
-> I couldn't find any ablations/analysis in the paper

-> No results on ModelNet which has been included in previous papers

-> The differences between baseline in terms of architecture/pre-training/losses used has not been made very clear

QUESTIONS:
-> How does the method compare on datasets such as ModelNet or CO3D? I would be interested in seeing comparisions against recent methods like these: https://arxiv.org/abs/2208.05963

->  Do baselines use the same backbone/pre-training? 

-> What happens if u drop the induction layer while still modelling the uncertainity?

-> How does increasing/decreasing the training data affect the results? (currently only 10%)

-> Why results on SYMSOL II are poor compared to the baseline? Anything specific about SYMSOL II?

LIMITATIONS:
Yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-30