Equivariant Single View Pose Prediction Via Induced and Restriction Representations
Authors: Owen Lewis Howell
David Klee
Ondrej Biza
Linfeng Zhao
Robin Walters
Venue: NeurIPS 2023 poster (2023)
Keywords: Equivarient Machine Learning
Pose Prediction
Computer Vision
TL;DR: Equivariant methods for single view pose estimation.

Abstract: Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in three dimensions to make predictions about novel images. However, imposing $SO(3)$-equivariance on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have a natural action on the two-dimensional plane. Specifically, it is possible that an element of $SO(3)$ will rotate an image out of plane. We show that an algorithm that learns a three-dimensional representation of the world from two dimensional images must satisfy certain consistency properties which we formulate as $SO(2)$-equivariance constraints. We use the induced representation of $SO(2)$ on $SO(3)$ to construct and classify architectures that have two-dimensional inputs and 
which satisfy these consistency constraints. We prove that any architecture which respects said consistency constraints can be realized as an instance of our construction. We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose a new algorithm that is a learnable generalization of previously considered methods. We test our architecture on three pose predictions task and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.

Equivariant Single View Pose Prediction Via Induced and Restricted Representations
Owen Howell +1 David Klee?, Ondrej Biza², Linfeng Zhao², and Robin Walters
Department of Electrical and Computer Engineering, Northeastern University Khoury College of Computer Sciences, Northeastern University
Abstract
Learning about the three-dimensional world from two-dimensional images is a fundamental problem in computer vision. An ideal neural network architecture for such tasks would leverage the fact that objects can be rotated and translated in threc-dimensions to make predictions about novel images. However, imposing -qquivaavva on two-dimensional inputs is difficult because the group of three-dimensional rotations does not have natural action the two-dimensional plane. Specifically, is possible that an element of SO(3) will rotate an image out of the plane. We show that algorithm that learns diree-dmmensionl representation of the world from two-dimensional images must satisfy certain consistency properties which formulate (e)))-qquvariianee constraints. We use the induced and restricted representations of SO(2) on SO(3) to construct and classify architectures that satisfy these consistency constraints. We prove our construction realizes all possible architectures that respect these constraints, We show that three previously proposed neural architectures for 3D pose prediction are special cases of our construction. We propose new model that generalizes previously considered methods and contains additional trainable parameters, We test our architecture on three pose prediction tasks and achieve SOTA results on both the PASCAL3D+ and SYMSOL pose estimation tasks.
Introduction
One of the fundamental problems in computer vision learning representations 3D objects from 2D images [133]. By understanding how image features correspond to a physical object, model can generalize better novel views of the object, for instance, when estimating the pose of an object. In general, neural networks that respect the symmetries of a problem are more noise robust and data efficient, while also less prone to over-fitting 4 Three-dimensional space has a natural symmetry group of three-dimensional rotations and three-dimensional translations, SE5(). While we would like leverage this symmetry to design improved neural architectures, serious challenges exist in incorporating 3D symmetry when applied to image data. Specifically, projection of a three- dimensional scene into a two-dimensional plane does not transform equivariantly under all elements of SE(3). This is because there is no priori model for how -wo-dimensional images transform under out-of ol-ne object rotations. Cohen and Welling [ showed how to design neural networks that explicitly SQuiiaaqui and accept images inputs. However, SO(2) -equiivrra methods ignore the fact that the group SO(3) acts on the space of pose configurations.
Equivariant neural networks are much more constrained than general multi-layer perceptrons. The requirement of cquivariance to a group places strict restrictions on the allowed linear maps and the
hoowello@@oorthesseee.ccm
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

allowed non-linear functions in each network layer [ Because of this, the possible structures of Geequivariant neural networks can be completely classified based on the representation theory of the group G kernels of G-equivariant networks [
4
7 g
For compact groups, it is possible to characterize the structure of all possible
(ff) E F
s(h f) E
We argue that any equivariant machine learning algorithm that builds a three-dimensional model of the world from two-dimensional images must satisfy a natural geomet- ric consistency property. This consistency property can be stated as SO(2) -qqiivarraeeee with respect to only the (0) subgroup acting along the camera viewing orienta- tion (see Figure 1). We give complete characterization of maps that satisfy this property. This derivation uses the so- called restricted representation of SO(2) since the group action restricted from the full SO(3). Using the Frobe- nius Reciprocity theorem, we show that this geometric constraint can also be derived using induced representa- tions, type of representation of SO(3) constructed from representations of SO(2). The classification theorems derived in [ g f assume that both the input and out- put layers are G-equivariant. The construction presented map H-equivariant functions to G--
h SO(2)
S
C
à c SO(2)
FEF
h FEF
Figure I: A map t: F. F from signals on IR² to signals on S². Let SO(2) the subgroup that consists of all in-plane rota- tions i.e about the axis defined by the red arrow). The map & must be equivariant with respect this S0(2) S SO(3) subgroup.
in f is different; equivariant functions. Our arguments using induced and restricted representations give natural generalization of equivariant maps between different groups. We derive analogies of the theorems presented in g g
for the induced and restricted representations.
1.1 Importance and Contribution
In this work, we will show how induced and restricted representations can be used to construct neural architectures that accept image data and leverage -qquivarian methods to avoid learning nuisance transformations in three-dimensional space.
We show that our construction satisfies two desirable theoretical properties, completeness and universality. Let H G. We focus on the case G SO(3) and let H SO but we give a theoretical analysis for general groups. Specifically, the induced representation construction is complete in that all group-valued functions on G can be induced from a set of group-valued functions on H. The construction is universal in that all multi-linear maps that map H- -qquiiaari functions to G-equivariant functions are specific cases of the induced representation, modulo isomorphism. Furthermore, we show that the architectures proposed in (10] M are special cases of our construction for the icosahedral group G As and the construction proposed in 12 is a special case of our construction for the diree-dimessioal rotation group G = SO(3). Our method achieves state of the art performance for orientation prediction on PASCAL33+ [3] and SYMSOL [4] datasets.
Contributions:
We propose a unified theory for learning three-dimensional representations from two- dimensional images. We show that algorithms that learn three-dimensional representations from two-dinensional images must satisfy certain consistency properties, which are equiva- lent to (0(2))stcerabilitty constraints.
We introduce a fully differentiable layer called an inductioon/retriictoon layer that maps signals on the plane into signals on the sphere. We show that the induction/restriction layer satisfies a natural consistency constraint and prove both a completeness and universal property for our construction.
Our method achieves SOTA performance for orientation prediction on PASCAL3D+ and SYMSOL datasets.

---

2 Related Work
Equivariant Learning Incorporating task symmetry into the design of neural networks has been
effective domains such as computer vision [5 T6], point cloud processing [17 18], and robotics [9]. Cohen and Welling [ introduced steerable kernels which are trainable layer that be used to build networks that are equivariant to 2D
[5] 20 and 3D transformations [2] 22). The majority of
past works have studied end-to-end equivariant models, where the input can be transformed by the action of the group and all layers are equivariant, in this work, we explore how to "fuse" SO(2) and SO(3) equivariant layers.
There has been growing interest in leveraging 3D symmetry from 2D inputs. 23 learned a 3D transformable latent space from images of single object. [25 trained convolutional network to predict pre-trained (3(3) equivariant embeddings, while [1] 10 12 mapped image features onto elements of the discrete group of SO(3), using structured viewpoints hand-coded projections, respectively. In contrast to prior work, we provide theoretical foundation for learned equivariant mappings from 2D 3D, which additionally guides us introduce more general and effective trainable operation.
Object Pose Estimation Predicting the 3D orientation of objects is important problem in fields like autonomous driving 266 robotics [27] and cryogenic electron microscopy [28]. Many works EQ have used regression approach, and others [ have identified ways to mitigate the discontinuities along the (3) manifold. More recent works have explored ways to model orientation distribution over 3D rotations, which handles object symmetries and captures uncertainty. [34], 36 predict parameters for Bingham, von Mises and Laplace distributions, respectively. These families of distributions can have limited expressivity, so other work explored using implicit networks [4] or the Fourier basis 22 to model more complex pose distributions. Inspired by [3]. 23)
29
355 and
parameterize the latent object pose a distribution SO(3) and then ask what constraints A.0.4 which a generalization of
need to be imposed to enforce SO(2) C SO(2) -qquivarrre
3 Background
We introduce the induced and restricted representations. For a more extensive review of representation theory, see
Let G be group and be vector space over C. A representation (p. V) of map p: G -> Hom[V, VV V) such that
Vg.g' E G, Vo VVVV
p'g-g')u= p(g) p(g')o
Concisely, a group representation is an embedding of a group into a set of matrices. The matrix embed- ding must obey the multiplication rule of the group. We now introduce the restricted representation and induced representation.
Restricted Representation Let H C G be subgroup. Let (p. V) be representation of G. The restricted representation of (p, V) from G' to H is denoted as Res G ((p. V)J. Intuitively, Res H ((p. V))
G
can be viewed as (p, V) evaluated on the subgroup H of G. Specifically,
vh E H, Vo V, ( p(h)v For more in depth discussion of the restricted representation, please see
Induced Representation The induced representation is a way construct representations of larger group G out of representations of a subgroup H C G. Let (p, V) be representation of H. The induced representation of (p, V) from H to is denoted as Ind [(p, V)]. Define the space of functions
F F= F f. V, vh H, f(gh) p(h-)))(g))
Then the induced representation is defined as (I,F) = [nd V)] where the induced action π acts the function space F via
Vg.g E G, VFEF, (T(2) (0 = 1 f'9
3

---

Please see A for an in depth discussion of the induced representation. The induced and restricted
representations are adjoint functors [37
Method
Convolutional networks or vision transformers are typically used to extract spatial feature maps from 2D images. For convenience ignore discretization and treat the feature maps as having continuous inputs f: IR2 - Rd. To leverage spatial symmetries in 3D, we would like to map our features f from plane onto sphere: g S TRD Klee et al. T2 proposed planar feature map is stretched over a hemisphere, but other possible mappings exist.
such mapping. where the
We formalize the equivariance property that every projection should have through the theory of induced and restricted representations. The constraints that we impose have an intuitive geometric interpretation. We give complete characterization of all possible linear and equivariant projections from planar features to a spherical representation. Our general formulation includes [1] as special case, and show that learnable equivariant projection leads to better predictive models.
4.1 Equivariant 2D to 3D Projection by Induced and Restricted Representations
We first derive the -Quuieequvaariance constraint for the most general linear mapping from images to spherical signals.
Image inputs We first describe F the space of image input signals. Let V and Vt be vector spaces. Let F the vector space of V-valued signals defined on the plane
F
Elements of F are sometimes called SE(()-steeraby feature fields 20 The group SE(2) Rº x SO()) of 2D translations and rotations acts F via representation unique factorization = hh, where E IR² is a translation and he E SO(2) is a rotation. Let (p, V) be an represttpttnnnnaainnn describing the transformation of the fibers of the features f. Then the action ı is defined
Each h E SE(2) has
VFEF, r P.R. h h SF((), a(h)- f()) p(he)s
so that (T. Ind
SE(2) 2012 (p. V) and (n, F) gives a representation of the group SE(2)
Spherical outputs We would like to map signals in F to functions from S into the vector space VT. Let Ft be the vector space of all such outputs defined as
=t - -
The group SO(3) acts on the vector space Ft via
vft E Ft, E g E SO(3),
x((g) (() = (1)
where pt (g) describes the SO(3) fiber representation.
-quiivaann image to sphere Let H = SO(2) be the SO(2) subgroup of SO(3) that corresponds in-plane rotations of the image. Our goal is classify H-equivariant linear maps : Ft. This equivalent to the constraint that
vh E H1 = SO(2), fEF, @(T(h)- = -
=
(I)
The constraint enforces equivariance with respect SO(2) transformations. By definition, the
evaluation of nt (h) at h SO(2) is the restricted representation n (1))) = Res (e2 [m]])))
S0(3)

---

Res (3)
Ind ()
()
(2
S0(3)
S2( :
5O(3)
P3, V3
D Ws)
(03, V3)
D Ws)
p2, V2
(D², W2)
(p2, V2)
DD W2)
P1; V1
(D', W1)
P1, Vi1)
(D), Wi)
Po: Vo)
(D), Wo)
(pn, Vo)
(D), Wo)
Figure 2: Left: Decomposition of the restricted representation Resso(2) of (0 (D', We)
SO(3) into (i(e)-irreusibles (pk, Vx)) SO(2). Not every S0(2) (0(2)ereppesnnttaion can be realized as the restriction of SO(3) representation. Right: Decomposition of the induced representation (0(3) for SO(2)-
irreuucibles (pt, V2) E (2( into SO(3) -ireduciiss (Df, Wr) E SO(3). Not every SO(3) cepreseeprsinnttatoon can be realized as the induction of an represseeentnnii..n.
4.2 Solving the Kernel Constraint
We use tools from 38 8 to solve for the space of possible maps satisfying the constraint I giving the trainable space for the image to sphere layer.
Our conclusion is that instead of mapping arbitrary SO(2)-input representations arbitrary SO output representations, the allowed input and output representations (p, V) and (p", VT) must satisfy additional constraints. Specifically, not every representation can be realized as the restriction of (3(3) to SO(2) representation Although in this paper focus orientation estimation, the equivariant framework in Section C.O.I is more general. In the Appendix D we formulate and solve analogous equivariance constraints for both 6DoF-pose estimation and monocular volume reconstruction.
The constraint Equation Tean be solved exactly using the results of 38 linear general map $ - Fi can be expanded as
Theorem
g
The most
[ = f dr x(ū, r))(r)
where K. IR2 x S² + HHmm[V, vtJ. Then, the form of k. can be written
<(ā,)) Σ Ye(r)
(2)
where Ye(f) is the vectorization of the E-type spherical harmonics and each Fe(r) is a standard 5O(2)-steerable kernel 19 ES that has inpul rrpreerrpres (p, V) and output S2(
representation (p', (VV (p, V) Ressor))((D) (DC, vd)..
The proof of this statement is given in Appendix F Note that similar to 18 6 the tensor product structure of the SO( and SO(3) irreducible representations determine the allowed input and output representations of the matrix valued harmonic coefficients (r))..
4.3 Including Non-Linearities
In Section 4.2 considered the most general linear maps that satisfied the generalized equivariance constraint. Adding non-linearities should allow for more expressiveness. Understanding non- linearities between equivariant layers is still an active area of research 39 42].
One way to include non-linearity is to apply standard SO(3) non-linearities after the linear induction layer. After applying the linear mapping described in C we apply additional spherical non-linearity
4 to the signal on 52, This is the method we employ for the results presented 6.2 shown in
is also possible to include tensor-product based non-linearity analogous to the results of [18 d

---

Theory
5.1 Universal Property
In section 4 we showed how the restriction representation arises naturally when constructing S0(3)- equivariant architectures for image data. However, there is no a priori choice of the hidden SO(3) representation. We show that with this choice, our construction satisfies a universal property and is unique up to isomorphism [4]
standard result in group theory establishes the following universal property of induced representa- tions, stated in [37]:
Let H. G. Let (p. V) be any HI-representation Let Indi (p. V) be the induced representation of V) from H to G. Then, there exists a unique H-equivariant linear map Φ V + Indfi V such that for any G-representation (a, W) and any H-equivariant linear map v : V + W, there is unique G-equivariant map V Ind V - W such that the diagram commutative.
Theorem 2.
This theorem can be applied to the construction proposed in to to prove universality property, similar to the results of for G-equivariant neural networks.
Let (p, V) be a H H-representation and let (a, W) be G- representation. Let V W where v an intertwiner of a the H -ereprreenntttonn and the restriction of the G-representation to an H -rrepreesetttton so that
(p, V)
Ind (p.
wh E H, Vpphh) Res Res (h)
(a, W)
that V E Hom V). f (o, W)). The universal prop- erty of the induced representation allows us to write any such V in a canonical form. Specifically, as illustrated in Figure 5.1 we can always uniquely decompose V VrT o t where v Homg HomgIInd (p. V). (a, W)] and vp V + Ind V is (a, W) independent.
Figure Commutative Dia-
gram for Uniqueness Property of Induced Representations.
(
(a, W)
1
Ind (p, V)
(p, V)
1 W)
p(h)
Ind pl(h) [ind al())
a(h) (())
p(h)
a(h) a(s)
~
Ind (p. V)
(0 v (a, W)
W
V)
(a. W)
Figure 4: Factorization Identity for Universal Property of Induced Representations
Factorization Property of H C Neural Networks We use the universal property of induced representations to show that all possible latent G-equivariant architectures can be written in terms of the induced representation. from a homogeneous space of group into some vector space 6 Let X! be set of homogeneous spaces of the group H and let X be set of homogeneous spaces of the group G. Let V and wC be a of vector spaces. Then, consider the function spaces
each layer of a equivariant neural network, we have a set functions
+ v.H },
=
W
The group H acts on the homogeneous spaces X and the group G acts on the homogeneous spaces X that the function spaces JHH and J& form representations of H and G, respectively
Suppose we wish to design a downstream G-equivariant neural network that accepts as signals functions that live in the vector space Fff and transform in the po representation of H. Thus,

---

(po. Fo") is H- Hrrepresentation, but not necessarily G-representation At some point, in the architecture, layer F must be H equivariant on the left and both H and Geeuivariinn on the right. Let us call the layer that both H and G-equivariant F..
(pn )!
" (1) "
(P) ""s Ind ((on. Fi") ot (o11FF)) "
11(0)
(DI Ff)
(PI: Ind ((or. [(o.. (0s, ".
Figure 5: Factorization of Generic Architecture Using Universal Property of Induced Representation. Any network that has input layer (P", that is --equivariant and output layer ( that is G-equivariant can factorized terms of the induced representation. The map v y 2o, where v is G Gequivariant and Po, is H equivarriat..
Suppose that Vp is intertwiner between Fi and (fI, Using the factor- ization property of induced representations 5. there
canonical basis of the space Homg Ind [(p,, FF (0), FR)] and we may write w
Hom (ps, F ), Res [(1,,, uniquely as V v Φ where Φ is an H H-equivariant map and & is a G-equivariant map. Thus, any boundary between H and G layers can be written an H-equivariant layer between F!!, and
followed by G-equivariant layer between [ [F and (0 In this way, induction is all you need and all possible latent G-equivariant architectures can be written terms of the induction representation.
Experiments
6.1 Datasets Evaluation Metries
We evaluate the performance of our method on three single-object pose estimation datasets. These datasets require making predictions in SO(3) from single 2D images. SYMSOL [4] consists of set of images of marked and unmarked geometric solids, taken from different vantage points. Training data is annotated with viewing direction. Some objects have symmetries so that there are multiple equivalent viewing directions, which requires learning distributions over poses. PASCAL33+ [3] a popular benchmark for object pose estimation composed of real images of objects from twelve categories. This dataset is challenging due to the large variation in object appearances and the presence of novel object instances in the test set. To be consistent with the baselines, we augment the training data with synthetic renderings 45| and evaluate performance the PASCALVOC. val set. For more details on the benchmark datasets and additional numerical experiments, see
When single ground truth rotation label is provided, we evaluate the method using the geodesic distance between the predicted and ground truth rotation matrices, reported as ei- ther median rotation error or accuracy at given rotation error threshold. For SYMSOL, which provides the full set of equivalent rotations asso- ciated with an image, we measure the accuracy of the learned pose distribution using average log- log-likelhhod.. This is also the accuracy metric used in [12].
B
R x S
à E g
Ft
R -R
5: + Rd f Ft
6.2 Implementation & Training Details
Figure 6: Diagram of an Equivariant Image Sphere Convolution. At each unit vector E 5² the kernel x( p) is dependent on the image point p= (x, y) R². Equivariance constraints put restrictions on the al- lowed form of x(A p). Similar to standard convolu- tion, the kernel has a user defined receptive field.
For the results presented in we use a ResNet encoder with weights pre-tra on ImageNet. With 224x224 images as input, this generates a 7x7 feature map with 2048 channels.
The filters the induction layer are instantiated using the e2nn (8) package. The maximum
7

---

frequency is set at & = 6. The output of the induction layer is 64-channeled S signal with fibers transforming in the trivial representation of SO(3). After the induction layer, spherical convolution operation is performed using filter that is parameterized in the Fourier domain, which generates an 8-channel signal over S3(3). spherical non-linearity applied by mapping the signal the spatial domain, applying a ReLU, then mapping back to the Fourier domain. One final spherical convolution with a locally supported filter is performed to generate a one-dimensional signal on SO(3). The output signal is queried using an SO(3) HEALPix grid (recursion level during training, 5 during evaluation) and then normalized using a softmax following [14]. S and (O(3) convolutions were performed using the e3nn 43 package. The network was initialized and trained using Py PyTorch [46]
In order create a fair comparison existing baselines, batch size (64), number of epochs (40), optimizer (SGD) and learning rate schedule (StepLR) were chosen to be the same as that of [[2] Numerical experiments were implemented on NVIDIA P-100 GPUs.
6.3 Comparison to Baselines
We compare our method's performance to competitive pose estimation baselines. We include regression methods, 2 30 33,, that perform well on datasets where objects have single valid pose
(e.g. are non-symmetric or symmetry is disambiguated in labels). We also baseline against methods that model pose with parametric families of distributions, B51 and the Fourier basis of
R4
36
an implicit model [14].
SO(3) [12. To make the comparison fair, methods use the same-sized ResNet backbone for each experiment, and we report results stated in the original papers where possible.
SYMSOL Results Performance the SYMSOL dataset reported in Table Our method achieves the highest average oog-likelhhood on SYMSOL Importantly, we observe a significant improvement over Klee et al. [12] all objects, which indicates that our induction layer is more effective than its hand-designed orthographic projection. On SYMSOL II, our method slightly underperforms Murphy al. [14, which has much higher expressivity the output since is implicit model. However, we demonstrate that our approach, which preserves the symmetry present in the images, is better with less data, as shown in Table 2
Table Average log likelihood (the higher the better 1) on SYMSOL II. Per 14 a single model trained all classes in SYMSOL and separate model is trained on each class in SYMSOL. II.
SYMSOLI I (t)
SYMSOLI (1) sphX
cone
cyl
ter
cube
ico
avg
cyl0
tetX
Deng et al. Prokudin et al.
3
-1.48
0.16
-0.95
0.27
-4.44
-2.45
2.57
1.12
2.99
3.61
35
-1.87
-3.34
-1.28
-1.86
-0.50
-2.39
0.48
-4.19
4.16
1.48
Gilitschenski
-0.43
3.84
0.88
-2.29
-2.29
-2.29
3.70
3.32
4.88
2.90
Murphy
1E!
4.10
4.26
5.70
4.81
1.28
7.30
6.91
8.49
Klee et al. Ours
12
3.41
3.75
3.10
4.78
3.27
2.15
4.84
3.74
5.18
5.61
5.11
4.91
4.22
6.10
5.73
4.69
6.20
7.10
6.01
5.62
Table 2: Average log likelihood on SYMSOL with 10% of training data.
10%
SYMSOL I (1)
10%
SYMSOL 11 (f)
avg
cone
cyl
cube
ico
avg
sphX
cylO
tetX
Murphy al. 14
-7.94
-1.51
-2.92
-6.90
10004
-18.34
40.3
-2.51
2.02
-1.70
Klee et al.
2.98
3.51
2.88
3.62
2.94
1.94
3,61
3.12
3.87
3.84
Ours
3.01
3.63
3.01
3.53
3.02
1.91
3.54
2.88
3.71
4.04
PASCAL3D+ Results Our method achieves state-of sfate-of-the-ar performance on PASCAL3D+ with an average median rotation eтror of 9.2 degrees, as reported in Table Even though object symmetries are consistently disambiguated in the labels, modeling pose as a distribution beneficial for noisy images where there is insufficient information to resolve the pose exactly. Because our induction layer produces representations on the Fourier basis of (O(3), naturally allows for capturing this uncertainty distribution over SO(3). While both method and [2 leverage SO(3) equivariant

---

layers to improve generalization, we find our method achieves higher performance. We believe our induction layer is more robust variations how the images are rendered or captured. which is important for PASCAL3D++ since the data is aggregated from many sources. Moreover, our method does restrict features to the hemisphere, which could be beneficial for objects, like bikes and chairs, that do not fully self-occlude their backsides.
Table 3: Rotation prediction PASCAL3D+. First column is the average over all categories.
Median rolation
degrees (J)
ovg
plane
bike
hoaz
bottle
bus
car
chair
table
mhike
sofa
tnain
Moblin et al.
11.5
10.1
15.6
24.3
7.8
3.3
5.3
13.5
12.5
12.9
13.8
7.4
11.7
Prokudia ct
85
12.2
9.7
15.5
45.6
5.4
2.9
4,5
13.1
12.6
11.8
9.1
4,3
12.0
Tulsiani
Mahendran et
10
10.1
8,5
14.8
20.5
7.0
3.1
5.1
9.5
11.3
14.2
10.2
5.6
11.7
et
3
13.0
12.3
Murphy ct
[0
10.3
10.8
12.9
23.4
8.8
3,4
5.3
10.0
7.3
13.6
9,5
6.4
12.3
Klee et 12
9.8
9.2
12.7
21.7
3.3
9.5
9.3
11.5
10.5
7.2
10.6
Yin 86
9.4
11.7
21.8
2.8
7.9
12.2
8.1
6.9
11.6
Ours (ReeNet-50)
10.2
9.2
13.1
30.6
6.7
3.1
4.8
8.7
3.4
11.6
11.0
5.8
10.6
Ours
Conclusion
In conclusion, have argued that any network that learns a diree-dmmensoonal model of the world from wwo-dimensional images must satisfy certain consistency properties. We have shown how these consistency properties translate into an SO(()-equivariance constraint. Using the induced representation we have derived an explicit form for any neural networks that satisfies said consistency constraint. We have proposed an induction/restriition layer, which is a learnable network layer that satisfies the derived consistency equation. We have shown that the induction layer satisfies both completeness property and universal property and, up to isomorphism, is unique. Furthermore, we have shown that the methods of 12 layer.
can be realized specific instances of the induction
The framework that we have developed is general and can be applied to other computer vision problems with different symmetries. For example, as was noted in 4 the cryogenic electronic microscopy orientation estimation problem has latent SO(3) symmetry but manifest SO(2) x Mn ~ O(2) (as opposed to an SO(2)) symmetry. With a slight modification the results presented in the main text allow for the construction of an induction layer that leverages this observation.
Future Work In many fiructue-frommmmon tasks, one has access multiple images of the same object, taken at either known or unknown vantage points. Our work considers only single view pose estimation. A natural generalization of our work is to include stereo measurements into the induced/restricted representation framework. [50 51 use transformer architectures to learn models
of three-dimensional objects from two-dimensional images. Furthermore, in this work have only considered supervised learning, but our framework naturally generalizes to unsupervised settings like 2 Another natural extension of work would be include transformers into the framework presented here, which only applies to convolutional networks.
In deep learning, we often wish to construct neural network that respects a latent symmetry G that does not have an explicit action on the input data space. We have show how the induced representation can be used to construct latent (G-equivariant neural networks. Our work provides systematic way construct neural architectures that accept any format of inputs and respect the latent symmetries of the problem.
Acknowledgments and Disclosure of Funding
Owen Howell thanks the National Science Foundation Graduate Research Fellowship Program (NSF-GRFP) for financial support. Ondrej Biza, David Klee and Robin Walters were supported by National Science Foundation (NSF) grants 2107256 and 2134178. Linfeng Zhao was additionally supported by NSF grant 2107256. Ondrej Biza also acknowledges support from NSF grants 1750649

---

and 1763878. Owen Howell thanks Dr. Thomas Sayre-Maccord for invaluable advice and logistics
help. Owen Howell further thanks Liam Pavlovic and Dr. David Rosen for useful discussions.
10

---

References
David Marr. Vision: computational investigation into the human representation and processing of visual information. MIT press. 2010.
Richard Harlley and Andrew Zisserman. Multiple View Geometry in Computer Vision. Cambridge University Press, 2 edition, 2004, doi: 10.1017/CBB9988051188 181666.
[3 Onur Ozyesil, Vladislav Voroninski. Ronen Basri, and Amit Singer. survey of structure from motion. 2017. URL http::tttss//////aeaeeeama wwwpaaawwww.r..aaaaaaaaaaaaaaaae wtt.oo/////aa/..oaae/a/a//o
[4] Michael M. Bronstein, Joan Bruna, Taco Cohen, and Petar Veličkovié. Geometric deep learning: Grids, groups, graphs, geodesics, and gauges, 2021. URL httll.ttts:///rrtttt http:///rrivvv...aaaaaa wtgg/aa/////////o 13478
[5] Taco S. Cohen and Max Welling, Stecrable cnns. axriv, 2016. doi: 1044850/ARXXV.1611.008882 URL /stps://arxiv..o wwtg/bs////ioooooo/..o
[6] Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups, 2018. URL
[7] S. Cohen, Mario Geiger, and Maurice Weiler. Intertwiners between indrccetteereeenttttttiiccoioo (with applica- tions to the theory of equivariant neural networks), 2018. URL http::t/aa.ii..orrrtttrr/ http://aaa.iiv..oraaa/aa/////0/ 10743
kemnlls, [8] Leon Lang and Maurice Weiler. A wigner-eckart theorem for group equivariant convolution kemels, 2020. URL htttp:://axxiy> org/ abs/2010 10992
[9] Taco S. Cohen and Max Welling. Group equivariant convolutional networks. axriv, 2016. doi: 10.48550/
ARXIV.1602.0757. URL https://arxii wttp://orrr/aabb/ababaeee / 07576
[10] David Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image icosahedral projection for SO(3) object reasoning from singlc-view images, 2022. URL https://arxiv. wwwerii.rrggeeee www.oogggg//////eooooooo.oe..e// /tlpb//22200000 08925
[11] Carlos Esteves, Yinshuang Xu, Christine Allcn-Blanchette, and Kostas Danilidis. Equivariant multi-view networks, 2019. URL URL
[12] David Klec, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere: Learning equivariant features for efficient pose prediction. In International Conference on Learning Representations, 2023. URL www./eeoouuue id= 2bDpAtr7PI
[13] Yu Xiang, Roozbeh Mottaghi, and Silvio Savarese. Beyond pascal: A benchmark for 3d object detection in the wild. In IEEE Winter Conference on Applications of Computer Vision, pages 75 82, 2014. doi: 10.
[14] Kieran Murphy, Carlos Esteves, Varun Jampani, Srikumar Ramalingam, and Ameesh Makadia. Implicit-pdf: Non-parametric representation of probability distributions the rotation manifold, 2022.
[15] Yann LeCun, Yoshua Bengio, al. Convolutional networks for images, speech, and time series. The handbook of brain theory and neural networks, 336) 1995.
[16] Peter Shaw, Jakob Uszkoreit, and Ashish Vaswani. Self-attention with relative position representations. arXiv preprint /803.02 2018
[17] Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: classification and segmentation. Proceedings of the IEEE conference on computer vision and pattern recognition, pages 652-660, 2017.
Deep learning point for 3d
[18] Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation- and translation translation-equivariaan equivariant neural networks for 3d point clouds, 2018.
[19] Dian Wang, Jung Ycon Park, Neel Sortur, Lawson L. S. Wong, Robin Walters, and Robert Platt. The surprising effectiveness of equivariant models in domains with latent symmetry, 2022. URL https
//arxiv.org// aba/ wttp///////.....///////g/ /ttp//2211110 09231
[20] Maurice Weiler and Gabriele Cesa. General e(2) -quiivarat steerable cans, 2021.
[21] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cons: Learning rotationally equivariant features in volumetric data, 2018
[22] Taco S. Cohen, Mario Geiger, Jonas Koehler, and Max Welling, Spherical enns, 2018.

---

[23] Luca Falorsi, Pim de Haan, Tim R. Davidson, Nicola De Cao, Maurice Weiler, Patrick Forré, and Taco S.
Cohen. Explorations in homeomorphic variational auto-encoding, 2018.
[24] Jung Yeon Park, Ondrej Biza, Linfeng Zhao, Jan Willem van de Meent, and Robin Walters. Learning symmetric embeddings for equivariant world models. arXiv preprint arXiv:2204. 11377,, 2022.
[25] Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Daniilidis, and Ameesh Makadia. Cross-domain 3d equivariant image embeddings. In International Conference on Machine Learning, pages 1812-1822. 1822. PMLR, 2019.
[26] Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32(11):1231-1237 2013.
[27] Yu Xiang. Tanner Schmidt, Venkatraman Narayanan, and Dieter Fox. Posecnn: A convolutional neural network for 6d object pose estimation in cluttered scenes. arXiv preprint 1xXiv:7711.00199 2017.
[28] Ellen D. Zhong, Tristan Bepler, Joseph H. Davis, and Bonnic Berger. Reconstructing continuous distribu- tions of 3d protein structure from cryo-em images, 2020.
[29] Shubham Tulsiani and Jitendra Malik. Viewpoints and keypoints, 2015.
[30] Siddharth Mahendran, Haider Ali, and Rene Vidal. A mixed classfiication-reressiin framework for 3d pose estimation from 2d images, 2018.
[31] Yi Zhou, Connelly Barnes. Jingwan Lu, Jimei Yang, and Hao Li, On the continuity of rotation representa- tions in neural networks, 2020.
[32] Romain Brégier. Deep regression on manifolds: 3d rotation case study, 2021.
[33] Shuai Liao, Efstratios Gavves, Cees G. M. Snoek. Spherical regression: Learning viewpoints, surface normals and 3d rotations on n-spheres, 2019.
[34] Haowen Deng, Mai Bui, Nassir Navab, Leonidas Guibas, Slobodan Ilie, and Tolga Birdal. Deep bingham networks: Dealing with uncertainty and ambiguity in pose estimation, 2020.
[35] Sergey Prokudin, Peter Gehler, and Sebastian Nowozin. Deep directional statistics: Pose estimation with uncertainty quantification. 2018
[36] Yingda Yin, Yang Wang, He Wang, and Baoquan Chen. A laplace-inspired distribution on SO(3) for probabilistic rotation estimation. In The Eleventh International Conference on Learning Representations, 2023. URL htttp:://pperreviii wett/forrm//ddiweetg0000000
[37] Tullio Ceccherini-Silbeessee Fabio Scarabotti, and Filippo Tolli. Harmonic Analysis Finite Groups: Representation Theory, Gelfand Pairs and Markov Chains. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2008. doi: 10.1017/C80978051 16619922...
[38] Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation equivariant cnns, 2018
[39] Daniel Franzen and Michael Wand. Nonlinearities in stecrable so(uieequivariaat cnns, 2021.
[40] Pim de Haan, Maurice Weiler, Taco Cohen, and Max Welling. Gauge equivariant mesh cnns: Anisotropic convolutions on geometric graphs, 2021.
[41] Adrien Poulenard and Leonidas Guibas. A functional approach to rotation equivariant non- linccriiiiisis for tensor field networks. In 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 13169- 13178, 2021. doi: 10. 10010099CVP44443..22200.000
[42] Yinshuang Xu, Jiahui Lei. Edgar Dobriban, and Kostas Daniilidis. Unitied bourier-based kernel and nonlinearity design equivariant networks on homogeneous spaces, 2022.
[43] Mario Geiger and Tess Smidt. c3nn: Euclidean neural networks, 2022.
[44]
Tom Leinster. Basic category theory, 2016.
[45] Hao Su, Charles R. Qi, Yangyan Li, and Leonidas Guibas. Render for enn: Viewpoint estimation in images using trained with rendered 3d model views, 2015.
12

---

[46] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf. Edward Yang. Zach Vitito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. Pytorch: An imperative style, segh-performance deep learning library, 2019.
[47] David Mohlin, Gerald Bianchi, and Josephine Sullivan. Probabilistic regression with huber distributions, 2021.
[48] Igor Gilitschenski, Roshni Sahoo, Wilko Schwarting, Alexander Amini, Sertac Karaman, and Daniela Rus. Deep orientation uncertainty learning based on bingham loss. In International Conference on Learning Representations, 2020. URL https://openrevvice net/forumm -dy-rrddddrrdddyar
[49] Gabriele Cesa, Arash Behboodi, Taco Cohen, and Max Welling. On the symmetries of the synchronization problem in cryo-EM: Multi-frequeet vector diffusion maps on the projective plane. In Alice H. Oh, Alckh Agarwal, Danielle Belgrave, and Kyunghyun Cho, editors, Advances in Neural Information Processing Systems, 2022. M http://pwrrrreieeee.ettrrorrn/ee het//foruundd--oweeLLiigm
[50] Ondrej Biza, Sjoerd van Steenkiste, Mchdi S. M. Sajjadi, Gamaleldin F. Elsayed, Aravindh Mahendram, and Thomas Kipf. Invariant slot attention: Object discovery with slot-centric reference frames, 2023.
[51] Mchdi S. M. Sajjadi, Daniel Duckworth, Aravindh Mahendran, Sjoerd van Steenkiste, Filip Pavetić, Mario Lučié, Leonidas J. Guibas, Klaus Greff, and Thomas Kipf. Object scene representation transformer, 2022.
[52] A. Zec. Group Theory in Nutshell for Physicists. In a Nutshell. Princeton University Press, 2016. ISBN 9780691162600, URL http:////.oooo..oooooooo..ooo//a
[53] J. P. Serre. Groupes finis, 2005. URL htt.aappp./aar.....oogaaaa/aa///
[54] Tullio Ceccherini-Sillene theory, page -99-425. Cambridge Studies in Advanced Mathematics. Cambridge University Press, 2018. doi: 10.101779781311655333..1....
Fabio Scarabotti, and Filippo Tolli. Induced representations and Mackey
[55] Zhirong Wu, Shuran Song. Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. shapenets: deep representation for volumetric shapes, 2015.
[56] Jiehong Lin, Hongyang Li, Ke Chen, Jiangbo Lu, and Kui Jia. Sparse steerable convolutions: An efficient learning equivvaant features for estimation and tracking object poses 3d space. In A. Beygelzimer, Y. Dauphin, P. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, 2021. URL wttpttt:/ooppeeeeeeeeee
net/forur (afforrmmd=Faaeae w- 10s7YQ
[57] Erik Jenner and Maurice Weiler. Stecrable partial diffcrential operators for equivariant neural networks, 2022.
[58] Jaime Spencer, Chris Russell, Simon Hadfield, and Richard Bowden. Deconstructing sulf-super monocular reconstruction: The design decisions that matter, 2022.
[59] Saurabh Saxena, Abhishek Kar, Mohammad Norouzi, and David J. Flect. Monocular depth estimation using diffusion models, 2023.
[60] Xingtong Liu, Ayushi Sinha, Masaru Ishii, Gregory D. Hager, Austin Reiter, Russell H. Taylor, and Mathias Unberath. Dense depth estimation in monocular endoscopy with self-supervised learning methods, 2019,
[61] Victor M. Batlle, J. M. M. Montiel, and Juan D. Tardos. Photometric single-view dense 3d reconstruction in endoscopy. 2022.
[62] Michaël Fonder, Damien Ernst, and Mare Van Droogenbroeck. M4depth: Monocular depth estimation for autonomous vehicles in unseen environments, 2022.
[63] Saro Passaro and C. Lawrence Zitnick. Reducing so(3) convolutions to so(2) for efficient equivariant gans, 2023.
[64] Kurt Homikk, Maxwell Stinchcombe, and Halbert White. Multilayer feedforward networks are uni- versal approximators. Neural Networks, 2(5):359 366, 1989. ISSN 0893 doi: https///oi.org/ 10. (111016/08936008008))))0000998 6080089990000-8 0893608089900208
URL
13


Reviews:
[{"review_id": "E1oYd5fHvW", "reviewer": "Reviewer_Nd1u", "summary": "The paper focuses on the task of 3d pose prediction. The paper proposes a set of consistency constraints for learning 3d representation and shows that few of the previously proposed neural architectures follow these constraints. Further using their constraints they propose a new architecture with an induction layer that maps feature maps to a feature sphere. Using this new architecture they show that their method can achieve state-of-the-art performance on PASCAL3D+ and Symsol 2 dataset.", "questions": "-> How does the method compare on datasets such as ModelNet or CO3D? I would be interested in seeing comparisions against recent methods like these: https://arxiv.org/abs/2208.05963\n\n->  Do baselines use the same backbone/pre-training? \n\n-> What happens if u drop the induction layer while still modelling the uncertainity?\n\n-> How does increasing/decreasing the training data affect the results? (currently only 10%)\n\n-> Why results on SYMSOL II are poor compared to the baseline? Anything specific about SYMSOL II?", "limitations": "Yes", "rating": 5, "confidence": 1, "soundness": 2, "presentation": 2, "contribution": 3, "strengths": "-> the paper proposes a unifying theory that encompases previously proposed architecture\n\n-> the paper achieves strong performance on PASCAL3D+ and SYMSOL II\n\n-> the paper is well written and described", "weaknesses": "-> I couldn't find any ablations/analysis in the paper\n\n-> No results on ModelNet which has been included in previous papers\n\n-> The differences between baseline in terms of architecture/pre-training/losses used has not been made very clear ", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for the useful feedback. Please find our point-by-point response below.\n\n$\\textcolor{red}{ \\text{ Weaknesses: }} $\n\n\n **I couldn't find any ablations/analysis in the paper** \n\nWe have added two ablations studies. The first study compares the performance an architecture that uses a spherical convolution followed by an SO(3) convolution and an architecture that utilizes just SO(3) convolutions. We have also added one additional ablation study that compares a naive linear layer vs our proposed layer. We compare both the performance and equivarience error of the linear layer and our proposed layer. These results are included in the attached pdf.\n\n**No results on ModelNet which has been included in previous papers**\n\nWe have included results on ModelNet-SO(3) in the appendix. Other reviewers pointed out that the performance of our architecture was not SOTA on ModelNet-SO(3). Using a residual connection, we can achieve better results on the ModelNet-SO(3) dataset. We address this point in more detail the attached pdf.\n\n\n**The differences between baseline in terms of architecture/pre-training/losses used has not been made very clear**\n\nIn order to ensure that the reader understands our work, we have added additional clarifications to the main paper.\nWe used the same backbone (ResNet-50 for SYMSOL and ModelNet, ResNet-101 for PASCAL3D+) and pre-training (ImageNet-1K) as the baselines. The loss was trained using cross-entropy on SO(3), which was also used in [14].\n\n\n$ \\textcolor{red}{ \\text{ Questions: }} $\n\n\n**How does the method compare on datasets such as ModelNet or CO3D?**\n\nAs mentioned before, we have included the results on ModelNet in the appendix. These results are also commented on in the attached pdf. We do not have time to evaluate our method on the CO3D dataset, but we agree this would be an interesting experiment. \n\n**Do baselines use the same backbone/pre-training?**\n\nYes. In order to make a fair comparison, we used the same backbone (ResNet-50 for SYMSOL and ModelNet, ResNet-101 for PASCAL3D+) and pre-training (ImageNet-1K) as the baselines. Interestingly, we are able to achieve very competitive results on PASCAL3D+ even when using the ResNet-50 backbone. This is somewhat surprising as the ResNet-50 backbone does not have the same representational power as the ResNet-101.\n\n**What happens if u drop the induction layer while still modeling the uncertainty?**\n\nWe are a bit unsure about what the reviewer means by this statement. The uncertainty is important to modeling objects with intrinsic symmetries. Many of the existing baselines use models which model uncertainty in some way, see for example [14].  In the attached pdf we replaced the induction/restriction layer with a linear layer and compared performance. Does this address the stated question?\n\n\n\n**How does increasing/decreasing the training data affect the results? (currently only 10 percent)**\n\nIn general, equivariant methods perform better than non-equivariant methods in the low-data regime. This is because equivariant methods can generalize to unseen data if said data is related to previously seen data by symmetry transformation. In the regime of very large amounts of data, equivariant and non-equivariant methods tend to have similar performance [4]. \n\n\n**Why results on SYMSOL II are poor compared to the baseline? Anything specific about SYMSOL II?**\n\nThis question was asked by multiple reviewers. Unlike SYMSOL I where a single model is trained on all class, in SYMSOL II a separate model is trained on each distinct class. We address this point further attached pdf."}, {"review_id": "CVlBW5WSCF", "reviewer": "Reviewer_mVf3", "summary": "The paper studies learning equivariant SO3 representations from 2D images. The authors first propose a generalized theory that shows SO2 equivariance constraint for image to spherical signals. Then they propose a construction algorithm called induction layer to implement equivariance. The proposed method outperforms relevant baselines on pose estimation on SYMBOL and PASCAL 3D+.", "questions": "Please see weakness section", "limitations": "The limitation is discussed. ", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper studies an important problem in 3D vision  (equivariant-SO3 from 2D input) and shows its empirical advantage in 3D pose estimation. This has a potential impact on lots of downstream tasks. \nThey provide a unified theory that shows previous works are their special cases. \nThe empirical results are convincing, especially results with 10% training data. \n", "weaknesses": "1. It is not clear how to construct the induction layer from the paper. More details may be provided on how to implement the layer, e.g. how to solve equation 2.\n2. The experiments are object-centric 6D pose estimation. I wonder if the method can generalizes to cluttered scenes like those in BOP challenges (https://bop.felk.cvut.cz/challenges/)\n\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nThank you for the feedback. Regarding your questions about our work, here is our response:\n\n$\\textcolor{red}{ \\text{Weaknesses: }} $\n\n**It is not clear how to construct the induction layer from the paper. More details may be provided on how to implement the layer, e.g. how to solve equation 2**\n\nWe have added an additional details on the form of the solution of the geometric constraint and how this is implemented in PyTorch. We also would like to mention that if reading code is more helpful than reading mathematics, our code is attached in the original submission. We have tried to make the code as user friendly as possible.\n\nWe have expanded line 244 in the main text:\n\n\"The filters in the induction layer were instantiated using the e2nn [39] package.\"\n\nBy theorem 2 in the main text, we can expand any linear map $\\Phi$ that satisfies the geometric constraint in equation 1 (line 143) as\n\n$ \\\\Phi (f) ( \\hat{n} ) = \\int_{r \\in \\mathbb{R}^{2} }dr \\text{ } \\kappa( \\hat{n} , r ) f(r) $\n\nwhere each $\\kappa$ can be written as\n$\\kappa( \\hat{n} , r ) = \\sum_{\\ell=0}^{\\infty} F_{\\ell}(r)^{T} Y_{\\ell}(\\hat{n}) $ where each $F_{\\ell}(r)$ is an $SO(2)$-steerable kernels that depend on chosen input and output representations (which are user inputs). The terms $F_{\\ell}(r)$ can be instantiated using the e2nn [39] package. Using the definition of $\\kappa$, the decomposition of $\\Phi(f)$ in terms of spherical harmonics is given by\n\n$ \\\\Phi ( f )( \\hat{n} ) = \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ } \\kappa( \\hat{n} , r ) f(r) = \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ } [ \\sum_{\\ell=0}^{\\infty} F_{\\ell}^{T}(r) Y_{\\ell}(\\hat{n}) ] f(r) = \\sum_{\\ell=0}^{\\infty} [    \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ }  F_{\\ell}^{T}(r) f(r)      ] Y_{\\ell}(\\hat{n})  $\n\nThus, the $\\ell$-th spherical harmonic coefficient of $\\Phi(f)$ is given by $\\Phi_{\\ell}(f) = [    \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ }  F_{\\ell}^{T}(r) f(r)      ] $. This can be computed as a tensor contraction. The inputs to the spherical convolution are then the set of $\\Phi_{\\ell}(f)$. Spherical convolutions are performed with the e3nn [44] package.\n\n\n**The experiments are object-centric 6D pose estimation. I wonder if the method can generalizes to cluttered scenes like those in BOP challenges**\n\nWe thank the reviewer for bringing this benchmark to our attention. In its current form, our method would be unable to deal with pose-estimation for multiple objects. Most equivarient methods have a similar problem as they are equivarient with respect to global transformations. Specifically, there is not a natural way to rotate a single object in a scene and not rotate the whole scene. One natural possible method to address pose-estimation in cluttered scenes would be to first segment objects then apply pose-estimation to each object. This is an interesting problem and will be left for future work.\n"}, {"review_id": "zliQFOHe9s", "reviewer": "Reviewer_beVj", "summary": "This work tackles the challenging problem of achieving SO(3) equivariance over 2D projected images for pose estimation tasks. They propose a unified 2-step framework by adding an induced layer to turn SO(2)-equivariant representations into spherical signal, then using SO(3)-equivariant convolution to get the final pose distribution prediction. They have laid a general mathematical derivation of SO(3) equivariance with restriction and induction representation that satisfies the universal property, which can well summarize existing works as special cases. Comparable results have been achieved in 3 (one from supp.) well-adopted public benchmark for pose estimation.", "questions": "1. It seems that S2-conv and SO(3) are both spherical convolutions according to Sec. 6.2, I am wondering what is the major difference between these two operations.\n\n2. The author mentions that the implicit function seems to have unique advantage in pose distribution estimation, can the propose framework be potentially extended to support implicit function prediction in the near future?", "limitations": "NA", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "- A novel and universal theory has been proposed to achieve SO(3) equivariance over the challenging 2D image-based pose estimation, which covers several previous works as special cases\n- The proposed framework is able to extend to 6D pose estimation and monocular density reconstruction problem, which impact is potentially to be significant for a lot of other 2D-to-3D learning tasks\n- Compared to the main paper, the additional full derivation in the supp. is clear and detailed", "weaknesses": "- My major concern is on the contribution significance, and the author might want to give a better clarification on the original contribution of the paper.\n  - The claimed major contribution of induced layer seems to mainly come from this [Image to Sphere(Klee et al.)](https://arxiv.org/abs/2302.13926) paper. \n  - Given we already have a lot of works tackling SO(2)-equivariant convolutions on image space, and SE(3)-equivariant operation on 3D space, it seems a trivial contribution to get a SO(3)-equivariant framework for 2D images by simply connecting existing tools from e2nn and e3nn package. Like one possible simple solution might be first lifting the 2D image using a pre-trained monocular depth, and then use SE(3) equivariant convolutions to do 6D pose estimation on the 3D space, thus we do not need to do this complex spherical signal lifting and follow-up neural operations. \n  - The theory on equivariant 6D pose estimation and monocular density volume seems interesting, but it is put in the supp. without any experimental support. \n\n- Though we have this nice equivariant property, the results are just comparable to previous SOTA methods, and on ModelNet40 it is obviously worse than the Klee et al. results, which doesn't seem to match with the conclusion drawn from the main paper.\n\n- Sec 4.1 seems to be poorly written, and the logic flow is very confusing to understand the connections between the 3 paragraphs, like the part mapping F to S2 is missing, and I couldn't find which one is the exact induced representation. \n\n- The theory on universal property in Sec. 5 should be put after Sec. 3, so that we can better understand the unique purpose of having both H-equivariant restriction representation and H-to-G induction representation. Also all designs in Sec. 4 can be better supported in a theoretical way before listing them out. Also the completeness property should be put in the main paper instead of only in the supp\n\n- Line 114 should be $R^D \\to S^2$, since we are mapping feature from a plane onto a sphere", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\nWe thank the reviewer for an especially detailed and thorough review. We provide a point-by-point response to the comments and questions below:\n\n$ \\textcolor{red}{ \\text{ Weaknesses: }} $\n\n**The claimed major contribution of induced layer seems to mainly come from this Image to Sphere(Klee et al.) paper**\n\nWe would respectfully disagree with this statement. Although much of our work is heavily inspired by [12], the constraint that we formulate is a geometric statement that should be applicable to architectures that learn consistent three-dimensional models of the world from two-dimensional images. This is a statement that is geometric in nature, and is not unique to [12].\n\n**Given we already have a lot of works tackling SO(2)-equivariant convolutions on image space, and SE(3)-equivariant operation on 3D space, it seems a trivial contribution to get a SO(3)-equivariant framework for 2D images by simply connecting existing tools from e2nn and e3nn package. Like one possible simple solution might be first lifting the 2D image using a pre-trained monocular depth, and then use SE(3) equivariant convolutions to do 6D pose estimation on the 3D space, thus we do not need to do this complex spherical signal lifting and follow-up neural operations.**\n\n\nWe would emphatically disagree with the reviewer on this point. There is an active area of research that aims to learn three-dimensional models of images. The problem of ‘stitching’ SO(2)-equivariant convolutions and SO(3)-equivariant convolutions is not trivial. We would ask the reviewer to find a previous paper which observes that the SO(2) $\\subseteq$ SO(3) subgroup need to ‘align’, which naturally means that the map between two dimensional and three dimensional features must be an intertwiner of restricted/induced representations.\n\nIn addition, the ‘lifting’ procedure proposed by the reviewer can be dependent on the camera model. For example, for some computer vision tasks, such as cryoEM, the image formation model involves Radon transformations, which is drastically different from the pinhole camera model that is applicable to telephoto-lenses. Our proposed construction allows for the camera model to be learned, instead of assumed. The only constraint imposed is geometric consistency.\n\n**The theory on equivariant 6D pose estimation and monocular density volume seems interesting, but it is put in the supp. without any experimental support.**\n\nWe did not consider 6D-pose estimation in detail because the benchmarks that we consider only measure orientation estimation. We agree that this is a natural continuation of our work and we believe that our work on orientation estimation is a promising first step in this direction.\n\n**Though we have this nice equivariant property, the results are just comparable to previous SOTA methods, and on ModelNet40 it is obviously worse than the Klee et al. results, which doesn't seem to match with the conclusion drawn from the main paper.**\n\nThe performance of our model on PASCAL3D+ and SYMSOL I is current state of the art. Using a residual connection, we can achieve better results on the ModelNet-SO(3) dataset. We address this point in more detail the attached pdf.\n\n\n**Sec 4.1 seems to be poorly written, and the logic flow is very confusing to understand the connections between the 3 paragraphs, like the part mapping F to S2 is missing, and I couldn't find which one is the exact induced representation**\n\nWe have added to section 4.1 and section E. Please see response to reviewer mVf3.\n\n**The theory on universal property in Sec. 5 should be put after Sec. 3, so that we can better understand the unique purpose of having both H-equivariant restriction representation and H-to-G induction representation. Also all designs in Sec. 4 can be better supported in a theoretical way before listing them out. Also the completeness property should be put in the main paper instead of only in the supp**\n\nWe cannot fit the full derivation of the completeness property in the main text. We will include a statement of the completeness property in the main text and then refer the reader to the proof in the appendix.\n\n**Line 114 should be R2 -> S2, since we are mapping feature from a plane onto a sphere**\n\nWe are interested in mapping the space of functions $f: \\mathbb{R}^{2} \\rightarrow \\mathbb{R}^{d}$ into the space of functions $ g : S^{2} \\rightarrow \\mathbb{R}^{d'} $. We have changed line 114 to\n\n\"For convenience we ignore discritization and treat the feature maps as having continuous inputs $f : \\mathbb{R}^{2} \\rightarrow \\mathbb{R}^{d}$. To leverage spatial symmetries in 3D, we would like to map features $f$ into the space of features defined on a sphere: $g : S^{2} \\rightarrow \\mathbb{R}^{d'}$. We thus need to consider the space of maps between two function spaces.\n\n$ \\textcolor{red}{ \\text{Questions: }} $\n\n**It seems that S2-conv and SO(3) are both spherical convolutions according to Sec. 6.2, I am wondering what is the major difference between these two operations.**\n\nSpherical convolutions and SO(3) convolutions are distinct operations. Spherical convolution (cite taco paper) is an operation that convolves two signals defined on $S^{2}$ (cite taco paper) and returns is a signal defined on SO(3). SO(3) convolution is an operation that takes as inputs two signals defined on the group SO(3) and the output of an SO(3) convolution is a signal defined on SO(3). In the appendix, we include an ablation study that compares mapping directly to the sphere and performing spherical convolutions vs. mapping directly to SO(3).\n\n**The author mentions that the implicit function seems to have unique advantage in pose distribution estimation, can the propose framework be potentially extended to support implicit function prediction in the near future?**\nYes it is possible to generalize our work to implicit function prediction. This observation was raised by a few of the reviewers and we address this point in the attached pdf.\n"}, {"review_id": "bvB60gOAYT", "reviewer": "Reviewer_Uxrh", "summary": "This is a theory paper. The paper presents the general form of H equivariant linear maps that lift the signals defined on R2 to signals defined on S2, which is defined by convolution with kernel constraints. The main paper focuses on the case where H=SO(2) and G=SO(3). Under such a setting, the authors evaluate an SO(3) distribution regression task and show better performance there.", "questions": "see weakness", "limitations": "The reviewer hasn't found explicit social impact claim.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper proposes a theoretical framework to study the group equivariance under projection, which is important when studying 3D reconstruction-related tasks\n- The formulation for the SO(2)-SO(3) setting is clean and easy to implement with existing equivariant network libraries.\n- The theory can be found somehow useful for the SO(3) distribution regression task.", "weaknesses": "- One main limitation of the proposed theory is that the authors only extensively demo the case of SO(2)—SO(3), but lack more general application settings on other groups, for example, at least SE(2) and SE(3), which has wider application settings for vision. Line 134 has an equation for the semi-direct product SE(2) but is not further illustrated as far as the reviewer understands.\n- Another inevitable drawback of studying the single view SO(2)—SO(3) case is that only rotation along the camera optical axis strictly obeys the equivariance since when the object is rotated in 3D or the camera is rotated towards another direction (viewpoint) the content of the image will change due to the nature of the projection, under such case, the single view has no way to guarantee any equivariance. The reviewer thinks studying the multi-view case is more reasonable to address the view content change due to the 3D rotation. This limitation is mentioned by the author and the reviewer thinks it’s relatively fair.\n- For equivariance theory paper, making it more rigorous, people always try to include the case of higher-order tensor features not just with trivial (zero-order) features. In Line 246, as far as the reviewer understands, the paper implementation only tests with zero order features, the reviewer is curious about the performance of including higher-order features.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "$\\textcolor{red}{\\text{Weaknesses}}$\n\n**One main limitation of the proposed theory is that the authors only extensively demo the case of SO(2)—SO(3), but lack more general application settings on other groups, for example, at least SE(2) and SE(3), which has wider application settings for vision. Line 134 has an equation for the semi-direct product SE(2) but is not further illustrated as far as the reviewer understands.**\n\nIn this work, we choose to focus on pose prediction tasks only. The semi-direct product SE(2) appears in line 134, because the method that we implement is invariant with respect to in-plane translations. We have added additional sentences to the paper to make this more clear.\n\n\n**Another inevitable drawback of studying the single view SO(2)—SO(3) case is that only rotation along the camera optical axis strictly obeys the equivariance since when the object is rotated in 3D or the camera is rotated towards another direction (viewpoint) the content of the image will change due to the nature of the projection, under such case, the single view has no way to guarantee any equivariance. The reviewer thinks studying the multi-view case is more reasonable to address the view content change due to the 3D rotation. This limitation is mentioned by the author and the reviewer thinks it’s relatively fair.**\n\n In the multi-view case, there is a more intricate set of equivarience constraints that must be satisfied. However, in this work, we chose only to consider single view pose-estimation. Although we acknowledge that much of current research focuses on multi-view problems, single view pose-estimation is an interesting problem in its own right and has applications to robotics and autonomous driving (cite works that consider single view).\n\n**For equivariance theory paper, making it more rigorous, people always try to include the case of higher-order tensor features not just with trivial (zero-order) features. In Line 246, as far as the reviewer understands, the paper implementation only tests with zero order features, the reviewer is curious about the performance of including higher-order features.**\n\nApologies, this should read 'spherical' not 'trivial'. The 'spherical' representation of $SO(3)$ consists of one copy of each $SO(3)$ irreducible. This choice of irreducibles was also used in [12]. We fixed this typo in the text.\n\n"}, {"review_id": "zIKkD2FMso", "reviewer": "Reviewer_hj9M", "summary": "This paper shows that algorithms which learn 3D representations from 2D images must satisfy certain consistency properties, which are equivalent to SO(2)-steerable constraints. To this end, a differentiable induction layer is proposed to map signals on the plane into signals on the sphere. The method is evaluated on PASCAL3D+ and SYMSOL datasets to verify the validity of performing orientation prediction.", "questions": "1) L33: what does \"group truth\" mean?\n\n2) What is the number of layers of the default ResNet in the experiments? Why does the ResNet-50 version have very poor performance for the boat category?", "limitations": "From the experimental results, the learned layer can be much worse than some hand-coded methods. However, this is not well addressed in the paper. This could be mentioned as a limitation.", "rating": 4, "confidence": 3, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1) A theoretical foundation for learned equivariant mappings from 2D to 3D is presented.\n\n2) The theoretical background and analyses presented in this paper is thorough.", "weaknesses": "Major:\n1) The implementation part is too short. It is unclear how the implementation details reflect the theoretical parts. Could the authors elaborate the connection between the theory and the implementation with more details? \n\n2) There is no experimental analysis of how the learning-based framework satisfies the mentioned consistency and the properties. There could be some visualizations and ablation studies to show the effectiveness of the proposed method.\n\n3) The paper mentioned that some previous methods are the special cases of the proposed learning-based method. However, from the experimental results, we can see that the proposed method does not always outperform the compared methods. In some cases, the learned layer's performance is even much worse than that of other methods. Moreover, it seems not convincing that the learned layer can avoid learning nuisance transformations. The analyses of failure cases might be helpful (for example, to explain why on some objects, the proposed method performs much worse than other methods).\n\n\nMinor:\n1) The size of figure 3 is too strange.\n\n2) Many references are not complete.\n\n3) Some references should be their conference/journal version instead of the arXiv version.\n\n4) Some typos:\n\n-- L35: \"then\"\n\n-- L64: \"a SO(2)-steerable constraints\"\n\n-- L158: \"an standard\"\n\n-- L195: \"a the\"\n\n-- L206: \"spaces .Then\"\n\n-- L209: \"respectively\"\n\n-- L230: \"directions. which\"\n\n-- L300: \"stero\"\n\n-- L306: \"have show\"\n\n-- Figure 7: \"a image\"\n\n-- Figure 7: \"a induced\"\n\n-- Figure 7: \"with and SO(3)\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "\n\n\nWe thank the reviewer for the thorough feedback. We address the questions posed by the reviewer below.\n\n$\\textcolor{red}{ \\text{Weaknesses: } }$\n\n**The implementation part is too short. It is unclear how the implementation details reflect the theoretical parts. Could the authors elaborate the connection between the theory and the implementation with more details?**\n\nUnderstood. This point was mentioned by multiple reviewers, we have added additional sentences on the actual implementation of our proposed method. Specifically, we have expanded line 244 in the main text:\n\n\"By theorem 2 in the main text, we can expand any linear map $\\Phi$ that satisfies the geometric constraint in equation 1 (line 143) as\n\n$ \\\\Phi (f) ( \\hat{n} ) = \\int_{r \\in \\mathbb{R}^{2} }dr \\text{ } \\kappa( \\hat{n} , r ) f(r) $\n\nwhere each $\\kappa$ can be written as\n$\\kappa( \\hat{n} , r ) = \\sum_{\\ell=0}^{\\infty} F_{\\ell}(r)^{T} Y_{\\ell}(\\hat{n}) $ where each $F_{\\ell}(r)$ is an $SO(2)$-steerable kernels that depend on chosen input and output representations (which are user inputs). The terms $F_{\\ell}(r)$ can be instantiated using the e2nn [39] package. Using the definition of $\\kappa$, the decomposition of $\\Phi(f)$ in terms of spherical harmonics is given by\n\n$ \\\\Phi ( f )( \\hat{n} ) = \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ } \\kappa( \\hat{n} , r ) f(r) = \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ } [ \\sum_{\\ell=0}^{\\infty} F_{\\ell}^{T}(r) Y_{\\ell}(\\hat{n}) ] f(r) = \\sum_{\\ell=0}^{\\infty} [    \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ }  F_{\\ell}^{T}(r) f(r)      ] Y_{\\ell}(\\hat{n})  $\n\nThus, the $\\ell$-th spherical harmonic coefficient of $\\Phi(f)$ is given by $\\Phi_{\\ell}(f) = [    \\int_{r \\in \\mathbb{R}^{2} } dr \\text{ }  F_{\\ell}^{T}(r) f(r)      ] $. This can be computed as a tensor contraction. The inputs to the spherical convolution are then the set of spherical harmonic coefficients $\\Phi_{\\ell}(f)$. Spherical convolutions are performed with the e3nn [44] package.\"\n\nWe hope that this addendum gives the reader a better understanding of how our proposed method is implemented.\n\n**There is no experimental analysis of how the learning-based framework satisfies the mentioned consistency and the properties. There could be some visualizations and ablation studies to show the effectiveness of the proposed method.**\n\nWe thank the reviewer for this excellent point. We have added one additional numerical experiment that replaces our proposed layer with a linear layer. We then train the linear layer on the SYMSOL I dataset. Post-training we measure the $SO(2)$-equivariant properties of the trained model. We compare the performance of a linear layer with our equivarient layer. We include an additional section on this numerical experiment in the attached pdf.\n\n**The paper mentioned that some previous methods are the special cases of the proposed learning-based method. However, from the experimental results, we can see that the proposed method does not always outperform the compared methods. In some cases, the learned layer's performance is even much worse than that of other methods. Moreover, it seems not convincing that the learned layer can avoid learning nuisance transformations. The analyses of failure cases might be helpful (for example, to explain why on some objects, the proposed method performs much worse than other methods).**\n\nOur method is agnostic with respect to the image formation model that the data is collected with. The exact image formation model in the ModelNet-SO(3) dataset is an orthographic projection. Thus, on the ModelNet-SO(3) dataset our model has to learn the correct image formation model while [12] already uses the correct image formation model. It should be noted that on the PASCAL3D+ dataset, where the image formation model is not described by orthographic projection, our method achieves SOTA results. By adding a residual connection to our method, we are able to achieve much better performance on the ModelNet-SO(3) dataset. We address this point in more depth in the attached pdf. \n\n\n$\\textcolor{red}{ \\text{Questions: } }$\n\n**L33: what does \"group truth\" mean?**\n\nWe apologize for the typo. That should read “ground truth”. This has been changed in the text.\n\n**What is the number of layers of the default ResNet in the experiments?**\n\nWe always choose the encoder to agree with existing baselines in order to create a fair comparison. On the PASCAL3D+ dataset the encoder was chosen to be ResNet-101. On the SYMSOL and ModelNet-SO(3) the encoder is a ResNet-50. \n\n**Why does the ResNet-50 version have very poor performance for the boat category?**\n\nAll existing baselines use the ResNet-101 on the PASCAL3D+ dataset. The poor performance of our model with ResNet-50 encoder probably reflects the fact that the encoder is not deep enough and the boat performance is one of the most difficult categories due to pose ambiguity. We decided to include the results using the ResNet-50 encoder because they illustrate the fact that even with a weaker encoder we can still achieve competitive results.\n\n$\\textcolor{red}{ \\text{ Minor: } }$\nWe thank the reviewer for their editing vigilance. We have addressed all of the minor issues and typos in the updated version of the paper.\n"}]

Author rebuttal:
We thank the reviewers for their comments on our manuscript. We are pleased that the majority of reviewers recognize the fact that our theory provides a unifying perspective for a set of relevant computer vision tasks:
- Reviewer Nd1u: "the paper proposes a unifying theory that encompases previously proposed architecture"
- Reviewer mVf3: "The paper studies an important problem in 3D vision (equivariant-SO3 from 2D input) and shows its empirical advantage in 3D pose estimation. This has a potential impact on lots of downstream tasks." 
- Reviewer beVj: "A novel and universal theory has been proposed to achieve SO(3) equivariance over the challenging 2D image-based pose estimation, which covers several previous works as special cases"
- Reviewer Uxrh: "The paper proposes a theoretical framework to study the group equivariance under projection, which is important when studying 3D reconstruction-related tasks"

We recognize the criticism of the reviewers in regards to clarity of presentation, numerical performance on benchmarks and dearth of ablation studies. In order to address these concerns, we have added an additional description of the implementation (found in section 4.2) of our proposed method. In the attached pdf, we include tables for additional numerical experiments. Specifically,

- We include an additional experiment on ModelNet-SO(3) dataset [33]. A major concern of many of the reviewers was that the performance of our architecture was worse that [12] on the ModelNet-SO(3). In some ways, this may be expected as the [12] assumes that the correct image formation model is an orthographic projection, which is the true image formation model used in the data generation of the ModelNet-SO(3) dataset. Our architecture needs to learn the correct image formation model. By including additional biases about the image formation model, we can achieve state of the art results on the ModelNet-SO(3) dataset. We added a residual connection to our induction/restriction layer that is an orthographic projection. This reflects the assumption that for the ModelNet-SO(3) model, the true image formation model is close to orthographic projection, which is common for pinhole camera models [2]. With this additional bias, our model achieves SOTA when averaged over each ModelNet-SO(3) category. These results are shown in the attached pdf in table 10.

- We include an ablation study where we replace our construction with a linear layer. We replaced the induction/restriction layer with a linear layer and trained on the SYMSOL I dataset [14]. We chose the SYMSOL dataset as it consists of rotated solids and any model that performs well on SYMSOL should be approximately equivarient. We choose the spherical layer to have fibers transforming in the $ \\rho _{spherical} = \bigoplus _{\ell=0}^{6} D^{\ell} $ SO(3)-representation. Post-training, we then tested the SO(2)-equivarience properties of the output spherical layer we found a percentage error of about %18 with
output SO(2)-representation approximately $\text{Res} _{SO(2)}^{SO(3)}( \\rho _{spherical}  ) $. This simple numerical experiment shows that the trained linear layer approximately satisfies the geometric constraint derived in the main text. These results are shown in table 12.

- A concern of the reviewers was the performance of our architecture on the SYMSOL II dataset. This phenomena is expected and also observed in [12]. Unlike SYMSOL I on the SYMSOL II dataset, a different model is trained on each class independently. Thus, the SYMSOL I task is more concerned with learning while the SYMSOL II is more concerted with representational power. The method proposed in [14] has significantly more parameters than both our method and [12]. Generalization of our proposed framework to implicit models is an interesting direction for future work. These results are shown in table 11.

- We include an ablation study which compares $S^{2}$ and $SO(3)$ convolutions. We rerun the experiments in the main text using an induction layer that maps images directly to SO(3). The direct induction to SO(3) slightly underperforms the induction to $S^{2}$ on the PASCAL3D+ dataset. We believe that this adds some depth to section E and section F.  These results are shown in table 13.

We addressed specific comments directly in the response section. 

Minor:
There is a mistake in the appendix in section J.4. Specifically, the constraints imposed by restricted/induced representations are conditions on the intertwiners between layers, not the content of irriducibles in each layer. Line 836- 845 should be deleted and replaced with: "Specifically, the linear map between boundary layers must satisfy,

$ \\Psi \in \text{Hom}_{H}(  ( \\rho _{i} , \mathcal{F} _{i}^{H}  ) ,   \text{Res} _{H}^{G}(  \sigma _{1} , \mathcal{F} _{1}^{G}  )   ) \cong \text{Hom} _{G}(   \text{Ind} _{H}^{G}( \\rho _{i} , \mathcal{F} _{i}^{H}  ) , (\sigma _{1} , \mathcal{F} _{1}^{G} )    )$

Specifically, if 
$( \rho _{i} , \mathcal{F} _{I}^{H} )$
and 
$( \sigma _{1} , \mathcal{F} _{1}^{G} ) $
decompose into irreducibles as
	
$ ( \rho _{i} ,	\mathcal{F} _{i}^{H} ) = \bigoplus _{ \rho \in \hat{H} } m _{i \rho } (\rho , V _{\rho} ) , \quad \quad ( \sigma _{1} , \mathcal{F} _{G}^{1} ) = \bigoplus _{ \sigma  \in \hat{G} } n _{1 \sigma } ( \sigma  , W _{\sigma}  )  $


Then, we can write the induced and restricted representations in terms of the branching and induction rules,
 
$ \text{Res} _{H}^{G}( ( \sigma _{1} , \mathcal{F} _{1}^{1})  ) = \bigoplus _{ \rho \in \hat{H} }( (\sum _{  \sigma \in \hat{G} } n _{1 \sigma } B _{ \sigma, \rho} ) ( \rho , V _{\rho} ) ) \quad \text{Ind} _{H}^{G}( ( \rho _{i} , \mathcal{F} _{i}^{H} )    ) = \bigoplus _{ \sigma \in \hat{G} }(  ( \sum _{  \rho \in \hat{H} } m _{ i, \rho } I _{\rho, \sigma}) ( \sigma , W _{\sigma} ) ) $

and intertwiners $\Psi$ can be computed by considering the decomposition of irreducibles."


Meta-review: The paper proposes a method for single view single object 3D pose estimation by mapping feature maps to feature spheres. The reviewers raised concerns regarding lack of ablations, lack of experiments in ModelNet, as well as decreased performance w.r.t specific baselines. The rebuttal addressed those concerns by providing experiments in ModelNet, ablation studies and explaining that the baseline knows the image formation model, which their method doesn’t and needs to infer it. Overall, the reviewers appreciate the idea of the paper and encourage publication.
Decision: Accept (poster)