Quaternion Equivariant Capsule Networks for 3D Point Clouds
Authors: Yongheng Zhao
Tolga Birdal
Jan Eric Lenssen
Emanuele Menegatti
Leonidas Guibas
Federico Tombari
Venue:  (2020)
Keywords: 3d
capsule networks
pointnet
quaternion
equivariant networks
rotations
local reference frame

Abstract: We present a 3D capsule architecture for processing of point clouds that is equivariant with respect to the SO(3) rotation group, translation and permutation of the unordered input sets. The network operates on a sparse set of local reference frames, computed from an input point cloud and establishes end-to-end equivariance through a novel 3D quaternion group capsule layer, including an equivariant dynamic routing procedure. The capsule layer enables us to disentangle geometry from pose, paving the way for more informative descriptions and a structured latent space. In the process, we theoretically connect the process of dynamic routing between capsules to the well-known Weiszfeld algorithm, a scheme for solving iterative re-weighted least squares (IRLS) problems with provable convergence properties, enabling robust pose estimation between capsule layers. Due to the sparse equivariant quaternion capsules, our architecture allows joint object classification and orientation estimation, which we validate empirically on common benchmark datasets. 



Under review as a conference paper at ICLR 2020
QUATERNION EQUIVARIANT CAPSULE NETWORKS FOR 3D POINT CLOUDS
Anonymous authors Paper under double-blind review
ABSTRACT
We present a 3D capsule architecture for processing of point clouds that is equiv- ariant with respect to the 5O(3) rotation group, translation and permutation of the unordered input sets. The network operates on a sparse set local reference frames, computed from input point cloud and establishes end-to-end equivari- ance through a novel 3D quaternion group capsule layer, including an equivariant dynamic routing procedure. The capsule layer enables to disentangle geometry from pose, paving the way for more informative descriptions and a structured la- tent space. In the process, theoretically connect the process of dynamic routing between capsules to the well-known Weiszfeld algorithm, a scheme for solving iterative re-weighted least squares (IRLS) problems with provable convergence properties, enabling robust pose estimation between capsule layers. Due to the sparse equivariant quaternion capsules, our architecture allows joint object clas- sification and orientation estimation, which we validate empirically on common benchmark datasets.
INTRODUCTION
is now well understood that in order to learn a compact representation of the input data, one
needs to respect the symmetries in the problem domain (Cohen al., 2019: Weiler al. 201 8a).
Arguably, one of the primary reasons of the success of 2D convolutional neural networks (CNN) is
the translation-invarre
of the 2D convolution acting on the image grid (Giles & Maxwell 1987
Kondor et al. 2018). Recent trends aim to transfer this success into the 3D domain in order to support many applications such as shape retrieval, shape manipulation, pose estimation, 3D object modeling and detection. There, the data is naturally represented as sets of 3D points or a point cloud (Qi jet al. 2a Unfortunately, extension of CNN architectures to 3D point clouds non-trivial due to two reasons: 1) point clouds are irregular and unstructured, 2) the group of transformation that we are interested in is more complex as 3D data is often observed under arbitrary non- non-commutatiie SO(3) rotations. As result, achieving appropriate embeddings requires 3D networks that work on points to be equivariant to these transformations, while also being invariant to the permutations of the point set.
In order to fill this important gap, we propose the quaternion equivariant point capsule network or QE-Newwork that is suited to process point clouds and is equivariant to SO(3) rotations compactly parameterized by quaternions (Fig. [ in addition to preserved translation and permutation equiv- ariance. Inspired by the local group equivariance (Lenssen et al. 2018 Cohen al.) 2019). we efficiently cover SO(3) by restricting ourselves to the sparse set of local reference frames (LRF) that collectively characterize the object orientation. The proposed capsule layers (Hinton et al.) 2011) deduces equivariant latent representations by robustly combining those local LRFs using the proposed Weiszfeld dynamic routing. Hence, our latent features specify to local orientations disen- tangling the pose from object existence. Such explicit storage is unique to our work and allows us to perform rotation estimation jointly with object classification. Our final architecture is a hierarchy of QE-networks, where we use classification error as the only training cue and adapt a Siamese version when the relative rotation is to be regressed. We neither explicitly supervise the network with pose annotations nor train by augmenting rotations. Overall, our contributions are:
We propose a novel, fully (O(3) -qquivaauvv capsule architecture that is tailored for simul- taneous classification and pose estimation of 3D point clouds. This network produces in-

---

Under review as a conference paper at ICLR 2020
(a) Input Point Cloud
Initial LRFS
(c) LRFS Prior QE-Netwark--
(d) Multi-channel LRFs Prior QE-Neworrk-2
Figure I: Our network operates by processing local reference frames (LRF) the object. Initial LRFs (b) are obtained by computing normal & tangent vectors on the point set in (a). (c) shows the LRFs randomly sampled from (a) and these inputs the first layer of network. Subsequently, we obtain multi-channe LRF that is a set of reference frames per pooling center (d). Holistically, our network aggregates the LRFs arrive at rotation equivariant capsules.
variant latent representations while explicitly decoupling the orientation into capsules, thus attaining equivariance. Note that equivariance results have not been previously achieved regarding the quaternion parameterization of the 3D special orthogonal group.
2. By utilizing LRFs on points, we reduce the space of orientations that we consider and hence can work sparsely subset of the group elements.
3. We theoretically prove the equivariance properties of our 3D network regarding the quater- nion group. Moreover, to the best of our knowledge, we for the first time establish a con- nection between the dynamic routing of Sabour et al. (2017) and Generalized Weiszfeld iterations (Aftab et al. 2015). By that, we theoretically argue for the convergence of the employed dynamic routing.
4. We experimentally demonstrate the capabilities of our network on classification and orien- tation estimation of 3D shapes.
2 PRELIMINARIES AND TECHNICAL BACKGROUND
In this paper will speak of the equivariance of point clouds under the actions of quaternions. We now provide the necessary background required for the grasp of this content.
2.1 EQUIVARIANCE
Definition 1 (Equivariant Map). For a G-space acting on X, the map & 9 x X + X said to be equivariant if its domain and co-domain are acted on by the same symmetry group (Cohen & Welling 2016 Cohen al. 2018at:
Φ = 0 ((x)
(1)
where g1 E 9 and g2 E G. Equivalently (TT x) = T(g2) ((x), where T(-) a linear
representation of the group G. Note that T()) does not have commute. It suffices for T(-) to be homomorphism: T(g1 o g2) = T(g1) o (g2).. this paper we use a stricter form of equivariance and consider g2 -1.
Definition 2 (Equivariant Network). An architecture or network is said to be equivariant if all of its layers are equivariant maps. Due to the transitivity of the equivariance, stacking up equivariant layers will result in globally equivariant networks e.g., rolating the input will produce output vectors which are transformed by the same rotation (Lenssen el al. 2018 Kondor & Trivedi 2018).
2.2 THE QUATERNION GROUP H,
The choice of 4-vector quaternions has multiple motivations: All 3-vector formulations suffer from infinitely many singularities as angle goes to O, whereas quaternions avoid those. 2. 3-vectors also suffer from infinitely many redundancies (the norm can grow indefinitely). Quaternions have single redundancy: 'manifold mean' on the Lie algebra requires iterative techniques with subsequent updates on the tangent space. Such iterations are computationally harmful for differentiable GPU implementation.
-q, a condition that is in practice easy to enforce. 3. Computing the actual
Definition 3 (Quaternion). A quaternion q is an element of Hamilton algebra HI, extending the complex numbers with three imaginary units j. k in the form: q 91 + q2i qai qak
2

---

Under review as a conference paper at ICLR 2020
Algorithm 1: Quaternion Equivariant Dynamic Routing
input Input points (x1, "x,,,.,, xx) R input capsules (LRFs) (qi iq....., q1) HH1 with L N K, N the number of capsules per point, activations a (an 0r. trainable transformations T [tr E [H| LxM
output: Updated frames Q = .......... 9M E H1 updated activations á = (â1..... إMه for All primary (input) capsules do for All latent (outpul) capsules j do
Vi.j +- q [i.j // compute votes
for All latent (oulput) capsules do q < A(vviJ... V. a) initialize output capsules for & iterations do
for All primary (input) capsules i do
W t- ag sigmoid( - 8(9,, Vi.)) // compute the current weight 9 + A((vig... ... .r.st. // see Eq
10
11
āj - sigmoid( Σ 5(1,, vis)) // recompute activations
12
(q), Q2 13. 94)", with (q1, 92, 93, 94) E R² and = scalar part and = (12, 93, q4) E R.,, the vector part. The conjugate of the quaternion q is given by q = - qui - gak. unit quaternion q E H with ||||| : q and q q. gives compact and numerically stable parametrization represent orientation of objects the unit sphere S avoiding gimbal lock and singularities (ussa et al. 2017]. Identifying antipodal points q and qq with the
ijk -. qI E denotes the
element, the unit quaternions form double covering group of SO (3). H1 closed under the non non-commutative multiplication or the Hamilton product:
(p E H1) (r E Hj) ipir Vp v PLV, + riVp Vp X v.
Definition 4 (Linear Representation of H1). We follow Birdal et al.
2018 and use the
43
T(q) A
Q2 Q1
q4
Q3
Q1
91
q3
Q2
91_
To be concise will capital letters to refer to the matrix representation of quaternions e.g.
T(q). = Tgg). Note that (.)), the injective homomorphism to the orthonormal matrix ring. by
construction satisfies the condition in Dfn. 7 Steenrod 1951 : det Q)
Q Q :1Q
|Q.,.II = |Q.iil = and Q - ②) is skew symmetric: Q Q 2qp I. is easy to verify these
properties. T linearizes the Hamilton product or the group composition: g A T(g)q Gq.
2.3 3D POINT CLOUDS
Definition 5 (Point Cloud). We define a 3D surface to be a differentiable 2-manifold embedded in the ambient 3D Euclidean space: M F and a point cloud to a discrete subset sampled on M X E {x, E E2 n
Definition 6 (Local Geometry). For a smooth point cloud (x:} E M C NR3 local reference frame (LRF) is defined as an ordered basis of the tangent space TxM, consisting of orthonor-
mal vectors: [(x) = [0 θ2 0z = 0 x 02). Usually the first component is defined to be the surface normal 0 n E 2 ||||| = and the second one is picked according to modality dependent heuristic.
Note that recent trends such (Cohen et al. 2019) acknowledge the ambiguity and either employ gauge (tangent frame) equivariant design or propagate the determination of a certain direction until the last layer (Poulenard & Ovsjanikov 2018). Here, will assume that O2 be uniquely and

---

Under review as a conference paper at ICLR 2020
ball
X2 BRxa
((): 4 c liI KaoifaM
(q.4= Be
& R
(I)
{q H1 KXNN
( Hy
polnt-to-transform map
[x]] X M
Dynamic Routing
a, c gRXoN
(ean())
Figure 2: Our quaternion equivariant (QE) network for processing local patches: Our input is a 3D point set X on which we query local neighborhoods {x; with precomputed LRFs {q:}. Essentially, we learn the parameters of fully connected network that continuously maps the canonicalized local point set to transformations t;, which are used to compute hypotheses (votes) from input poses. By special dynamic routing procedure that uses the activations determined in previous layer, we arrive at latent capsules that are composed of set of orientations q: and new activations á Thanks to the decoupling of local reference frames, Og is invariant and orientations qi are equivariant to input rotations. All the operations and hence the entire QE-network equivariant achieving guaranteed disentanglement of the rotation parameters. Hat symbol (q) refers to "estimated'.
repeatably computed, reasonable assumption for the point sets we consider (Petrelli & Di Stefano 2011]. For the cases where this does not hold, we will rely on the network's robustness. We will explain our method of choice in Sec. 4 and visualize LRFs of airplane object in Fig. [
3 S0(3)-EQUIVARIANT 3D CAPSULE NETWORKS
Disentangling orientation from representations requires guaranteed equivariances and invariances. Yet, the original capsule networks of [Sabour et al. (2017) cannot achieve equivariance to general groups. To this end, Lenssen al. (2018) proposed to manifold-mean and special aggregation that makes sure that the trainable transformations get pose-aligned points as input. We will extend this idea the non-abelian SO(3) and design capsule networks sparsely operating on a of LRFs computed on local neighborhoods of points, parameterized by quaternions. In the following, we first explain our novel capusle layers, the main building block of our architecture. We then show how to stack those layers via a simple aggregation resulting in an SO(3) -qqiivaaava 3D capsule network that yields invariant representations (or activations) as well equivariant rotations (latent capsules).
3.1 QUATERNION EQUIVARIANT CAPSULE LAYERS
To construct equivariant layers on the group of rotations, we are required to define left-equivariant averaging operator A that is invariant under permutations of the group elements, as well as distance metric that remains unchanged under the action of the group. For these, make the following choices:
Definition (Geodesic Distance). The Riemannian (geodesic) distance in the manifold of rotations lead to the following geodesic distance 8(.) dquar (-):
d(qi, 92) = diguar (qi, q2) cos -1 "(Kq), q2)l)
(3)
Definition 8 (Quaternion Mean pu())). For set of Q rotations S {q:} and associated weights w {w;}, the weighted mean operator A(S, w H1" " x R" + HI," defined through the
following maximization procedure Maarkly et al. 2007
arg max Mq gega
(4)
w.q.q.- The average quaternion is the eigenvector of M corresponding the maximum eigenvalue. This operation lends itself both analytic Magnus 1985) and automatic differentiation (Laue et al. 2018).
where M E 4R4444 is defined as: M
Σ
Theorem 1. Quaternions, the employed mean A(S, w) and geodesic distance 8(.) enjoy the fol- lowing properties:

---

also be Under review as a conference paper at ICLR 2020
QB Natwoekk
C Clacificatton
QE Netwonls
QE- Netoork
QF Network
xx
QE Network
Gx
X M2
Doveccangle
Downssniple
Estimation
Figure 3: Our entire capsule architecture. We hierarchically send all the local patches to our Q- network as shown in Fig. 2 field, gradually reducing the LRFs into single capsule per class. We use classification and pose estimation (in the siamese case) as supervision cues to train the point-to-transforn maps.
At each level the points are pooled in order to increase the receptive
1. A(g S, w) is lest-equivariant: A(g S. w) g A(S, w). A((go(1) ..... ...()).
2. Operator A is invariant under permulations:
Aliq....,,qQh, w).
3. The transformations g E Hi, preserve the geodesic distance 8(-) given in Dfn.
Proof. The proof's are given in the supplementary material.
We also note that the above mean is closed form, differentiable and can be implemented batchwise. We are ready to construct the group dynamic routing (DR) by agreement that is equivariant thanks to Thm. J The core idea is to route from or assign the primary capsules that constitute the input LRF set, to the latent capsules by an iterative clustering which respects the group structure. At each step, we assign the weighted group mean to each output capsule. The weights w + o oxx, y) are inversely propotional to the distance between the vote quaternion and the new quaternion (cluster center). See Alg. [ for details. In the following, we analyze our variant of routing as an interesting case of the affine, Riemannian Weiszfeld algorithm (Aftab al. 2015 2014).
Lemma 1. For yx 5(x. y(x,y)0-2 the equivariant routing procedure given in Alg. iis variant of the affine subspace Wieszfeld algorithm (Aftab et al. 2015 computing the Ly geometric median.
2014) that is robust algorithm for
Proof Sketch. The proof follows from the definition of Weiszfeld iteration (Aftab al., 2014 and the mean and distance operators defined in Sec. 3.1 We first show that computing the weighted mean is equivalent to solving the normal equations in the iteratively reweighted least squares (IRLS) scheme (Burrus 2012). Then, the inner-most loop correspond to the IRLS or Weiszfeld iterations. We provide the detailed proof in supplementary material.
Note that, in practice one is quite free choose the weighting function ar (-) long as is inversely proportional to the geodesic distance and concave (Aftab & Hartley 2015). We leave the analyses of the variants of these algorithms as future work. The original dynamic routing formulated as a clustering procedure with a KL divergence regularization. This holistic view paves the way better routing algorithms (Wang & Liu| 2018). Our perspective is akin yet more geometric due to the group structure of the parameter space. Thanks to the connection to Weiszfeld algorithm. the convergence behavior of our dynamic routing can be directly analyzed within the theoretical framework presented by Aftab et al. (2014) 2015).
Theorem 2. Under mild assumptions provided in the appendix, the sequence of the DR-iterates generated by the inner-most loop almost surely converges critical point.
Proof Sketch. Proof, given in the appendix, is a direct consequence of Lemma and directly exploits the connection the Weiszfeld algorithm.
3.2 EQUIVARIANT 3D POINT CAPSULE NETWORK ARCHITECTURE
The essential ingredient of our architecture, QE-Network, is shown in Fig. 2 We also provide a corresponding pseudocode in Alg. 3 of suppl. material. The input of the QE-Netork are local patch of points with coordinates x; C K rotations parametrized as quaternions qi H, K and activations a; C x qi also represents input primary capsules and local reference frames. N is the number of input capsule channels per point and it is equal to the number of output capsules
5

---

Under review as a conference paper at ICLR 2020
Table 1: Classification accuracy ModelNet40 dataset (Wu et al., 2013) for different methods as well as ours. We also report the number of parameters optimized for each method. X/Y means that we train with X and test with Y.
PN
PN++ KD-treeNet Point2Seq Sph.CNNs PRIN PPF
Ours (Var.)
Ours
NR/NR
88.45
89.82
86,20
92.60
80.13
70.16
85.27
74.43
NR/AR
12.47
21.35
8.49
10.53
43.92
68.85
70.16
11.75
74.07
# Params 3.5M
1.5M
3.6M
1.8M
0.5M
1.5M 3.5M
0.4M
0.4M
(M) from the last layer. In the initial layer, q: represents the pre-computed LRFs and N is equal to 1. Given points x, and rotations q. we compute the quaternion average μ4 in channel-wise as the initial pose candidates: 14, C H, N". These candidates are used to bring the receptive field in multiple canonical orientations by rotating the points: x (M; -1 o x;) R x3 Since the points in the local receptive field lie in continuous IR3, training a discrete set of pose transformations f.y based on local coordinates is not possible. Instead, we employ a point-to-trane woint-to-transform network t(-) RN x3 + RM x4 that maps the point in multiple canonical orientations to transformations. The network is shared over all points to compute the transformations t,, (t(x)) ), .... f(x C IR Kx MxM x4 which are used to calculate the votes for dynamic routing as Va.j qi o tt.y. The network t(-) consists of fully-connected layers that regresses the transformations, similar to common operators for continuous convolutions (Schütt et al.) [017]] Wang et al.; 2018 [Fey et al., 2018). It is the continuous alternative to directly optimizing transformations lying in a grid kernel, as it is done in the original dynamic routing by /Saoour et al. (2017). Note that t(-) predicts quaternions by unit-
normalizing the regressed output: fijj C H Xxxx Although Riemannian layers of [ééiggneul& & Ganea 20188 18) or spherical predictions Liao et al. (2019) can improve the performance, the simple strategy works reasonably for our case. After computing the votes, we utilize the input activation a; to iteratively refine the output capsules (weighted average of votes) q: and activations o; by routing by agreement as shown in Alg.
In order to gradually increase the receptive field, we stack QE-networks creating a deep hierarchy, pooling the points and the LRFs before each layer. Note that we are allowed to do so thanks to the properties of equivariance. In particular, we input N 64 patches to our architecture that composed of two Qe-networks. We call the centers of these patches pooling centers. In the first layer, each of those centers is linked to their immediate vicinity leading to K 9-star local connectivity from which we compute the 64 x 64 x intermediary capsules. The input LRFs of the first layer sampled from pre- neecaallll LRF-set and the input activation is set to 1. The LRFs in the second layer | = 2 are the output capsules of the first layer. = and are routed to the output capsules that are as many the number of classes C, M2 = C. The activation of the second layer is updated by the output of the first layer as well. This construction is shown in Fig. 3 Specifically, for = we use K 9, _Mi 1, MM- = 64 and for / 2. K- 64, N 64, Mi C 40. This way, in this last layer all the pooling centers act as single patch (K = 64). A single QE-network acts on this patch to create the final C' capsules and C¹ activations. More details are reported in Alg. of the appendix.
4 EXPERIMENTAL EVALUATIONS
Implementation Details We implement our network in PyTorch and use the ADAM opti- mizer (Kingma& & Ba 2014) with learning rate of 0.001. The print-transformation mapping net- work is implemented by two FC-layers composed of 64 hidden units. We set the initial activation of the input LRF to 1.0. In each layer, we use 3 iterations of DR. For classification we use the spread loss (Sabour et al. 2018) and the rotation loss is identical to 8(-).
Surface normals are computed by local plane fits (Hoppe et al.) 1992]. We compute the second axis of the LRF, 0z by FLARR (Petrelli & Di Stefano, 2012). that uses the normalized projection of the point within the periphery of the support showing the largest distance, onto the tangent plane
of the center: O2 Pu--p Pp]l Note that using other LRFs such as SHOT (Tombari et al. 2010 or the more modern GFrames of Melzi et al. (2019) is possible. We found FLARE to be sufficient for our experiments. Prior to all operations, we flip all the LRF quaternions such that they lie on the northern hemisphere {qu E S² > 03.
6

---

Under review as a conference paper at ICLR 2020
Table 2: Error of rotation estimation in different categories of ModelNet10. Right side of the table denotes the objects with rotational symmetry, which we include for completeness, PCA-S refers to running PCA only on a resampled instance, while PCA-SR applies both rotations and resampling.
Method
Avg. No_Sym
Chair
Bed
Sofa
Toilet
Monitor
Table
Desk
Dresser
NS
Bathtub
Mcan LRF
0.41
0.35
0.32
0.36 0.34
0.41
0.34
0.45
0.60
0.50
0.46
0.32
PCA-S
0.40
0,42
0.60
0.53 0.46
0.32
0.12
0,47
0.23
0.33
0.43
0.55
PCA-SR
0.67
0.67
0.69
0.70 0.67
0.68
0.61
0.67
0.67
0.67
0.66
0.70
PointNetLK
0.37
0.38
0.43
0.31
0.40
0.40
0.31
0,40
0.33
0.39
0.38
0.34
IT-net
0.27
0.19
0.10
0.22 0.17
0.20
0.28
0.31
0.41
0.44
0.40
0.39
Ours
0.27
0.17
0.11
0.20
0.16
0,18
0,19
0,43
0,40
0.48
0.33
0.31
Ours (siamese)
0.20
0.09
0.08
0.10
0.08
0.11
0.08
0,40
0.35
0.34
0.32
0.30
3D Shape Classification. We use ModelNer40 dataset of (Wu et al., 2015 Qi et al. 2017b) to
assess classification performance. Each shape is composed by 10K points. We assign the LRFs to a subset of the uniformly sampled points, .V = 512 (Birdal & Tic 2017). We train the networks without any rotation augmentation (NR) and put them to test under arbitrary SO(3) rotations (AR). Our results are shown in Tab. [ along with that of PointNet (PN) (Qi et al. 2017a), PointNet++ (PN++) (Qi et al., ) KD-treeNet (Li et al. 2018a). Point2Seq (Liu et al. 2019b). Spherical CNNs (Esteves et al.) 2018), PRIN (You et al.) 2018) and PPF-FoldNet (PPF) (Deng et al. 201 8a). We also present a version of our algorithm ( (Var) that avoids the canonicalization within the QE- network. This is a non-equivariant network that we still train without data augmentation. While this version gets comparable results to the state of the art for the NR/NR case, it cannot handle random SO(3) variations (AR). Note that PPF uses the point-pair-feature (Birdal & Ilie 2015) encoding and hence creates invariant input representations. For the scenario of NR/AR, our equivariant version outperforms all the other methods, including equivariant spherical CNNs (Esteves et al. 2018) by significant gap of at least 5% even when (Esteves et al. 2018) uses the mesh. The object rotational symmetries in this dataset are responsible for a significant portion of the errors we make. It is worth mentioning that we also trained TENs (Thomas et al. 2018) for that task, but their memory demand made it infeasible to scale to this application.
Number of Parameters. Use of LRFs helps to restrict the rotation group to certain elements and thus we can use networks with significantly less parameters (as low as 0.44M) compared to others as shown in Tab. [ Number of parameters in our network depends upon the number of classes, c.g. for ModelNetI0 we have 0.047M parameters.
Rotation estimation in 3D point clouds. Our network can estimate both the canonical and rela- tive object rotations without oose-supervision. To evaluate this desired property, we used the well classified shapes on ModelNet10 dataset, a sub-dataset of Modelnet40 (Wu et al., 2015). We gen- erate multiple instances per shape by transforming the instance with five arbitrary SO(3) rotations. As we are also affected by the sampling of the point cloud, we resample the mesh five times and generate different pooling graphs across all the instances of the same shape. Our QE-architecture can estimate the pose in two ways: 1) by directly using the output capsule with the highest activa- tion, 2) by a siamese architecture that computes the relative quaternion between the capsules that are maximally activated as shown in Fig. 4 Both modes of operation are free of the data augmentation and we give further schematics of the latter in our appendix Fig. 5 Our results against the baselines including naive averaging of the LRFs (Mean LRF) and principal axis alignment (PCA) are re- ported in Tab. 2 as the relative angular error (RAE). We further include results of PointNetL Aokii al.) (2019) and IT-Net (uaa et al. 2018). two state of the art 3D networks that iteratively aligns two point sets. These methods are in nature similar to iterative closest point (ICP) algorithm (Besl & McKay, 1992) but 1) do not require an initialization (first iteration estimates the pose). 2) learn data driven updates. Methods that use mesh inputs such as Spherical CNNs (Esteves et al. (2088) cannot be included here as the random sampling of the same surface would not affect those. We also avoid methods that are just invariant to rotations (and hence cannot estimate the pose) such as Tensorfield Networks (Thomas et al. 2018). Finally, note that IT-net (Yuan et al., 2018) and PointLK need to train lot of epoches (e.g. 500) with random SO(3) rotation augmentation in order to get the models that cover the full SO(3). whereas we train only for ~ 100 epochs. We include more details about the baselines in the appendix under Fig. 8
7

---

Under review as a conference paper at ICLR 2020
[Q)
Q Roxt bx the Tky S 17 Thavsnr
I
Shape
Aligument
Figure 4: Shape alignment the monitor (left) and toilet (right) objects via our siamese equiv- ariant capsule architecture. The shapes are assigned to the the maximally activated class. The corresponding pose capsule provides the rotation estimate.
RAE between the ground truth and the prediction is computed as the relative angle in degrees: d(qi.9 92)/TT Note that resampling and random rotations render the job of all methods difficult. However, both our version that tries find canonical alignment and the siamese variant which seeks a relative rotation are better than the baselines. As pose estimation of objects with rotational symmetry is challenging task we also report results on the non-symmetric subset (No.Sym).
Robustness against point and LRF resampling. Density changes in the local neighborhoods of the shape are an important cause of error for our net- work. Hence, we ablate by applying random re- samplings (patch-wiss wise dropout) to the objects in ModelNet10 dataset and repeating the pose estima- tion and classification as described above. The first part (LRF-10K) of Tab. 3 shows our findings against gradual increases of the number of patches. Here, sample 2K LRFs from the 10K LRFs computed on an input point 10K. 100% dropout corresponds to 2K points in all columns. On second ablation, reduce the amount of points which we compute the LRFs, to 2K and 1K respectively. As we can see from the table, our network is robust towards the changes in the LRFs as well as the density of the points.
Table 3: Ablation study on point density.
LRF Input LRF-1OK
[LRF-2K LRF-IK
Dropout (50% 66% 75% 100%)
100%
100%
Class. Err 77.8 83.3 83.4
87.8
85.46
79.74
Angle. Err 0.34 0.27 0.25
0.09
0.10
0.12
5 RELATED WORK
Deep learning on point sets. The capability to process raw, unordered point clouds within a neu- ral network is introduced by the prosperous PointNet (Qi et al. 2017a) thanks to the point-wise convolutions and the permutation invariant pooling functions. Many works have extended Point- Net primarily to increase the local receptive field size (Qi et al. 2017b; et al.) 2018b; Shen et al.) 2018 Wang et al. 2019). Point-clouds are generally thought of as sets. This makes any permutation- invariant network that can operate on sets an amenable choice for processing points (Zaheer et al.) 2017 Rezatofighi et al., (017). Unfortunately, common neural network operators in this category solely equivariant to permutations and translations but to other groups.
Equivariance in Neural Networks. The early attempts achieve invariant data representa- tions usually involved data augmentation techniques to accomplish tolerance to input transforma- tions (Maturana Scherer 2015) Qi al. 2016 2017a). Motivated by the difficulty associated with augmentation efforts and acknowledging the importance of theoretically equivariant or invari- ant representations, the recent years have witnessed leap in theory and practice of equivariant neural networks (Bao & Song 2019 Kondor & Trivedi 2018).
While laying out the fundamentals of the group convolution, G-CNNs (Cohen Welling 2016) guaranteed equivariance with respect to finite symmetry groups. Similarly, Steerable CNNs CChhe & Welling 2017) and its extension to 3D voxels (Worrall Brrostow 2018) considered discrete symmetries only. Other works opted for designing filters linear combination of harmonic basis functions, leading to frequency domain filters (Worrall et al.) [017] Weiler et al. (018b). Apart from suffering from the dense coverage of the group using group convolution, filters living in the
8

---

Under review as a conference paper at ICLR 2020
frequency space are less interpretable and less expressive than their spatial counterparts, as the basis does not span the full space of spatial filters.
Achieving equivariance in 3D is possible by simply generalizing the ideas of the 2D domain to 3D by voxelizing 3D data. However, methods using dense grids (Chakraborty et al. 2018 Cohen
& Welling 2017) suffer from increased storage costs, eventually rendering the implementations infeasible. An extensive line of work generalizes the harmonic basis filters to (3(3) by using e.g.,
spherical harmonic basis instead of circular harmonics (Cohen et al. 2018b Esteves et al. 2018
Cruz-Mota et al. 20122). In addition to the same downsides as their 2D, these approaches have in common that they require their imput to be projected the unit sphere (Jiang et al. 2019), which
poses additional problems for unstructured point clouds. A related line of research are methods which define a regular structure on the sphere to propose equivariant convolution operators (Liu
et al.
2019a Boomsma & Frellsen 2017
To learn rotation equivariant representation of 3D shape, one can either act on the input data or on the network. In the former case, one either presents augmented data to the network (Qi et al. 2017aa Maturana & Scherer 2015) or ensures otation-invariange in the input (Deng et al. 201 8a b Khoury et al. 2017). In the latter case one can enforce equivariance in the bottleneck so as to achieve an invariant latent representation of the input (Mehr et al., [0188) Thomas et al. 2018 Spezialetti et al.) 2019). Further, equivariant networks for discrete sets of views (Esteves al. 2019b) and cross- domain views (Esteves et al.) 2019a) have been proposed. Here, we aim for a different way of embedding equivariance in the network by means of an explicit latent rotation parametrization in addition to the invariant feature.
Marcos et al. (2017) developed Vector Field Networks, which was followed by the 3D Tensor Field Networks (TFN) (Thomas et al. 2018) that are closest to our work. Based upon a geometric algebra framework, the authors did achieve localized filters that are equivariant to rotations, translations and permutations. Moreover, they are able to cover the continuous groups. However, TFN are designed for physics applications, is memory consuming and a typical implementation is neither likely to handle the datasets we consider nor can provide orientations in an explicit manner.
Capsule Networks. The idea of capsule networks was first mentioned by Hinton et al. (201I), before Sabour et al.] (2017) proposed the dynamic routing by agreement, which started the recent line of work investigating the topic. Since then, routing by agreement has been connected to several well-known concepts, e.g. the EM algorithm (Sabour et al. (2018), , clustering with KL divergence regularization Wang & Liu (2018) and equivariance (Lenssen et al. 2018). They have been extended autoencoders (Kosiorek al. 2019) and GANs Jaiswal et al. (2019). Further, capsule networks have been applied for specific kinds of input data, e.g. graphs (Xinyi & Chen 2019), 3D point
clouds (Zhao al. 2019) or medical images (Afshar al. 2018
6 CONCLUSION AND DISCUSSION
In this work, we have presented a new framework for achieving permutation invariant and SO(3) cquivariant representations on 3D point clouds. Proposing variant of the capsule networks, we operate on sparse set of rotations specified by the input LRFs thereby circumventing the effort to cover the entire S0().. Our network natively consumes a compact representation of the group of 3D) rotations- quaternions, and we have theoretically shown its equivariance. We have also established convergence results for our Weiszfeld dynamic routing by making connections to the litcrature of robust optimization. Our network is among the few for having explicit proup-valued latent space and thus naturally estimates the orientation of the input shape, even without a supervision signal.
Limitations. In the current form our performance is severely affected by the shape symmetries.
The length of the activation vector depends on the number of classes and for achieving sufficiently descriptive latent vectors we need to have a significant number of classes. On the other side, this allows us to perform with merit on problems where the number of classes are large. Although, we have reported robustness to those, the computation of LRFs are still sensitive to the point density changes and resampling. LRFs themselves are also ambiguous and sometimes non-unique.
Future work. Inspired by Cohen et al. (009) and Poulenard & Ovsjanikov (2018) our feature work will involve establishing invariance to the direction in the tangent plane. We also plan to apply our network in the broader context of 3D object detection under arbitrary rotations and look for equivariances among point resampling.

---

Under review as a conference paper at ICLR 2020
REFERENCES
P. Afshar, A. Mohammadi, and K. N. Plataniotis. Brain tumor type classification via capsule net- works. In 2018 25th IEEE International Conference on Image Processing (ICIP), 2018.
Khurrum Aftab and Richard Hartley. Convergence of iteratively re-weighted least squares to robust m-estimators. In 2015 IEEE Winter Conference on Applications of Computer Vision. IEEE, 2015.
Khurrum Aftab, Richard Hartley, and Jochen Trumpf. Generalized weiszfeld algorithms for lq optimization. JEEE Iransactions on pattern analysis and machine intelligence, 37(4), 2014.
Khurrum Aftab, Richard Hartley, and Jochen Trumpf. l closest-point affine subspaces using the generalized weiszfeld algorithm. International Journal of Computer Vision, 114(1):1-15, 2015.
Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey, Pointnetlk: Robust & efficient point cloud registration using pointnet. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 7163-7172, 2019.
Erkao Bao and Linqi Song. 1xXiv:9906.07722 2019.
Equivariant neural networks and equivarification. arXiv preprint
Gary Bécigneul and Octavian- -ugen Ganea. Riemannian adaptive optimization methods. arXiv preprint arXiv:1810.0077 2018.
Paul Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pp. 586-606. International Society for Optics and Photonics, 1992.
Tolga Birdal and Slobodan Ilie. Point pair features based object detection and pose estimation revisited. In 2015 International Conference on 3D Vision, pp. 527-535. IEEE, 2015.
Tolga Birdal and Slobodan Ilic. A point sampling algorithm for 3d matching of irregular geometries. In JEEE/RSJ International Conference on Intelligens Robots and Systems (IROS). IEEE, 2017.
Tolga Birdal, Umut Simsekli, Mustafa Onur Eken, and Slobodan Ilie. Bayesian pose graph optimiza- tion via bingham distributions and tempered geodesic meme. Advances Neural Information Processing Systems, pp. 308-319, 2018.
Wouter Boomsma and Jes Frellsen. Spherical convolutions and their application in molecular mod- elling. In Advances in Neural Information Processing Systems 30, pp. 3433-3443. 2017.
C Sidney Burrus. Iterative reweighted least squares. OpenStax CNX. Available online: http://oxx. 220p:ttmme99220007322224449-9- 7fe7226b788, 12, 2012.
Benjamin Busam, Tolga Birdal, and Nassir Navab. Camera pose filtering with local regression geodesics on the riemannian manifold of dual quaternions. In IEEE International Conference on Computer Vision Workshop (ICCVW), October 2017.
Rudrasis Chakraborty, Monami Banerjee, and Baba C Vemuri. H-cnns: Convolutional neural net- works for riemannian homogeneous spaces. arXiv preprint 1805v:1800505555 2018.
Taco Cohen and Max Welling. Group equivariant convolutional networks. In International confer- ence on machine learning, pp. 2990-2999, 2016.
Taco Cohen, Mario Geiger, and Maurice Weiler. A general theory of equivariant cnns on homoge- neous spaces. arXiv preprint axXiv:1811.02017, 2018a.
Taco Cohen and Max Welling. Steerable cnns. International Conference on Learning Represen- tations (ICLR), 2017.
Taco Cohen, Mario Geiger, Jonas Köhler, and Max Welling. Spherical cnns. 2018b.
Taco S Cohen, Maurice Weiler, Berkay Kicanaoglu, and Max Welling, Gauge equivariant convolu- tional networks and the icosahedral enn. arXiv preprint arXiv: arXiv: 1902.0465. 2019.
10

---

Under review as a conference paper at ICLR 2020
Javier Cruz-Mota, Iva Bogdanova, Benoît Paquier, Michel Bierlaire, and Jean- -hi-Ppilipp Thiran. Scale invariant feature transform on the sphere: Theory and applications. International Journal of Computer Vision, 98(2):217-241, June 2012.
Haowen Deng, Tolga Birdal, and Slobodan llie. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In European Conference on Computer Vision (ECCV), 2018a.
Haowen Deng, Tolga Birdal, and Slobodan llic. Ppfnet: Global context aware local features for robust 3d point matching. In Conference on Computer Vision and Pattern Recognition, 2018b.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis. Learning so (3) equivariant representations with spherical cnns. In Proceedings of the European Conference on Computer Vision (ECCV). pp. 52-68, 2018.
Carlos Esteves, Avneesh Sud, Zhengyi Luo, Kostas Danillidis, and Ameesh Makadia. Cross- 3d equivariant image embeddings. In International Conference on Machine Learning (ICML), 2019a.
Carlos Esteves, Yinshuang Xu, Christine Blenchlln,he,te, and Kostas Daniilidis. Equivariant multi- view networks. arXiv preprini 1rXiv:1904000993 2019b.
Matthias Fey, Jan Eric Lenssen, Frank Weichert, and Heinrich Müller. Splinecnn: Fast geometric deep learning with continuous b-spline kernels. The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018.
C Lee Giles and Tom Maxwell. Learning, invariance, and generalization in high-order neural net-
works. Applied optics, 26(23):4972-4978, 1987.
Geoffrey E Hinton, Alex Krizhevsky, and Sida D Wang. Transforming auto-encoders. Interna- tional Conference on Artificial Neural Networks, pp. 44-51. Springer, 2011.
Hugues Hoppe, Tony DeRose, Tom Duchamp, John McDonald, and Werner Stuetzle. Surface construction from unorganized points, volume 26.2. ACM, 1992.
Ayush Jaiswal, Wael AbdAlmageed, Yue Wu, and Premkumar Natarajan. Capsulegan: Generative adversarial capsule network. In Computer Vision ECCV 2018 Workshops, pp. 526-535. Springer International Publishing, 2019.
Chiyu Max Jiang, Jingwei Huang, Karthik Kashinath, Prabhat, Philip Marcus, and Matthias Niess- ner. Spherical CNNs on unstructured grids. In International Conference on Learning Represen- tations, 2019.
Mare Khoury, Qian- Yi Zhou, and Vladlen Koltun. Learning compact geometric features. In Pro- ceedings of the IEEE International Conference on Computer Vision, pp. 153-161, 2017.
Diederik Kingma and Jimmy Ba. Adam: A method for stochastic optimization, arXiv preprint arXiv:1412.6900 2014.
Risi Kondor and Shubhendu Trivedi. On the generalization of equivariance and convolution in neural networks to the action of compact groups. arXiv preprint arXiv: (802:08000 2018
Risi Kondor, Zhen Lin, and Shubhendu Trivedi. Clebsch-gordan nets: fully fourier space spherical convolutional neural network. In Advances in Neural Information Processing Systems, 2018.
Adam R. Kosiorek, Sara Sabour, Yee Whye Teh, and Geoffrey E. Hinton. Stacked capsule autoen- coders. arXiv preprint 1xXiv:1906.06818, 2019.
Sören Laue, Matthias Mitterreiter, and Joachim Giesen. Computing higher order derivatives of matrix and tensor expressions. In Advances in Neural Information Processing Systems, 2018.
Jan Eric Lenssen, Matthias Fey, and Pascal Libuschewski. Group equivariant capsule networks. In Advances in Neural Information Processing Systems, pp. 8844-8853, 2018.
Jiaxin Li, Ben Chen, and Gim Hee Lee. So-net: orga-irga network for point cloud analysis. In Proceedings of the IEEE conference on computer vision and pattern recognition, 2018a.
11

---

Under review as a conference paper at ICLR 2020
Yangyan Li, Rui Bu, Mingchao Sun, Wei Wu, Xinhan Di, and Baoquan Chen. Pointenn: Convolu- tion on x-transformed points. In Advances in Neural Information Processing Systems, 2018b.
Shuai Liao, Efstratios Gavves, and Cees GM Snoek. Spherical regression: Learning viewpoints, sur- face normals and 3d rotations on n-spheres. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9759-9767, 2019.
Min Liu, Fupin Yao, Chiho Choi, Sinha Ayan, and Karthik Ramani. Deep learning 3d shapes using alt-az anisotropic 2-sphere convolution. In International Conference on Learning Representations (ICLR), 2019a.
Xinhai Liu, Zhizhong Han, Yu-Shen Liu, and Matthias Zwicker. Point2sequence: Learning the shape representation of 3d point clouds with an attention-based sequence to sequence network. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pp. 8778-8785, 2019b. 2019b.
David G Luenberger, Yinyu Ye, al. Linear and nonlinear programming. Springer, 1984.
Jan R Magnus. On differentiating cigenvalues and eigenvectors. Econometric Theory, 1(2), 1985.
Diego Marcos, Michele Volpi, Nikos Komodakis, and Devis Tuia. Rotation equivariant vector field networks. In The IEEE International Conference on Computer Vision (ICCV), Oct 2017.
Landdis Markley, Yang Cheng, John Lucas Crassidis, and Yaakov Oshman. Averaging quaternions. Journal of Guidance, Control, and Dynamics, 30(4):1193-)) 11999999-991,--11111111111991991 2007.
Daniel Maturana and Sebastian Scherer. Voxnet: A 3d convolutional neural network for real-time object recognition. In Intelligent Robots and Systems (IROS). IEEE, 2015.
Eloi Mehr, André Lieutier, Fernando Sanchez Bermudez, Vincent Guitteny, Nicolas Thome, and Matthieu Cord. Manifold learning in quotient spaces. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 9165-9174, 2018.
Simone Melzi, Riccardo Spezialetti, Federico Tombari, Michael M. Bronstein, Luigi Di Stefano, and Emanuele Rodola. Gframes: Gradient-based local reference frame for shape matching. In The IEEE Conference Computer Vision and Pattern Recognition (CVPR), June 2019.
Alioscia Petrelli and Luigi Di Stefano. On the repeatability of the local reference frame for partial shape matching. In 2011 International Conference on Computer Vision. IEEE, 2011.
Alioscia Petrelli and Luigi Di Stefano. A repeatable and efficient canonical reference for surface matching. In 2012 Second International Conference on 3D Imaging, Modeling, Processing, Visu- alizution & Transmission, pp. 403-410. IEEE, 2012.
Adrien Poulenard and Maks Ovsjanikov. Multi-directional geodesic neural networks via equivariant convolution. In SIGGRAPH Asía 2018 Technical Papers, pp. 236. ACM, 2018.
Charles R Qi, Hao Su, Matthias Nießner, Angela Dai, Mengyuan Yan, and Leonidas Guibas. Volumetric and multi-view enns for object classification 3d data. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 5648-5658 2016.
Charles R Qi, Hao Su, Kaichun Mo, and Leonidas J Guibas. Pointnet: Deep learning on point sets for classification and segmentation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 652-660, 2017a.
Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: Deep hierarchical feature learning on point sets in metric space. In Advances in neural information processing systems, pp. 5099-5108, 2017b.
S Hamid Rezatofighi, Anton Milan, Ehsan Abbasnejad, Anthony Dick, Ian Reid, et al. Deepsetnet: Predicting sets with deep neural networks. In 2017 IEEE International Conference on Computer Vision (ICCV), pp. 5257-5266. IEEE, 2017.
Sara Sabour, Nicholas Frosst, and Geoffrey E Hinton. Dynamic routing between capsules. In Advances in neural information processing systems, pp. 3856-3866, 2017.
12

---

Under review as a conference paper at ICLR 2020
Sara Sabour, Nicholas Frosst, and G Hinton. Matrix capsules with em routing. In 6th International Conference on Learning Representations, ICLR, 2018.
Kristof Schütt, Pieter-Jan Kindermans, Huziel Enoc Sauceda Felix, Stefan Chmiela, Alexandre Tkatchenko, and Klaus-Robert Müller. Schnet: A continuous-fille convolutional neural network for modeling quantum interactions. In Advances in Neural Information Processing Systems. 2017.
Yiru Shen, Chen Feng, Yaoqing Yang. and Dong Tian. Mining point cloud local structures by kernel correlation and graph pooling. Proceedings of the IEEE conference on computer vision and paltern recognition, pp. 4548-4557 2018.
Riccardo Spezialetti, Samuele Salti, and Luigi Di Stefano. Learning an effective equivariant 3d descriptor without supervision. arXiv preprint arXiv:/909.000 068877 2019.
Norman Earl Steenrod. The topology fibre bundles, volume 14. Princeton University Press, 1951.
Nathamiel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang. Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-ant and translation-equivaraanl neural networks for 3d point clouds. arXiv preprint a8X.v:/802.082//, 2018.
Federico Tombari, Samuele Salti, and Luigi Di Stefano. Unique signatures of histograms for local surface description. European conference on computer vision, pp. 356-369. Springer, 2010.
Dilin Wang and Qiang Liu. An optimization view on dynamic routing between capsules, 2018. URL htttps:///penrevleew.. net forum/orummmm http://oorommiidddtjeeee...
Shenlong Wang, Simon Suo, Wei-Chiu Ma, Andrei Pokrovsky, and Raquel Urtasun. Deep paramet- ric continuous convolutional neural networks. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. In Advances in Neural Information Processing Systems, pp. 10381- 2018a
Maurice Weiler, Fred A. Hamprecht, and Martin Storath. Learning steerable filters for rotation cquivariant cnns. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), June 2018b.
Daniel Worrall and Gabriel Brostow. Cubenet: Equivariance to 3d rotation and translation. In The European Conference on Computer Vision (ECCV), September 2018.
Daniel E. Worrall, Stephan J. Garbin, Daniyar Turmukhambetov, and Gabriel J. Brostow. Harmonic networks: Deep translation and rotation equivariance. In The IEEE Conference on Computer Vision and Pattern Recognition (CVPR), July 2017.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang. and Jianxiong Xiao. 3d shapenets: A deep representation for volumetric shapes. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1912-1920, 2015.
Zhang Xinyi and Lihui Chen. Capsule graph neural network. International Conference on Learn- ing Representations (ICLR), 2019, URL openreview .et. forttt/orum22 Mil-Byy88BBRCh
Yang You, Yujing Lou, Qi Liu, Yu- Yu-Wing Tai, Weiming Wang, Lizhuang Ma. and Cewu Lu. Prin: Pointwise
network. arXiv preprint arXiv:1811.0933 2018.
Wentao Yuan, David Held, Christoph Mertz, and Martial Hebert. Iterative transformer network for 3d point cloud. arXiv preprint 1rXiv:1811.11200 2018
Manzil Zaheer, Satwik Kottur, Siamak Ravanbakhsh, Barnaba Poczos, Ruslan R Salakhutdinov, and Alexander J Smola. Deep sets. Advances in Neural Information Processing Systems. 2017.
Yongheng Zhao, Tolga Birdal, Haowen Deng, and Federico Tombari. 3d point capsule networks. In Conference on Computer Vision and Puttern Recognition (CVPR), 2019.
13

---

Under review as a conference paper at ICLR 2020
A PROOF OF PROPOSITION 1
Before presenting the proof we recall the three individual statements contained in Prop.
1. A(g S, w) is left-equivariant: A(g o S, w) - A(S, w).
2. Operator A is invariant under permutations: A(f ((())))))...,,)()))) w.)
A( A(qq w).
3. The transformations g Hi preserve the geodesic distance 6(-).
Proof. We will prove the propositions in order.
1. We start by transforming each element and replace q by (g o q:) of the cost in Eq (4)
q MM Σ
=
w.q.q )a
(5)
1=1
= =q aT{²
m m(gog)(goq))))
(6)
q
w, w.G...
(7)
1=1
q GGM, + GMMG
q G
M + +
(8)
q G((M ..... + G
(9)
q GMG q
(10)
= Mp.
(11)
where M, w.q.a. and p G.q. From orthogonallity of G it follows p
= g o = and hence g A(S, w) A(go w).
2. The proof follows trivially from the permutation invariance of the symmetric summation operator the outer products in Eq g
3. It is sufficient to show that |q q2
(g qı) (g 92)l for any g HH;
(g o q1) (g q2)l |ql G Gqzl
(12)
|q Iqpl
(13)
qq qal-
(14)
where g gog q= Gq. The result is a direct consequence of the orthonormality of G.
B PROOF OF LEMMA
We will begin by recalling some preliminary definitions and results that aid us to construct the connection between the dynamic routing and the Weiszfeld algorithm.
Definition (Affine Subspace). A dd-dimensional affine subspace of RN is obtained by translation of d-dimensional linear subspace V C. RN such that the origin is included in S:
$
Σ O.X,
2a.11.
(15)
Simplest choices for S involve points, lines and planes of the Euclidean space.
14

---

Under review as a conference paper at ICLR 2020
Definition 10 (Orthogonal Projection onto an Affine Subspace), An orthogonal projection of point x E RN onto an affine subspace explained by the pair (A, c) is defined as:
(x) _ projs(x)
(16)
denotes the translation to make origin inclusive and A is projection matrix typically defined via the orthonormal bases of the subspace.
Definition 11 (Distance to Affine Subspaces), Distance from given point x to set of affine subspaces {Si, S2 ... } can be writlen as Aftub et al. (2015):
K C(x) Σ d(x,,
k
Σ x projs, (x)]]
(17)
Given that all the antipodal counterparts are mapped to the northern hemisphere, we will now think of the unit quaternion or versor as the unit normal of a four dimensional hyperplane h, passing through the origin:
Lemma 2.
h,(x) = x qu qa:= 0.
(18)
qa is added term to compensate for the shift. When qa 0 the origin is incident the hyperplane. With this perspective, quaternion forms an affine subspace with = 4, for which the projection
operator takes the form:
projs, (p)= (I-..q.)
=
(19)
Proof. We consider Eq 19) for the case where = O and A = (I-qq) ). The former follows from the fact that subspaces by construction pass through the origin. Thus, we only need to show that the matrix A = I-qq is an orthogonal projection matrix onto the affine subspace spanned by To this end, it is sufficient to validate that A symmetric and idempotent: A AA A A. Note that by construction the unit-ness of the quaternion, we arrive at the proof:
is symmetric matrix and hence A itself. Using this property and
A A A = (I qq T( - qq )
(20)
(I - qq ([ qq T
(21)
2qq I
qq qq
(22)
I 2qq
qq
(23)
qq
A
(24)
is easy to verify that the projections are orthogonal to the quaternion that defines the subspace by showing projs (q) q 0:
q (q) = q Aq = q T - qqppppp )qq = q T (q qq q) = (q
(q--aa-o) - q) = 0.
(25)
(26)
Also note that this choice corresponds tr(qq - a4 1.
Lemma 3. The quaternion mean we suggest to use in the main paper Markley et al. 2007 is
equivalent to the Euclidean Weiszfeld mean on the affine quaternion subspaces.
Proof. We now recall and summarize the L Weiszfell Algorithm on affine subspaces Aftab et al. 2015), which minimizes a q-norm variant of the cost defined in Eq (17):
(x)
= d(x, S:) = x projs, (x)
(27)
1=1
Defining M. = -1 A,, Alg. summarizes the iterative procedure.
Note that when q = the algorithm reduces the computation of non-weighted mean (w; 1 Vi), and a closed form solution exists for Eq (29 and is given by the normal equations:
w. M,
w. w,M,c,)
(30)
1=1
15

---

Under review as a conference paper at ICLR 2020
Algorithm 2: by Weiszfeld Algorithm on Affine Subspaces Aftab et al. (2015).
input: An initial guess that does not lie any of the subspaces (S:). Projection operators IT; the norm parameter !
x" Xo
while not converged do Compute the weights w' {w:}:
w = (MM(x'-c;ill =i= 1...k
(28)
Solve:
x ++1 = arg min XERA
i (x - cc)l
(29)
For the case of our quaternionic subspaces с = 0 and we seek the solution that satisfies:
( M. M.) x
M. x
(31)
It is well known that the solution to this equation under the constraint ||x|| = 1 lies in nullspace
of M
Σ M, and can be obtained by taking the singular vector of M that corresponds to the 1=1
largest singular value. Since M, is idempotent, the same result can also be obtained through the cigendecomposstion:
q = arg max qMq
(32)
qEsa
which gives us the unweighted Quaternion mean Markley et al. (2007).
C PROOF OF THEOREM
Once the Lemma D is proven, we only need to apply the direct convergence results from the literature.
Consider set of points Y = [y1.... yx) where K > 2 and y, HHI Due to the compactness, we can speak of ball ((,, p) encapsulating y,. We also define the {x H,, Cx(x) < Cy(o)), the region where the loss decreases.
We first state the assumptions that permit our theoretical result. These assumptions are required by the works that establish the convergence of such Weiszfeld algorithms Luenberger et al. (1984); Aftab & Hartley (2015): Aftab et al. (2014):
HI. y1 .K should not lie on a single geodesic of the quaternion manifold.
H2. D is bounded and compact. The topological structure of SO(3) imposes bounded convexity
radius of < </2
H3. The minimizer Eq 29 is continuous.
H4. The weighting function (-) is concave and differentiable.
H5. Initial quaternion (in our network chosen randomly) does not belong to any the subspaces.
Note that H5 is not a strict requirement as there are multiple ways to circumvent (simplest being a re-initialization). Under these assumptions, the sequence produced by Eq (29) will converge to critical point unless x' y, for any t and i Afftab et al. (2014). For 1, this critical point is one of the subspaces specified in Eq (18) and thus a geometric median.
O
Note that due to the assumption H2, we cannot converge from any given point. For randomly initialized networks this indeed a problem and does not guarantee practical convergence. Yet, in experiments we have not observed any issue with the convergence of our dynamic routing. As our result is one of the few ones related to the analysis of DR, we still find this to an important first step.
16

---

Under review as a conference paper at ICLR 2020
For different choices of algorithm is shown to converge for a larger class of weighting choices as long as the aforementioned conditions met. That is why in practice we use simple sigmoid function.
2, the weights take different forms. In fact, this IRLS type of
D OUR SIAMESE ARCHITECTURE AND THE ALGORITHM
For estimation of the relative pose with supervision, we benefit from a Siamese variation of our network. In this case, latent capsule representations of two point sets X and Y jointly contribute to pose regression as shown Fig.
QU- Netheeeekkk
Elass Bcation
QB Natwoark
Nemwork
Neework
QU- Ketwork
Cx
Irtermediate Copasile
Puse Fotimattin
xc Mr
Dowmommple
YcM,
Quateraion Architeture
Cty
Cy
Elasstfication
Figure 5:
Our siamese architecture used in the estimation of relative poses. We shared net-
work to process two distinct point clouds (X, Y) to arrive at the latent representations (Cx, axx and (Cy, (Cy respectively. We then look for the highest activated capsules both point and compute the rotation from the corresponding capsules. Thanks to the rotations disentangled into capsules, this final step simplifies relative quaternion calculation.
We show additional results from the computation of local reference frames and the multi-channel capsules deduced from our network Fig. 0
Finally, the overall algorithm of our network is summarized under Alg.
Algorithm 3: Quaternion Equivariant Network
input Input points of one patch (x1, ... x E RRX3 x3 input capsules (LRFs)
{qi..... 4r E H.,4, with L No. K, No is the number of capsules per point, activations a .t,"
output: Updated frames 2 = (q)..... .. E H, updated activations â = (â), ... âm )
for Each input channel of all the primary capsules channels N do
(((()) A(Q(n))) // Input quaternion average, for Each point x; of this patch do
L x ((n()==1 ox; rotate point in canonical orientation
{x]] E RKXN<x3// Points in multiple (N9) canonical frames
for Each point x of this patch do
L t +++(xx) // Point to Transform, 1()) g N=X xxxxx
10 T = [t.e KXN5XM XM + [t} E HI| L.M
Ô ④) DynamicRouting (X, 0, see Alg.
Alg. summarizes the overall pipeline of our QE-net depicted in Fig. We use multiple layers in a hierarchical architecture. In the first layer, the input primary capsules are represented by LRFs computed with FLARE algorithm Petrelli & Di Stefano 2012). Therefore, the number of input capsule channels No the first layer is equal to I. Its activation is also defaulted to 1. The output of former layer is propagated the input of the latter, creating the hierarchy.
17

---

Under review as a conference paper at ICLR 2020
Point Chod
Inittal Life
LAF's Prier NieNewoorle
Muitl-chanmel L Rife Prior Qf-Metoorrk-2
Figure 6: Additional intermediate results on car (first row) and chair (second row) objects. This figure supplements Fig. of the main paper.
night stand table
monitor
bathtub
dresser
chair
tollet sofa
bed
desk
sofa
-80
Classification Accuracy (96
toilet
bed
65
monitor
40
night_stamp stand
bathtub
20
dresser
desk
Figure 7: Confusion matrix on ModelNet10 for classification.
E ADDITIONAL DETAILS ON EVALUATIONS
Details on the evaluation protocol. For Modelnet40 dataset used in Tab. I we used the official split with 9,843 shapes for training and 2,468 different shapes for testing. For rotation estimation in Tab. 2 we used the official Modelenet10 dataset split with 3991 for training and 908 shapes for testing. 3D point clouds (10K points) are randomly sampled from the mesh surfaces of each shape ct al. (2017a b). The objects in training and testing dataset different, but they are from the same categories so that they can be oriented mcaningfully. During training, we did not augment the dataset with random rotations. All the shapes trained with single orientation (well-aligned). We call this trained with NR. During testing, we randomly generate multiple arbitrary 50(3) rotations for each shape and evaluate the average performance for all the rotations. This is called test with AR. This protocol is used in both our algorithms and the baselines.
Confusion of classification in ModelNet. We now report the confusion matrix the task of classification the all the objects of ModelNet10. The classification and rotation estimation affects one another. As we can see from Fig. / the first five categories that exhibit less rotational symmetry has the higher classification accuracy than their rotationally symmetric counterparts.
18

---

Under review as a conference paper at ICLR 2020
Distribution of errors reported in Tab. 2 We now provide more details the emror attained by our algorithm as well as the state of the art. To this end, we report, Fig. g the histogram of errors that fall within quantized ranges of orientation errors. It is noticeable that our Siamese architecture behaves best in terms of estimating the objects rotation. For completeness, we also included the results of the variants presented in our ablation studies: Ours-2kLRRF, Ours- 1 They evaluate the model on the re-calculated LRFs in order to show the robustness towards to various point densities. We have also modified IT-Net and PointNetLK only predict rotation because the original works predict both rotations and translations. Finally, note here that we do not use data augmentation for training our networks (see AR), while both for PointNetLK and for IT-Net do use augmentation.
100
90
Percentage (%)
80
70
60
50
30
20
10
Ours
Ours-sia
Ours-2kLRF
Ours-1kLRF
IT-net
PointNetLK
Mean-LRF
PCA
< 15"
23.18%
58.16%
53.64%
41.36%
49.68%
45.40%
16.24%
29.80%
< 30
49.53%
79.12%
76.88%
66.40%
68.40%
49.28%
33.84%
32.00%
69.87%
87.48%
86.68%
78.76%
79.48%
55.84%
57.64%
33.80%
Figure 8: Cumulative error histograms of rotation estimation ModelNet10. Each (< 8P) of this extended table shows the percentage of shapes that have rotation error less than 0. The colors of the bars correspond to the rows they reside in. The higher the emror contained in the first bins (light blue) the better. Vice versa, the more the errors are clustered toward the 60% the worse the performance of the method.
19


Reviews:
[{"review_id": "rkgmo9psKS", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a rotation-equivariant model for use with point-clouds.  Capsule networks with activation and rotation pose estimate in the form of a quaternion are applied to local patches of 3D points, and combined up through progressively more coarser receptive fields.  Rotation equivariance is obtained by looking for agreement between input point frames (or capsules), by iteratively clustering using the quaternion mean.  The method is evaluated on ModelNet40, obtaining much improved results in the case where rotation orientation is arbitrary.\n\nUnfortunately, I found too many important parts of the method difficult to understand, enumerated below, and am also not very clear on the details of how they fit together.  At a high level, the approach of aggregating pose agreement with quaternion averages makes intuitive sense, the quaternion average step itself is described well, and the experimental results seem to corroborate this (results on NR/AR condition are very good, and also near identical to NR/NR condition, a major verification of the method).  However, the rest of the pieces of the method still leave me guessing too much.\n\n*  It isn't clear what computes the activations alpha.  I don't see a description for how to compute activations alpha that are the input to Alg. 1, unless these are simply the activations of the previous layer's capsules, and constant for the initial point frames?\n\n*  Likewise, the transform t() could use more description.  Sec 3.2 says it is R3->R4, indicating that the input to t() is a single point.  Does this mean that the mere presence of a point in a location R3 (relative to a mostly-canonicalized pose) counts as a vote for a rotation pose?  And if so, how does does this work?  Or does the regressor actually take multiple points as input?  According to the picture in Fig. 1, it looks like it takes all points in the local patch as input.\n\n*  Algorithm 1 outer for loop for i doesn't seem quite right.  It suggests that the body is performed either independently or in sequence for each i.  I believe what may actually be intended, is that all i are used at once in parallel, so that the v_ij votes assignment is found for *all* i in order to use in finding the cluster mean for each j.  That is, \"v_i,j = Q_i * t_i,j for all i\", rather than having the outermost loop over i.  Is this correct?\n\n*  Much space is dedicated to background explanations of quaternions, geodescic distance, etc.  Given my difficulty in understanding much of the system, it seems like some of could be shortened or put into the appendix, and more of the main text devoted to more detailed explanations of each component in the system (e.g. alpha and t), and how the capsule agreement step (QE DR) makes use of these.\n\n\nAlso, some more detailed comments/suggestions:\n\n* Alg.1:  output alpha^hat_K, seems K should be M here?\n* Table 1:  \"Right hand side denotes symmetric objects\" seems it actually shoudl be in the caption for Table 2.\n", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper."}, "extra_text": {}}, {"review_id": "rJgTqxck5S", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This work builds a capsule network, for use with point cloud data, that has units that are equivariant to SO(3) 3D rotations.  I found that the authors made the case for rotation equivariance well and I liked the analysis of the dynamic routing approach and its mapping to the Generalized Weiszfeld Iterations. \n\nAs a caveat to the review I should point out that while I am familiar with capsule networks, they are not my main area of expertise (I do work on geometry however) so this should be taken into consideration - apologies if there is literature I have missed. \n\nMy main concern is that group equivariance for Capsule networks has been studied before - I acknowledge that SO(3) can present additional challenges but would have expected a work that specifically targeted SO(3) to investigate the relative benefits of the different approaches to encoding the lie algebra and not just quaternions. How do we know that quaternions are the best fit for the Capsule framework? Could the authors show empirical results that indicate that other representations for the equivariance do not work as well? My experience from optimization is that the best representation can often be application specific and it would be very helpful to understand their relative merits for this sort of architecture.\n\nThe preparation of the local patches and the estimation of the rotations seems very important to these applications to me and would be worthy of more discussion and an empirical presentation of their different merits. I know the authors discussed the issue of the second axis but it would seem worth including an experiment to detect the sensitivity in a similar manner to the point cloud density.\n\nMy biggest concern about the paper is the presentation of the results. Please could the authors provide appropriate error bars for the tables - this is the least action necessary to at least start to estimate significance. Ideally something like histograms or violin plots would be more useful. The use of a single dataset makes it hard to identify the efficacy of the approach - especially when it is not clear how the training and test data are set-up. Are the same objects used in training and test but with different rotations? Or different objects from the same categories. What is the protocol for the AR test set?\n\nI agree that table 1 confirms the equivariance so that is great but it is less clear that the network is suitable for the pose task since the baselines seem very simple for the pose estimation experiment (and the related work covers a lot of prior work in this area) and again, without error bars it is hard to judge significance.\n\nIt is nice that the authors discuss limitations and I agree with the comment on symmetries - would it be possible to include an illustrative example to confirm these suspicions?\n\n\nOther notes:\n\nI have no issue with the proofs - they all seemed to make sense to me.\n\nI found Figure 1 (and to a lesser extent Figure 3) to be unclear and uninformative - the notation doesn't match the main part of the paper (also where does C come from in the caption?) and its not clear what the illustrations are meant to represent. Would it not be much more clear to include something like Algorithm 1 instead of Figure 1 which is much more clear and easy to replicate the precise process.\n\nI think there is a lot of material in the appendix that should really be in the main paper - I'm happy for the proofs to be in the appendix but I think it is a bit wrong to essentially violate the page restrictions by moving important related work into the appendix. The whole of section 2 is standard text-book information about quaternions - would it not be more appropriate for that to be in the appendix and the related work to be in the main paper?\n\nCould the authors indicate the use of the Frobenius norm in eqn (5)? There also seems to be a mixing of notations for inner products through-out the paper and it might be helpful to standardize this?", "extra_scores": {"experience_assessment": "I do not know much about this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper thoroughly.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I carefully checked the derivations and theory."}, "extra_text": {}}, {"review_id": "SklKk4uy9B", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes a quaternion equivariant network for 3d point clouds. It builds on the capsule networks and dynamic routing, but instead of using arbitrary 4x4 transforms, they propose to use quaternions to represent 3D rotations.\n\nInvariances and equivariances are very important in DNN for classification/detection, so extending these properties to 3d point clouds based applications is interesting and important. The proposed approach is novel and seems to be equivariant to SO(3) rotations, translations, and set permutations.\n\nHowever, there are a couple of major negatives in the paper. The first is that the details of the architecture are very unclear. A large amount of space is allocated to discussing equivariant properties but not enough is dedicated to the actual implementations of the network. In particular,  only on page 5 in section 3.2 is the QE network detailed discussed.\n\n- How does one obtain Q_i from points X_i ?\n- How are the points subsampled to create LRFs? is there a clustering first step? If so, how would that clustering be invariant to global rotation?\n- Does the method require a known centroid of the object point clouds in order to compute the quaternions of the LRFs?\n- Theorem 1 has shown that equation 6 of Alg. 1 is equivariant, but are the multiple interactions of DR equivariant? In particular, after k iterations of DR as presented in algorithm 1, with nonlinear sigmoid activations, are the output capsule representations equivariant?\n- How do you guarantee that the LRF’s are the same under any global rotation+translations of the point cloud? Or do you assume the same LRFs are known?\n\nThe main selling point for equivarince and learning the separation of pose and object class representation is in the hope of more accurate classification. However, empirical classification results are far from the state-of-the-art. \n- As a side question, there are other methods listed on the ModelNet40 website with performances in the high 90’s, are there reasons those results are not included as a part of Table 1? ", "extra_scores": {"experience_assessment": "I have read many papers in this area.", "review_assessment:_thoroughness_in_paper_reading": "I read the paper at least twice and used my best judgement in assessing the paper.", "review_assessment:_checking_correctness_of_experiments": "I assessed the sensibility of the experiments.", "review_assessment:_checking_correctness_of_derivations_and_theory": "I assessed the sensibility of the derivations and theory."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper presents a capsule network to handle 3d point clouds which is equivariant to SO(3) rotations. It also provides the theoretical analysis to connect the dynamic routing approach to the Generalized Weiszfeld Iterations. The equivariant property of the method is demonstrated on classification and orientation estimation tasks of 3D shapes.
While the technical contribution of the method is sound, the main concern raised by the reviewers was the lack of details in the presentation of methodology and results. Although the authors have made substantial efforts to update the paper, some reviewers were still not convinced and thus the scores remained the same. The paper was on the very borderline, but because of the limited capacity, I regret that I have to recommend rejection. 
Invariances and equivariances are indeed important topics in representation learning, for which the capsule network is known as one of the promising approaches but still not well investigated compared to other standard architectures. I encourage authors to resubmit the paper taking in the reviewers' comments.
Decision: Reject