SE(3)-bi-equivariant Transformers for Point Cloud Assembly
Authors: Ziming Wang
Rebecka Jörnsten
Venue: NeurIPS 2024 poster (2024)
Keywords: equivariant neural networks
SE(3)-bi-equivariant transformer
point cloud assembly
Primary area: machine_vision
TL;DR: A SE(3)-bi-equivariant and correspondence-free method for point cloud assembly.

Abstract: Given a pair of point clouds, the goal of assembly is to recover a rigid transformation that aligns one point cloud to the other. This task is challenging because the point clouds may be non-overlapped, and they may have arbitrary initial positions. To address these difficulties, we propose a method, called $SE(3)$-bi-equivariant transformer (BITR), based on the $SE(3)$-bi-equivariance prior of the task:it guarantees that when the inputs are rigidly perturbed, the output will transform accordingly.   Due to its equivariance property, BITR can not only handle non-overlapped PCs, but also  guarantee robustness against initial positions. Specifically, BITR first extracts features of the inputs using a novel $SE(3) \times SE(3)$-transformer, and then projects the learned feature to group $SE(3)$ as the output. Moreover, we theoretically show that swap and scale equivariances can be incorporated into BITR, thus it further guarantees stable performance under scaling and swapping the inputs. We experimentally show the effectiveness of BITR in practical tasks.

SE(3)-i-equivarriat Transformers for Point Cloud Assembly
Ziming Wang, Rebecka Jörnsten Department of Mathematical Sciences
University Gothenburg and Chalmers University of Technology zimingwa@chalmerr..co jornsten@chalmers. se
Abstract
Given a pair of point clouds, the goal of assembly is to recover rigid transformation that aligns point cloud to the other. This task is challenging because the point clouds may be non-overlapped, and they may have arbitrary initial positions. To address these difficulties, transformer (BITR), based on the SE(3)- SE(33)-ieequivarriacc prior of the task: it guarantees that when the inputs are rigidly perturbed, the output will transform accordingly. Due to its equivariance property. BITR can not only handle non- overlapped PCs, but also guarantee robustness against initial positions. Specifically, BITR first extracts features of the inputs using a novel SFF(3 x $ ( ttrassfraooo, and then projects the learned feature to group SE(3) as the output. Moreover, we theoretically show that swap and scale equivariances can be incorporated into BITR, thus further guarantees stable performance under scaling and swapping the inputs. We experimentally show the effectiveness of BITR in practical tasks.
propose method, called SE(()--ieequivariin
Introduction
Point cloud (PC) assembly is a fundamental machine learning task with a wide range of applications such biology [12] archeology (36), robotics [28 20 and computer vision (23]. As shown in Fig. I given a pair of 3-D PCs representing two shapes, i.e., a source and a reference PC, the goal of assembly to find rigid transformation, that the transformed source PC is aligned to the reference PC. This task is challenging because the input PCs may have random initial positions that far from the optimum, and may be non-overlapped, e.g., due occlusion or erosion of the object.
Input PCs
Assembled PCs
Input PCs
Assembled PCs
(a) Assembly of overlapped PCs
(b) Assembly of non-overlapped PCs
Figure 1: Two examples of PC assembly. Given pair of PCs, the proposed method BITR transforms the source PC (red) to align the reference PC (blue). The input PCs may be overlapped or non-overlapped (b)
Most of existing assembly methods correspondence--ased 2 the input PCs are aligned, the points corresponding to the same physical position should be close to
they the fact that when
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

each other. For example, in Fig. (aa the points at the head of the airplane in the source and reference PCs should be close in the aligned PC. Specifically, these methods first estimate the correspondence between PCs based on feature similarity distance, and then compute the transformation by matching the estimated corresponding point pairs. As a result, these methods generally have difficulty handling PCs with no correspondence, i.e,, non-overlapped PCs, such as Fig. T(b) In addition, they are often sensitive to the initial positions of PCs.
To address these difficulties, we propose method, called (i-))-ii-quiivarian transformer (BITR), based on the S (3 bi-eiiuuuiiarianee prior of the task: when the input PCs are perturbed by rigid trans- formations, the output should transform accordingly. A formal definition of (3) (-l))---iiivvee bi-equivariance can be found in Def. 3.1 Our motivation for using the (3 bi-equuvaariance prior is threefold: First, the strong training guidance provided by symmetry priors is known to lead to large perfor- mance gain and high data efficiency. For example, networks with a cunnllation-euivarianne prior, Le.. convolutional neural networks (CNNs), are known to excel at image segmentation [18]. Thus, (F(3))-ii-quivarinn
prior should lead to similar practical benefits in PC assembly tasks. Second, SE(3) (b-e))--equuivrrian methods are theoretically guaranteed to be "global", i.e., their performances are independent of the initial positions. Third, the (3 bi-eequivariance prior does not rely on correspondence, i.e., it can be used to handle PCs with no correspondence.
Specifically, the proposed BITR is an end-to- end trainable model consisting of two steps: it first extracts SO(3) x -euuiiaaurari fea- tures from the input PCs, and then obtains a rigid transformation by projecting the fea- tures into (3)... For the first step, we de- fine a S. (3) x (E (ta3)-rransorrmer acting on the 6-D merged PC by extending the SE3))- transformer Mi: For the second step, we use SVD-type projection inspired by Arun's method [2]. In addition, we theoretically show that rcale--quivariance and wwap-qquivariane can be incorporated into BITR via weight con- straining techniques, which further guarantees that the performance is not influenced by scal- ing swapping inputs. An illustration of BITR is presented in Fig.
Point cloud merge
Feature extraction
X
X (Lxx3) 30
(M
8 Ey 60
1 - Z
r x3)
(3)
t
Cy
y
(M x
TrassTere
SSC3ysPrgociin
An overview of BITR. The input 3-D PCs X and Y are first merged into a 6-D PC Z by concatenating the extracted key points X and Y. Then, 7 is fed into a SE(3) x to obtain equivariant features f. fx and fy. These features are finally projected to SE(3) as the output.
Figure 2:
In summary, the contribution of this work is as follows:
We present a bb-uuu--b-e PC assembly method, called BITR. 3 BITR can assemble PCs without correspondence, and guarantees stable performance with arbitrary initial positions. In addition, the SE(3) x $E(3)) (tr())))tanffommrr used in BITR is the first SE(3) x SE(3) -qqiivaraa stecrable network to the best of our knowledge.
Theoretically, we show that scale and swap equivariance can be incorporated in to BITR by weight-consttaining thus it guarantees stable performance under scaling and swapping the inputs.
We show experimentally that BITR can effectively assemble PCs in practical tasks.
2 Related works
A special case of PC assembly is PC registration, where the correspondence between input PCs is assumed to exist. A seminal work in this task was conducted by [2] which provided closed-form solution to the problem with known correspondence. To handle PCs with unknown correspondence, most of the subsequent works extend [2] by first estimating the correspondence by comparing distances [4) or features 25 23 41 of the PCs, and then aligning the PCs by aligning the estimated features [42] 10
corresponding points. Notably, to obtain S (3) bib.-iuvvariaanee.......... 50(3)-invarin)
43 have been investigated for correspondence estimation. However, since these methods require a sufficient number of correspondences, they have difficulty handling PCs where the correspondence does not exist. In addition, they often have difficulty handling PCss with large initial errors 445
CCode is available at: https: /et:::/ggttuee wttptttwww...omm//emm2m6666

---

The proposed BITR is related to the existing registration methods because it can be seen as generalization Arun's method [2]. However, contrast these methods, BITR is correspondence- free, i.e., it is capable of handling PCs with no correspondence. In addition, it theoretically guarantees stable performance under arbitrary initial position.
Recently, some works have been devoted to new PC assembly task, called fragment reassembly, whose goal is reconstruct a complete shape from two fragments. Unlike registration task, this lask generally does not assume the existence of correspondence. [7] first studied this task, and they addressed it as a pose estimation problem, where the pose of each fragment relative canonical pose is estimated via a regression model. (38] further improved this method by considering the -qiiavaauvvreianc of each fragment. addition, [1] proposed a simulated dataset for this task. In contrast to these methods, the proposed BITR does not rely on the canonical pose, i.e., it directly estimates the relative pose. As result, BITR conceptually simpler and it can handle the shape whose canonical pose is unknown.
Another related research direction is equivariant networks. Due their ability incorporate 3D
rotation and translation symmetry priors, these networks have been extensively used in modelling 3D
data 29
[ and recently they have been used for robotic manipulation task (27 28 40. In
particular, proposed a tensor field network (TFN) for PC processing, and (B)
||]
further improved TFN by introducing the attention mechanism. On the other hand, the theory of equivariant networks was developed in [66 []. BITR follows this line of research because the
(E)) x SE(3)-transformee used in BITR a direct generalization of SE (3))-raanffrrmmrr [T]. and it is the first SF(3) x -eeuiiaraaar steerable network to the best of our knowledge.
3 Preliminaries
This section briefly reviews Arun's method and the concept of equivariance, which will be used in BITR.
3.1 Group representation and equivariance
Given group G, its representation a group homomorphism p - GL(V), where V is linear space. When G is the 3D rotation group SO(3), it is convenient to consider its irreps (irreducible orthogonal representation) Pp : (O(() + GL(V,), where p E N is the degree of the irreps, and VV) = 2p + 1. For r E G. Pr() E 1) (2p++) is known as the Wigner-D matrix. For example, (r) = 1 for all r 5O(3); Pi (r) e 3 is the rotation matrix of More details can be found in (5] and the references therein.
In this work, we focus the group G of two independent rotations, i.e., G SO(3) SO(3), where x represents the direct product. Similar to S0(3), we also consider the irreps of G. A useful fact is that all irreps of G can be written the combinations of the irreps of SO(3): the degree-(p, q) irreps of G is Pp.q Pp @ Pq : 50(3) x SO(3) + GL(V, @ Vg), where p. q E N. Pp and P. are irreps of SO(3), and @ tensor product (Kronecker product for matrix). For example, Po.0(11 X (2) = R; (1.((1 72) E 33x3 is the rotation matrix of ri: (r, X r2) = (r()) ) @ pi (1() R 999 is the Kronecker product of the rotation matrices of rı and r2.
Given two representations p G GL(V) and T G GL(W), map O V + W satisfying & (p(g)zz = (<) for all g E G and æ E called Geequivariant. When &b is parametrized by neural network, we call & an equivariant neural network, and we call the feature extracted by & cquivariant feature. Specifically, a degree-p equivariant feature transforms according to Pp under the action of SO(3), and degree-(p, q) equívariant feature transforms according to Pp Pg under the action of 50(3) x S0(3). For simpler notations, we omit the representation homomorphism p. ie., write instead of p(r), when p is clear from the text.
3.2 Arun's method
Consider PC assembly problem with known one-to-one correspondence: Let Y J. C R and X { corresponding point pairs. What is the optimal rigid transformation that aligns X to Y??
1N C R be pair of PCs consisting of N points, and let [(I., yo) ] 1=1 be their
3

---

2 provided a closed-for solution to this problem. It claims that the optimal solution (r)
(3) in terms of mean square errors is
SVD(F) and t= d=m(Y) -rm(X)
(1)
where
F= Σ D.ee"
(2)
is the correlation matrix, m²-) represents the mean value, I; = m(X) and y; yi m( represent the centralized points, and (VD(-) represents the singular value decomposition projection. The definition of SVD projection can be found in Def. C.S)
Arun's solution enjoys rich equivariance properties. Formally, we have the following proposition: Definition 3.1. Consider a map &b : x S - SE(3) where S is the set of 3D PCs. Given X, E S let g ((X,))
is SE(3) (E5(3)-i-equiivee equivaivvaian if ((g1x.992Y) 92991 19 E SE(3).
+ is swap-equivariant if ((Y,X)= g-
+ is enuleeequivariant if ((cX, (r, (r, ct), Vc R++
Proposition 3.2. Under mild assumption C.2) Arun's algorithm
is SEi()-bi-equiivriinn,,
swap-equivariant and euui-eqquva
In other words, Arun's 's method guarantees to perform consistently 1) with arbitrary rigid perturbations on X and Y, i.e., it is global. 2) X and are swapped (aligning Y to the fixed X or aligning X to the fixed Y), and 3) in arbitrary scale. Details of Prop. 3.2 can be found at Appx. C.1
We regard Arun's method extracts a (egree-(1, 1) 50(3) x 5O(3)) translation-invarian feature, i.e., the correlation matrix (2) and then obtains output g SF) using -yl)-based projection. This observation immediately leads to more general $
prototype of (E(3)-bi (ite)))--qquivriann PC assembly methods: it first
methods where the handcrafted feature is replaced by the more expressive learned equivariant features, thus, the correspondence is no longer necessary. We will develop this idea in the proposed BITR in the next section, and further show that the scale and swap equivariance of Arun's method can also be inherited.
4 SF bi-quiaaieuu transformer
This section presents the details of the proposed BITR. BITR follows the same principle as Arun's method 2 it first extracts (O(3) S0(3)-equivarianl features as generalization of the correlation matrix 2 and then projects the features to SE(3) similarly to (I.. Specifically, we first propose SE(3) (E(3)-transfomme for feature extraction in Sec. 4.2 Since this transformer is defined on 6-D space, i.e., it does not directly handle the given 3-D PCs, it relies on pre-processing step described in Sec. [.3] where the input 3-D PCs are merged into 6-1) PC. Finally, the Arun-type (F(3))rrojection is presented in Sec. 4.4
An overview of BITR is presented in Fig.
4.1 Problem formulation
Yet { R and X W C R be the PCs sampled from the reference and source shape respectively. The goal of assembly is to find a rigid transformation g E (3),)), so that the transformed PC g.X {rx;+t + + aligned to Y. Note that we do not assume that X and Y are overlapped, i.e., we do not assume the existence of corresponding point pairs.
4.2 SE(3) x SE(3) -ransforreer
To learn SO(3)
translation invariint features generalizing! F [ this subsection brief introduction (E(33)-tansformm transformer as generalization of (transsraan II. We present
proposes a (E(3) x (3) (3((3)-rransform) [11] in Appx. for completeness.
According to the theories developed in [S, define a S (3)
transformer, we first need to define the feature map of transformer layer tensor field, and specify the action

---

of SE(3) x SE(3) on the field. Since (3(3) x SE(3) can be decomposed as (T(3) x T(3)) x (SO(3) x (3(3)) where T(3) is the 3-D translation group, the tensor field should be defined in the 6-D Fuclidean space R = T(3) x T(3), and the features attached to each location should be SO(3) x equivvarrn and T()) x ( -3)-invariinn. Formally, we define the tensor field as
[(z) Σ Luō(z-
where Z = {au S IR is 6-D PC, å is the Dirac delta function, fu. = f 2 is the feature attached to Zus where f is the degree-(p, 9) equivariant component. We then specify the action of SE(3) x (3) the base space R as
(g) x 92)(2) = (g1=') @ (92=")
E IR
(4)
where = 2 ( ())2 22, E TR are the first and last three components of z, 4 represents direct sum (concatenate), and Di = 4) E SF(33 for = 1.2. Thus, the action of SF(33) SFF(3 on the degree-(p, 9) component of is
((g) x 92)l = (ppa(n x r2)) fo ((n) x 92)
(2) (=))
With the above preparations, we now define SE(3) SE(3) transffrmer layer in message passing formulation similar to SE transforrr.
Foul 2u) = Wo +
Σ
(5)
self-intermction
VERNN(U)) Mu) atlention message
Here, notations o (01, 02) and i (i1, i2) for simplicity. For example, f represents the degree-o feature for.e,, (zu) E 1gex(2ox + +1)s is the collection of all degree-o features at Zu.
where is the number channels of the degree-o features, and W E IR a learnable parameter
for seff-ineeractin. KNN(-) represents the k nearest neighborhood, and attention Ouv is computed according to
exp (Q K.v)
Quaz
r' C''ENNN()) (u) exp (Q K,,
(6)
where Q, K and V are known as the query, key and value respectively. They are defined as
Qu Wgrm(zu), K.n + We (zvu) f( (=v), Vol = W (#vw) fid (zv)
(7)
where 2pu = Ze zu. Wi is learnable parameter for Q and the convolutional kernel Wo in
Q
V and K both take the form of
01
02
vec(M)(()))
Σ
Σ
Pr.Jo [[=='11111 11:1 Di.
YJ2
(8)
2
J=
radial component
angular component
0, where the learnable radial component 4 PJ IR IR R is parametrized by neural network, the non-learnable angular component is determined by the 2-nd order Ciebsch-Gordan constant Q and the spherical harmonics Y, : 3R3 + 222++1, and vec(-) is the vectorize function reshaping a matrix to vector. Formulation (8) is derived in Appx. B
transsoeran layer and a SE(3) x S (r(3)-tranfformer layer. In the special case where only SFF(3) (eu)3v-aqua is considered, iee. all features are of degree (p, 0) (or (0,4)), a SF((3) x trransfroo layer becomes S FF3)- transformer layer.
Note that the kernel g is the main difference between
Finally, we adopt the equivariant Relu (Elu) layer similar our network. Given an input degree-i feature fer with e channels, an Elu layer is defined as
the point-wise non-linear layer in
F F
(p,, 2 0
Elt(F+)
(F
F. [TTT TPT
otherwise.
where F3. WiFe and F, = W Wn, W. E Rexe are learnable weights and || is the channel-wise vector norm. Note that when i = (1,0) or = (0, 1), our definition (9) becomes the same as [9] By interleaving transformer layers and Elu layers, we can finally build complete SE(3) x eqqiivarin transformer model.
5

---

4.3 Point cloud merge
To utilize the transformer model defined in Sec. 4.2, we need construct a 6-D PC its input. To this end, we first extract key points from the raw 3-D PCs, and then concatenate them to a 6-D PC to merge their information. Thus, the resulting 6-D PC is not only small in size but also contains the key information of the raw PCs pairs.
(Du 10 from X and Y respectively, and then obtain Z = {Tu, + Tu Ju=1- Note that we do not require X ()) to be a subset of X (Y). Specifically, represent the coordinates of the key points convex combination of the raw PCs:
Formally, we extract L ordered key points X = { and Y
X SoftMar(F2)X,, = ==ffMMax(EE))
(10)
where X E RMx3 and Y E IR represent the coordinates of X and Y respectively, and SoftMax()) represents the row-wise softmax. F E RLXM and Fy TR.2N are the weights of each point in X and Y respectively, and they are degree-O, i.., rolation-invariant, features computed by shared )3) -transsorooorem gg:
=
FP = (Y)
(11)
Furthermore, inspired by 39 we fuse the features of X and Y in &g before the last layer, so that their information is merged more effectively, ie.. the selection of X or Y depends on both X and Y. Specifically, the fused features are
{ Sounxx x (xw) fia. (x @ Pooly (Ji. (Ww)) + fixx (Ix) (#n) Y (Ne) Jm. (yv) @ Pooly (Jm.x (zw) @ ftyy (No).
(12)
where we only consider degree-O and degree-1 features. f.x and f.y represent the features of X and Y. and Pool is the average pooling over the PC.
Note that X and Y obtained in Eqn. 10 are permutation invariant. For example, according to Eqn. 10 the i-th key point of X is (after softmax normalization). When X is permutated by then the i-th key point can be written as 3/ = E3 Fig. where j' (j). It is easy to see that J = 3. because both j and j' iterate through {1, M) in the summation.
FijYz. where Fij is the i-th channel of F at zy
4.4 SE(3)
We now obtain the final output by projecting the feature extracted by the SE(3) x ( -transformer to S3).). Formally, let f be the output tensor field of the SE(3) x the final output g (r, t) E SE(3) using an Arun-type projection follows:
We compute
SVI(FF)
(13)
t (m(P) ((m)-(mm(X) tx),
where f = unvec(F) E p3x3 tx E R33 and ty E IR3 are equivariant features computed as = Pool, (ft.t), tx = Pool,, ty = Pool, (f
We note that projection
13 extends Arun's projection 11 in two aspects. First, although i in [1) and
(2) are both degree-(1, features, f more flexible than because / a learned feature while handcrafted, and f is correspondence-free while F is correspondence-based Second, projection [13 explicitly considers non-zero offsets fx and ty, which allow solutions where the centers of PCs do not match.
In summary, BITR computes the output g for PCs X and Y according
4p (X,
(14)
where Øs 5 x F SE)3) x (E(33)transfommn) (with the PC merge step), 4p F SE(3)
represents projection (I3). and I' is the set of tensor field. We finish this section with a proposition that BITR is indeed SEB bi--eqivvariin
Proposition 4.1. Under mild assumption
C.2
BITR 14 is (113))bb--quivriin

---

Swap-equivariance and scale-qquivariance
This section seeks to incorporate swap and scale equivariances into the proposed BITR model. These two equivariances are discussed in Sec. 1.1 and Sec. 5.2 respectively.
5.1 Incorporating swap-equivariance
This subsection seeks to incorporate swap-equivariance to BITR, i.e., to ensure that swapping X and Y has the correct influence on the output. To this end, we need to treat the group of swap as Z/2Z = {1,s} where x² L.e, 8 represents the swap of and Y, and properly define the action of z/22 on the learned features.
Formally, we define the action of 2/22 on field f [ follows. We first define the action of on the base space I² as swapping the coordinates of X and Y: s(2) + 2¹, where 2 o 2 and =1, 2 E 3 are the coordinates of X and Y respectively. Then we define the action of on feature as (s()) P- (z) (faP2 (s(=)))² " where we regard a degree- q) feature f& as matrix of shape (22 by abuse of notation, and (-)= represents matrix transpose.
Intuitively, according to the above definition, (egrre-(1, 1). (1, 0) and (0. features will become (the transpose of) degree-(1, 1). (0, 1) and (1, 0) features respectively under the action of s, i.e., f will be transposed, xx and by will be swapped. This is exactly the transformation needed to ensure swap-equivariant outputs. We formally state this observation in the following proposition.
Proposition 5.1. For tensor field and projection Dp
13 4p (s(J)) = (
Now the remaining problem is how to make a SE(3) x (333)-transforme Z/2z-cquivariant. A natural solution is to force all layers in the SE(3) x transsfrrmm to be 2/2Z 2/2Z-equvvaiiant The following proposition provides a concrete way to achieve this.
Proposition 5.2. Let represent the swap of index, e.g. iso = (0), 02), then ô (02, 1) For transformer layer (5). if the inffrinteraation weight satisfies W W the weight of query 7
satisfies W = W and the radial function satisfies JJ,,,,, (1)= 1=21)) = IJ,n,, (1|=21), ||= for all i, o, J1, J2. 2² and 2², then the transformer layer is 2//22-qquivarian.
1.0
2) For Elu layer 9 i W and W w for all i, then the Elu layer is 2
More details, including the complete matching property (Prop. C.II. can be found in Appx. C
5.2 Incorporating scale-equivariacce
This subsection seeks incorporate scale equivariance to BITR, i.e., ensure that when X and Y are multiplied by scale constant E R_,, the output result transforms correctly. To this end, we need to consider the scale group (R24, x), i.e., the multiplicative group of R,, and properly define the (R4, x "equivarrance of the learned feature. For simplicity, we abbreviate group (I,+, x) as R..
We now consider the action of R+ on field (3) We call a degree-p -4-qqvivrriant field (p N) transforms as (c(f)) (2) = ff 2) under the action of R4,, where 2 E IR6 and c E R4. We immediately observe that degree-1 R4 equivvaiaan features lead to scale-equivariant output. Intuitively, if F, lx and ly are 1 IR. -equivariia features, then they will become cř, clx and dty under the action of c, and the projection step will cancel the scale of 7 while keeping the scale of Ix and ly, which is exactly the desirable results. Formally, we have the following proposition.
Proposition 5.3. Let 4p be projection (13) f be a degree-1 equivvrraa tensor field, and (r, t) (f) We have 4p (c(f)) (r,ct) Vcc E Rt
The remaining problem is how to ensure that a S3()) x SF (ranss-rrnsfoommer is -Ruievviaariant and its output is of 1 that scaling the input can lead the proper scaling of output. Here provide a solution based on the following proposition.
Definition 5.4. (p : IR X + R is degree-p function if c((z,, cy) = c':p(,, y) for all c E R,- Proposition 5.5. 1) Denote yp and yov the radial functions used in and V respectively. Let уи be a degree-O funcrion,, fin be degree-O R- -_--qqvivariant input field. For transformer layer ( if field.
7

---

is function and the self- ieffrinteraation weight W = 0, then the output field fout is degree-1 R. equivariant If ip degree-O function, then the output field fou. is degree-O R eqquivariant
2) For Elu layer
( if the imput field is degree-p RR, requivariat,, then the output field is also degree-p R2 equivariant.
More discussions can be found in Appx. C.4
Experiments and analysis
This section experimentally evaluates the proposed BITR. After describing the experiment settings in Sec. 6.1] first present simple example in Sec. S highlight the equivariance of BITR. Then we eva nate BITR on assembling the shapes in ShapeNer (6]. BB dataset [31]. 7Scenes [32 and
ASL 22 from Sec. 6.3.I to Sec. 6.4 We finally apply BITR visual manipulation tasks in Sec. 6.6
6.1 Experiment settings
We extract L = 32 key points for each PC, The SE(3) (ta3)-rransforrmrr and the SE(33 x SE(3)- transformer both contain 2 layers with 4 channels. We consider k = 24 nearest neighborhoods for message passing. We only consider low degree equivariant features, i.e., p.q E {0. 13 for efficiency. We train BITR using Adam optimizer [15] with learning rate 1e We use the loss function = || gge I1|2 + li - t| where (r. t) are the output transformation, (Tge. tgt) are the corresponding ground truth. We evaluate all methods by isotropic rotation and translation errors: Ar = (180/m n jaccos (1/2 2 (Ir(rr])) - 1))), and A = l1lgt - 4]] where tr(-) is the trace of a matrix. We do not random rotation and translation augmentations
23 More details are in Appx. D.1
6.2 A proof-of-concept example
To demonstrate the equivariance property of BITR, we train BITR on the bunny shape 333 We prepare the dataset similar to 41: In each training iteration, we first construct the raw PC S by uniformly sampling 2048 points from the bunny shape and adding 200 random outliers from [-1, then obtain PCs {Xp, Yp) by dividing S into parts of ratio (3%% using random plane P. We train BITR to reconstruct S using randomly rotated and translated {Xp, Yp}. To construct the test set, generate new sample (X Yy+), and additionally construct 3 test samples by 1) swapping, 2) scaling (factor 2) and 3) randomly rigidly perturbing {x Yr.--
The assembly results of BITR these four test samples are shown in Fig. We observe that BITR performs equally well in all cases. Specifically, the differences between the rotation errors in these four cases are small (less than le 3). The results suggest that BITR is indeed robust against these three perturbations, which verifies its swap-equivariance, scale-equivariance and (E equivariance. More experiments can be found in the appendix: numerical verification of Def. is presented Appx. [.2] an ablation study of swap and scale equivariances are presented in Appx. D.3 and the verification of the complete-matching property C.II is presented in Appx. D.4
(a) Original (Ar = 11.275)
(b) Swapped Ar = 11.275
(c) Rigidly perturbed (Ar 11.276)
(d) Scaled (Ar = 11.276)
Figure 3: The results of BITR on test example (a) and the swapped b scaled (d) and rigidly perturbed (c) inputs. The red, yellow and blue colors represent the source, transformed source and reference PCs respectively.

---

6.3 Results on ShapeNet
6.3.1 Single shape assembly
In this experiment, we evaluate BITR on assembling PCs sampled from a single shape. When the inputs PCs are overlapped, this setting is generally known as PC registration. We construct a dataset similar to [4 for a shape in the airplane class of ShapeNet (6], we obtain each of the input PCs by uniformly sampling 1024 points from the shape, and keep ratio s of the raw PC by cropping it using a random plane. We vary from 0.7 to 0.3, Note the PCs may be non-overlapped when < 0.5.
We compare BITR against the state-of-the-ar registration methods GEO 2 and ROI 43, and the state-of-the-an fragment reassembly methods NSM 7 and LEV 38 For NSM and LEV, additionally provide the canonical pose for each shape. Note that LEV and ROI are
methods. For 2 0.5, we report the results of BITR fine-tuned by ICP 45 (BITR+ICP). Note that BITR+ICP is S (3) biieuuiiiuuiaa and seale-equivariant but not swap-equivariant.
We present the results in Fig. We observe that the performance of all methods decrease as s decreases. Mcanwhile, BITR outperforms all baseline methods when 8 is small (s S. 0.5). On the other hand, when is large (s 0.5), BITR performs worse than GEO, but it still outperforms other baselines. Nevertheless, since the results of BITR are sufficiently close optimum (Ar S 20), the ICP refinement can lead to improved results that are close GEO. More details can be found in Appx. D.5
CCO
GCO
0.
100
0.5
0.4
PC
Figure 4: Assembly results on the airplane dataset. denotes methods which require the true canonical poses of the input PCs.
6.3.2 Inter-class assembly
To evaluate BITR on non-overlapped PCs, we extend the exper- iment in Sec. 6.3. to inter-class assembly. We train BITR to place a car shape on the right of motorbike shape, so that their directions are the same and their distance is 1. We consider 1.0 and 0.7. Note that this task is beyond the scope of registration methods, since the input PCs are nonooverlapped. result of BITR is shown in Fig. More details can be found in Appx. D.6
Figure 5: A result of BITR on as- sembling a motorbike and
6.4 Results on fragment reassembly
This subsection evaluates BITR on fragment reassembly task. We compare BITR against NSM (7], LEV [38) and DGL 44 on the 2-fragment WineBottle class of the BB dataset 31. The data preprocessing step is described in Appx. D.7]
We test the trained BITR 3 times using differ- random samples, and report the mean and standard deviation of (Ar, At) in Tab. 1 We ob- serve that BITR outperforms all baseline meth- ods: BITR achieves the lowest rotation errors, and its translation error is comparable to DGL, which is lower than other baselines by a large margin. We provide some qualitative compar- isons in Appx. D.7
Table 1: Reassembly results on 2-fragment WineBottle We report the mean and std of the error of BITR.
Ar
At
DGL
101.3
0.09
NSM
101
0.18
LEV
98.3
0.17
BITR (Ours)
8.4 (0.9)
0.07 (0.008)
6.5 Results on real data
This subsection evaluates BITR indoor dataset 7Scenes [22 and the outdoor scenes in ASL dataset [22]. We present the results on 7Scenes in this section, and leave the results on ASL and some qualitative results to Appx. D.8
For the 7Scenes dataset, we arbitrarily rotate and translate all frames, and train BITR to align all adjacent frames. We use the data from the first 6 scenes the training set, and the data from 7-th

---

scene as the test set. To train BITR, we use random clipping augmentation similar to Sec. 6.3.1] we keep ratio of each PCs by clipping them using random plane, where is uniformly distributed in (0.5, 1.0]. We compare BITR against GEO (23 ROI 43 ICP [45] and OMN 33 where OMN is
recently proposed correspondence- free registration method.
The results on 7Scenes are reported in Tab. 2 We ob- serve that BITR can produce results that are close the optimum (Ar x 25) from random initialization (Ar O 180), and extra refinements like ICP and OT can further improve the results (Ar = 10). This observation consistent with that Sec. 6.3.] In particular, BITR with the OT refinement is compara- ble with GEO and ROI, which use highly complicated features specifically designed for registration tasks and an OT-like refinement process. On the other hand, ICP and OMN fails in this task due to their sensitive- ness to initial positions.
Table 2: Results on 7Scenes. We report mean and std of Ar and At.
Ar
A
ICP
73.2
2.4 (0.2)
OMN
129.02 (2.15)
0.98 (0.06)
GEO
(0.02)
0.2 (0.08)
ROI
(0.0)
0.2 (0.0)
BITR (Ours)
26.7 (0.0)
0.8 (0.0)
BITR+ICP (Ours)
11.1 (0.0)
0.3 (0.0)
BITR+OT (Ours)
9.5
0.3 (0.0)
6.6 Results on visual manipulation
This subsection investigates the potential of BITR in manipulation tasks. Following [26]. we consider two tasks: mug-hanging and bowl-placing. For both tasks, X represents an object grasped by robotic arm, i.e., a cup or bowl, Y represents the fixed environment with target, i.e., stand or plate, and we train BITR to align X to Y, so that the cup can be hung to the stand, or the bowl can be placed on the plate.
Fig. presents the results of BITR bowl-placing. We observe that although BITR is not originally designed for manipulation tasks, it can place the bowl in reasonable position relative to the plate. However, also notice that BITR may produce unrealistic results, e-g., the PCs may collide. Thus, post-processing steps (30) or extra regularizers [13] may be necessary in practical applications. More
results and discussions can be found in Appx. D.9
(a) A success case of bowl-placing
(b) A failure case of bowl-placing
Figure 6: The results of BITR on bowl-placing. We present the input PCs (left panel) and the assembled results (right panel). BITR can generally place the bowl (red) on the plate (green) (a) but it sometimes produces unrealistic results where collision exists (b)
7 Conclusion
This work proposed a PC assembly method, called BITR. The most distinguished feature of BITR is that it is correspondence-fres free, SE(33-bbi-quiivarinn,, scale-equivariant and swap-equivariant. We experimentally demonstrated the effectiveness of BITR.
BITR in its current form has two main limitations. First, BITR is computationally inefficient because each degree of feature is computed independently without parallel. This issue was also observed in SE(3) -equivrriit networks, and was recently addressed by [21] A promising future research direction develope similar acceleration techniques for BITR. Second, since BITR is deterministic, i.e., it only predicts one result for given input, it cannot handle symmetric PCs. Although this feature does not cause any difficulty in this work (there is no strictly symmetric PCs in this work due to noise, random sampling, etc), it may be problematic in future applications such as molecule modelling where symmetric PCs exist, e.g., benzene rings. To address this issue, we plan to generalize BITR to generative model in the future. More discussions can be found in Appx. E
10

---

Acknowledgments
This work is funded by the Swedish Research Council through grant agreement no. 2019-03686 and Chalmers AI Research Initiative (CHAIR). The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725, The authors would like thank Jan Gerken at Chalmers and Nan Xue Ant Group for useful discussions, and thank the anonymous reviewers for their insightful comments and suggestions.
References
[1] Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 2018.
[2] K Somani Arun, Thomas Huang, and Steven Blostein. Least-squares fitting of 3-d point sets. IEEE Transactions on pattern analysis and machine intelligence, (5):698-700, 1987.
[3] Jimmy Lei Ba. Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization, arXiv preprint arXiv:1607.06450, 2016.
[4] Paul J Besl and Neil McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures, volume 1611, pages 586-606. Spie, 1992.
[5] Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build equevvarari steerable cnns. International Conference on Learning Representations, 2022.
[6] Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang. Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Ree 3D Model Repository. Technical Report a11.v:1512.00011 [cs.GR], Stanford University Princeton University Toyota Technological Institute at Chicago, 2015.
[7] Yun-Chun Chen, Haoda Li, Dylan Turpin, Alec Jacobson, and Animesh Garg. Neural shape mating: Self-supervised object assembly with adversarial shape priors. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 12724-12733, 2022.
[8] Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.
[9] Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-qquivariant networks. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 12200-12209, 2021.
[10] Haowen Deng. Tolga Birdal, and Slobodan Ilie. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV), pages 602-618, 2018
[11] Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling, Se(3)-transformers:: 3d roto- translation equivariant attention networks. Advances in neural information processing systems, 33:1970-1981, 2020.
[12] Pablo Gainza, Sarah Wehrle. Alexandra Van Hall-Beauvais, Anthony Marchand, Andreas Scheck, Zander Harteveld, Stephen Buckley, Dongchun Ni, Shuguang Tan, Freyr Sverrisson, al. De novo design of protein interactions with learned surface fingerprints. Nature, pages 1-9, 2023.
[13] Octavian-Fugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent se equivvrrrat models for end-to-end rigid protein docking, arXiv preprint arXiv:2111.07786 2021.
[14] Zan Gojeic, Caifa Zhou, Jan D Wegner, Leonidas J Guibas, and Tolga Birdal. Learning multiview point cloud registration. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 1759-1769, 2020.

---

[15] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980, 2014.
[16] Leon Lang and Maurice Weiler. A wigner-eckart theorem for group equivariant convolution kernels. In International Conference on Learning Representations, 2021.
[17] Cheng- Wei Lin, Tung-I Chen, Hsin- Hsin-Ying Lee, Wen-Chin Chen, and Winston Hsu. Coarse-to- fine point cloud registration with se (3)equivariant representations. In 2023 IEEE international conference on robotics and automation (ICRA), pages 2833-2840. IEEE, 2023.
[18] Jonathan Long, Evan Shelhamer, and Trevor Darrell, Fully convolutional networks for se- mantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages
2015.
[19] Benjamin Midtvedt, Jesús Pineda, Fredrik Skärberg, Erik Olsén, Harshith Bachimanchi, Emelie Wesén, Elin Esbjörner, Erik Selander, Fredrik Höök, Daniel Midtvedt, et al. Single-shot self-supervised object detection in microscopy. Nature communications, 13(1):7492, 2022.
[20] Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task- specific cross-pose estimation for robot manipulation. In Conference on Robot Learming, pages 1783- 1792. PMLR, 2023.
[21] Saro Passaro and C Lawrence Zitnick. Reducing so (3) convolutions to so (2) for efficient equivariant gnns. In International Conference on Machine Learning, pages 27420-27438. PMLR, 2023.
[22] François Pomerlcau, Ming Liu, Francis Colas, and Roland Siegwart. Challenging data sets for point cloud registration algorithms. The International Journal of Robotics Research, 31(14):1705-11 1711, 2012.
[23] Zheng Qin, Hao Yu, Changjian Wang. Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11143-11152, 2022.
[24] rustyls. Is. pytorchscatter, 2023.
[25] Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpfhh for 3d registration. In 2009 IEEE international conference on robotics and automation, pages 3212-3217. IEEE, 2009.
[26] Hyunwoo Ryu, Hong Lee, Jeong- Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: Se(3)-equivariant mnasey-based models for end-to-end visual robotic manipulation learning. In The Eleventh International Conference on Learning Representations, 2023.
[27] Hyunwoo Ryu, Jiwoo Kim, Hyunseok An, Junwoo Chang, Joohwan Seo, Taehan Kim, Yubin Kim, Chaewon Hwang, Jongeun Choi, and Roberto Horowitz. Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 18007-18018, 2024.
[28] Hyunwoo Ryu, Hong-in Lee, Jeong-Hood Lee, and Jongeun Choi. Equivariant descriptor fields: Se (3)-equivariant energy-based based models for end-to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.0888 083211 2022.
[29] Victor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning, pages 9323 -3339.33.222 PMLR, 2021.
[30] Johannes Schauer and Andreas Nüchter. Collision detection between point clouds using an efficient kd tree implementation. Advanced Engineering Informatics, 29(3):440-45,, 2015.
[31] Silvia Sellán, Yun-Chun Chen, Ziyi Wu, Animesh Garg, and Alec Jacobson. Breaking bad: dataset for geometric fracture and reassembly. Advances in Neural Information Processing Systems, 35:38885-38899 2022.
12

---

[32] Jamie Shotton. Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference computer vision and puttern recognition, pages 2930-29311 2013.
[33] Standford.
The stanford 3d scanning repository.
[Online]. Available:
[34] Nathaniel Thomas, Tess Smidt, Steven Keames, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-cquivrriant neural networks for 3d point clouds. arXiv preprint arXiv:/802.082 19, 2018.
[35] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems, 30, 2017.
[36] Ariana M. Villegas-Suaree Cristian Lopez, and Ivan Sipiran. Matchmakernet: Enabling fragment matching for cultural heritage analysis. In Proceedings the the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops, pages 1632-1641, October 2023.
[37] Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data, Advances in Neural Information Processing Systems, 31, 2018.
[38] Ruihai Wu, Chenrui Tie, Yushi Du, Yan Zhao, and Hao Dong. Leveraging se (3) equivariance for learning 3d geometric shape assembly. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 14311-14320 2023.
[39] Hao Xu, Shuaicheng Liu, Guangfu Wang, Guanghui Liu, and Bing Zeng. Omnet: Learning overlapping mask for partial-to-pp partial point cloud registration. Proceedings of the IEEF/CVF International Conference on Computer Vision, pages 3132-3141,200 2021.
[40] Jingyun Yang, Congyue Deng, Jimmy Wu, Rika Antonova, Leonidas Guibas, and Jeannette Bohg. Equivact: Sim equiivarat visuomotor policies beyond rigid object manipulation. arXiv preprint 2tt.::2311..000000 2023.
[41] Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF conference on computer vision and paltern recognition, pages 1 11824-1 1833, 2020.
[42] Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang, Benjamin Busam, and Slobodan Ilie. Riga: Rotation-invarian and globally-aware descriptors for point cloud registration. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2024.
[43] Hao Yu, Zheng Qin, Hou, Mahdi Salch, Dongsheng Li, Benjamin Busam, and Slobodan Ilic. Rotation-invariant transformer for point cloud matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5384-5393, 2023.
[44] Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems, 33:6315-63222 2020.
[45] Zhengyou Zhang. Iterative point matching for registration of free-form curves and surfaces. International Journal of Computer Vision, 13(2):119-152, 1994.
[46] Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: modern library for 3D data processing. arXiv: 099447, 2018.
13

---

A -Euiiarquvaariat Transformers
well-known (E(3) -equivrrra network is SE(3) transformer [T], which adapts the powerful transformer structure [35] to (O settings. In this model, the feature map f of each layer is defined as a tensor field supported on a 3-D PC:
[(x) Σ Ju8( J.);
(15)
where is the Dirac function, X x= (I )u R a point set, and fu is the feature attached Za- Here, feature fa takes the form of fa @pfl. where the component f E Vp is the degree-p feature, ie., it transforms according to Py under the action of SO(3). For example, when fu represents the norm vector of a point cloud. then fu fu E R We also write the collection of all degree-p features zy as FP (2+) [exx(2p+1), where e the number of channels.
For each transformer layer, the degree-k output feature at point z; is computed by performing message passing:
(x = F' (2x))
Σ Σ
(16)
(EN(u)V(u) (u) attention message
self-interaction
where (u) represents the neighborhood u, W R is the learnable weight for self-interaction, represents the number of channels, and
exp (oe)
Obee En exp (Q,, K Kuv*)
(17)
is the attention from v to u. Here, key K, value V and query Q are
Qu W (z)u, Kuv
θ W (I. Fn) (#r), Vlk. W (x. - -x)) fth (#r)
(18)
where W e (2 is learnable weight, and the kernel W (x) ((2++))((2++1) is defined as
vec(MN((()))
Σ
(11))) (1///11)
(19)
J=|k-1| radial component angular componcnt
where vec(-) is the vectorize function, the learnable radial component s R. > R is parametrized by neural network, and the non-learnable angular component is determined by Clebsch-Gordan constant Q and the spherical harmonic Y, R - 2/++1
B Derive of the Convolutional Kernel
To derive the kernel (8) for SE(3) x +SE(3) transformer layer, we consider the equivariant convo- layer:
lution as a simplified version of the SE(3) x
(W * (z.)
Σ
yoo (z. - zu) (zv);
(20)
VEKMN(s)\(W)
i.e., we only consider the message Vo. while fixing the self-interaction weight 0 and attention
Quv = 1 in (5. To ensure the SE(3) x SE(3) equivarraae of convolution (20). i.e.,
((9 x 92) f)) (z) (w ((g2 x g2)))) (z),
(21)
the kernel W must satisfy a constraint:
pa(r12) @ (o(r12))(2 = W (r12z),
(22)
where we abbreviate ri T2 as T12. abbreviate vec()oo.t E +2(2+ +1)(o++1) as W.
and assume ||=ا|| = ||~||| = for simpler notations. Equation (2 is generally known as the kernel
constraint, and its necessity and sufficiency can be proved in verbatim way as Theorem 2 in [37].
14

---

so we omit the proof here. Now we can obtain the kernel g by solving this constraint. Note that direct formulation given Theorem 2.1 in [5], but here we derive it in less abstract way.
We first observe that (²) = Yú @ Vi, & Yo, (=*) @ Yog (=²) is a special solution to equation 22). where ()() E TR2++1 is the column vector consisting of degree-J harmonics with order m O,,, J as each row element, because
p:(r12) 2 (x())))W"
(pi (12) Po. (r)) Por Yy (2))))))),())))) Yo (,) (o)))))
= (Pp,()r)) (pa((2)
(Pon
-1,,(1 YI, (r22") 2 Yo, (r12) You (12= -r1(r)22).
Then we point out that (="))="")""""" (x²)
my J2.J2.:1 is an orthogonal and complete basis for two variable square-integrable functions (2 S x S + R, where 5 the 2-D sphere. Thus, we can write each component of (z) as linear combination of this basis. Specifi- Br. (x*) & (z"). and let (m1, M2, m3, m4) be the index of the ele- ment Y. (e") y (2 (z Yo (2²) in W and (J1, J2, me) be the index the element
cally, let V(z)
me (=²) in V. We have W = PV, where yJ. (2 y J2
P Pmi. ma ma. Jo.
Y, (z')x² (z 2 Yoo (33) (2))
Yith (2 (='))"" (3)) (x ydz
ymu (z?)8 ( (z?)dz?) (z)))?=
-(1, mi, 01, ma J1, ms) (iz, m2, 02 my (() Here, c(Ji) and c(J2) are coefficients related to J1 and J2 respectively. (i, m, mg]J, ms) is known (1 01, ma J1, ms) (iz, m2, 02, mm ma) is known as the 2-nd order Clebsch-Gordan coefficient, and we represent it as Q. In other words, we have P = Qc, where is block diagonal matrix, and each block c(J1)c(J2)1. Therefore, we have (ps(rie) po(r12) (othe))>P(s>>)>))))))>>) PV(2) PV(r122)
(23)
the Clebsch-Gordan coefficient, the product
(p((ria) po(r12) QcV(2) QcV(r12=)
QT ((((1e) @ po(112)
QcV(2) cV(r122).
Since
[
+ PJ. (112) cV(2)) = cV(112)) J1.J2
(24)
for arbitrary 2, we further have
e (pi(r12) & (
+ PJr.ss Jr.J,
(25)
by equating the above two equations. Finally, we stack this decomposition back the kernel constraint (2). and obtain
① Prnnsrra) W J1.J2
(26)
which suggests that
= (V(())=cV(C W(z) = Qc'V(xr,
(27)
(28)
where c' is the coefficient matrix of the same shape as e, and each coefficient is arbitrary. Specifically,
W(z) c' J2
Σ Y, (x*) & (2²)
(29)
J1.J2
We note that J is parametrized by neural network in model g
15

---

C Proofs and theoretical results
C.1 The proof of the equivariance of Arun's method
We first establish the result on the uniqueness of Arun's method. We begin with the result of the uniqueness of singular vectors.
Lemma C.1. Let A UEVT E Rax3 be the SVD decomposition. If a singular value o, is distinct, then the corresponding singular vectors Uj and Vj can be determined up a sign. If 0j + 0. UjVi is unique.
Proof. If 3 + O, we have
A = V==V"==V=V"""""""""
(30)
i... the eigenvalues of AA A are o7, i 3, and Vi are the corresponding eigenvectors. Since oj is distinct and all o; 2 O, o3 is distinct. As result, the eigenspace corresponding to of has dim 1, i..., it is spanned by Vj. Since V, is a unit vector, can be determined up a sign. In addition, have AV, U,Uj. thus Un can also be determined up sign. UjVF is unique since flipping the sign of V always leads to the flipping of sign of Uj. and vice versa.
If o 0, we can still repeat the above argument determine Vj up sign, and determine Uj up to sign using the above argument for AAT..
Note that in Arun's algorithm, we use the SVD projection defined as follows.
(VD U diag(1, 1, (.igg(()))"
(31)
where A = ÚLVT. U.V E 0(3) is the SVD decomposition of A. In other words, if der(UV/T) 1. then take ÜVT, otherwise we flip the sign of U3 V3. An important observation is that this SVD projection is unique under a mild assumption.
Assumption C.2. In the SVD decomposition, 03 is distinct, i.e., σ1 2 02 > 03 2 0.
Proposition C.3 (Uniqueness SVD projection). Let A = UVVT _g3x3 be the SVD decompo-
sition, where Σ = diag(a 02, 02) is diagonal matrix, and U, V E 0(3). If assumption C.2 is satisfied, then SVD(A) is unique.
Proof. Let A -ËEV be another SVD decomposition of A, i.e.
U + +oUU2V + +ogUaVV Ui + o2Üsi + +og0sVf
(32)
the goal to prove
= =U,V + +02 + (VV")) We first show that Ui VT U2VV2 = U.F -0,Y..
V + U2V +
(33)
1) If 03 = 0 and σ 02, then
UNV + UUVV U. + -Z σi
A.
(34)
2) 03 0 and T 0g. then og and 02 are distinct and nonzero. According to lemma C.I U.VL" Ü, V, and V2V V2 thus the summation of these two terms equal.
3) If 0g > 0 and 01 > d2, the argument the same as 2)
4) If 0 and 0 d2, then 03 is distinct and nonzero. According lemma C.1 UsV = 7.
Thus,
V. + +UVV = U.V + +UVY
- σ1 (1 - (3-o3UVVz)
(35)
Thus have UNV + UUVV UUvT +U.VT.
16

---

Now we prove that the last term in Equ.
33 is equal. We have U3Vy = =U3V3' and we have
shown that V U2 = = + +2
thus we have
der(U]VV + U.V + U.VV')
=de((UiV) + U2V2 = )U,13' [Vei
=de((U) (=der( 1/2, v3l²)) =deelUi U2V22 + UzV3))
(36)
By multiplying this term with Ü2V3 = =U3V3 we have
den(U,VF + U2 + V.5V V0y = der(U.VV + U2V2 +
(37)
which suggests that last term in Eqn. 33 is equal. In summary, we have proved 33
Finally, we can prove the uniqueness of Arun's method under the same assumption.
Proposition C.4 (Uniqueness of Arun's method). Under assumption unique.
C.2 Arun's method
is
Proof. According to Prop. C,3 unique.
SVD(F) is unique. As result, l = mYY) rm((X) is also
Throughout this work, we assume that assumption C.2 holds, so that the uniqueness of Arun's method and BITR can be guaranteed.
For the rest of this appendix, we use simpler notation of SVD projection, where we simply absorb the sign matrix into and V:
U
SO(3),
(38)
VT (diag(1, (1, (. E SO(3),
(39)
and thus obtain the following equivalent definition. Definition C.5. The SVD projection of matrix A is defined as
Σ drsignnee
(40)
SVD(A) UVT
(41)
where USVT and U, V E SO(3).
Finally, we can discuss the equivariance of Arun's method.
Lemma C.6. Let
SVD(A) = UVT
(42)
be the SVD projection of A, i.e., A can be decomposed A UEVT, E is diagonal matrix and U. V E 50(3). We have
SVD(r2ArT) (A)
(43)
SVD(AT) = (SVI(A))) SVD(cA) = (A)(
(44)
(45)
for arbitrary A E GL(3), r1.12 SO(3), and E R+.
The Proof of Lemma. C.6
We have
r2ArT -UUEV" (1)
(46)
where r2U SO(3), V SO(3) and is diagonal matrix. In other words, 46 is SVD
decomposition of matrix Art thus the SVD projection can be computed as
SVD(r2Arf) (r, vY" (2V SV)D
(47)
which proves the first part of this lemma. We omit the other two statements of this lemma since they can be proved similarly.
17

---

The Proof of Prop. 3.2 Let D be Arun's method and ( Y = (x(X,)),(x.,)))
We directly verify these three equivariances according to Def. 3.1 We only prove the SE(3)-
bi-qquivariance and swap-equivariance and omit the proof of scale-qquivariance, because it can be proved similarity.
1) Siquu--i- We compute & (g) X, g292) for the perturbed PCs gi X {riz 4 f and geY = (r23// + 12 as follows. We first compute F(g1 92 as
(0,X,92Y) Σ (F: r2 Σ Ene,
)
=f
(48)
Then we compute r via SVD:
r(n X, Y) = SVD(F( X. = SVD(r2F(XX - r2SVD(F(X, = (X Y)r
(49)
Since the mean values are also perturbed as m(giX) (X) and mgg2Y) g2 we compute X 92 as
where the 3-rd equality holds due to Lemma. C.6
t(91X, = m r(g) x22)..X.2 = gY) (X, X) 9im( - ((x, t2.
(50)
In summary.
(xxx s Xx = (v2r(X,Y)rr" (r) 41 rx((X, (2) -92991
(51)
which proves the bii(()--bieeuuiaariance of Arun's method.
2) swap-equivariance
We compute (Y, as follows. We first compute X(Y,X) as
X(Y, = Σ I:D
Σ T. Je = (F(
(52)
Then we compute x(Y,X) via SVD:
r(,, = SVD(F(Y, X)) = SVD ((((X) -(VDD=XX
(53)
We can finally compute X(Y
x) = m(X) - (Y, (Y = m(x) -(x))=r(x,x))'m((
((x,)). (54)
In summary.
4 A = , = =-1
(55)
which proves the swap-equivariance of Arun's method.
C.2 The proof of the equivariance of BITR
of BITR, we first verify that each layer in SE(3) x dransfrreee indeed SE(3) x (E3))vaauvvarian Intuitively, the equivariance of a trans- former layer (5) is result of the kernel design, i.e., the kernel constraint (22), and the invariance of attention. We state this result the following lemma for completeness, and we note that similar techniques are also used in the construction of -33anntoroooorm [II
Before proving the SE(3)-bi
Lemma C.7. The transformer layer 5 is SE(3) SE (ee()3))eiivrian...
Proof. We abbreviate r1 12 as 112, and 9i x 82 as 912- Let be a transformer layer 5 we seek to prove (Lf)) ((()12)(f)) for the input tensor field f and 91, 92 SE(3). We compute the
18

---

RHS of the equation as
(Moozlf1)) (zu)
-10(22(F)) (zu)+ +
Σ
a(gı2()), Ru, 21)) (x)(11(()))
ExEmpp(o12(())
=Wo zu) (p
+
Σ
a(gi2(f), 2us 2.) -2--2z)ppp (1)p((112) (912
2n Emppp(sna(())
- zul (Po((1z))
Σ
algz(s); Rus Fuz )
(012% - (
Empp(f)
Note that here we have (2) (9 2 (FI(2)))) because we write F in the channel-fing form, Le., the shape of Fo c (201 + 1) + 1). The LHS of the equation computed as
(L12(L())) (zu) (zx)
=WFF (9 2u) po(r12))
Σ
aff.gi2 Za: 2v Po(r12)Wo 912 zu) (2))
Zu Esup(())
We now verify that these 3 terms equal respectively.
By design, the kernel W satisfies the kernel constraint (22
Po(ri2)We (r12) (r12) W(rizz).
(56)
-1 In addition, let z 912 Zu- We have
g12 2n = g122g 912 ri2(2z - --,) r12(2. 912 91222)) zu).
(57)
Thus, we can obtain the equality by combining the above two equations.
We compute Q and K for B and B respectively. We have
Q(112()) Fur 2un
W)(nneP)'(z +
War (po(ri)))
Q(1-92 Ru; zv)
(zu)):
Σ W (9122r - 2n) (112(()) (1122))
K(g12(J)s Zu, 9122n)
(912 - (2)
i
and
K(J Fo)
Σ (zv 9r2 (z))
(p((12)) (91220 - - (())())))))))))
where the last equation holds due to the kernel constraint 22 and the orthogonality of p. (r12).
Specifically, (r(r2) )- = Poz (ri) ((1)) 6 Pog (r2) is orthogonal matrix because Poy and Poz are orthogonal representations, i.e.. Por (r1) and Poz (r2) are orthogonal matrices.
19

---

Thus, we have
(Q122(()), Zv. ((912(()))))))))))))))(()))))))) Eus 0122.))
Σ WgF ( (zw)) (Po(ri2))
T
[ W =
T
Σ W Fo (1 (zu)) E (Po(12))
-
= (Q 912 Su. 20), K(f, 912 Rus 2n))
Finally,
A' trivial. In summary, we have shown (L())) ((())))()). which proves the S3 x (3))---viaarianee of the transformer layer (5)
On the other hand, an Elu layer (5) is also SE(3) x -quuieaauiva,in,,,, because this layer is piece- wise linear. We state this statement formally in the following lemma, and omit the proof. We note that a similar argument was used in [9].
Lemma C.8. The Elu layer
is SE(3) SE(3) eqquivaran.
Now can prove the SF((3 biuqqiivaree of BITR.
The Proof of Prop. 4.1 Let g pp 4 (X, Y), and fout (X) We prove this proposition by showing
92 g9l = +p 2s(giX,922) 92
(58)
We compute the RHS of the equation follows. First, since each point in X and Y is convex combination of X and Y respectively, we have
X - (XX
(59)
Then we have [gg X. 92 = (g1 x 92) 92) Y) because X Y. When (gi 92 Y) is fed to the SE(3) x SE(3) -transfforrerr the output feature will be (0) X 92) by design (This is verified Lemma. C.7 and Lemma. C.&). In particular, for degree-(1, 1) feature degree-(11))) (1,0)
feature 1x and degree- (0, 1) feature ly, we have
(n (xx) tx (x(0xX,02)) = (X,) (y = r2ty (X, (X.Y). Y).
(60)
Therefore, we have f(g1 X.91) = à unvee(F). Finally, we compute projection & similar C.1
by applying unvec(-) to the first equation, i.e.,
r(91X,g2Y) ,
(61)
and
t(g.X,928) = +
( - ( x2)x +
= =
= +22
- -X x))((())m)))))xx(x()))))))))))))
((x,)) tz2
In summary,
&p Os = (r2r(X, (xr - (), which proves the 33((3)-iiequivvriin) of BITR.
+2 12) = 9299 (62)
We finally prove Prop. 5.1 and Prop. 5.3 similarly to Prop. 3.2
The Proof of Prop. 5.7 We compute (p(s(f)) as follows. First, we have
(s( (z) = (s())) (s(S))20 (=) = (s(=)), ( (z) =({))()())))
(63)
20

---

by definition. Then we compute the equivariant features f(s(f)) (f(f))" tx(s(f)) = ty (f), ty (s(f)) tx()).
(64)
where we treat xx and ły vectors. Finally, the output can be computed as
= r(s(f)) = =VV((()))) - (SVX(((()))))(((())())))""""))") (((())
(65)
according to lemma C.6 and
t(s(f)) -mt
= =m(X)+ + +x((f) -x(f)(((f))) (m)) (m())
= =f((
In other words, +r(((s)) - (rT, --= ---1)=9- which proves this proposition.
The Proof of Prop. 5.3 We compute pp(c([)) as follows. First, since f is a degree-1 R+- equivariant tensor field, satisfies c(f)(=) cf(c-12) Therefore, we have
f(c(f)) (cf(f)), tx(c(f)) ctx(f), tr(c(f)) =x
(66)
by definition. Then, we compute the output
r(c(f)) = = = SVV(((((())) (SVD(((()) (r(f))
(67)
according to lemma C. and
Mc(s))
+ ty(c))))))))))))-))))))))))))))) ( = =cm( + (f) (((()(((()((m) + (f)
ct(())
In other words, +p(c([)) = (r, el), which proves this proposition.
C.3 More results of Sec. 5.1
C.3.1
The proof of Prop. 5.2
In this subsection, we use slightly different notations than other parts of the paper. We regard the kernel W as a 4-D tensor of shape (20) + 1) x (2og + 1) x (2i1 + 1) x (2ig + 1). and we regard the feature fc as 2-D tensor of shape (2i + 1) x (2iz + 1), i.e., a matrix, when it multiplied by a kernel. Therefore, the multiplication W fa is treated as a tensor product where all two dimensions of ft are treated rows, i.e., the result of W f is of shape (20) + x (2og + 1). Similarly, regard the collection of features F as a 3-D tensor of shape c X (201 + 1) x (202 + 1). When is
multiplied by a self-interaction weight 8o E Rlxe the result is W E IR (2or (1x
We first make the following observation on the symmetry of the kernel.
400 PJn., (1l 11. 1|2 Lemma C.9. If the radial function yp satisfies
Pa.n (11= (I) ()) for all 21,
3.0
22, Jr. Jz. then kernel W 5 satisfies
(oid()) = -woa (s(2))
TiaTss
(68)
where Tig represents the transpose of the i-th and j-th dimension of a tensor.
Proof. According to and the discussion in Sec. B the (m3, m4 m1, m2)-th element of (o.)(2)
01421
PJ. (1| 11, 1|2 (1) mi, m3, ms)
J1= 01 -1 J1=|ي iy
Ji
J2
(12, m2, m4 J2, m)) Y (2 /21 I]) y J2 2=2/11=]1), (69)
21

---

and the (m4, Ma. m2, mi) >-th element of W (s(z)) is
ΣΣΣ
0g ogti
Σ
P Ja 1|, (12, m2, 02, 02, ma mo)
J= 02 12 J= 11 mg
J2 my=-j
(11 m1. 01, my J1. ms) (2² (22/|2=11) (=1/11=1
(70)
Since the angular components in 669 and (70) are the same, and the radial components are equal by assumption, we immediately conclude that (69) and (70) are equal. In other words,
(x)
W (s(z)) Ma..2g..m...1
(71)
mL. 11 1712
which proves this lemma.
Then proceed to the proof of Prop. 5.2
The Proof of Prop. 5.2 Let I. transformer layer the input tensor field J.
we seek to prove s(T(()) = I(s(J)) for
To this end, we expand the RHS of the equation as
(L(S(N)) (=w)
(
+
Σ
a(s(f), (t - (2u)((()=2)ss())))
tEI(a())) (f)ppla(())
Tos -W (
+
Σ
a({(J), 20) - zv)
(El(s()) ss Empp(s(())
wo (p)((((())))
als(J) Fu, s(zo)) woi (s(oo) - zu) ())(1((z))
RET(S) ESupp())
where If) is the set of all degrees of field f. and supp(f) is the support of field f. The last equation holds because we replace 2, by s(2)), and replace by i. We expand the LHS of the equation
(s(T(1)) (z.) -((()) (s(z)))
(((())))((EE)
a(f))(z)) (Mo s(Zw)){((E))
(ETU)
Zu Esupp((
We now verify that these three terms equal respectively.
By assumption, o W thus we immediately have
T = =(wor>(s(zw)))
(o(((ss(2w)))
T2a
(72)
Since a = (Q, K), we compute Q and K for B and B respectively. As for Q, we
have
+ we (3(((()))) T
Q(s(ss Zui s(zx))
(((s(F))((x)== =
oEI(s(())
oEI(s(()
- (W&FF(s(zx)))
oEI(s())
22

---

where the last equation holds because W = Wi by assumption. In addition,
Q(f.s(zu),z) 2o) = ①
(((2o))(z)))
oET(S)
As for K, we have
K(o(f), Rus s(zo))
Σ
- - (s(2))
oEI(a(f)) tEf(s(()
Σ oEI(s(f)) AET(F)
Wx - ((Ew))
where the last equation holds due the constraint of radial function and Lemma. i by i. In addition,
C.9 and replace
(Z),)()
Σ Wxx - OET(S) KET(F)
((x)) s(zu))
We observe that Q(s(J), Ru, s(z))) is just the transpose and re-ordering of each component of (Zu,), zo), and this is also true for K(s(J), &u, s(z.)) and ((J, s(zu), zn), which sug- gests that their inner products are the same. Specifically, let Qo = W/Q po (b(z)) and K
EiEr W (Ze (Ez)) We have
a(s(f), Eus ((zo)) {
(Q")" T
+ OEI(a([))
T (KO)
(Ko)
Σ
(Q
Σ (Q", K') OEI(S)
OEI(a([))
OEI(a([))
and
alf, s(z), 2n) = + Qo
K
K
Σ (Q", K%)
OET(F)
OET(S)
nET(S)
which suggests that a(s; s ((z.), = (s((s() Zu; s(zn)).
6.0 C C": By assumption, the radial function satisfies Pir,,,
(1|= 11.
2.6 (1| 1==1|1
12.h
T12734
for all i, o, J1, J2, z and 22, thus (z)((2) (s())s result,
(s(2))
according to Lemma.
C.9
( = ((w))()(2)))zz))) TiaTsa ((z))) = (o) (F((E)
(73)
In summary, we have shown the Z equivviivvre of a transformer layer (5) which is the first part of Prop. 5.2 We omit the proof of the second part of this proposition, ie., the equivariance of an Elu layer, as a simple extension of the proof of
A'
Remark C.10. The weight sharing technique in Prop. [5.2 reduces the number of learnable parameters of SE(3) x SE(3) transffrrre by about half, thus it makes the model more efficient. In practice, the weight sharing technique can be achieved by sharing weights between different o, é and J directly, except for the symmetric case, i.e., o = Ô, = 7 and J1 = Jz, where the requirement of the radial function becomes
0.4 PR.,,
(I|2 11. 11 o (11 (1 11. II)-
(74)
We represent the radial function in this case as p(x,8) v) 1 (0(2,)) v) p(y x)), where 6 is neural network, which guarantees the symmetry of sp
C.3.2 The complete-matching property
Consider the following complete matching problem.
23

---

The complete matching problem Given pair of PCs X and Y, where Y is generated by unknown rigidly transformation of X, ie.. Y = g.X and g E SE(3) is unknown, how infer g?
The complete matching problem is a prototype of the registration problem, ie., in the simplest case (no outlier, noise, no partial visibility..), how to derive the relative transformation between two fully overlapped PCs? Despite its simplicity, this problem is non- con-trivial, because we do not know the correspondence between X and Y, i.e., Arun's method does not apply.
surprising fact is that g can be exactly recovered using a (E(3) -i--qquivariant and swap-equivariant assembly method. We state this fact formally follows.
Proposition C.11 (Conplete-matching property). Let ④ be a swap-equivariam and (E(3)-bi- equivariant assembly method. Then
$ (XX, = 91,
(75)
for arbitrary PC X and 91 E SE(3)
Proof. First, since & is swap-equivariant, then O X X) = I. Then, since & is SE(3)
equivariant, we have $(X, X)x(X X) (X, (X) X) = 91-
According to the main text, BITR is swap-equivariance (Prop. 5.2) thus, we can derive concrete algorithm, called untrained BITR (U-BITR), for the complete matching problem:
(76)
where 4g is BITR model. Prop. C.II suggests that Ly solves the complete matching problem without training. We numerically evaluate this property in Appx. D.4, and leave the theoretical analysis of U-BITR to future research.
C.4 The proof of Prop. 5.5
5.5
First, we have the following straightforward lemma for the kernel.
Lemma C.12. If the radial function yp : R x R + is degree-p function, then kernel W satisfies
5
W(cz) = c))(s), Ve E R..
(77)
Then directly verify Prop. 5.5
The proof of Prop. 5.3 Let be a transformer layer (5). f be a degree-O -qqiivrirai input field,
рK be degree-O function. We seek to prove: If yv is degree-O function, then c(I(f)) L(c(f)) and L(f) is degree-O R4, equiivaaai;, 2) If yov is a degree-1 function and W = 0, then
c(L(f)) = L(e(f)) and L(f) is degree-1 R+ equiivarria
To begin with, we expand L(c(f)) as
(M(c()) (E.)(())))) (((((F))((
Σ
a(e(f), Zu, z 2x) (
Empple(()
(c
Σ
2u Wo (aa -u))
Expple(())
W 2u
Σ
a(c(f) Zus c(2.) W Wer -2a)
Zo Esupp(()
and expand c(L(f))
(d(z(f))) (zw) de (L(F)) (c-1zz)
d (c (zx)) +
Σ
alJ,c-1 zn, 2n) Wa - cc '''==)) (
1'
Enpp(f)
24

---

We first point out that B
B'
because
kcc(f)
W -2)
+ E
-
(z))s
i(ffc c "us Zu),
where the second equation holds because чи is degree-O function by assumption, and Lemma. C.12 claims that the corresponding kernel WK scale-invariant. addition,
Qc((), Zu, Zu, c(z)))
A (((c((P))(zu))
+ war Q c Eus zo).
In other words, both and K are ceale-invariant. Thus, conclude that the attention is also scale-invariant:
a(c(f), c., (Q(clf), 2ur ((z))),((((()) Zus c(z)))) (K(J, ( 20), Q c Zu; 2u))
Now we discuss these two situations:
1) If pv is degree-1 function and W = 0, then
cW (z. c 2x,) (c(zo) - zu)
according to Lemma. C.12 and A the RHS the action on the degree- 14 -equivariia field, thus we have L(c(f) = c(L(f))
A' O. In other words,
where
and L(f) is degree-1 equivaarai.
2) If sov is degree-O function, then
W (2. - c 2z)) We -2,) 2n)
according to Lemma. C.12 Thus, L(c(f)((2 ((f(c-12). where the RHS is the action of e on the degree-0 R4 eqquiirrar field, thus we have L(c(f)) = c(L(f)), and L(f) is degree-0
-equevvvvrri
In summary, we have shown that the equivariance of the transformer layer. We omit the proof equivariance of the Elu layer it can be proved similarly.
We can now construct a special structure of BITR for producing degree-1 IR, -equivaaari output:
deg-0 + deg-0 + deg-0 - deg-1
(78)
In other words, all tensor fields are degree-O -euuivvvarr, except for the final output which is degree-1 -uuuquviaaia More specifically, we consider two types of layers, i.e., deg-0 - deg-0 and deg-0- deg-1.
Prop. 5.5 suggests that a transformer layer with degree-p radial function can generate a degree-p R_ equivariant field when the input field degree-D Reqqqivarriat (Elu layers do not influence the R -quuivvaaiianc Therefore, we only need to consider degree-O and degree-1 radial functions for our purpose. In practice, represent degree-1 functions using neural networks consisting of linear and relu [ layers, and we represent degree-O functions using linear, relu and layer normalization layers. Specifically, We represent degree-O functions neural network db-
Φo Linear + Relu > LayerNorm + Linear - Relu + LayerNorm.
(79)
and represent degree I-functions neural network Φ1:
Φ1 Linear Relu - Linear + Relu.
(80)
Remark C.13. There are two major difficulties in developing general ccale-equivariane theory for BITR. First, the attention term is not scale-invariant due to the existence the soft-max operation, iee, SeftMax( A) & SeftMax(cA) for vector A and E [R., That is the reason why we only accept degree-0 -qqiiivari input in our current theory. Second, when p {0, 1), we are not aware of any general way represent degree-p functions using neural networks.
25

---

D More experiment results
D.1 More training details
We run all experiments using a Nvidia T4 GPU card with 16G memory. The batch size is set to the largest possible value that can be fitted into the GPU memory. We set bs 16 for the airplane dataset, and bs = 4 for the wine bottle dataset. We train BITR until the validation loss does not decrease. For the airplane dataset, we train BITR 10000 epochs, and the training time is about 8 days when = 0.7. For the wine bottle dataset, we train BITR 1000 epochs, and the training time is about 12 hours. The FLOPS is 14.5G in forward pass (including the computation of harmonic functions), and the model contains 0.17M parameters.
For Sec. 6.3.1 and 6.3.2
the normal vector computed by Open3D 46 as the input feature of contaans test samples. The wine bottle dataset used in Sec. [6.4] contains 331 training and 41 test samples. We adopt the few-shot learning setting in the manipulation tasks in Sec. 6.6 we use 30 training and 5 test samples for mug-hanging; we use 40 training samples and 10 test samples for bowl-placing.
BITR. The airplane dataset used in Sec.66.31 contains 715 random training samples and 103 random
D.2 More results of Sec. 6.2
We quantitatively verify the equivariance of BITR according to Def. 3.1 Specifically, we compute
bb ( 92 ( (), 1|r
(81)
A swop ,v, (9 -
(82)
Acale |rB(cX,cY) r Y) || [It cx (
(83)
to verify the Sbujiybbiequivvariancc, swap-qqiivariance and scale-equivariance of BITR, where & represent the BITR model, (r=()),( = ( (.) are the output of BITR, and all g (r, t) E SE are written as
g =
ERAA
(84)
Note that if BITR is perfectly equivariant, these three errors should always be O.
The quantitative results the experiment are summarized Tab. 3 where we can see that errors below the numerical precision of float numbers, i.e., less than 1e-5, The results suggest that BITR is indeed (i-))-bi-quivvarann,, eruivrruu and ccalc-equivariant.
Table Verification of the equivariance of BITR.
bi
A
Swop
Ascale
5e
2e
5e-1
D.3
Ablation study
To show the practical effectiveness of our theory scale and swap equivariances, we consider an ablation study. We use the same data as Sec. [6.2 and remove the weight sharing technique in dcc. to break wwap-equivariance, and force yov all layers to be degree-1 function to break the scalo-equivariance.
We evaluate the trained model on 100 test samples, and report the mean and standard deviation of Ar in Tab. [ where we can see that removing an equivariance of the model leads the failure in the corresponding test case, which is consistent with our theory.
Table 4: Ablation study scale and swap equivariances. We report mean and std of Ar
original (17.1, 6.0)
rigidly perturbed
swapped
scaled
BITR (full model)
(17.1, 6.0)
(17.1,
(17.1, 6.0)
BITR (w/ swap)
(15.3, 7.0)
(15.3, 7.0)
77.33.0.00
(15.3, 7.0)
BITR (w/o scale)
(15.0, 8.0)
(15.0, 8.0)
(15.0, 8.0)
(125.4, 15.0)
26

---

D.4 Evaluation of the complete-matching property
This experiment numerically evaluates the robustness of the complete-matching property (Prop. C.I1 against resampling, noise and partial visibility. We first sample X and Y of size 1024 from the bunny shape, and a random E (3)3), then we use a random initialized U-BITR to match to gY. We consider different settings: X and Y are exactly the same; 2) X and Y are different random samples; 3) Gaussian noise std 0.01 is added to X and ;;4) 4) Ratio 8 of X and Y is kept by cropping using random plane.
We repeat the experiment times, and report the results U-BITR Tab, [ We observe that the transformation is perfectly recovered when X = Y. which is consistent with the complete-matching property. Meanwhile, cropping the PCs leads to large decrease of the accuracy, while noise and resampling have less effect. This is consistent with expectation because cropping the PCs has larger effect on the shape of PCs. We provide qualitative results in Fig. 7
Table 5: Results of complete matching using U-BITR.
Ar
At
x=Y
0.0 (0.0)
0.0 (0.0)
Resampled
20.7 (16.4)
0.18 (0.16)
Noisy
5.0 (2.8)
0.02 (0.008)
Cropping s = 0.9
99.1 (40.49)
0.58 (0.4)
(a) Complete- matching
(b) Noisy data (Ar 2.8) (c) Resampled
data
(Ar
(Ar = 20.1)
Figure 7: Qualitative results of U-BITR complete matching. The blue and red PCs are the reference and the transformed source PCs respectively. Note that U-BITR only random initialized (not trained).
D.5
More results of Sec. 6.3.1
We report the training process of BITR in Fig. where we can see that the loss value, Ar and gradually decrease during training expected.
We represent the input PCs using light colors, and represent the 32 learned key points using dark colors and large points. As we explained in Sec. 4.33 the key points are the convex hull of the input PCs, and they are NOT subset of the input PC. In addition, can be seen, the key points of the inputs do not overlap.
Some qualitative results BITR are presented in Fig. 9
D.6 More results of Sec.
We generate the raw training samples by sampling motorbike and car shapes from the training set of ShapeNet. Then we centralize them, and move the car by [0, 0, 1]. Note the shapes in ShapeNet are already pre-aligned. The test samples generated from the test set of ShapeNet in the same way.
We repeat the test process times, and report the results Fig. We observe that BITR achieves lower rotation error than LEV, and their translation errors are comparable. Meanwhile, NSM fails in this experiment. Note that we do not report the results of registration methods, because their loss functions are undefined due to the lack of correspondence.
27

---

50
0.4
0.3
Lo8
-1
7
0.2
0.3
0,5
30-
1939
Epoch
Epoch
Epoch
Figure The training process of BITR on the airplane dataset with 0.4. All metrics are measured on the validation set.
0.7
(b) 0.5
(c) 0.4
(d) 0.3
Figure 9: The PC registration results of BITR on the airplane dataset. The input PCs are represented using light colors, and the learned key points are represented using dark and large points.
LEY*
LEV
80
0.6-
40
1.0
0.7
1.0
PC size ratio
PC size ratio
Figure 10: Assembly results of car and motorbike. denotes methods which require the true canonical poses of the input PCs.
D.7 More results of Sec. 6.4
For BITR, we first obtain raw PCs by applying grid sampling with grid size 0.005 to the shape, and then randomly sample 5% points from the raw PCs as the training and test samples. The sizes of the resulting PCs are around 1000, which is close to the data used in the baseline methods. The data is pre-processed following (38 for the baseline methods.
The random sampling process in our method causes the randomness of test error. We quantify the randomness by evaluating the test 3 times, and report the mean and std of the errors in Tab.
Fig. presents 5 examples of reassembling wine bottle fragments, where we observe that the proposed BITR can reassemble most of the shapes correctly, while the baseline methods generally have difficulty predicting the correct rotations.
D.8 More results of Sec.
We preprocess the 7Scenes dataset by applying grid sampling with grid size 0.1. The training and test sets contain 278 and 59 samples.
We consider the 5 outdoor scenes in the ASL dataset: mountain, winter, summer, wood autumn, wood summer. We preprocess all data by applying grid sampling with grid size 0.9. We arbitrarily rotate and translate all data, and train BITR to align the i-th frame to the (i + 2)-th frame. We use the first
28

---

(a) DGL
(b) NSM
(c) LEV
(d) BITR (Ours)
Figure 11: Results of reassembling wine bottle fragments. We compare the proposed BITR with
DGL 44 NSM 7 and LEV (38] Zoom in to see the details.
20 frames as the training set, and the other frames as the test set. This leads to a training set of size 105, and test set of size 48
We report the results Tab. 6 Our observation consistent of that in the 7Scenes dataset: the result BITR is close to the optimum, thus refinement can lead to improved results. Note that BITR+ICP outperforms baselines in this task. addition, GEO causes the out-of-memory error on our 16G GPU. An assembly result (wood summer) is presented in Fig. 12
29

---

Table 6: Results on the outdoor scenes of ASL. We report mean and std of Ar and At.
Ar
Al
ICP
(8.0)
9.4 (1.0)
OMN
110.0 (5.0)
2.0 (1.0).
GEO
ROI
(0.0)
(0.0)
BITR (Ours)
10.6 (0.0)
1.4 (0.0)
BITR+ICP (Ours)
0.7 (0.0)
0.8 (0.0)
(a) Input
BITR
(c) BITR+ICP
Figure 12: An assembly result of BITR 7Senne (1-st row) and ASL (2-nd row). BITR can produce results that are close to the optimum, and a ICP refinement leads to the improved results.
(a) Mug- Bugnhanging
Figure 13: The result BITR on mug- nug-hanging.
D.9 More results of Sec. 6.6
All data this experiment is generated using PyBullet R. where the objects in training and test sets different in shape and position.
Note that we do not report the quantitative results for this experiment because the metric (Ar; At) ambiguous due to the non-uniqueness of the correct solution. For example, for correct bowl-placing result, the bowl is allowed to rotate horizontally even for this correct assembly. A suitable metric is the success rate of the manipulation in physical hardware experiments as (27]. but measuring this metric is beyond the scope this work. In addition, the assembly methods such NSM 7 and LEV BS not applicable because the canonical pose not known, and we do not report the result of any registration method because the correspondence does not exist. We present result of BITR mug- mug-hanging in Fig. [3
the plate, so the metric Ar can be very large
E Limitations and future research directions
In our current implement, we have accelerated most of the layers of BITR using the "scatter" func- tion 24). However, BITR is still relatively slow due to the independent computation of convolutional
30

---

kernels, i.e. the harmonic function and the independent multiplication of radial and angular compo- nent in each degree. This is reflected by a low GPU utility ratio (about 20%). We expect to see large speed gain (about x5) if the above- mentionet computation is implemented using CUDA kernel, i.e, GPU utility ratio can be close to 100%. On the other hand, a rotation-basee technique was recently introduced to reduce the computation cost of SE(3)-equivarian networks 22]. A promising future direction is to extend this technique to BITR.
To explain the limitation of BITR in handling symmetric PCs, we consider the following example. For pair of PCs X and Y, there exists a non-identity rigid transformation J2 such that g₂Y Y. and g E (3) is proper transformation that assembles g.X to Y, then 929 is an equally good transformation that gives the same assembly result. In other words, the optimal transformation is non-unique which cannot be modelled by BITR. Actually, we notice that the result of BITR is undefined in this case. Specifically, let ΦB () we have
g # Y)= a X, 92 Y) u B 929
(85)
due to 3F(33-bbiequivvarineee Then we have g2 1, which contradicts the assumption.
To address this issue, we plan to extend BITR to generative model in future research, i.e., it should assign a likelihood value to each prediction. For the example considered above, it should assign equal probability and 92g.
Apart from the -eetionee two limitations, there are several directions for future investigation. First, it is important to generalize BITR to multi-PC assembly tasks where more than 2 PCs considered [14 (8). Second, we expect the U-BITR model to be useful in self-suue 3D shape retrieval detection models, such those in image object detection/retrrivaal [9
31

---

NeurIPS Paper Checklist
1. Claims
Question:
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We have pointed out that the main contribution of the paper is (E equivariant, swap-equivariant, and scale-equivarian PC assembly method.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
We discuss limitations in Appx.
E
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g.. independence assumptions, noiseless settings, model well-specification. asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
32

---

Justification:
All proofs can be found in Appx.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yss]
Justification: The architecture BITR is described Sec.
and 5
More details are in
Appx.D.. D.1
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, No answer this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how reproduce that algorithm. (b) If the contribution is primarily
model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either be way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome deseribe the particular way they provide for reproducibility, In the case of closed-source models, it may be that access to the model is limited in some way (e.g., registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
33

---

Answer: [Yes]
Justification:
The code will available at htttps://githuui http::///ggcchhb..ccm//////z//// http////...oommmmm2//////2555/
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.cm
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/ /mips.ccc/puliic/uuii http:p//b.....iiieee/deeeeeaaaaa CoodeSubeeiieeeeeeeeee ssi asionnolle onnolicy)))) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification:
This is included in Sec. 6.1 and more can be found in Appx. D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [Yes]
Justification:
We report the mean and standard deviation for our experiments.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, statistical significance tests, at least for the experiments that support main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
34

---

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of enrors is not verified.
For asymmetric distributions, the authors should be careful to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative eтror rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of exccution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification:
The information regarding the computation resource is provided in Appx. D.I
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs well estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't make into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics
Answer: [Yes]
Justification:
The research conducted in the paper conform the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [NA]
This work focus on the equivalence of the point cloud assembly task. We do not see any social impact that needs to be specifically addressed.
Justification:
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
35

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: Our work is about assembly 3D point cloud. It does not generate new image/language or other types of data of high risk.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recogmize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We cited all dataset we used: bunny [33], BB [1I, 7scenes [32], shapenet [6]. and the data generated by PyBullet (
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
36

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
The code of this paper will be released when the paper is accepted.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details the dataset/code/moodl as part their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
37


Reviews:
[{"review_id": "2eo6K9gfel", "reviewer": "Reviewer_P5ME", "summary": "This paper proposes a novel architecture that extends the SE(3) transformers to become bi-equivariant for the task of rigid point cloud assembly. That is the authors enforce a single assembly for all rigid transformations of the source or target point clouds that is learnable by the network. They also propose how to extend the method for scale and swap consistent assembly of the point clouds. The authors aim to design a method that is correspondence-free to deal with the cases where the point clouds have zero overlap. They first learn (the same number of) keypoints on the original point clouds and concatenate them channel wise. After extracting the bi-equivariant features they utilize the Arun's algorithm to align the point clouds. Some experiments in small-scale datasets of zero overlap showcase benefits of the method.", "questions": "- Line 28: \"Sensitive to the initial poses of PCs\". Is there any reference in the literature for that?\n- In practice we do not know id the point clouds have zero overlap or some overlap. How can you discriminate such cases?\n- The zero overlap case could in principle involve many solutions. For example a chair next to a table. Enforcing the configuration that the training data suggest and restricting it with equivariant constraints can hurt performance as the dataset could have other possible configurations for other training samples that are inconsistent with the constraint.\n- The setup of zero overlap has to be formulated more properly.  \n- In the method the authors extract the same amount of keypoints for the X,Y point clouds. Is that restricitve in cases one point cloud is larger than the other?\n- Shouldn't there be some sorting of the keypoints before concatenation? See Weaknesses too.\n- Why is BITR+ICP bi-equivariant? \n- How does the method scale to large point clouds with complicated distributions as in 3DMatch? \n- Are normals given as features in this and the rest of the methods? Are the normals computed before or after the cut?\n- Is the method trained per-class or for the whole dataset?", "limitations": "See the section in Weaknesses and then Questions.", "rating": 4, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "- The authors aim to address a significant problem of pairwise PC registration when the PC parts have close to zero / zero overlap. \n- The derivations of the layer parametrization seem correct (although missing details on second order CG coefficients).\n- The experiments show some performance gain in certain cases.", "weaknesses": "- **On the assumptions/setup** : The setup of zero overlap is not addressed properly when equivariant constraints are enfroced. In the zero overlap case multiple correct poses is inherent. For example a chair next to a table can be placed in many \"correct\" ways. Some of them will be in the training data. But when two different configurations are presented in the training data that breaks the equivariant assumption. Or symmetrically, enforcing equivariance from the first configuration can never reduce the error of the second configuration. I believe the problem of zero overlap has to be formulated more carefully and these consequences should be discussed. \n- **On the method**:\n    1. **Correspondence-free methods should take care of the permutations of the points**: Concatenating the two point clouds after processing them independently does not properly take care of the permutation of points. In fact an individual permutation of the points in X,Y would result in permuted keypoints in Eq.(10). When the keypoints are concatenated each keypoint will match a different keypoint which means that the bi-equivariant transformer will view the permuted case as completely different features.    \n    2. **Assumptions on the sizes of the parts**: Extracting the same amount of keypoints from both point clouds might be restrictive when sizes differ by a lot. \n\n- **On the experiments**:  Fewer experiments are performed in this paper than what is standard in the literature. Also the scale of the datasets is much smaller in number and sizes.\n   - Low-overlap settings (<10%) exist in KITTI, 3DMatch  that the authors do not consider. This also raises the question of the scalability of the method. \n   - The sizes of the parts also answer the question regarding extracting fixed number of keypoints from the two point clouds. E.g. for large point clouds as in 3DMatch how many keypoints are enough?   \n\n- **References**:  Missing literature. A lot of point cloud processing papers utilize the SE(3) transformer (some are even bi-equivariant) for registration, reconstruction, docking etc. Others address equivariant registration with other networks. Some suggestions:\n    - On Point Cloud Registration:\n       - M.Zhu et al. \"Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representations.\" \n       - C.Lin et al  \"Coarse-to-Fine Point Cloud Registration with SE(3)-Equivariant Representations.\"\n    - On PC Processing using SE(3)-transformer and/or bi-equivariance:\n        - E. Chatzipantazis et al.   \"SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space\".\n        - Y.Peng et al \"SE(3)-Diffusion: An Equivariant Diffusion Model for 3D Point Cloud Generation.\"\n        - O. Ganea et al. \"INDEPENDENT SE(3)-EQUIVARIANT MODELS FOR END-TO-END RIGID PROTEIN DOCKING.\"\n        - C.Lin et al. \"SE(3)-Equivariant Point Cloud-Based Place Recognition\".", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort. We address the concerns below.\n\n1. The setup of zero overlap is not addressed properly when equivariant constraints are enforced...\n\nThe uniqueness and overlap ratio are completely different concepts. There can be non-unique assembly even in the fully overlapped case. Consider, for example, aligning a sphere to another sphere of the same size. The uniqueness is already discussed in Appx.E.\n\n\n2. ...permutation of the points in X,Y would result in permuted keypoints in Eq.(10). \n\nThis is not true. Eqn 10 is a quite common technique to get invariant key points, see the definition of $y_{1k}$ and $y_{2k}$ in Independent SE(3)-equivariant Models for End-to-end Rigid Protein Docking,\nwhich is a paper mentioned by the reviewer.\n\n3. Extracting the same amount of keypoints from both point clouds might be restrictive when sizes differ by a lot.\n \nWe do not see any restriction. If the reviewer thinks there is a restriction, evidence should be provided.\n \n \n4. standard in the literature. Also the scale of the datasets is much smaller in number and sizes.\n \nOur method is an assembly method where the input is not necessarily overlapped, not just a registration task for overlapped input. Those datasets might be standard for registration, but not for assembly. \nIf the reviewer thinks the scalability of our method is an issue, then we insist a more scalable baseline method on this task must be provided. \n\n \n5. References\n\nWe have now added citations \n\"Coarse-to-Fine Point Cloud Registration with SE(3)-Equivariant Representations\" and \"Independent SE(3)-equivariant Models for End-to-end Rigid Protein Docking\" in line 88 (..modelling 3D data []).\nWe will not cite other papers because they are not related to the subject.\n \n \n\n6. Line 28: \"Sensitive to the initial poses of PCs\". Is there any reference in the literature for that?\n\nIt's a well-known fact. For example, the last sentence of the 2nd paragraph in [36] \"All these methods are sensitive to the initial positions.\"\n\n7. In practice we do not know id the point clouds have zero overlap or some overlap. How can you discriminate such cases?\n\nWe don't think there is an easy way to discriminate this. That's actually an important reason why our method is more flexible than registration methods: \nthe registration methods will fail silently when the point clouds are not overlapped.\n\n8 and 9. The zero overlap case could in principle involve many solutions. \n\nSee 1.\n\n\n10. In the method the authors extract the same amount of keypoints for the X,Y point clouds. Is that restrictive in cases one point cloud is larger than the other?\n\nSee 3\n\n\n10. Shouldn't there be some sorting of the keypoints before concatenation? See Weaknesses too.\n\nSee 2.\n\n11. Why is BITR+ICP bi-equivariant?\n\nBecause ICP is distance-based.\n\n\n12. How does the method scale to large point clouds with complicated distributions as in 3DMatch?\n\nFor a \"complicated\" indoor dataset, see experiments on 7Scenes. See Appx E for computational cost.\n\n\n13. Are normals given as features in this and the rest of the methods? Are the normals computed before or after the cut?\n\n1. Normals are use in ROI, not other mehtods. 2. After.\n\n14. Is the method trained per-class or for the whole dataset?\n\nper-class", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "yBUoHxHZCY", "reviewer": "Reviewer_mtoz", "summary": "The paper addresses the problem of assembling point clouds. Given two partial, potentially unmatched point clouds, the objective is to determine the rigid transformation that best aligns them in relation to the unknown complete shape. The proposed approach is a learning-based method, utilizing an architecture that adheres to symmetry constraints inspired by Arun's method. The effectiveness of the method is demonstrated through a toy experiment and synthetic data, with evaluations on real data included in the appendix.", "questions": "Does the suggsted bi-equivariance architecture generalize better than a single-equivariance network trained on shape completion?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "The paper tackles a challenging task.\n\nDeriving the required properties of the suggested network from Arun's method is an elegant solution. The theoretical analysis supporting the suggested method seems to be solid.\n\nThe provided toy experiment seems to be convincing.", "weaknesses": "The presentation quality could be improved. For example, figure 2 is hard to understand. The description of the toy experiment is not easy to follow as well.\n\nEvaluating the benefits of bi-equivariance. An alternative to bi-equivariance, is to train a regular equivarince network on the task of shape completion and then align the two partial inputs according to the predicted completed shape (using one of the baselines that use correspondences).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort. We address the concerns below.\n\n1. The presentation quality could be improved. For example, figure 2 ... The description of the toy experiment..\n\nWe apologize for the lack of clarity. We kept all descriptions concise to fit in the limited 9-page constraint. However, we have now added more descriptions to the caption of Fig 2 and revised the description of the toy example. We have now cited [37] to provide more visualization of the generated data. \n\nWe have made the following modification (the added content is marked bold):\n- (The caption of Fig.2)  \"The input 3-D PCs X and Y are first merged into a 6-D PC Z **by concatenating the extracted key points $\\tilde{X}$ and $\\tilde{Y}$\"**....\n- (Line 298)..We train BITR on the bunny shape. **We prepare the dataset similar to [37]:** In each training iteration.....We train BITR to ~~reconstruct S using~~ **randomly rotated and translated** $\\\\{X_P, Y_P\\\\}$...\n\n2. Evaluating the benefits of bi-equivariance. An alternative to bi-equivariance, is to train a regular equivariance network on the task of shape completion and then align the two partial inputs according to the predicted completed shape.....  Does the suggested bi-equivariance architecture generalize better than a single-equivariance network trained on shape completion?\n\nWe have evaluated the benefits of bi-equivariance against non-equivariant methods [21] and SE(3)-equivariant method [35]. We observed some improvement against these methods in our experiments.\n\nOn the other hand, we are not aware of any assembly/registration method that is based on shape completion, nor do we think this can be implemented easily. Because the two generated completed shapes may not be consistent (there is no correspondence between the generated part), unless both of the inputs are taken into consideration in the generation. But that will require assembling the inputs first, or at least using SE(3)-bi-equivariant features, which falls back to the assembly task. In summary, we do not think point cloud completion based methods can be used in this task.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "W1RhFCpyWf", "reviewer": "Reviewer_fq5D", "summary": "This paper proposes an SE(3)-bi-equivariant approach for point cloud assembly, addressing the difficulty of assembling non-overlapping point clouds where traditional correspondence matching methods struggle. The proposed BITR (BI-equivariant TRansformer) solves this problem by exploiting the symmetry of the point cloud assembly task as a powerful inductive bias, instead of relying on point correspondence.\n\nSpecifically, BITR first merges two distinct 3D point clouds into a 6D point cloud with features that are the tensor product (also known as the Kronecker product or outer product) representation of two SE(3) irreps. These merged 6D point clouds are then processed through novel SE(3)xSE(3)-transformer layers derived from the Wigner-Eckart theorem of G-steerable kernels [5,17], and finally through an SE(3) projection layer that resembles Arun’s method.\n\nThe paper also includes theoretical analysis showing that BITR effectively inherits the symmetry properties of the classical Arun’s method in three aspects: 1) SE(3)-bi-equivariance, 2) Swap equivariance, and 3) Scale equivariance. This enables effective point cloud assembly without assuming known point correspondence. These benefits are also validated through extensive comparison with state-of-the-art baselines.", "questions": "1. Why include type-1 features in the last layer before softmax in Eq (12)? Why not use only type-0 irreps? Why not use type-2 or higher?\n\n2. How is locality (neighborhood) defined for 6D point clouds? Directly using KNN for 6D point clouds to construct a neighborhood graph does not make sense to me.\n\n3. The key points of X and Y are ordered and thus not permutation-invariant. Could this have any negative consequences?", "limitations": "1. The proposed method is not generative and thus cannot handle scenarios where multiple assembly poses are equally valid.\n\n2. The $SE(3)\\times SE(3)$-transformer layer in BITR relies on the Clebsch-Gordon tensor product, which is notorious for its computational complexity, scaling with $O(L^6)$ where $L$ is the maximum degree. The paper proposes a 2nd-order Clebsch-Gordon tensor product, which likely has similar or worse complexity. The lack of efficient CUDA kernels exacerbates this problem.\n\nThese limitations are appropriately discussed by the authors in Appendix E. The reviewer believes that these limitations are not fundamental and could be addressed in future research. For example, recently proposed eSCN Convolution [4] and Gaunt tensor product [5] have reduced the computational complexity of the C-G tensor product from $O(L^6)$ to $O(L^3)$. Extending these state-of-the-art $SE(3)$-equivariant mechanisms into a bi-equivariant form would be an interesting future direction.\n\n[4] Passaro et al., \"Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs,\" ICML 2023\n\n[5] Luo et al., \"Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products,\" ICLR 2024", "rating": 9, "confidence": 4, "soundness": 4, "presentation": 2, "contribution": 4, "strengths": "1. To the best of the reviewers' knowledge, this is the first method to achieve SE(3)-bi-equivariance in every layer. While several existing works in biology [1] and robotics [2,3] have addressed SE(3)-bi-equivariance, they only managed it in an ad hoc, template-matching style in the final layer. In contrast, the proposed $SE(3)\\times SE(3)$-transformer layer is inherently bi-equivariant, theoretically grounded in the Wigner-Eckart theorem, and thus expected to be more general and expressive than previous approaches.\n\n2. Experimental results show that the proposed BITR significantly improves point cloud assembly accuracy compared to previous methods. In particular, BITR retains accuracy when the overlap is small, whereas other methods do not.\n\n3. Although not included in the main sections, U-BITR presented in Appendix C is also innovative and promising\n\n4. The proposed method has the potential for broader applications beyond point cloud assembly. It could be beneficial in various areas that require inferring the relative SE(3) pose between two point clouds (or graphs), such as protein docking, robotic manipulation, and camera pose estimation.\n\nOverall, this paper presents a novel and theoretically solid approach. I am confident that it is a breakthrough in bi-equivariant modeling for SE(3) pose inference problems, including point cloud assembly tasks. The empirical results are sufficient to support the claimed benefits.\n\n[1] Ganea et al., \"Independent SE(3)-equivariant Models for End-to-end Rigid Protein Docking,” ICLR 2022\n\n[2] Ryu et al., “Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation,” CVPR 2024\n\n[3] Huang et al., “Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D,” ICLR 2024", "weaknesses": "1. The paper may be challenging for readers without a background in representation theory and SE(3)-equivariant neural networks. The proposed method is complicated and might be difficult to implement and extend.\n\n2. ~~Experimental validation is rather confined to simple problems in which object parts are clipped by a random plane. It isn’t sufficiently verified for more general and realistic assembly tasks in which the parts are not cleanly cut by a plane.~~ Addressed.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thanks for the careful reading of our paper.\nWe are happy you like the extended U-BITR model which is not even included in the main text due to space limitations.\nThanks for bringing up [4,5], we have read through these papers, and they indeed seem useful for accelerating our method.\nWe now address your concerns as follows.\n\n1. The paper may be challenging for readers without a background in representation theory and SE(3)-equivariant neural networks. The proposed method is complicated and might be difficult to implement and extend.\n\nWe agree that some non-trivial preliminaries are required to read this paper. To alleviate this, we included a brief introduction of SE3-transformers in Appx.A to make the material self-contained. We also (briefly) introduced representation theory in Sec.3.2. [5] is cited and readers can find more complete information there. As for the implementation, we will release the code upon accept to facilitate the research in this area.\n\n2. Experimental validation is rather confined to simple problems in which object parts are clipped by a random plane. It isn’t sufficiently verified for more general and realistic assembly tasks in which the parts are not cleanly cut by a plane.\n\nWe have tested our method on datasets which are not generated by cutting. For example, the BB, 7Scene and ASL datasets from Sec 6.4 to Sec. 6.6, where BB is generated by physics simulation and the other two are real datasets. Also, the data in the manipulation experiment is not generated by cutting.\n\n\n3. Why include type-1 features in the last layer before softmax in Eq (12)? Why not use only type-0 irreps? Why not use type-2 or higher?\n\nWe only use degree-$\\\\{0, 1\\\\}$ features in our network due to efficiency considerations. In the merging layer, we merge these features. We will add a sentence below Eqn 12 to make this more clear: \"here we only include degree-0 and degree-1 features, but higher degree features can be used similarly\"\n\n4. How is locality (neighborhood) defined for 6D point clouds? Directly using KNN for 6D point clouds to construct a neighborhood graph does not make sense to me.\n\nWe use KNN in 6D space. Our rationale is that a small distance in 6D space means small distances in the first and last 3D components ($\\tilde{X}$ and $\\tilde{Y}$), i.e. the key point coordinates of X and Y. So KNN in 6D is similar to doing KNN for $\\tilde{X}$ and $\\tilde{Y}$ and then taking the intersection of the edges of these two graphs, and it has the advantage of avoiding empty edges for all points.\n\n\n\n5. The key points of X and Y are ordered and thus not permutation-invariant. Could this have any negative consequences?\n\nNo, they are permutation invariant. The coordinate of each key point $k_j$ is computed as $k_j = \\sum_{i} F_{ji}X_i$, where $F_{ji}$ is an invariant feature at node $i$ (after softmax normalization). So if we permute $i$, we still get the same $k_j$  for all $j$.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "NrDpnObyik", "reviewer": "Reviewer_1H6H", "summary": "This paper introduces a new network to solve the point cloud assembly task, where a 3D transformation is predicted to “align” two point clouds. The proposed network, BTIR, is designed to enforce the symmetries of the point cloud assembly task into the network layers. Specifically, it enforces SE(3) bi equivariance, scale equivariance and swap equivariance through weight constraints. The network is evaluated on several point cloud assembly tasks. It significantly outperforms existing equivariant methods in some settings.", "questions": "- In the limitations, it is mentioned that the method is deterministic. How does it solve the wine bottle task in Table 1? Is the loss computed up to the to next symmetry or is the symmetry broken in the loss?\n- It is not clear whether the method is limited to type 0 and 1 features or if they were only mentioned for simplicity. Do the layers introduced in the paper work for higher order features ? Is there a reason other than computational effort that higher order features are not used?\n- It should be mentioned that the enforcing scale equivariance on type 1 features was explored in another paper [1].\n- Is this the first instance of biequivariant filters being used to process two inputs simultaneously? If so this should be stated more clearly as it has many interesting possibly applications even if it is more computationally costly. \n\n[1] Yang, Jingyun, et al. \"Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation.\" arXiv preprint arXiv:2310.16050 (2023).", "limitations": "Limitations are discussed in Appendix E. If possible these limitations should be included in the main paper if only briefly.", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- The paper is well-written. Motivating the idea as a learnable extension of Arun’s method was intuitive and easy to follow. \n- The paper introduces a SE(3) biequivariant network, BTIR, to solve point cloud assembly tasks. The network is composed of novel SE(3) biequivariant group convolution layers. \n- The paper provides weight constraints that enforce scale and swap equivariance. An ablation study is performed that demonstrates the added generalization provided by these constraints. \n- The paper includes several PC alignment and assembly experiments where the proposed BTIR network is shown to perform well.", "weaknesses": "- There are several works from the robotic manipulation community that are related to BTIR. These should be discusses in RW and perhaps included as baselines, especially the bowl placing and mug hanging tasks. [1,2]\n- The definition of point cloud assembly in the introduction is a bit imprecise. What does it mean for two non-overlapped point clouds to be “aligned”, when the points themselves will not overlap? Presumably what aligned means depends on who generates the dataset. \n- Some of the writing is inaccurate or unclear. On page 2, “[correspondence based methods] are often sensitive to initial positions of PCs”; most correspondence based approaches use Aruns method which is not sensitive to initial poses. On page 2, “SE(3)-bi-equivariance does not rely on correspondence”; do you mean your BTIR method does not? \n- The results on real data and visual manipulation are of interest and should be included in more detail in the main paper. They are more challenging and would better demonstrate that the proposed method is generally effective. Also, more discussion should be included about these results. For instance, why is it that BTIR without ICP is relatively bad on 7Scenes but better on outdoor scenes of ASL?\n- The experimental results are somewhat limited. More baselines should be included (point cloud alignment is a very well studied problem) and more challenging or varied datasets used. For instance, compare performance across multiple shapenet classes rather than just the wine bottle. Very little space in the paper is currently dedicated to the experiments and discussion. Given BTIR is more computationally heavy than other methods, it should be clear from the experiments that it is robust and effective enough to motivate its use. \n\n[1] Pan, Chuer, et al. \"Tax-pose: Task-specific cross-pose estimation for robot manipulation.\" Conference on Robot Learning. PMLR, 2023.\n[2] Ryu, Hyunwoo, et al. \"Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort. We address the concerns below.\n\n1. Robotic manipulation papers including Diffusion-edfs, Tax-pose and Sim3\n\nYes. Those are related papers. We have now cited them, but as noted in line 889, we are not able to conduct quantitative comparisons, because the correct solution is not unique in this task. For example, for a correct assembly, the bowl is allowed to rotate horizontally in the plate, so $\\Delta r$ can be very large even for the correct assembly. A proper metric is the success rate of the manipulation in physical hardware experiments as in [1,2], but measuring this metric is beyond the scope of this work.\n\nWe have now added the following sentence (bold) in line 88: \"... modelling 3D\ndata, **and recently they have been used for robotic manipulation task [26 , 27, 38]** \". Since the Tax-pose paper does not use equivariant network, we feel that it is more suitable to mention it in line 16 (robotics [27, 20]) in our introduction.  (27 is the Diffusion-edfs paper and 20 is the Tax-pose paper, and 38 is the Sim3 paper)\n\n2. The definition of point cloud assembly... is a bit imprecise. .. Presumably what aligned means depends on who generates the dataset.\n\nYes. Depending on the specific dataset, it can mean reconstructing the shape, robotic manipulation, or even protein binding. We feel the word \"align\" is general enough to cover these meanings, and we provided some examples and citations in Line 16 to make the meanings concrete. \n\n3. ... most correspondence-based approaches use Aruns method which is not sensitive to initial poses. \n\nAlthough Arun's method is robust to initial poses, most of the correspondence-based methods are sensitive because the features they use are not invariant (or equivariant). We mentioned this in line 69 in Sec.2.\n\n4. ...“SE(3)-bi-equivariance does not rely on correspondence”; do you mean your BTIR method does not?\n\nSorry for the confusion. We mean the SE(3)-bi-equivariance prior (the equation in definition 3.1) does not rely on correspondence. We have now made this sentence clearer (the added content is marked bold): \"$SE(3)$-bi-equivariance **prior** does not rely on correspondence, i.e., it can **be used to** handle PCs with no correspondence.\"\n\n5. The results on real data and visual manipulation...should be included in more detail in the main paper... Also, more discussion should be included.. why is it that BTIR without ICP is relatively bad on 7Scenes but better on outdoor scenes of ASL?\n\nWe agree that the experiments are important, but due to the 9-page limit of the main text, we have to place them in the appendix and refer to them in the main text to keep the overall structure complete. \n\nOn the other hand, we have now made it clearer that BITR+OT should be compared with GEO and ROI, because they also include an OT-type refinement process. The result of BITR is used to show that the model can generate results that are close to the optimum (the errors can vary in different datasets). \n\nSpecifically, we re-organize the paragraph starting at line 848 as follows:\n\"We report the results in Tab. 5. We observe that BITR can produce results that are close to the optimum ($\\Delta r \\approx 25$) from a random initialization ($\\Delta r \\in U [0, 180]$), and extra refinements like ICP and OT can further improve the results ($\\Delta r \\approx 10$). This observation is consistent with that in Sec. 6.3.1. In particular, BITR with the OT refinement is comparable with GEO and ROI, which use highly complicated features specifically designed for registration tasks and an OT-like refinement process. On the other hand, ICP and OMN fail in this task due to their sensitivity to initial positions. An example of an assembly result of BITR is presented in Fig. 11.\"\n\n6. ...More baselines should be included ... and more challenging or varied datasets used. For instance, compare performance across multiple shapenet classes... \n\n- As for the registration baselines, we have included the state-of-the-art methods GEO [21] and ROI [39] as two strong baselines. We also considered the classic method ICP [41], and a recent correspondence-free method OMN [36] in Sec 6.5 for completeness. We think the comparisons with these methods are sufficient to support the argument that our method is at least comparable with the existing registration methods.\n\n- On the other hand, as for the dataset, we applied our method to different types of datasets, including indoor (7Scene), outdoor (ASL), objects (ShapeNet), and manipulation. We think these experiments can show the versatility of our method in practice. However, we are not able to train on larger datasets due to high computational cost as discussed in Appx.E. For example, training on one class of ShapeNet takes several days as noted in Appx.D.1, so we do not have enough time to train on multiple classes at this moment. We leave the task of scaling BITR to future research. Probably this can be done with the aid of the techniques mentioned by reviewer fq5D.\n\n7. ...the method is deterministic. How does it solve the wine bottle task in Table 1?\n\nThe input pieces of a bottle are not symmetric (there is no restriction on the complete bottle shape), so our method can be used without difficulty. Actually, as noted in line 910, we do not need to worry about the symmetry problems for the Lidar type data because strict symmetry never exists due to noise. (Symmetry is a problem when modelling molecules, such as a benzene ring, which is strictly symmetric.)\n\n\n\n8. About the degree of features.\n\nOur layer works for higher order features. We did not use higher order features due to efficiency consideration. We will add a sentence below Eqn 12 to make this clearer: \"here we only include degree-0 and degree-1 features, and higher degree features can be used similarily\"\n\n9. Is this the first instance of bi-equivariant filters being used to process two inputs simultaneously? \n\nYes. We have stated it in line 92.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "uZeA6ogXTD", "reviewer": "Reviewer_1793", "summary": "This paper presents an end-to-end framework for pointcloud correspondence and relative pose estimation with bi-equivariance to per-part poses. The framework is also equivariant to scaling and swapping part orders. It also designes a transformer architecture with $\\mathrm{SE}(3)\\times\\mathrm{SE}(3)$-equivariance. The effectiveness of the framework is shown with experiments on the standard ShapeNet benchmark as well as some other applications.", "questions": "I don't have questions about the method. But I would really like to see experimental comparisons with the pointcloud registration works.", "limitations": "Limitations are well-discussed in the paper.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "- The problem studied in this paper is well-suited for equivariance, and the paper provides a decent problem formulation with clear explanations.\n- The network architecture in the paper is well-designed, with the bi-equivariant transformer, and the module with the classic pose matching algorithm.\n- The experiments are extensive in the sense that they not only show standard comparisons on ShapeNet, but also have more studies with different setups and other applications.", "weaknesses": "- To my understanding, \"shape assembly\" usually refers to assembling parts of a shape that exactly matches their boundaries (like jigsaw), and usually with more than just two parts. I feel \"shape registration\" is a more proper description of the task in the paper.\n- Following my previous comment, I think the method should also be compared to prior works on shape registration, including classic optimization-based methods like ICP, and more recent learning-based methods such as [1, 2] below.\n\t- A minor point: to compare to these works and show the advantages brought by equivariance, it may be a good setting to compare their data efficiency. I believe that with less training data or less pose augmentation, the proposed framework can have less performance drop than these non-equivariant prior works.\n- [3] is also a related work, especially to the robot manipulation task in Sec. 6.6 (and Suppl. Sec. D.9).\n\n[1] Wang, Y., & Solomon, J. M. (2019). Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3523-3532).\n\n[2] Wang, Y., & Solomon, J. M. (2019). Prnet: Self-supervised learning for partial-to-partial registration. Advances in neural information processing systems, 32.\n\n[3] Ryu, H., Kim, J., An, H., Chang, J., Seo, J., Kim, T., ... & Horowitz, R. (2024). Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18007-18018).", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the time and effort. We address the concerns below.\n\n\n1.  I feel \"shape registration\" is a more proper description of the task in the paper.\n\nWe use the word \"assembly\" following [7], where two pieces of point cloud are matched together. We avoid using the word \"registration\" because we are afraid this word will cause some confusion: registration is a classic task which seeks to align overlapped point clouds using correspondence, so aligning non-overlapped point clouds seems not well-defined for this task.\n\n\n2.  Should be compared to prior works on shape registration, including classic optimization-based methods like ICP, and more recent learning-based methods such as [1, 2] below.\n\nWe have already included two state-of-the-art registration methods [21] and [39] as baselines (they are more recent than the papers mentioned by the reviewer). These two methods are learning based and have an optimization-type finetuning process. ICP is not compared in the main text because it generally fails when the pose is randomly initialized (e.g. when the initial rotation error is larger than 45 degrees). We have included ICP in table 5 and table 6 in the appendix for clarity.\n\n3.  with less training data or less pose augmentation, the proposed framework can have less performance drop than these non-equivariant prior works...\n\nWe showed that one important advantage of our method brought by equivariance is that it can be used when correspondence does not exist (Fig.4 when $s < 0.5$). Of course, you are correct about the data efficiency, because we do not use pose augmentations as noted in line 296 (they have no influence on our method), but they are necessary for learning-based methods like [21]. \n\n4. [3] is also a related work, especially to the robot manipulation task in Sec. 6.6 (and Suppl. Sec. D.9).\n\nYes, this is a related paper. We have now cite this paper, but we are not able to conduct quantitative comparisons, because the correct solution is not unique in this task (as noted in line 889). For example, for a correct assembly, the bowl is allowed to rotate horizontally in the plate, so $\\Delta r$ can be very large even for the correct assembly. A proper metric is the success rate of the manipulation in physical hardware experiments as in [3], but measuring this metric is beyond the scope of this work.", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: The paper introduces a bi-equivariant transformer model to solve the task of pointcloud assembly, where two pointclouds with no overlapping regions need to be registered. Although most of the reviewers provided an initial positive assessment of the work, one reviewer recommended rejecting the paper on the grounds that the method is not equivariant. After a careful read of the paper and authors' response, we concluded that this is due to a misunderstanding of Eq.10 of the paper by the reviewer and the method is in fact equivariant. Therefore, although several reviewers raised the concern of an evaluation focused only on small datasets or missing baselines, we will follow the recommendation of the other reviewers and recommend the paper for acceptance. However, in the final version of the paper, the authors should include the feedback received from reviewers and improve clarity in the method's description (especially regarding the equivariant nature of the method and why Eq.10 makes the method equivariant) and the experiments section.
Decision: Accept (poster)