Title: SE(3)-bi-equivariant Transformers for Point Cloud Assembly
Venue: NeurIPS 2024
Accepted: yes
Keywords: equivariant neural networks

SE(3)-bi-equivariant transformer

point cloud assembly
Abstract: Given a pair of point clouds, the goal of assembly is to recover a rigid transformation that aligns one point cloud to the other. This task is challenging because the point clouds may be non-overlapped, and they may have arbitrary initial positions. To address these difficulties, we propose a method, called $SE(3)$-bi-equivariant transformer (BITR), based on the $SE(3)$-bi-equivariance prior of the task:it guarantees that when the inputs are rigidly perturbed, the output will transform accordingly.   Due to its equivariance property, BITR can not only handle non-overlapped PCs, but also  guarantee robustness against initial positions. Specifically, BITR first extracts features of the inputs using a novel $SE(3) \times SE(3)$-transformer, and then projects the learned feature to group $SE(3)$ as the output. Moreover, we theoretically show that swap and scale equivariances can be incorporated into BITR, thus it further guarantees stable performance under scaling and swapping the inputs. We experimentally show the effectiveness of BITR in practical tasks.

SE(3)-bi-equivariant Transformers for Point Cloud Assembly

Abstract

Given a pair of point clouds, the goal of assembly is to recover a rigid transformation that aligns one point cloud to the other. This task is challenging because the point clouds may be non-overlapped, and they may have arbitrary initial positions. To address these difficulties, we propose a method, called SE (3) -bi-equivariant transformer (BITR), based on the SE (3) -bi-equivariance prior of the task: it guarantees that when the inputs are rigidly perturbed, the output will transform accordingly. Due to its equivariance property, BITR can not only handle nonoverlapped PCs, but also guarantee robustness against initial positions. Specifically, BITR first extracts features of the inputs using a novel SE (3) × SE (3) -transformer, and then projects the learned feature to group SE (3) as the output. Moreover, we theoretically show that swap and scale equivariances can be incorporated into BITR, thus it further guarantees stable performance under scaling and swapping the inputs. We experimentally show the effectiveness of BITR in practical tasks.

1 Introduction

Point cloud (PC) assembly is a fundamental machine learning task with a wide range of applications such as biology [12], archeology [36], robotics [28, 20] and computer vision [23]. As shown in Fig. 1, given a pair of 3-D PCs representing two shapes, i.e ., a source and a reference PC, the goal of assembly is to find a rigid transformation, so that the transformed source PC is aligned to the reference PC. This task is challenging because the input PCs may have random initial positions that are far from the optimum, and may be non-overlapped, e.g ., due to occlusion or erosion of the object.

(a) Assembly of overlapped PCs

(b) Assembly of non-overlapped PCs

InputPCs

InputPCs

Specifically, the proposed BITR is an end-toend trainable model consisting of two steps: it first extracts SO (3) × SO (3) -equivariant features from the input PCs, and then obtains a rigid transformation by projecting the features into SE (3) . For the first step, we define a SE (3) × SE (3) -transformer acting on the 6-D merged PC by extending the SE (3) -transformer [11]; For the second step, we use a SVD -type projection inspired by Arun's method [2]. In addition, we theoretically show that scale-equivariance and swap-equivariance can be incorporated into BITR via weight constraining techniques, which further guarantees that the performance is not influenced by scaling or swapping inputs. An illustration of BITR is presented in Fig. 2.

Pointcloudmerge

Featureextraction

X

(M × 3)

(L×3)

↑

(3×3)

④

Z←

→tx(3)

r (3 ×3)

t (3)

i(9×7)

ty (3)

Y

Y

(N × 3)

(L×3)

SE(3)-Transformer

SE(3)x SE(3)-Transformer

SE(3)-Projection

In summary, the contribution of this work is as follows:

-We present a SE (3) -bi-equivariant PC assembly method, called BITR. 1 BITR can assemble PCs without correspondence, and guarantees stable performance with arbitrary initial positions. In addition, the SE (3) × SE (3) -transformer used in BITR is the first SE (3) × SE (3) -equivariant steerable network to the best of our knowledge.

-Theoretically, we show that scale and swap equivariance can be incorporated in to BITR by weight-constraining, thus it guarantees stable performance under scaling and swapping the inputs.

-We show experimentally that BITR can effectively assemble PCs in practical tasks.

2 Related works

A special case of PC assembly is PC registration, where the correspondence between input PCs is assumed to exist. A seminal work in this task was conducted by [2], which provided a closed-form solution to the problem with known correspondence. To handle PCs with unknown correspondence, most of the subsequent works extend [2] by first estimating the correspondence by comparing distances [4], or features [25, 23, 41] of the PCs, and then aligning the PCs by aligning the estimated corresponding points. Notably, to obtain SE (3) -bi-equivariance, SO (3) -invariant features [42, 10, 43] have been investigated for correspondence estimation. However, since these methods require a sufficient number of correspondences, they have difficulty handling PCs where the correspondence does not exist. In addition, they often have difficulty handling PCs with large initial errors [45].

The proposed BITR is related to the existing registration methods because it can be seen as a generalization of Arun's method [2]. However, in contrast to these methods, BITR is correspondencefree, i.e ., it is capable of handling PCs with no correspondence. In addition, it theoretically guarantees stable performance under arbitrary initial position.

Recently, some works have been devoted to a new PC assembly task, called fragment reassembly, whose goal is to reconstruct a complete shape from two fragments. Unlike registration task, this task generally does not assume the existence of correspondence. [7] first studied this task, and they addressed it as a pose estimation problem, where the pose of each fragment relative to a canonical pose is estimated via a regression model. [38] further improved this method by considering the SE (3) -equivariance of each fragment. In addition, [31] proposed a simulated dataset for this task. In contrast to these methods, the proposed BITR does not rely on the canonical pose, i.e ., it directly estimates the relative pose. As a result, BITR is conceptually simpler and it can handle the shape whose canonical pose is unknown.

Another related research direction is equivariant networks. Due to their ability to incorporate 3D rotation and translation symmetry priors, these networks have been extensively used in modelling 3D data [29, 9, 37, 17, 13], and recently they have been used for robotic manipulation task [27, 28, 40]. In particular, [34] proposed a tensor field network (TFN) for PC processing, and SE (3) -transformer [11] further improved TFN by introducing the attention mechanism. On the other hand, the theory of equivariant networks was developed in [16, 5]. BITR follows this line of research because the SE (3) × SE (3) -transformer used in BITR is a direct generalization of SE (3) -transformer [11], and it is the first SE (3) × SE (3) -equivariant steerable network to the best of our knowledge.

3 Preliminaries

This section briefly reviews Arun's method and the concept of equivariance, which will be used in BITR.

3.1 Group representation and equivariance

Given a group G , its representation is a group homomorphism ρ : G → GL ( V ) , where V is a linear space. When G is the 3D rotation group SO (3) , it is convenient to consider its irreps (irreducible orthogonal representation) ρ p : SO (3) → GL ( V p ) , where p ∈ N is the degree of the irreps, and dim ( V p ) = 2 p + 1 . For r ∈ G , ρ p ( r ) ∈ R (2 p +1) × (2 p +1) is known as the Wigner-D matrix. For example, ρ 0 ( r ) = 1 for all r ∈ SO (3) ; ρ 1 ( r ) ∈ R 3 × 3 is the rotation matrix of r . More details can be found in [5] and the references therein.

In this work, we focus on the group G of two independent rotations, i.e ., G = SO (3) × SO (3) , where × represents the direct product. Similar to SO (3) , we also consider the irreps of G . A useful fact is that all irreps of G can be written as the combinations of the irreps of SO (3) : the degree-( p, q ) irreps of G is ρ p,q = ρ p ⊗ ρ q : SO (3) × SO (3) → GL ( V p ⊗ V q ) , where p, q ∈ N , ρ p and ρ q are irreps of SO (3) , and ⊗ is tensor product (Kronecker product for matrix). For example, ρ 0 , 0 ( r 1 × r 2 ) = 1 ∈ R ; ρ 1 , 0 ( r 1 × r 2 ) ∈ R 3 × 3 is the rotation matrix of r 1 ; ρ 1 , 1 ( r 1 × r 2 ) = ρ 1 ( r 1 ) ⊗ ρ 1 ( r 2 ) ∈ R 9 × 9 is the Kronecker product of the rotation matrices of r 1 and r 2 .

Given two representations ρ : G → GL ( V ) and τ : G → GL ( W ) , a map Φ : V → W satisfying Φ( ρ ( g ) x ) = τ ( g )Φ( x ) for all g ∈ G and x ∈ V is called G -equivariant. When Φ is parametrized by a neural network, we call Φ an equivariant neural network, and we call the feature extracted by Φ equivariant feature. Specifically, a degreep equivariant feature transforms according to ρ p under the action of SO (3) , and a degree-( p, q ) equivariant feature transforms according to ρ p ⊗ ρ q under the action of SO (3) × SO (3) . For simpler notations, we omit the representation homomorphism ρ , i.e ., we write r instead of ρ ( r ) , when ρ is clear from the text.

3.2 Arun's method

Consider a PC assembly problem with known one-to-one correspondence : Let Y = { y u } N u =1 ⊆ R 3 and X = { x u } N u =1 ⊆ R 3 be a pair of PCs consisting of N points, and let { ( x u , y u ) } N u =1 be their corresponding point pairs. What is the optimal rigid transformation that aligns X to Y ?

[2] provided a closed-form solution to this problem. It claims that the optimal solution g = ( r, t ) ∈ SE (3) in terms of mean square errors is

where is the correlation matrix, m ( · ) represents the mean value, ¯ x i = x i -m ( X ) and ¯ y i = y i -m ( Y ) represent the centralized points, and SVD ( · ) represents the singular value decomposition projection. The definition of SVD projection can be found in Def. C.5.

Arun's solution enjoys rich equivariance properties. Formally, we have the following proposition:

Definition 3.1. Consider a map Φ : S × S → SE (3) where S is the set of 3D PCs. Given X,Y ∈ S , let g = Φ( X,Y ) .

-Φ is SE (3) -bi-equivariant if Φ( g 1 X,g 2 Y ) = g 2 gg -1 1 , ∀ g 1 , g 2 ∈ SE (3) .

-Φ is swap-equivariant if Φ( Y, X ) = g -1 .

-Φ is scale-equivariant if Φ( cX,cY ) = ( r, ct ) , ∀ c ∈ R + .

Proposition 3.2. Under a mild assumption (C.2), Arun's algorithm (1) is SE (3) -bi-equivariant, swap-equivariant and scale-equivariant.

In other words, Arun's method guarantees to perform consistently 1) with arbitrary rigid perturbations on X and Y , i.e ., it is global, 2) if X and Y are swapped (aligning Y to the fixed X or aligning X to the fixed Y ), and 3) in arbitrary scale. Details of Prop. 3.2 can be found at Appx. C.1.

We regard Arun's method as a prototype of SE (3) -bi-equivariant PC assembly methods: it first extracts a degree(1 , 1) SO (3) × SO (3) -equivariant translation-invariant feature, i.e ., the correlation matrix ¯ r (2), and then obtains an output g ∈ SE (3) using a SVD -based projection. This observation immediately leads to more general SE (3) -bi-equivariant methods where the handcrafted feature ¯ r is replaced by the more expressive learned equivariant features, thus, the correspondence is no longer necessary. We will develop this idea in the proposed BITR in the next section, and further show that the scale and swap equivariance of Arun's method can also be inherited.

4 SE (3) -bi-equivariant transformer

This section presents the details of the proposed BITR. BITR follows the same principle as Arun's method [2]: it first extracts SO (3) × SO (3) -equivariant features as a generalization of the correlation matrix ¯ r (2), and then projects the features to SE (3) similarly to (1). Specifically, we first propose a SE (3) × SE (3) -transformer for feature extraction in Sec. 4.2. Since this transformer is defined on 6-D space, i.e ., it does not directly handle the given 3-D PCs, it relies on a pre-processing step described in Sec. 4.3, where the input 3-D PCs are merged into a 6-D PC. Finally, the Arun-type SE (3) -projection is presented in Sec. 4.4. An overview of BITR is presented in Fig. 2.

4.1 Problem formulation

Let Y = { y v } N v =1 ⊆ R 3 and X = { x u } M u =1 ⊆ R 3 be the PCs sampled from the reference and source shape respectively. The goal of assembly is to find a rigid transformation g ∈ SE (3) , so that the transformed PC gX = { rx i + t } i M =1 is aligned to Y . Note that we do not assume that X and Y are overlapped, i.e ., we do not assume the existence of corresponding point pairs.

4.2 SE (3) × SE (3) -transformer

To learn SO (3) × SO (3) -equivariant translation-invariant features generalizing ¯ r (1), this subsection proposes a SE (3) × SE (3) -transformer as a generalization of SE (3) -transformer [11]. We present a brief introduction to SE (3) -transformer [11] in Appx. A for completeness.

According to the theories developed in [5], to define a SE (3) × SE (3) -equivariant transformer, we first need to define the feature map of a transformer layer as a tensor field, and specify the action

of SE (3) × SE (3) on the field. Since SE (3) × SE (3) can be decomposed as ( T (3) × T (3)) glyph[multicloseright] ( SO (3) × SO (3)) where T (3) is the 3-D translation group, the tensor field should be defined in the 6 -D Euclidean space R 6 ∼ = T (3) × T (3) , and the features attached to each location should be SO (3) × SO (3) -equivariant and T (3) × T (3) -invariant. Formally, we define the tensor field as

where Z = { z u } L u =1 ⊆ R 6 is a 6-D PC, δ is the Dirac delta function, f u = ⊕ p,q f p,q u is the feature attached to z u , where f p,q u is the degree-( p, q ) equivariant component. We then specify the action of SE (3) × SE (3) on the base space R 6 as

where z = z 1 ⊕ z 2 , z 1 , z 2 ∈ R 3 are the first and last three components of z , ⊕ represents direct sum (concatenate), and g i = ( r i , t i ) ∈ SE (3) for i = 1 , 2 . Thus, the action of SE (3) × SE (3) on the degree-( p, q ) component of f is

With the above preparations, we can now define a SE (3) × SE (3) -transformer layer in a message passing formulation similar to SE (3) -transformer:

Here, we use notations o = ( o 1 , o 2 ) and i = ( i 1 , i 2 ) for simplicity. For example, f o represents the degreeo feature f o 1 ,o 2 . F o ( z u ) ∈ R c × (2 o 1 +1)(2 o 2 +1) is the collection of all degreeo features at z u , where c is the number of channels of the degreeo features, and W o ∈ R 1 × c is a learnable parameter for self-interaction. KNN ( · ) represents the k nearest neighborhood, and attention α uv is computed according to

where Q , K and V are known as the query, key and value respectively. They are defined as

where z vu = z v -z u , W i Q is a learnable parameter for Q , and the convolutional kernel W o , i ( z ) in V and K both take the form of

where the learnable radial component ϕ o , i J 1 ,J 2 : R × R → R is parametrized by a neural network, the non-learnable angular component is determined by the 2-nd order Clebsch-Gordan constant Q and the spherical harmonics Y J : R 3 → R 2 J +1 , and vec ( · ) is the vectorize function reshaping a matrix to a vector. Formulation (8) is derived in Appx. B.

Note that the kernel (8) is the main difference between a SE (3) -transformer layer and a SE (3) × SE (3) -transformer layer. In the special case where only SE (3) -equivariance is considered, i.e ., all features are of degree ( p, 0) (or (0 , q ) ), a SE (3) × SE (3) -transformer layer becomes a SE (3) -transformer layer.

Finally, we adopt the equivariant Relu (Elu) layer similar to [9] as the point-wise non-linear layer in our network. Given an input degreei feature F i with c channels, an Elu layer is defined as

where F µ = W i µ F i and F ν = W i ν F i . W µ , W ν ∈ R c × c are learnable weights and ‖ · ‖ is the channel-wise vector norm. Note that when i = (1 , 0) or i = (0 , 1) , our definition (9) becomes the same as [9]. By interleaving transformer layers and Elu layers, we can finally build a complete SE (3) × SE (3) -equivariant transformer model.

4.3 Point cloud merge

To utilize the transformer model defined in Sec. 4.2, we need to construct a 6-D PC as its input. To this end, we first extract key points from the raw 3-D PCs, and then concatenate them to a 6-D PC to merge their information. Thus, the resulting 6-D PC is not only small in size but also contains the key information of the raw PCs pairs.

Formally, we extract L ordered key points ˜ X = { ˜ x u } L u =1 and ˜ Y = { ˜ y u } L u =1 from X and Y respectively, and then obtain Z = { ˜ x u ⊕ ˜ y u } L u =1 . Note that we do not require ˜ X ( ˜ Y ) to be a subset of X ( Y ). Specifically, we represent the coordinates of the key points as a convex combination of the raw PCs:

where X ∈ R M × 3 and Y ∈ R N × 3 represent the coordinates of X and Y respectively, and SoftMax ( · ) represents the row-wise softmax. F 0 X ∈ R L × M and F 0 Y ∈ R L × N are the weights of each point in X and Y respectively, and they are degree0 , i.e ., rotation-invariant, features computed by a shared SE (3) -transformer Φ E :

Furthermore, inspired by [39], we fuse the features of X and Y in Φ E before the last layer, so that their information is merged more effectively, i.e ., the selection of ˜ X or ˜ Y depends on both X and Y . Specifically, the fused features are

where we only consider degree0 and degree1 features. f · ,X and f · ,Y represent the features of X and Y , and Pool is the average pooling over the PC.

Note that ˜ X and ˜ Y obtained in Eqn. 10 are permutation invariant. For example, according to Eqn. 10, the i -th key point of X is ˜ x i = ∑ j F ij X j , where F ij is the i -th channel of F 0 X at x j (after softmax normalization). When X is permutated by σ , then the i -th key point can be written as ˜ x ′ i = ∑ j ′ F ij ′ X j ′ , where j ′ = σ ( j ) . It is easy to see that ˜ x ′ i = ˜ x i because both j and j ′ iterate through { 1 , ..., M } in the summation.

4.4 SE(3)-projection

We now obtain the final output by projecting the feature extracted by the SE (3) × SE (3) -transformer to SE (3) . Formally, let f be the output tensor field of the SE (3) × SE (3) -transformer. We compute the final output g = ( r, t ) ∈ SE (3) using an Arun-type projection as follows:

where ˆ r = unvec (˜ r ) ∈ R 3 × 3 , t X ∈ R 3 and t Y ∈ R 3 are equivariant features computed as

We note that projection (13) extends Arun's projection (1) in two aspects. First, although ˆ r in (13) and ¯ r (2) are both degree(1 , 1) features, ˆ r is more flexible than ¯ r because ˆ r is a learned feature while ¯ r is handcrafted, and ˆ r is correspondence-free while ¯ r is correspondence-based. Second, projection (13) explicitly considers non-zero offsets t X and t Y , which allow solutions where the centers of PCs do not match.

In summary, BITR computes the output g for PCs X and Y according to

where Φ S : S × S → F is a SE (3) × SE (3) -transformer (with the PC merge step), Φ P : F → SE (3) represents projection (13), and F is the set of tensor field. We finish this section with a proposition that BITR is indeed SE (3) -bi-equivariant.

Proposition 4.1. Under a mild assumption (C.2), BITR (14) is SE (3) -bi-equivariant.

5 Swap-equivariance and scale-equivariance

This section seeks to incorporate swap and scale equivariances into the proposed BITR model. These two equivariances are discussed in Sec. 5.1 and Sec. 5.2 respectively.

5.1 Incorporating swap-equivariance

This subsection seeks to incorporate swap-equivariance to BITR, i.e ., to ensure that swapping X and Y has the correct influence on the output. To this end, we need to treat the group of swap as Z / 2 Z = { 1 , s } where s 2 = 1 , i.e ., s represents the swap of X and Y , and properly define the action of Z / 2 Z on the learned features.

Formally, we define the action of Z / 2 Z on field f (3) as follows. We first define the action of s on the base space R 6 as swapping the coordinates of ˜ X and ˜ Y : s ( z ) = z 2 ⊕ z 1 , where z = z 1 ⊕ z 2 , and z 1 , z 2 ∈ R 3 are the coordinates of ˜ X and ˜ Y respectively. Then we define the action of s on feature f as ( s ( f ) ) p,q ( z ) = ( f q,p ( s ( z )) ) T , where we regard a degree-( p, q ) feature f p,q u as a matrix of shape R (2 p +1) × (2 q +1) by abuse of notation, and ( · ) T represents matrix transpose.

Intuitively, according to the above definition, degree(1 , 1) , (1 , 0) and (0 , 1) features will become (the transpose of) degree(1 , 1) , (0 , 1) and (1 , 0) features respectively under the action of s , i.e ., ˆ r will be transposed, t X and t Y will be swapped. This is exactly the transformation needed to ensure swap-equivariant outputs. We formally state this observation in the following proposition.

Proposition 5.1. For a tensor field f and a projection Φ P (13), Φ P ( s ( f )) = (Φ P ( f )) -1 .

Now the remaining problem is how to make a SE (3) × SE (3) -transformer Z / 2 Z -equivariant. A natural solution is to force all layers in the SE (3) × SE (3) -transformer to be Z / 2 Z -equivariant. The following proposition provides a concrete way to achieve this.

Proposition 5.2. Let ˜ · represent the swap of index, e.g ., if o = ( o 1 , o 2 ) , then ˜ o = ( o 2 , o 1 ) . 1) For a transformer layer (5), if the self-interaction weight satisfies W o = W ˜ o , the weight of query (7) satisfies W o Q = W ˜ o Q , and the radial function satisfies ϕ i , o J 1 ,J 2 ( ‖ z 1 ‖ , ‖ z 2 ‖ ) = ϕ ˜ i , ˜ o J 2 ,J 1 ( ‖ z 2 ‖ , ‖ z 1 ‖ ) for all i , o , J 1 , J 2 , z 1 and z 2 , then the transformer layer is Z / 2 Z -equivariant.

2) For an Elu layer (9), if W i ν = W ˜ i ν and W i µ = W ˜ i µ for all i , then the Elu layer is Z / 2 Z -equivariant.

More details, including the complete matching property (Prop. C.11), can be found in Appx. C.3.1.

5.2 Incorporating scale-equivariance

This subsection seeks to incorporate scale equivariance to BITR, i.e ., to ensure that when X and Y are multiplied by a scale constant c ∈ R + , the output result transforms correctly. To this end, we need to consider the scale group ( R + , × ) , i.e ., the multiplicative group of R + , and properly define the ( R + , × ) -equivariance of the learned feature. For simplicity, we abbreviate group ( R + , × ) as R + .

We now consider the action of R + on field f (3). We call f a degreep R + -equivariant field ( p ∈ N ) if it transforms as ( c ( f )) ( z ) = c p f ( c -1 z ) under the action of R + , where z ∈ R 6 and c ∈ R + . We immediately observe that degree1 R + -equivariant features lead to scale-equivariant output. Intuitively, if ˜ r , t X and t Y are degree1 R + -equivariant features, then they will become c ˜ r , ct X and ct Y under the action of c , and the projection step will cancel the scale of ˜ r while keeping the scale of t X and t Y , which is exactly the desirable results. Formally, we have the following proposition.

Proposition 5.3. Let Φ P be projection (13), f be a degree1 R + -equivariant tensor field, and ( r, t ) = Φ P ( f ) . We have Φ P ( c ( f )) = ( r, ct ) ∀ c ∈ R + .

The remaining problem is how to ensure that a SE (3) × SE (3) -transformer is R + -equivariant and its output is of degree1 , so that scaling the input can lead to the proper scaling of output. Here we provide a solution based on the following proposition.

Definition 5.4. ϕ : R × R → R is a degreep function if ϕ ( cx, cy ) = c p ϕ ( x, y ) for all c ∈ R + .

Proposition 5.5. 1) Denote ϕ K and ϕ V the radial functions used in K and V respectively. Let ϕ K be a degree0 function, f in be a degree0 R + -equivariant input field. For transformer layer (5), if ϕ V

is a degree1 function and the self-interaction weight W = 0 , then the output field f out is degree1 R + -equivariant; If ϕ V is a degree0 function, then the output field f out is degree0 R + -equivariant.

2) For Elu layer (9), if the input field is degreep R + -equivariant, then the output field is also degreep R + -equivariant.

More discussions can be found in Appx. C.4.

6 Experiments and analysis

This section experimentally evaluates the proposed BITR. After describing the experiment settings in Sec. 6.1, we first present a simple example in Sec. 6.2 to highlight the equivariance of BITR. Then we evaluate BITR on assembling the shapes in ShapeNet [6], BB dataset [31], 7Scenes [32] and ASL [22] from Sec. 6.3.1 to Sec. 6.4. We finally apply BITR to visual manipulation tasks in Sec. 6.6.

6.1 Experiment settings

We extract L = 32 key points for each PC. The SE (3) -transformer and the SE (3) × SE (3) -transformer both contain 2 layers with c = 4 channels. We consider k = 24 nearest neighborhoods for message passing. We only consider low degree equivariant features, i.e ., p, q ∈ { 0 , 1 } for efficiency. We train BITR using Adam optimizer [15] with learning rate 1 e -4 . We use the loss function L = ‖ r T r gt -I ‖ 2 2 + ‖ t gt -t ‖ 2 2 , where ( r, t ) are the output transformation, ( r gt , t gt ) are the corresponding ground truth. We evaluate all methods by isotropic rotation and translation errors: ∆ r = (180 /π ) accos ( 1 / 2 ( tr ( rr T gt ) -1) )) , and ∆ t = ‖ t gt -t ‖ where tr ( · ) is the trace of a matrix. We do not use random rotation and translation augmentations as [23]. More details are in Appx. D.1.

6.2 A proof-of-concept example

To demonstrate the equivariance property of BITR, we train BITR on the bunny shape [33]. We prepare the dataset similar to [41]: In each training iteration, we first construct the raw PC S by uniformly sampling 2048 points from the bunny shape and adding 200 random outliers from [ -1 , 1] 3 , then we obtain PCs { X P , Y P } by dividing S into two parts of ratio (30% , 70%) using a random plane P . We train BITR to reconstruct S using randomly rotated and translated { X P , Y P } . To construct the test set, we generate a new sample { X ˜ P , Y ˜ P } , and additionally construct 3 test samples by 1) swapping, 2) scaling (factor 2 ) and 3) randomly rigidly perturbing { X ˜ P , Y ˜ P } .

The assembly results of BITR on these four test samples are shown in Fig. 3. We observe that BITR performs equally well in all cases. Specifically, the differences between the rotation errors in these four cases are small (less than 1 e -3 ). The results suggest that BITR is indeed robust against these three perturbations, which verifies its swap-equivariance, scale-equivariance and SE (3) -biequivariance. More experiments can be found in the appendix: a numerical verification of Def. 3.1 is presented in Appx. D.2, an ablation study of swap and scale equivariances are presented in Appx. D.3, and the verification of the complete-matching property C.11 is presented in Appx. D.4.

(a) Original

(∆

r

= 11

.

275)

(b) Swapped

(∆

r

= 11

.

275)

(c) Rigidly perturbed

(∆

r

= 11

.

276)

(d) Scaled

(∆

r

= 11

.

276)

6.3 Results on ShapeNet

6.3.1 Single shape assembly

In this experiment, we evaluate BITR on assembling PCs sampled from a single shape. When the inputs PCs are overlapped, this setting is generally known as PC registration. We construct a dataset similar to [41]: for a shape in the airplane class of ShapeNet [6], we obtain each of the input PCs by uniformly sampling 1024 points from the shape, and keep ratio s of the raw PC by cropping it using a random plane. We vary s from 0 . 7 to 0 . 3 . Note the PCs may be non-overlapped when s < 0 . 5 .

We compare BITR against the state-of-the-art registration methods GEO [23] and ROI [43], and the state-of-the-art fragment reassembly methods NSM [7] and LEV [38]. For NSM and LEV, we additionally provide the canonical pose for each shape. Note that LEV and ROI are SE (3) -equivariant methods. For s ≥ 0 . 5 , we report the results of BITR fine-tuned by ICP [45] (BITR+ICP). Note that BITR+ICP is SE (3) -bi-equivariant and scale-equivariant, but not swap-equivariant.

We present the results in Fig. 4. We observe that the performance of all methods decrease as s decreases. Meanwhile, BITR outperforms all baseline methods when s is small ( s ≤ 0 . 5 ). On the other hand, when s is large ( s > 0 . 5 ), BITR performs worse than GEO, but it still outperforms other baselines. Nevertheless, since the results of BITR are sufficiently close to optimum ( ∆ r ≤ 20 ), the ICP refinement can lead to improved results that are close to GEO. More details can be found in Appx. D.5.

6.3.2 Inter-class assembly

To evaluate BITR on non-overlapped PCs, we extend the experiment in Sec. 6.3.1 to inter-class assembly. We train BITR to place a car shape on the right of motorbike shape, so that their directions are the same and their distance is 1 . We consider s = 1 . 0 and 0 . 7 . Note that this task is beyond the scope of registration methods, since the input PCs are non-overlapped. A result of BITR is shown in Fig. 5. More details can be found in Appx. D.6.

6.4 Results on fragment reassembly

This subsection evaluates BITR on a fragment reassembly task. We compare BITR against NSM [7], LEV [38] and DGL [44] on the 2 -fragment WineBottle class of the BB dataset [31]. The data preprocessing step is described in Appx. D.7.

We test the trained BITR 3 times using different random samples, and report the mean and standard deviation of (∆ r, ∆ t ) in Tab. 1. We observe that BITR outperforms all baseline methods: BITR achieves the lowest rotation errors, and its translation error is comparable to DGL, which is lower than other baselines by a large margin. We provide some qualitative comparisons in Appx. D.7.

6.5 Results on real data

This subsection evaluates BITR on an indoor dataset 7Scenes [32] and the outdoor scenes in ASL dataset [22]. We present the results on 7Scenes in this section, and leave the results on ASL and some qualitative results to Appx. D.8.

For the 7Scenes dataset, we arbitrarily rotate and translate all frames, and train BITR to align all adjacent frames. We use the data from the first 6 scenes as the training set, and the data from 7 -th

09

ROI

GEO

0.4

ROI

GEO

LEV*

LEV*

40

NSM*

BITR

BITR+ICP

NSM*

BITR

BITR+ICP

20

0

0.7

0.6

0.5

0.4

0.3

0.0

0.7

0.6

0.5

0.4

0.3

PC sizeratio

PC size ratio

scene as the test set. To train BITR, we use a random clipping augmentation similar to Sec. 6.3.1: we keep ratio s of each PCs by clipping them using a random plane, where s is uniformly distributed in [0 . 5 , 1 . 0] . We compare BITR against GEO [23], ROI [43], ICP [45] and OMN [39], where OMN is a recently proposed correspondence-free registration method.

The results on 7Scenes are reported in Tab. 2. We observe that BITR can produce results that are close to the optimum ( ∆ r ≈ 25 ) from a random initialization ( ∆ r ∈ U [0 , 180] ), and extra refinements like ICP and OT can further improve the results ( ∆ r ≈ 10 ). This observation is consistent with that in Sec. 6.3.1. In particular, BITR with the OT refinement is comparable with GEO and ROI, which use highly complicated features specifically designed for registration tasks and an OT-like refinement process. On the other hand, ICP and OMN fails in this task due to their sensitiveness to initial positions.

6.6 Results on visual manipulation

This subsection investigates the potential of BITR in manipulation tasks. Following [26], we consider two tasks: mug-hanging and bowl-placing. For both tasks, X represents an object grasped by a robotic arm, i.e ., a cup or a bowl, Y represents the fixed environment with a target, i.e ., a stand or a plate, and we train BITR to align X to Y , so that the cup can be hung to the stand, or the bowl can be placed on the plate.

Fig. 6 presents the results of BITR on bowl-placing. We observe that although BITR is not originally designed for manipulation tasks, it can place the bowl in a reasonable position relative to the plate. However, we also notice that BITR may produce unrealistic results, e.g ., the PCs may collide. Thus, post-processing steps [30] or extra regularizers [13] may be necessary in practical applications. More results and discussions can be found in Appx. D.9.

(a) A success case of bowl-placing

(b) A failure case of bowl-placing

7 Conclusion

This work proposed a PC assembly method, called BITR. The most distinguished feature of BITR is that it is correspondence-free, SE (3) -bi-equivariant, scale-equivariant and swap-equivariant. We experimentally demonstrated the effectiveness of BITR.

BITR in its current form has two main limitations. First, BITR is computationally inefficient because each degree of feature is computed independently without parallel. This issue was also observed in SE(3)-equivariant networks, and was recently addressed by [21]. A promising future research direction is to develope similar acceleration techniques for BITR. Second, since BITR is deterministic, i.e ., it only predicts one result for a given input, it cannot handle symmetric PCs. Although this feature does not cause any difficulty in this work (there is no strictly symmetric PCs in this work due to noise, random sampling, etc), it may be problematic in future applications such as molecule modelling where symmetric PCs exist, e.g ., benzene rings. To address this issue, we plan to generalize BITR to a generative model in the future. More discussions can be found in Appx. E.

Acknowledgments

This work is funded by the Swedish Research Council through grant agreement no. 2019-03686 and Chalmers AI Research Initiative (CHAIR). The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS), partially funded by the Swedish Research Council through grant agreement no. 2022-06725. The authors would like to thank Jan Gerken at Chalmers and Nan Xue at Ant Group for useful discussions, and thank the anonymous reviewers for their insightful comments and suggestions.

References

Abien Fred Agarap. Deep learning using rectified linear units (relu). arXiv preprint arXiv:1803.08375 , 2018.

K Somani Arun, Thomas S Huang, and Steven D Blostein. Least-squares fitting of two 3-d point sets. IEEE Transactions on pattern analysis and machine intelligence , (5):698-700, 1987.

Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. Layer normalization. arXiv preprint arXiv:1607.06450 , 2016.

Paul J Besl and Neil D McKay. Method for registration of 3-d shapes. In Sensor fusion IV: control paradigms and data structures , volume 1611, pages 586-606. Spie, 1992.

Gabriele Cesa, Leon Lang, and Maurice Weiler. A program to build e (n)-equivariant steerable cnns. In International Conference on Learning Representations , 2022.

Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report arXiv:1512.03012 [cs.GR], Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.

Yun-Chun Chen, Haoda Li, Dylan Turpin, Alec Jacobson, and Animesh Garg. Neural shape mating: Self-supervised object assembly with adversarial shape priors. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 12724-12733, 2022.

Erwin Coumans and Yunfei Bai. Pybullet, a python module for physics simulation for games, robotics and machine learning. 2016.

Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J Guibas. Vector neurons: A general framework for so (3)-equivariant networks. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 12200-12209, 2021.

Haowen Deng, Tolga Birdal, and Slobodan Ilic. Ppf-foldnet: Unsupervised learning of rotation invariant 3d local descriptors. In Proceedings of the European conference on computer vision (ECCV) , pages 602-618, 2018.

Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se(3)-transformers: 3d rototranslation equivariant attention networks. Advances in neural information processing systems , 33:1970-1981, 2020.

Pablo Gainza, Sarah Wehrle, Alexandra Van Hall-Beauvais, Anthony Marchand, Andreas Scheck, Zander Harteveld, Stephen Buckley, Dongchun Ni, Shuguang Tan, Freyr Sverrisson, et al. De novo design of protein interactions with learned surface fingerprints. Nature , pages 1-9, 2023.

Octavian-Eugen Ganea, Xinyuan Huang, Charlotte Bunne, Yatao Bian, Regina Barzilay, Tommi Jaakkola, and Andreas Krause. Independent se (3)-equivariant models for end-to-end rigid protein docking. arXiv preprint arXiv:2111.07786 , 2021.

Zan Gojcic, Caifa Zhou, Jan D Wegner, Leonidas J Guibas, and Tolga Birdal. Learning multiview 3d point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1759-1769, 2020.

Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Leon Lang and Maurice Weiler. A wigner-eckart theorem for group equivariant convolution kernels. In International Conference on Learning Representations , 2021.

Cheng-Wei Lin, Tung-I Chen, Hsin-Ying Lee, Wen-Chin Chen, and Winston H Hsu. Coarse-tofine point cloud registration with se (3)-equivariant representations. In 2023 IEEE international conference on robotics and automation (ICRA) , pages 2833-2840. IEEE, 2023.

Jonathan Long, Evan Shelhamer, and Trevor Darrell. Fully convolutional networks for semantic segmentation. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 3431-3440, 2015.

Benjamin Midtvedt, Jesús Pineda, Fredrik Skärberg, Erik Olsén, Harshith Bachimanchi, Emelie Wesén, Elin K Esbjörner, Erik Selander, Fredrik Höök, Daniel Midtvedt, et al. Single-shot self-supervised object detection in microscopy. Nature communications , 13(1):7492, 2022.

Chuer Pan, Brian Okorn, Harry Zhang, Ben Eisner, and David Held. Tax-pose: Task-specific cross-pose estimation for robot manipulation. In Conference on Robot Learning , pages 17831792. PMLR, 2023.

Saro Passaro and C Lawrence Zitnick. Reducing so (3) convolutions to so (2) for efficient equivariant gnns. In International Conference on Machine Learning , pages 27420-27438. PMLR, 2023.

François Pomerleau, Ming Liu, Francis Colas, and Roland Siegwart. Challenging data sets for point cloud registration algorithms. The International Journal of Robotics Research , 31(14):1705-1711, 2012.

Zheng Qin, Hao Yu, Changjian Wang, Yulan Guo, Yuxing Peng, and Kai Xu. Geometric transformer for fast and robust point cloud registration. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11143-11152, 2022.

rusty1s. pytorchscatter. 2023.

Radu Bogdan Rusu, Nico Blodow, and Michael Beetz. Fast point feature histograms (fpfh) for 3d registration. In 2009 IEEE international conference on robotics and automation , pages 3212-3217. IEEE, 2009.

Hyunwoo Ryu, Hong in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: Se(3)-equivariant energy-based models for end-to-end visual robotic manipulation learning. In The Eleventh International Conference on Learning Representations , 2023.

Hyunwoo Ryu, Jiwoo Kim, Hyunseok An, Junwoo Chang, Joohwan Seo, Taehan Kim, Yubin Kim, Chaewon Hwang, Jongeun Choi, and Roberto Horowitz. Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 18007-18018, 2024.

Hyunwoo Ryu, Hong-in Lee, Jeong-Hoon Lee, and Jongeun Choi. Equivariant descriptor fields: Se (3)-equivariant energy-based models for end-to-end visual robotic manipulation learning. arXiv preprint arXiv:2206.08321 , 2022.

Vıctor Garcia Satorras, Emiel Hoogeboom, and Max Welling. E (n) equivariant graph neural networks. In International conference on machine learning , pages 9323-9332. PMLR, 2021.

Johannes Schauer and Andreas Nüchter. Collision detection between point clouds using an efficient kd tree implementation. Advanced Engineering Informatics , 29(3):440-458, 2015.

Silvia Sellán, Yun-Chun Chen, Ziyi Wu, Animesh Garg, and Alec Jacobson. Breaking bad: A dataset for geometric fracture and reassembly. Advances in Neural Information Processing Systems , 35:38885-38898, 2022.

Jamie Shotton, Ben Glocker, Christopher Zach, Shahram Izadi, Antonio Criminisi, and Andrew Fitzgibbon. Scene coordinate regression forests for camera relocalization in rgb-d images. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 2930-2937, 2013.

Standford. The stanford 3d scanning repository. [Online]. Available: http://graphics.stanford.edu/data/3Dscanrep/ .

Nathaniel Thomas, Tess Smidt, Steven Kearnes, Lusann Yang, Li Li, Kai Kohlhoff, and Patrick Riley. Tensor field networks: Rotation-and translation-equivariant neural networks for 3d point clouds. arXiv preprint arXiv:1802.08219 , 2018.

Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Łukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in neural information processing systems , 30, 2017.

Ariana M. Villegas-Suarez, Cristian Lopez, and Ivan Sipiran. Matchmakernet: Enabling fragment matching for cultural heritage analysis. In Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV) Workshops , pages 1632-1641, October 2023.

Maurice Weiler, Mario Geiger, Max Welling, Wouter Boomsma, and Taco S Cohen. 3d steerable cnns: Learning rotationally equivariant features in volumetric data. Advances in Neural Information Processing Systems , 31, 2018.

Ruihai Wu, Chenrui Tie, Yushi Du, Yan Zhao, and Hao Dong. Leveraging se (3) equivariance for learning 3d geometric shape assembly. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 14311-14320, 2023.

Hao Xu, Shuaicheng Liu, Guangfu Wang, Guanghui Liu, and Bing Zeng. Omnet: Learning overlapping mask for partial-to-partial point cloud registration. In Proceedings of the IEEE/CVF International Conference on Computer Vision , pages 3132-3141, 2021.

Jingyun Yang, Congyue Deng, Jimmy Wu, Rika Antonova, Leonidas Guibas, and Jeannette Bohg. Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation. arXiv preprint arXiv:2310.16050 , 2023.

Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 11824-11833, 2020.

Hao Yu, Ji Hou, Zheng Qin, Mahdi Saleh, Ivan Shugurov, Kai Wang, Benjamin Busam, and Slobodan Ilic. Riga: Rotation-invariant and globally-aware descriptors for point cloud registration. IEEE Transactions on Pattern Analysis and Machine Intelligence , 2024.

Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic. Rotation-invariant transformer for point cloud matching. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5384-5393, 2023.

Guanqi Zhan, Qingnan Fan, Kaichun Mo, Lin Shao, Baoquan Chen, Leonidas J Guibas, Hao Dong, et al. Generative 3d part assembly via dynamic graph learning. Advances in Neural Information Processing Systems , 33:6315-6326, 2020.

Zhengyou Zhang. Iterative point matching for registration of free-form curves and surfaces. International Journal of Computer Vision , 13(2):119-152, 1994.

Qian-Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3D: A modern library for 3D data processing. arXiv:1801.09847 , 2018.

A SE(3)-equivariant Transformers

A well-known SE (3) -equivariant network is SE (3) -transformer [11], which adapts the powerful transformer structure [35] to SO (3) -equivariant settings. In this model, the feature map f of each layer is defined as a tensor field supported on a 3-D PC:

where δ is the Dirac function, X = { x u } M u =1 ⊆ R 3 is a point set, and f u is the feature attached to x u . Here, feature f u takes the form of f u = ⊕ p f p u , where the component f p u ∈ V p is the degreep feature, i.e ., it transforms according to ρ p under the action of SO (3) . For example, when f u represents the norm vector of a point cloud, then f u = f 1 u ∈ R 3 . We also write the collection of all degreep features at x u as F p ( x u ) ∈ R c × (2 p +1) , where c is the number of channels.

For each transformer layer, the degreek output feature at point x i is computed by performing message passing:

where N ( u ) represents the neighborhood of u , W l ∈ R 1 × c is the learnable weight for self-interaction, c represents the number of channels, and

is the attention from v to u . Here, key K , value V and query Q are

where W l Q ∈ R 1 × (2 l +1) is a learnable weight, and the kernel W lk ( x ) ∈ R (2 l +1) × (2 k +1) is defined as

where vec ( · ) is the vectorize function, the learnable radial component ϕ lk J : R + → R is parametrized by a neural network, and the non-learnable angular component is determined by Clebsch-Gordan constant Q and the spherical harmonic Y J : R 3 → R 2 J +1 .

B Derive of the Convolutional Kernel

To derive the kernel (8) for a SE (3) × SE (3) -transformer layer, we consider the equivariant convolution as a simplified version of the SE (3) × SE (3) -transformer layer:

i.e ., we only consider the message V o , i uv while fixing the self-interaction weight W o = 0 and attention α uv = 1 in (5). To ensure the SE (3) × SE (3) -equivariance of convolution (20), i.e .,

the kernel W must satisfy a constraint:

where we abbreviate r 1 × r 2 as r 12 , abbreviate vec ( W o , i ) ∈ R (2 i 1 +1)(2 i 2 +1)(2 o 1 +1)(2 o 2 +1) as W , and assume ‖ z 1 ‖ = ‖ z 2 ‖ = 1 for simpler notations. Equation (22) is generally known as the kernel constraint, and its necessity and sufficiency can be proved in a verbatim way as Theorem 2 in [37],

so we omit the proof here. Now we can obtain the kernel (8) by solving this constraint. Note that a direct formulation is given in Theorem 2.1 in [5], but here we derive it in a less abstract way.

We first observe that W ∗ ( z ) = Y i 1 ( z 1 ) ⊗ Y i 2 ( z 2 ) ⊗ Y o 1 ( z 1 ) ⊗ Y o 2 ( z 2 ) is a special solution to equation (22), where Y J ( · ) ∈ R 2 J +1 is the column vector consisting of degreeJ harmonics with order m = -J, ... 0 , ..., J as each row element, because

Then we point out that { Y m 1 J 1 ( z 1 ) Y m 2 J 2 ( z 2 ) } m 1 ,m 2 ,J 2 ,J 1 is an orthogonal and complete basis for two variable square-integrable functions L 2 ( X,Y ) : S 2 × S 2 → R , where S 2 is the 2-D sphere. Thus, we can write each component of W ∗ ( z ) as a linear combination of this basis. Specifically, let V ( z ) = ⊕ J 1 ,J 2 Y J 1 ( z 1 ) ⊗ Y J 2 ( z 2 ) , and let ( m 1 , m 2 , m 3 , m 4 ) be the index of the element Y m 1 i 1 ( z 1 ) Y m 2 i 2 ( z 2 ) Y m 3 o 1 ( z 1 ) Y m 4 o 2 ( z 2 ) in W ∗ , and ( J 1 , m 5 , J 2 , m 6 ) be the index of the element Y m 5 J 1 ( z 1 ) Y m 6 J 2 ( z 2 ) in V . We have W ∗ = P V , where

Here, c ( J 1 ) and c ( J 2 ) are coefficients related to J 1 and J 2 respectively. 〈 i, m 1 , o, m 3 | J, m 5 〉 is known as the Clebsch-Gordan coefficient, the product 〈 i 1 , m 1 , o 1 , m 3 | J 1 , m 5 〉〈 i 2 , m 2 , o 2 , m 4 | J 2 , m 6 〉 is known as the 2-nd order Clebsch-Gordan coefficient, and we represent it as Q . In other words, we have P = Q c , where c is a block diagonal matrix, and each block is c ( J 1 ) c ( J 2 ) I . Therefore, we have

Since

for arbitrary z , we further have

by equating the above two equations. Finally, we stack this decomposition back to the kernel constraint (22), and obtain

where c ′ is the coefficient matrix of the same shape as c , and each coefficient is arbitrary. Specifically,

We note that c ′ J 1 ,J 2 is parametrized by a neural network in our model (8).

which suggests that

C Proofs and theoretical results

C.1 The proof of the equivariance of Arun's method

We first establish the result on the uniqueness of Arun's method. We begin with the result of the uniqueness of singular vectors.

glyph[negationslash]

i.e ., the eigenvalues of A T A are σ 2 i , i = 1 , 2 , 3 , and V i are the corresponding eigenvectors. Since σ j is distinct and all σ i ≥ 0 , σ 2 j is distinct. As a result, the eigenspace corresponding to σ 2 j has dim 1 , i.e ., it is spanned by V j . Since V j is a unit vector, it can be determined up to a sign. In addition, we have AV j = σ j U j , thus U j can also be determined up to a sign. U j V T J is unique since flipping the sign of V j always leads to the flipping of sign of U j , and vice versa.

If σ j = 0 , we can still repeat the above argument to determine V j up to a sign, and determine U j up to a sign using the above argument for AA T .

Note that in Arun's algorithm, we use the SVD projection defined as follows.

where A = ˆ U ˆ Σ ˆ V T , ˆ U, ˆ V ∈ O (3) is the SVD decomposition of A . In other words, if det ( ˆ U ˆ V T ) = 1 , then we take ˆ U ˆ V T , otherwise we flip the sign of ˆ U 3 ˆ V 3 . An important observation is that this SVD projection is unique under a mild assumption.

Assumption C.2. In the SVD decomposition, σ 3 is distinct, i.e ., σ 1 ≥ σ 2 > σ 3 ≥ 0 .

Proposition C.3 (Uniqueness of SVD projection) . Let A = U Σ V T ∈ R 3 × 3 be the SVD decomposition, where Σ = diag ( σ 1 , σ 2 , σ 3 ) is a diagonal matrix, and U, V ∈ O (3) . If assumption C.2 is satisfied, then SVD ( A ) is unique.

Proof. Let A = ˜ U ˜ Σ ˜ V T be another SVD decomposition of A , i.e .,

the goal is to prove

We first show that U 1 V T 1 + U 2 V T 2 = ˜ U 1 ˜ V T 1 + ˜ U 2 ˜ V T 2 .

1) If σ 3 = 0 and σ 1 = σ 2 , then

2) If σ 3 = 0 and σ 1 > σ 2 , then σ 1 and σ 2 are distinct and nonzero. According to lemma C.1, U 1 V T 1 = ˜ U 1 ˜ V T 1 and U 2 V T 2 = ˜ U 2 ˜ V T 2 , thus the summation of these two terms is equal.

3) If σ 3 > 0 and σ 1 > σ 2 , the argument is the same as 2)

4) If σ 3 > 0 and σ 1 = σ 2 , then σ 3 is distinct and nonzero. According to lemma C.1, U 3 V T 3 = ˜ U 3 ˜ V 3 T . Thus,

Thus we have U 1 V T 1 + U 2 V T 2 = ˜ U 1 ˜ V T 1 + ˜ U 2 ˜ V T 2 .

Lemma C.1. Let A = U Σ V T ∈ R 3 × 3 be the SVD decomposition. If a singular value σ j is distinct, then the corresponding singular vectors U j and V j can be determined up to a sign. If σ j = 0 , U j V T J is unique.

glyph[negationslash]

Proof. If σ j = 0 , we have

Now we prove that the last term in Eqn. (33) is equal. We have ˜ U 3 ˜ V T 3 = ± U 3 V T 3 , and we have shown that U 1 V T 1 + U 2 V T 2 = ˜ U 1 ˜ V T 1 + ˜ U 2 ˜ V T 2 , thus we have

By multiplying this term with ˜ U 3 ˜ V T 3 = ± U 3 V T 3 , we have

which suggests that last term in Eqn. (33) is equal. In summary, we have proved (33).

Finally, we can prove the uniqueness of Arun's method under the same assumption.

Proposition C.4 (Uniqueness of Arun's method) . Under assumption C.2, Arun's method (1) is unique.

Proof. According to Prop. C.3, r = SVD (¯ r ) is unique. As a result, t = m ( Y ) -r m ( X ) is also unique.

Throughout this work, we assume that assumption C.2 holds, so that the uniqueness of Arun's method and BITR can be guaranteed.

For the rest of this appendix, we use a simpler notation of SVD projection, where we simply absorb the sign matrix into ˆ U and ˆ V :

thus obtain the following equivalent definition.

Definition C.5. The SVD projection of matrix A is defined as

where A = U Σ V T and U, V ∈ SO (3) .

Finally, we can discuss the equivariance of Arun's method.

Lemma C.6. Let

be the SVD projection of A , i.e ., A can be decomposed as A = U Σ V T , Σ is a diagonal matrix and U, V ∈ SO (3) . We have

for arbitrary A ∈ GL (3) , r 1 , r 2 ∈ SO (3) , and c ∈ R + .

The Proof of Lemma. C.6. We have

where r 2 U ∈ SO (3) , r 1 V ∈ SO (3) and Σ is a diagonal matrix. In other words, (46) is a SVD decomposition of matrix r 2 Ar T 1 , thus the SVD projection can be computed as

which proves the first part of this lemma. We omit the other two statements of this lemma since they can be proved similarly.

The Proof of Prop. 3.2. Let Φ A be Arun's method and Φ A ( X,Y ) = ( r ( X,Y ) , t ( X,Y )) = g . We directly verify these three equivariances according to Def. 3.1. We only prove the SE (3) -bi-equivariance and swap-equivariance and omit the proof of scale-equivariance, because it can be proved similarity.

1) SE (3) -bi-equivariance We compute Φ A ( g 1 X,g 2 Y ) for the perturbed PCs g 1 X = { r 1 x i + t 1 } N i =1 and g 2 Y = { r 2 y i + t 2 } N i =1 as follows. We first compute ¯ r ( g 1 X,g 2 Y ) as

Then we compute r via SVD :

where the 3-rd equality holds due to Lemma. C.6. Since the mean values are also perturbed as m ( g 1 X ) = g 1 m ( X ) and m ( g 2 Y ) = g 2 m ( Y ) , we compute t ( g 1 X,g 2 Y ) as

In summary,

which proves the SE (3) -bi-equivariance of Arun's method.

2) swap-equivariance We compute Φ A ( Y, X ) as follows. We first compute ¯ r ( Y, X ) as

Then we compute r ( Y, X ) via SVD :

We can finally compute t ( Y, X ) as

In summary,

which proves the swap-equivariance of Arun's method.

C.2 The proof of the equivariance of BITR

Before proving the SE (3) -bi-equivariance of BITR, we first verify that each layer in a SE (3) × SE (3) -transformer is indeed SE (3) × SE (3) -equivariant. Intuitively, the equivariance of a transformer layer (5) is a result of the kernel design, i.e ., the kernel constraint (22), and the invariance of the attention. We state this result in the following lemma for completeness, and we note that similar techniques are also used in the construction of SE (3) -transformer [11].

Lemma C.7. The transformer layer (5) is SE (3) × SE (3) -equivariant.

Proof. We abbreviate r 1 × r 2 as r 12 , and g 1 × g 2 as g 12 . Let L be a transformer layer (5), we seek to prove ( g 12 )( L ( f )) = L (( g 12 )( f )) for the input tensor field f and g 1 , g 2 ∈ SE (3) . We compute the

RHS of the equation as

Note that here we have g 12 F o ( z ) = F o ( g -1 12 z )( ρ o ( r 12 )) T because we write F in the channel-first form, i.e ., the shape of F o is c × (2 o 1 +1)(2 o 2 +1) . The LHS of the equation is computed as

We now verify that these 3 terms are equal respectively.

C = C ′ : By design, the kernel W satisfies the kernel constraint (22)

In addition, let z ′ u = g -1 12 z u . We have

Thus, we can obtain the equality by combining the above two equations.

B = B ′ : We compute Q and K for B and B ′ respectively. We have

and

where the last equation holds due to the kernel constraint (22) and the orthogonality of ρ o ( r 12 ) . Specifically, ρ o ( r 12 ) = ρ o 1 ( r 1 ) ⊗ ρ o 2 ( r 2 ) is an orthogonal matrix because ρ o 1 and ρ o 2 are orthogonal representations, i.e ., ρ o 1 ( r 1 ) and ρ o 2 ( r 2 ) are orthogonal matrices.

Thus, we have

Finally, A = A ′ is trivial. In summary, we have shown ( g 12 )( L ( f )) = L (( g 12 )( f )) , which proves the SE (3) × SE (3) -equivariance of the transformer layer (5).

On the other hand, an Elu layer (5) is also SE (3) × SE (3) -equivariant, because this layer is piecewise linear. We state this statement formally in the following lemma, and omit the proof. We note that a similar argument was used in [9].

Lemma C.8. The Elu layer (9) is SE (3) × SE (3) -equivariant.

Now we can prove the SE (3) -bi-equivariance of BITR.

The Proof of Prop. 4.1. Let g = Φ P ◦ Φ S ( X,Y ) , and f out = Φ S ( X,Y ) . We prove this proposition by showing

We compute the RHS of the equation as follows. First, since each point in ˜ X and ˜ Y is a convex combination of X and Y respectively, we have

Then we have Z ( g 1 X,g 2 Y ) = ( g 1 × g 2 ) Z ( X,Y ) because Z = ˜ X ⊕ ˜ Y . When Z ( g 1 X,g 2 Y ) is fed to the SE (3) × SE (3) -transformer, the output feature will be ( g 1 × g 2 ) f out by design (This is verified in Lemma. C.7 and Lemma. C.8). In particular, for degree(1 , 1) feature ˜ r , degree(1 , 0) feature t X and degree(0 , 1) feature t Y , we have

Therefore, we have ˆ r ( g 1 X,g 2 Y ) = r 2 ˜ r ( X,Y ) r -1 1 by applying unvec ( · ) to the first equation, i.e ., ˆ r = unvec (˜ r ) . Finally, we compute projection Φ P similar to proof C.1:

and

In summary,

which proves the SE (3)

We finally prove Prop. 5.1 and Prop. 5.3 similarly to Prop. 3.2.

The Proof of Prop. 5.1. We compute Φ P ( s ( f )) as follows. First, we have

by definition. Then we compute the equivariant features as

where we treat t X and t Y as vectors. Finally, the output can be computed as

according to lemma C.6, and

In other words, Φ P ( s ( f )) = ( r T , -r T t ) = g -1 , which proves this proposition.

The Proof of Prop. 5.3. We compute Φ P ( c ( f )) as follows. First, since f is a degree1 R + -equivariant tensor field, f satisfies c ( f )( z ) = cf ( c -1 z ) . Therefore, we have

by definition. Then, we compute the output as

according to lemma C.6, and

In other words, Φ P ( c ( f )) = ( r, ct ) , which proves this proposition.

C.3 More results of Sec. 5.1

C.3.1 The proof of Prop. 5.2

In this subsection, we use slightly different notations than other parts of the paper. We regard the kernel W o , i as a 4-D tensor of shape (2 o 1 +1) × (2 o 2 +1) × (2 i 1 +1) × (2 i 2 +1) , and we regard the feature f i as a 2-D tensor of shape (2 i 1 +1) × (2 i 2 +1) , i.e ., a matrix, when it is multiplied by a kernel. Therefore, the multiplication W o , i f i is treated as a tensor product where all two dimensions of f i are treated as rows, i.e ., the result of W o , i f i is of shape (2 o 1 +1) × (2 o 2 +1) . Similarly, we regard the collection of features F o as a 3-D tensor of shape c × (2 o 1 +1) × (2 o 2 +1) . When it is multiplied by a self-interaction weight W o ∈ R 1 × c , the result is W o F o ∈ R (2 o 1 +1) × (2 o 2 +1) .

We first make the following observation on the symmetry of the kernel.

Lemma C.9. If the radial function ϕ satisfies ϕ i , o J 1 ,J 2 ( ‖ z 1 ‖ , ‖ z 2 ‖ ) = ϕ ˜ i , ˜ o J 2 ,J 1 ( ‖ z 2 ‖ , ‖ z 1 ‖ ) for all z 1 , z 2 , J 1 , J 2 , then kernel W (5) satisfies

where T ij represents the transpose of the i -th and j -th dimension of a tensor.

Proof. According to (5) and the discussion in Sec. B, the ( m 3 , m 4 , m 1 , m 2 ) -th element of W o , i ( z ) is

and the ( m 4 , m 3 , m 2 , m 1 ) -th element of W ˜ o , ˜ i ( s ( z )) is

Since the angular components in (69) and (70) are the same, and the radial components are equal by assumption, we immediately conclude that (69) and (70) are equal. In other words,

which proves this lemma.

Then we proceed to the proof of Prop. 5.2.

The Proof of Prop. 5.2. Let L be a transformer layer (5), we seek to prove s ( L ( f )) = L ( s ( f )) for the input tensor field f .

To this end, we expand the RHS of the equation as

where I ( f ) is the set of all degrees of field f , and supp ( f ) is the support of field f . The last equation holds because we replace z v by s ( z v ) , and replace i by ˜ i . We expand the LHS of the equation as

We now verify that these three terms are equal respectively.

A = A ′ : By assumption, W o = W ˜ o , thus we immediately have

B = B ′ : Since α = 〈 Q , K 〉 , we compute Q and K for B and B ′ respectively. As for Q , we have

where the last equation holds because W o Q = W ˜ o Q by assumption. In addition,

As for K , we have

where the last equation holds due to the constraint of radial function and Lemma. C.9, and we replace i by ˜ i . In addition,

We observe that Q ( s ( f ) , z u , s ( z v )) is just the transpose and re-ordering of each component of Q ( f, s ( z u ) , z v ) , and this is also true for K ( s ( f ) , z u , s ( z v )) and K ( f, s ( z u ) , z v ) , which suggests that their inner products are the same. Specifically, let Q o = W o Q F o ( s ( z u )) and K o = ∑ i ∈ I ( f ) W o , i K ( z v -s ( z u )) f i ( z v ) . We have

and

which suggests that α ( f, s ( z u ) , z v ) = α ( s ( f ) , z u , s ( z v )) .

for all i , o , J 1 , J 2 , z 1 and z 2 , thus W o , i ( z ) = ( W ˜ o , ˜ i ( s ( z )) ) 12 34 according to Lemma. C.9. As a result,

In summary, we have shown the Z / 2 Z -equivariance of a transformer layer (5), which is the first part of Prop. 5.2. We omit the proof of the second part of this proposition, i.e ., the equivariance of an Elu layer, as it is a simple extension of the proof of A = A ′ .

Remark C.10 . The weight sharing technique in Prop. 5.2 reduces the number of learnable parameters of a SE (3) × SE (3) -transformer by about half, thus it makes the model more efficient. In practice, the weight sharing technique can be achieved by sharing weights between different o , i and J directly, except for the symmetric case, i.e ., o = ˜ o , i = ˜ i and J 1 = J 2 , where the requirement of the radial function becomes

We represent the radial function in this case as ϕ ( x, y ) = 1 2 ( φ ( x, y ) + φ ( y, x )) , where φ is a neural network, which guarantees the symmetry of ϕ .

C.3.2 The complete-matching property

Consider the following complete matching problem.

The complete matching problem Given a pair of PCs X and Y , where Y is generated by a unknown rigidly transformation of X , i.e ., Y = gX and g ∈ SE (3) is unknown, how to infer g ?

The complete matching problem is a prototype of the registration problem, i.e ., in the simplest case (no outlier, no noise, no partial visibility..), how to derive the relative transformation between two fully overlapped PCs? Despite its simplicity, this problem is non-trivial, because we do not know the correspondence between X and Y , i.e ., Arun's method does not apply.

Asurprising fact is that g can be exactly recovered using a SE (3) -bi-equivariant and swap-equivariant assembly method. We state this fact formally as follows.

Proposition C.11 (Complete-matching property) . Let Φ be a swap-equivariant and SE (3) -biequivariant assembly method. Then

for arbitrary PC X and g 1 ∈ SE (3) .

Proof. First, since Φ is swap-equivariant, then Φ( X,X )Φ( X,X ) = I . Then, since Φ is SE (3) -biequivariant, we have Φ( X,g 1 X )Φ( X,X ) = g 1 Φ( X,X )Φ( X,X ) = g 1 .

According to the main text, BITR is swap-equivariance (Prop. 5.2), thus, we can derive a concrete algorithm, called untrained BITR (U-BITR), for the complete matching problem:

where Φ B is BITR model. Prop. C.11 suggests that Φ U solves the complete matching problem without training. We numerically evaluate this property in Appx. D.4, and leave the theoretical analysis of U-BITR to future research.

C.4 The proof of Prop. 5.5

First, we have the following straightforward lemma for the kernel.

Lemma C.12. If the radial function ϕ : R × R → R is a degreep function, then kernel W (5) satisfies

Then we directly verify Prop. 5.5.

The proof of Prop. 5.5. Let L be a transformer layer (5), f be a degree0 R + -equivariant input field, ϕ K be a degree0 function. We seek to prove: 1) If ϕ V is a degree0 function, then c ( L ( f )) = L ( c ( f )) and L ( f ) is degree0 R + -equivariant; 2) If ϕ V is a degree1 function and W = 0 , then c ( L ( f )) = L ( c ( f )) and L ( f ) is degree1 R + -equivariant.

To begin with, we expand L ( c ( f )) as

and expand c ( L ( f )) as

We first point out that B = B ′ , because

where the second equation holds because ϕ K is a degree0 function by assumption, and Lemma. C.12 claims that the corresponding kernel W K is scale-invariant. In addition,

In other words, both Q and K are scale-invariant. Thus, we conclude that the attention is also scale-invariant:

Now we discuss these two situations:

1) If ϕ V is a degree1 function and W = 0 , then

according to Lemma. C.12, and A = A ′ = 0 . In other words, L ( c ( f ))( z ) = c 1 L ( f )( c -1 z ) , where the RHS is the action of c on the degree1 R + -equivariant field, thus we have L ( c ( f )) = c ( L ( f )) , and L ( f ) is degree1 R + -equivariant.

2) If ϕ V is a degree0 function, then

according to Lemma. C.12. Thus, L ( c ( f ))( z ) = L ( f )( c -1 z ) , where the RHS is the action of c on the degree0 R + -equivariant field, thus we have L ( c ( f )) = c ( L ( f )) , and L ( f ) is degree0 R + -equivariant.

In summary, we have shown that the equivariance of the transformer layer. We omit the proof of the equivariance of the Elu layer as it can be proved similarly.

We can now construct a special structure of BITR for producing degree1 R + -equivariant output:

In other words, all tensor fields are degree0 R + -equivariant, except for the final output which is degree1 R + -equivariant. More specifically, we consider two types of layers, i.e ., deg0 → deg0 and deg0 → deg1 .

Prop. 5.5 suggests that a transformer layer with degreep radial function can generate a degreep R + -equivariant field when the input field is degree0 R + -equivariant (Elu layers do not influence the R + -equivariance). Therefore, we only need to consider degree0 and degree1 radial functions for our purpose. In practice, we represent degree1 functions using neural networks consisting of linear and relu [1] layers, and we represent degree0 functions using linear, relu and layer normalization [3] layers. Specifically, We represent degree0 functions as neural network φ 0 :

and represent degree 1 -functions neural network φ 1 :

Remark C.13 . There are two major difficulties in developing a general scale-equivariance theory for BITR. First, the attention term is not scale-invariant due to the existence of the soft-max operation, i.e ., SoftMax ( A ) = SoftMax ( cA ) for vector A and c ∈ R + . That is the reason why we only accept degree0 R + -equivariant input in our current theory. Second, when p / ∈ { 0 , 1 } , we are not aware of any general way to represent degreep functions using neural networks.

glyph[negationslash]

D More experiment results

D.1 More training details

We run all experiments using a Nvidia T4 GPU card with 16 G memory. The batch size is set to the largest possible value that can be fitted into the GPU memory. We set bs = 16 for the airplane dataset, and bs = 4 for the wine bottle dataset. We train BITR until the validation loss does not decrease. For the airplane dataset, we train BITR 10000 epochs, and the training time is about 8 days when s = 0 . 7 . For the wine bottle dataset, we train BITR 1000 epochs, and the training time is about 12 hours. The FLOPS is 14 . 5 G in a forward pass (including the computation of harmonic functions), and the model contains 0 . 17 M parameters.

For Sec. 6.3.1 and 6.3.2, we use the normal vector computed by Open3D [46] as the input feature of BITR. The airplane dataset used in Sec. 6.3.1 contains 715 random training samples and 103 random test samples. The wine bottle dataset used in Sec. 6.4 contains 331 training and 41 test samples. We adopt the few-shot learning setting in the manipulation tasks in Sec. 6.6: we use 30 training and 5 test samples for mug-hanging; we use 40 training samples and 10 test samples for bowl-placing.

D.2 More results of Sec. 6.2

We quantitatively verify the equivariance of BITR according to Def. 3.1. Specifically, we compute

to verify the SE (3) -bi-equivariance, swap-equivariance and scale-equivariance of BITR, where Φ B represent the BITR model, ( r B ( · ) , t B ( · )) = Φ B ( · ) are the output of BITR, and all g = ( r, t ) ∈ SE (3) are written as

Note that if BITR is perfectly equivariant, these three errors should always be 0 .

The quantitative results of the experiment are summarized in Tab. 3, where we can see that all errors are below the numerical precision of float numbers, i.e ., less than 1 e -5 . The results suggest that BITR is indeed SE (3) -bi-equivariant, swap-equivariant and scale-equivariant.

D.3 Ablation study

To show the practical effectiveness of our theory on scale and swap equivariances, we consider an ablation study. We use the same data as in Sec. 6.2, and remove the weight sharing technique in Sec. 5.1 to break swap-equivariance, and force ϕ V in all layers to be a degree1 function to break the scale-equivariance.

We evaluate the trained model on 100 test samples, and report the mean and standard deviation of ∆ r in Tab. 4, where we can see that removing an equivariance of the model leads to the failure in the corresponding test case, which is consistent with our theory.

D.4 Evaluation of the complete-matching property

This experiment numerically evaluates the robustness of the complete-matching property (Prop. C.11) against resampling, noise and partial visibility. We first sample X and Y of size 1024 from the bunny shape, and a random g ∈ SE (3) , then we use a random initialized U-BITR to match X to gY . We consider different settings: 1) X and Y are exactly the same; 2) X and Y are different random samples; 3) Gaussian noise of std 0 . 01 is added to X and Y ; 4) Ratio s of X and Y is kept by cropping using a random plane.

We repeat the experiment 3 times, and report the results of U-BITR in Tab. 5. We observe that the transformation is perfectly recovered when X = Y , which is consistent with the complete-matching property. Meanwhile, cropping the PCs leads to large decrease of the accuracy, while noise and resampling have less effect. This is consistent with our expectation because cropping the PCs has larger effect on the shape of PCs. We provide qualitative results in Fig. 7.

(a) Complete-matching

(

∆

r

= 0

)

(b) Noisy data (

∆

r

= 2

.

8

)

(c) Resampled

data

(

∆

r

= 20

.

1

)

D.5 More results of Sec. 6.3.1

We report the training process of BITR in Fig. 8, where we can see that the loss value, ∆ r and ∆ t gradually decrease during training as expected.

Some qualitative results of BITR are presented in Fig. 9. We represent the input PCs using light colors, and represent the 32 learned key points using dark colors and large points. As we explained in Sec. 4.3, the key points are in the convex hull of the input PCs, and they are NOT a subset of the input PC. In addition, as can be seen, the key points of the inputs do not overlap.

D.6 More results of Sec. 6.3.2

We generate the raw training samples by sampling motorbike and car shapes from the training set of ShapeNet. Then we centralize them, and move the car by [0 , 0 , 1] . Note the shapes in ShapeNet are already pre-aligned. The test samples are generated from the test set of ShapeNet in the same way.

We repeat the test process 3 times, and report the results in Fig. 10. We observe that BITR achieves lower rotation error than LEV, and their translation errors are comparable. Meanwhile, NSM fails in this experiment. Note that we do not report the results of registration methods, because their loss functions are undefined due to the lack of correspondence.

0.5

80

0.6

0.4

0.5

60

0.3

0.4

40

0.2

0.3

0.1

20

0.2

0

2000

4000

6000

8000

10000

0

2000

4000

6000

8000

10000

0

2000

4000

6000

8000

10000

Epoch

Epoch

Epoch

(a)

s

= 0

.

7

(b)

s

= 0

.

5

(c)

s

= 0

.

4

(d)

s

= 0

.

3

120

LEV*

1.0

LEV*

NSM*

NSM*

BITR

BITR

0.8

80

0.6

40

0.4

0

1.0

0.7

1.0

0.7

PC size ratio

PC sizeratio

D.7 More results of Sec. 6.4

For BITR, we first obtain raw PCs by applying grid sampling with grid size 0 . 005 to the shape, and then randomly sample 5% points from the raw PCs as the training and test samples. The sizes of the resulting PCs are around 1000 , which is close to the data used in the baseline methods. The data is pre-processed following [38] for the baseline methods.

The random sampling process in our method causes the randomness of test error. We quantify the randomness by evaluating on the test set 3 times, and report the mean and std of the errors in Tab. 1.

Fig. 11 presents 5 examples of reassembling wine bottle fragments, where we observe that the proposed BITR can reassemble most of the shapes correctly, while the baseline methods generally have difficulty predicting the correct rotations.

D.8 More results of Sec. 6.5

We preprocess the 7Scenes dataset by applying grid sampling with grid size 0 . 1 . The training and test sets contain 278 and 59 samples.

We consider the 5 outdoor scenes in the ASL dataset: mountain, winter, summer, wood autumn, wood summer. We preprocess all data by applying grid sampling with grid size 0 . 9 . We arbitrarily rotate and translate all data, and train BITR to align the i -th frame to the ( i +2) -th frame. We use the first

(a) DGL

(b) NSM

(c) LEV

(d) BITR (Ours)

20 frames as the training set, and the other frames as the test set. This leads to a training set of size 105 , and a test set of size 48

We report the results in Tab. 6. Our observation is consistent of that in the 7Scenes dataset: the result BITR is close to the optimum, thus a refinement can lead to improved results. Note that BITR+ICP outperforms all baselines in this task. In addition, GEO causes the out-of-memory error on our 16 G GPU. An assembly result (wood summer) is presented in Fig. 12.

(a) Input

(b) BITR

(c) BITR+ICP

(a) Mug-hanging

D.9 More results of Sec. 6.6

All data in this experiment is generated using PyBullet [8], where the objects in training and test sets are different in shape and position.

Note that we do not report the quantitative results for this experiment because the metric (∆ r, ∆ t ) is ambiguous due to the non-uniqueness of the correct solution. For example, for a correct bowl-placing result, the bowl is allowed to rotate horizontally in the plate, so the metric ∆ r can be very large even for this correct assembly. A suitable metric is the success rate of the manipulation in physical hardware experiments as in [27], but measuring this metric is beyond the scope of this work. In addition, the assembly methods such as NSM [7] and LEV [38] are not applicable because the canonical pose is not known, and we do not report the result of any registration method because the correspondence does not exist. We present a result of BITR on mug-hanging in Fig. 13.

E Limitations and future research directions

In our current implement, we have accelerated most of the layers of BITR using the 'scatter' function [24]. However, BITR is still relatively slow due to the independent computation of convolutional

kernels, i.e ., the harmonic function and the independent multiplication of radial and angular component in each degree. This is reflected by a low GPU utility ratio (about 20% ). We expect to see a large speed gain (about × 5 ) if the above-mentioned computation is implemented using CUDA kernel, i.e ., GPU utility ratio can be close to 100% . On the other hand, a rotation-based technique was recently introduced to reduce the computation cost of SE(3)-equivariant networks [21]. A promising future direction is to extend this technique to BITR.

To explain the limitation of BITR in handling symmetric PCs, we consider the following example. For a pair of PCs X and Y , if there exists a non-identity rigid transformation g 2 such that g 2 Y = Y , and g ∈ SE (3) is a proper transformation that assembles gX to Y , then g 2 g is an equally good transformation that gives the same assembly result. In other words, the optimal transformation g is non-unique, which cannot be modelled by BITR. Actually, we notice that the result of BITR is undefined in this case. Specifically, let ˜ g = Φ B ( X,Y ) , we have

due to SE (3) -bi-equivariance. Then we have g 2 = 1 , which contradicts the assumption.

To address this issue, we plan to extend BITR to a generative model in future research, i.e ., it should assign a likelihood value to each prediction. For the example considered above, it should assign equal probability to g and g 2 g .

Apart from the above-mentioned two limitations, there are several directions for future investigation. First, it is important to generalize BITR to multi-PC assembly tasks where more than 2 PCs are considered [14, 38]. Second, we expect the U-BITR model to be useful in self-supervised 3D shape retrieval or detection models, such as those in image object detection/retrieval [19].

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We have pointed out that the main contribution of the paper is a SE (3) -biequivariant, swap-equivariant, and scale-equivariant PC assembly method.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss limitations in Appx. E.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All proofs can be found in Appx. C.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The architecture BITR is described in Sec. 4 and 5. More details are in Appx. D.1.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The code will be available at https://github.com/wzm2256/BiTr Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: This is included in Sec. 6.1, and more can be found in Appx. D.1.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: We report the mean and standard deviation for our experiments.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The information regarding the computation resource is provided in Appx. D.1

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research conducted in the paper conform the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This work focus on the equivalence of the point cloud assembly task. We do not see any social impact that needs to be specifically addressed.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: Our work is about assembly 3D point cloud. It does not generate new image/language or other types of data of high risk.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: We cited all dataset we used: bunny [33], BB [31], 7scenes [32], shapenet [6], and the data generated by PyBullet [8].

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The code of this paper will be released when the paper is accepted.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents an end-to-end framework for pointcloud correspondence and relative pose estimation with bi-equivariance to per-part poses. The framework is also equivariant to scaling and swapping part orders. It also designes a transformer architecture with $\\mathrm{SE}(3)\\times\\mathrm{SE}(3)$-equivariance. The effectiveness of the framework is shown with experiments on the standard ShapeNet benchmark as well as some other applications.\n\nSTRENGTHS:\n- The problem studied in this paper is well-suited for equivariance, and the paper provides a decent problem formulation with clear explanations.\n- The network architecture in the paper is well-designed, with the bi-equivariant transformer, and the module with the classic pose matching algorithm.\n- The experiments are extensive in the sense that they not only show standard comparisons on ShapeNet, but also have more studies with different setups and other applications.\n\nWEAKNESSES:\n- To my understanding, \"shape assembly\" usually refers to assembling parts of a shape that exactly matches their boundaries (like jigsaw), and usually with more than just two parts. I feel \"shape registration\" is a more proper description of the task in the paper.\n- Following my previous comment, I think the method should also be compared to prior works on shape registration, including classic optimization-based methods like ICP, and more recent learning-based methods such as [1, 2] below.\n\t- A minor point: to compare to these works and show the advantages brought by equivariance, it may be a good setting to compare their data efficiency. I believe that with less training data or less pose augmentation, the proposed framework can have less performance drop than these non-equivariant prior works.\n- [3] is also a related work, especially to the robot manipulation task in Sec. 6.6 (and Suppl. Sec. D.9).\n\n[1] Wang, Y., & Solomon, J. M. (2019). Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3523-3532).\n\n[2] Wang, Y., & Solomon, J. M. (2019). Prnet: Self-supervised learning for partial-to-partial registration. Advances in neural information processing systems, 32.\n\n[3] Ryu, H., Kim, J., An, H., Chang, J., Seo, J., Kim, T., ... & Horowitz, R. (2024). Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18007-18018).\n\nQUESTIONS:\nI don't have questions about the method. But I would really like to see experimental comparisons with the pointcloud registration works.\n\nLIMITATIONS:\nLimitations are well-discussed in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new network to solve the point cloud assembly task, where a 3D transformation is predicted to “align” two point clouds. The proposed network, BTIR, is designed to enforce the symmetries of the point cloud assembly task into the network layers. Specifically, it enforces SE(3) bi equivariance, scale equivariance and swap equivariance through weight constraints. The network is evaluated on several point cloud assembly tasks. It significantly outperforms existing equivariant methods in some settings.\n\nSTRENGTHS:\n- The paper is well-written. Motivating the idea as a learnable extension of Arun’s method was intuitive and easy to follow. \n- The paper introduces a SE(3) biequivariant network, BTIR, to solve point cloud assembly tasks. The network is composed of novel SE(3) biequivariant group convolution layers. \n- The paper provides weight constraints that enforce scale and swap equivariance. An ablation study is performed that demonstrates the added generalization provided by these constraints. \n- The paper includes several PC alignment and assembly experiments where the proposed BTIR network is shown to perform well.\n\nWEAKNESSES:\n- There are several works from the robotic manipulation community that are related to BTIR. These should be discusses in RW and perhaps included as baselines, especially the bowl placing and mug hanging tasks. [1,2]\n- The definition of point cloud assembly in the introduction is a bit imprecise. What does it mean for two non-overlapped point clouds to be “aligned”, when the points themselves will not overlap? Presumably what aligned means depends on who generates the dataset. \n- Some of the writing is inaccurate or unclear. On page 2, “[correspondence based methods] are often sensitive to initial positions of PCs”; most correspondence based approaches use Aruns method which is not sensitive to initial poses. On page 2, “SE(3)-bi-equivariance does not rely on correspondence”; do you mean your BTIR method does not? \n- The results on real data and visual manipulation are of interest and should be included in more detail in the main paper. They are more challenging and would better demonstrate that the proposed method is generally effective. Also, more discussion should be included about these results. For instance, why is it that BTIR without ICP is relatively bad on 7Scenes but better on outdoor scenes of ASL?\n- The experimental results are somewhat limited. More baselines should be included (point cloud alignment is a very well studied problem) and more challenging or varied datasets used. For instance, compare performance across multiple shapenet classes rather than just the wine bottle. Very little space in the paper is currently dedicated to the experiments and discussion. Given BTIR is more computationally heavy than other methods, it should be clear from the experiments that it is robust and effective enough to motivate its use. \n\n[1] Pan, Chuer, et al. \"Tax-pose: Task-specific cross-pose estimation for robot manipulation.\" Conference on Robot Learning. PMLR, 2023.\n[2] Ryu, Hyunwoo, et al. \"Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.\n\nQUESTIONS:\n- In the limitations, it is mentioned that the method is deterministic. How does it solve the wine bottle task in Table 1? Is the loss computed up to the to next symmetry or is the symmetry broken in the loss?\n- It is not clear whether the method is limited to type 0 and 1 features or if they were only mentioned for simplicity. Do the layers introduced in the paper work for higher order features ? Is there a reason other than computational effort that higher order features are not used?\n- It should be mentioned that the enforcing scale equivariance on type 1 features was explored in another paper [1].\n- Is this the first instance of biequivariant filters being used to process two inputs simultaneously? If so this should be stated more clearly as it has many interesting possibly applications even if it is more computationally costly. \n\n[1] Yang, Jingyun, et al. \"Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation.\" arXiv preprint arXiv:2310.16050 (2023).\n\nLIMITATIONS:\nLimitations are discussed in Appendix E. If possible these limitations should be included in the main paper if only briefly.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-11"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes an SE(3)-bi-equivariant approach for point cloud assembly, addressing the difficulty of assembling non-overlapping point clouds where traditional correspondence matching methods struggle. The proposed BITR (BI-equivariant TRansformer) solves this problem by exploiting the symmetry of the point cloud assembly task as a powerful inductive bias, instead of relying on point correspondence.\n\nSpecifically, BITR first merges two distinct 3D point clouds into a 6D point cloud with features that are the tensor product (also known as the Kronecker product or outer product) representation of two SE(3) irreps. These merged 6D point clouds are then processed through novel SE(3)xSE(3)-transformer layers derived from the Wigner-Eckart theorem of G-steerable kernels [5,17], and finally through an SE(3) projection layer that resembles Arun’s method.\n\nThe paper also includes theoretical analysis showing that BITR effectively inherits the symmetry properties of the classical Arun’s method in three aspects: 1) SE(3)-bi-equivariance, 2) Swap equivariance, and 3) Scale equivariance. This enables effective point cloud assembly without assuming known point correspondence. These benefits are also validated through extensive comparison with state-of-the-art baselines.\n\nSTRENGTHS:\n1. To the best of the reviewers' knowledge, this is the first method to achieve SE(3)-bi-equivariance in every layer. While several existing works in biology [1] and robotics [2,3] have addressed SE(3)-bi-equivariance, they only managed it in an ad hoc, template-matching style in the final layer. In contrast, the proposed $SE(3)\\times SE(3)$-transformer layer is inherently bi-equivariant, theoretically grounded in the Wigner-Eckart theorem, and thus expected to be more general and expressive than previous approaches.\n\n2. Experimental results show that the proposed BITR significantly improves point cloud assembly accuracy compared to previous methods. In particular, BITR retains accuracy when the overlap is small, whereas other methods do not.\n\n3. Although not included in the main sections, U-BITR presented in Appendix C is also innovative and promising\n\n4. The proposed method has the potential for broader applications beyond point cloud assembly. It could be beneficial in various areas that require inferring the relative SE(3) pose between two point clouds (or graphs), such as protein docking, robotic manipulation, and camera pose estimation.\n\nOverall, this paper presents a novel and theoretically solid approach. I am confident that it is a breakthrough in bi-equivariant modeling for SE(3) pose inference problems, including point cloud assembly tasks. The empirical results are sufficient to support the claimed benefits.\n\n[1] Ganea et al., \"Independent SE(3)-equivariant Models for End-to-end Rigid Protein Docking,” ICLR 2022\n\n[2] Ryu et al., “Diffusion-EDFs: Bi-equivariant Denoising Generative Modeling on SE(3) for Visual Robotic Manipulation,” CVPR 2024\n\n[3] Huang et al., “Fourier Transporter: Bi-Equivariant Robotic Manipulation in 3D,” ICLR 2024\n\nWEAKNESSES:\n1. The paper may be challenging for readers without a background in representation theory and SE(3)-equivariant neural networks. The proposed method is complicated and might be difficult to implement and extend.\n\n2. ~~Experimental validation is rather confined to simple problems in which object parts are clipped by a random plane. It isn’t sufficiently verified for more general and realistic assembly tasks in which the parts are not cleanly cut by a plane.~~ Addressed.\n\nQUESTIONS:\n1. Why include type-1 features in the last layer before softmax in Eq (12)? Why not use only type-0 irreps? Why not use type-2 or higher?\n\n2. How is locality (neighborhood) defined for 6D point clouds? Directly using KNN for 6D point clouds to construct a neighborhood graph does not make sense to me.\n\n3. The key points of X and Y are ordered and thus not permutation-invariant. Could this have any negative consequences?\n\nLIMITATIONS:\n1. The proposed method is not generative and thus cannot handle scenarios where multiple assembly poses are equally valid.\n\n2. The $SE(3)\\times SE(3)$-transformer layer in BITR relies on the Clebsch-Gordon tensor product, which is notorious for its computational complexity, scaling with $O(L^6)$ where $L$ is the maximum degree. The paper proposes a 2nd-order Clebsch-Gordon tensor product, which likely has similar or worse complexity. The lack of efficient CUDA kernels exacerbates this problem.\n\nThese limitations are appropriately discussed by the authors in Appendix E. The reviewer believes that these limitations are not fundamental and could be addressed in future research. For example, recently proposed eSCN Convolution [4] and Gaunt tensor product [5] have reduced the computational complexity of the C-G tensor product from $O(L^6)$ to $O(L^3)$. Extending these state-of-the-art $SE(3)$-equivariant mechanisms into a bi-equivariant form would be an interesting future direction.\n\n[4] Passaro et al., \"Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs,\" ICML 2023\n\n[5] Luo et al., \"Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products,\" ICLR 2024", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-12"}

comments: SUMMARY:
The paper addresses the problem of assembling point clouds. Given two partial, potentially unmatched point clouds, the objective is to determine the rigid transformation that best aligns them in relation to the unknown complete shape. The proposed approach is a learning-based method, utilizing an architecture that adheres to symmetry constraints inspired by Arun's method. The effectiveness of the method is demonstrated through a toy experiment and synthetic data, with evaluations on real data included in the appendix.

STRENGTHS:
The paper tackles a challenging task.

Deriving the required properties of the suggested network from Arun's method is an elegant solution. The theoretical analysis supporting the suggested method seems to be solid.

The provided toy experiment seems to be convincing.

WEAKNESSES:
The presentation quality could be improved. For example, figure 2 is hard to understand. The description of the toy experiment is not easy to follow as well.

Evaluating the benefits of bi-equivariance. An alternative to bi-equivariance, is to train a regular equivarince network on the task of shape completion and then align the two partial inputs according to the predicted completed shape (using one of the baselines that use correspondences).

QUESTIONS:
Does the suggsted bi-equivariance architecture generalize better than a single-equivariance network trained on shape completion?

LIMITATIONS:
yes
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a novel architecture that extends the SE(3) transformers to become bi-equivariant for the task of rigid point cloud assembly. That is the authors enforce a single assembly for all rigid transformations of the source or target point clouds that is learnable by the network. They also propose how to extend the method for scale and swap consistent assembly of the point clouds. The authors aim to design a method that is correspondence-free to deal with the cases where the point clouds have zero overlap. They first learn (the same number of) keypoints on the original point clouds and concatenate them channel wise. After extracting the bi-equivariant features they utilize the Arun's algorithm to align the point clouds. Some experiments in small-scale datasets of zero overlap showcase benefits of the method.\n\nSTRENGTHS:\n- The authors aim to address a significant problem of pairwise PC registration when the PC parts have close to zero / zero overlap. \n- The derivations of the layer parametrization seem correct (although missing details on second order CG coefficients).\n- The experiments show some performance gain in certain cases.\n\nWEAKNESSES:\n- **On the assumptions/setup** : The setup of zero overlap is not addressed properly when equivariant constraints are enfroced. In the zero overlap case multiple correct poses is inherent. For example a chair next to a table can be placed in many \"correct\" ways. Some of them will be in the training data. But when two different configurations are presented in the training data that breaks the equivariant assumption. Or symmetrically, enforcing equivariance from the first configuration can never reduce the error of the second configuration. I believe the problem of zero overlap has to be formulated more carefully and these consequences should be discussed. \n- **On the method**:\n    1. **Correspondence-free methods should take care of the permutations of the points**: Concatenating the two point clouds after processing them independently does not properly take care of the permutation of points. In fact an individual permutation of the points in X,Y would result in permuted keypoints in Eq.(10). When the keypoints are concatenated each keypoint will match a different keypoint which means that the bi-equivariant transformer will view the permuted case as completely different features.    \n    2. **Assumptions on the sizes of the parts**: Extracting the same amount of keypoints from both point clouds might be restrictive when sizes differ by a lot. \n\n- **On the experiments**:  Fewer experiments are performed in this paper than what is standard in the literature. Also the scale of the datasets is much smaller in number and sizes.\n   - Low-overlap settings (<10%) exist in KITTI, 3DMatch  that the authors do not consider. This also raises the question of the scalability of the method. \n   - The sizes of the parts also answer the question regarding extracting fixed number of keypoints from the two point clouds. E.g. for large point clouds as in 3DMatch how many keypoints are enough?   \n\n- **References**:  Missing literature. A lot of point cloud processing papers utilize the SE(3) transformer (some are even bi-equivariant) for registration, reconstruction, docking etc. Others address equivariant registration with other networks. Some suggestions:\n    - On Point Cloud Registration:\n       - M.Zhu et al. \"Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representations.\" \n       - C.Lin et al  \"Coarse-to-Fine Point Cloud Registration with SE(3)-Equivariant Representations.\"\n    - On PC Processing using SE(3)-transformer and/or bi-equivariance:\n        - E. Chatzipantazis et al.   \"SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space\".\n        - Y.Peng et al \"SE(3)-Diffusion: An Equivariant Diffusion Model for 3D Point Cloud Generation.\"\n        - O. Ganea et al. \"INDEPENDENT SE(3)-EQUIVARIANT MODELS FOR END-TO-END RIGID PROTEIN DOCKING.\"\n        - C.Lin et al. \"SE(3)-Equivariant Point Cloud-Based Place Recognition\".\n\nQUESTIONS:\n- Line 28: \"Sensitive to the initial poses of PCs\". Is there any reference in the literature for that?\n- In practice we do not know id the point clouds have zero overlap or some overlap. How can you discriminate such cases?\n- The zero overlap case could in principle involve many solutions. For example a chair next to a table. Enforcing the configuration that the training data suggest and restricting it with equivariant constraints can hurt performance as the dataset could have other possible configurations for other training samples that are inconsistent with the constraint.\n- The setup of zero overlap has to be formulated more properly.  \n- In the method the authors extract the same amount of keypoints for the X,Y point clouds. Is that restricitve in cases one point cloud is larger than the other?\n- Shouldn't there be some sorting of the keypoints before concatenation? See Weaknesses too.\n- Why is BITR+ICP bi-equivariant? \n- How does the method scale to large point clouds with complicated distributions as in 3DMatch? \n- Are normals given as features in this and the rest of the methods? Are the normals computed before or after the cut?\n- Is the method trained per-class or for the whole dataset?\n\nLIMITATIONS:\nSee the section in Weaknesses and then Questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-30"}