SE(3)-bi-equivariant Transformers for Point Cloud Assembly

Authors: ziming  wang
rebecka  jörnsten
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-14 15:35:27

Abstract: Given a pair of point clouds, the goal of assembly is to recover a rigid transformation that aligns one point cloud to the other. This task is challenging because the point clouds may be non-overlapped, and they may have arbitrary initial positions. To address these difficulties, we propose a method, called $SE(3)$-bi-equivariant transformer (BITR), based on the $SE(3)$-bi-equivariance prior of the task:it guarantees that when the inputs are rigidly perturbed, the output will transform accordingly.   Due to its equivariance property, BITR can not only handle non-overlapped PCs, but also  guarantee robustness against initial positions. Specifically, BITR first extracts features of the inputs using a novel $SE(3) \times SE(3)$-transformer, and then projects the learned feature to group $SE(3)$ as the output. Moreover, we theoretically show that swap and scale equivariances can be incorporated into BITR, thus it further guarantees stable performance under scaling and swapping the inputs. We experimentally show the effectiveness of BITR in practical tasks.

Main paper: 
 introduction: 
 Point cloud (PC) assembly is a fundamental machine learning task with a wide range of applications such as biology [12] , archeology [36] , robotics [28, 20] and computer vision [23] . As shown in Fig. 1 , given a pair of 3-D PCs representing two shapes, i.e., a source and a reference PC, the goal of assembly is to find a rigid transformation, so that the transformed source PC is aligned to the reference PC. This task is challenging because the input PCs may have random initial positions that are far from the optimum, and may be non-overlapped, e.g., due to occlusion or erosion of the object. Most of existing assembly methods are correspondence-based [2, 23, 4] : they use the fact that when the input PCs are aligned, the points corresponding to the same physical position should be close to each other. For example, in Fig. 1 (a), the points at the head of the airplane in the source and reference PCs should be close in the aligned PC. Specifically, these methods first estimate the correspondence between PCs based on feature similarity or distance, and then compute the transformation by matching the estimated corresponding point pairs. As a result, these methods generally have difficulty handling PCs with no correspondence, i.e., non-overlapped PCs, such as Fig. 1 (b). In addition, they are often sensitive to the initial positions of PCs.To address these difficulties, we propose a method, called SE(3)-bi-equivariant transformer (BITR), based on the SE(3)-bi-equivariance prior of the task: when the input PCs are perturbed by rigid transformations, the output should transform accordingly. A formal definition of SE(3)-bi-equivariance can be found in Def. 3.1. Our motivation for using the SE(3)-bi-equivariance prior is threefold: First, the strong training guidance provided by symmetry priors is known to lead to large performance gain and high data efficiency. For example, networks with a translation-equivariance prior, i.e., convolutional neural networks (CNNs), are known to excel at image segmentation [18] . Thus, SE(3)-bi-equivariance prior should lead to similar practical benefits in PC assembly tasks. Second, SE(3)-bi-equivariant methods are theoretically guaranteed to be "global", i.e., their performances are independent of the initial positions. Third, the SE(3)-bi-equivariance prior does not rely on correspondence, i.e., it can be used to handle PCs with no correspondence. Specifically, the proposed BITR is an end-toend trainable model consisting of two steps: it first extracts SO(3) × SO(3)-equivariant features from the input PCs, and then obtains a rigid transformation by projecting the features into SE (3) . For the first step, we define a SE(3) × SE(3)-transformer acting on the 6-D merged PC by extending the SE(3)transformer [11] ; For the second step, we use a SVD-type projection inspired by Arun's method [2] . In addition, we theoretically show that scale-equivariance and swap-equivariance can be incorporated into BITR via weight constraining techniques, which further guarantees that the performance is not influenced by scaling or swapping inputs. An illustration of BITR is presented in Fig. 2 .In summary, the contribution of this work is as follows:-We present a SE(3)-bi-equivariant PC assembly method, called BITR. 1 BITR can assemble PCs without correspondence, and guarantees stable performance with arbitrary initial positions. In addition, the SE(3) × SE(3)-transformer used in BITR is the first SE(3) × SE(3)-equivariant steerable network to the best of our knowledge. -Theoretically, we show that scale and swap equivariance can be incorporated in to BITR by weight-constraining, thus it guarantees stable performance under scaling and swapping the inputs. -We show experimentally that BITR can effectively assemble PCs in practical tasks. 
related works: 
 A special case of PC assembly is PC registration, where the correspondence between input PCs is assumed to exist. A seminal work in this task was conducted by [2] , which provided a closed-form solution to the problem with known correspondence. To handle PCs with unknown correspondence, most of the subsequent works extend [2] by first estimating the correspondence by comparing distances [4] , or features [25, 23, 41] of the PCs, and then aligning the PCs by aligning the estimated corresponding points. Notably, to obtain SE(3)-bi-equivariance, SO(3)-invariant features [42, 10, 43] have been investigated for correspondence estimation. However, since these methods require a sufficient number of correspondences, they have difficulty handling PCs where the correspondence does not exist. In addition, they often have difficulty handling PCs with large initial errors [45] .The proposed BITR is related to the existing registration methods because it can be seen as a generalization of Arun's method [2] . However, in contrast to these methods, BITR is correspondencefree, i.e., it is capable of handling PCs with no correspondence. In addition, it theoretically guarantees stable performance under arbitrary initial position.Recently, some works have been devoted to a new PC assembly task, called fragment reassembly, whose goal is to reconstruct a complete shape from two fragments. Unlike registration task, this task generally does not assume the existence of correspondence. [7] first studied this task, and they addressed it as a pose estimation problem, where the pose of each fragment relative to a canonical pose is estimated via a regression model. [38] further improved this method by considering the SE(3)-equivariance of each fragment. In addition, [31] proposed a simulated dataset for this task. In contrast to these methods, the proposed BITR does not rely on the canonical pose, i.e., it directly estimates the relative pose. As a result, BITR is conceptually simpler and it can handle the shape whose canonical pose is unknown.Another related research direction is equivariant networks. Due to their ability to incorporate 3D rotation and translation symmetry priors, these networks have been extensively used in modelling 3D data [29, 9, 37, 17, 13] , and recently they have been used for robotic manipulation task [27, 28, 40] . In particular, [34] proposed a tensor field network (TFN) for PC processing, and SE(3)-transformer [11] further improved TFN by introducing the attention mechanism. On the other hand, the theory of equivariant networks was developed in [16, 5] . BITR follows this line of research because the SE(3) × SE(3)-transformer used in BITR is a direct generalization of SE(3)-transformer [11] , and it is the first SE(3) × SE(3)-equivariant steerable network to the best of our knowledge. 
preliminaries: 
 This section briefly reviews Arun's method and the concept of equivariance, which will be used in BITR.Given a group G, its representation is a group homomorphism ρ : G → GL(V ), where V is a linear space. When G is the 3D rotation group SO(3), it is convenient to consider its irreps (irreducible orthogonal representation) ρ p : SO(3) → GL(V p ), where p ∈ N is the degree of the irreps, and dim(V p ) = 2p + 1. For r ∈ G, ρ p (r) ∈ R (2p+1)×(2p+1) is known as the Wigner-D matrix. For example, ρ 0 (r) = 1 for all r ∈ SO(3); ρ 1 (r) ∈ R 3×3 is the rotation matrix of r. More details can be found in [5] and the references therein.In this work, we focus on the group G of two independent rotations, i.e., G = SO(3) × SO (3) , where × represents the direct product. Similar to SO(3), we also consider the irreps of G. A useful fact is that all irreps of G can be written as the combinations of the irreps of SO(3): the degree-(p, q) irreps of G is ρ p,q = ρ p ⊗ ρ q : SO(3) × SO(3) → GL(V p ⊗ V q ), where p, q ∈ N, ρ p and ρ q are irreps of SO(3), and ⊗ is tensor product (Kronecker product for matrix). For example, ρ 0,0 (r1 × r 2 ) = 1 ∈ R; ρ 1,0 (r 1 × r 2 ) ∈ R 3×3 is the rotation matrix of r 1 ; ρ 1,1 (r 1 × r 2 ) = ρ 1 (r 1 ) ⊗ ρ 1 (r 2 ) ∈ R 9×9is the Kronecker product of the rotation matrices of r 1 and r 2 .ρ : G → GL(V ) and τ : G → GL(W ), a map Φ : V → W satisfying Φ(ρ(g)x) = τ (g)Φ(x) for all g ∈ G and x ∈ V is called G-equivariant.When Φ is parametrized by a neural network, we call Φ an equivariant neural network, and we call the feature extracted by Φ equivariant feature. Specifically, a degree-p equivariant feature transforms according to ρ p under the action of SO(3), and a degree-(p, q) equivariant feature transforms according to ρ p ⊗ ρ q under the action of SO(3) × SO(3). For simpler notations, we omit the representation homomorphism ρ, i.e., we write r instead of ρ(r), when ρ is clear from the text.Consider a PC assembly problem with known one-to-one correspondence: Let Y = {y u } N u=1 ⊆ R 3 and X = {x u } N u=1 ⊆ R 3 be a pair of PCs consisting of N points, and let {(x u , y u )} N u=1 be their corresponding point pairs. What is the optimal rigid transformation that aligns X to Y ? [2] provided a closed-form solution to this problem. It claims that the optimal solution g = (r, t) ∈ SE(3) in terms of mean square errors is -EQUATIONΦ is SE(3)-bi-equivariant if Φ(g 1 X, g 2 Y ) = g 2 gg -1 1 , ∀g 1 , g 2 ∈ SE(3). -Φ is swap-equivariant if Φ(Y, X) = g -1 .-Φ is scale-equivariant if Φ(cX, cY ) = (r, ct), ∀c ∈ R + . Proposition 3.2. Under a mild assumption (C.2), Arun's algorithm (1) is SE(3)-bi-equivariant, swap-equivariant and scale-equivariant.In other words, Arun's method guarantees to perform consistently 1) with arbitrary rigid perturbations on X and Y , i.e., it is global, 2) if X and Y are swapped (aligning Y to the fixed X or aligning X to the fixed Y ), and 3) in arbitrary scale. Details of Prop. 3.2 can be found at Appx. C.1.We regard Arun's method as a prototype of SE(3)-bi-equivariant PC assembly methods: it first extracts a degree-(1, 1) SO(3) × SO(3)-equivariant translation-invariant feature, i.e., the correlation matrix r (2), and then obtains an output g ∈ SE(3) using a SVD-based projection. This observation immediately leads to more general SE(3)-bi-equivariant methods where the handcrafted feature r is replaced by the more expressive learned equivariant features, thus, the correspondence is no longer necessary. We will develop this idea in the proposed BITR in the next section, and further show that the scale and swap equivariance of Arun's method can also be inherited. 
se(3)-bi-equivariant transformer: 
 This section presents the details of the proposed BITR. BITR follows the same principle as Arun's method [2] : it first extracts SO(3) × SO(3)-equivariant features as a generalization of the correlation matrix r (2), and then projects the features to SE(3) similarly to (1) . Specifically, we first propose a SE(3) × SE(3)-transformer for feature extraction in Sec. 4.2. Since this transformer is defined on 6-D space, i.e., it does not directly handle the given 3-D PCs, it relies on a pre-processing step described in Sec. 4.3, where the input 3-D PCs are merged into a 6-D PC. Finally, the Arun-type SE(3)-projection is presented in Sec. 4.4. An overview of BITR is presented in Fig. 2 .Let Y = {y v } N v=1 ⊆ R 3 and X = {x u } M u=1 ⊆ R 3be the PCs sampled from the reference and source shape respectively. The goal of assembly is to find a rigid transformation g ∈ SE(3), so that the transformed PC gX = {rx i + t} M i=1 is aligned to Y . Note that we do not assume that X and Y are overlapped, i.e., we do not assume the existence of corresponding point pairs.To learn SO(3) × SO(3)-equivariant translation-invariant features generalizing r (1), this subsection proposes a SE(3) × SE(3)-transformer as a generalization of SE(3)-transformer [11] . We present a brief introduction to SE(3)-transformer [11] in Appx. A for completeness.According to the theories developed in [5] , to define a SE(3) × SE(3)-equivariant transformer, we first need to define the feature map of a transformer layer as a tensor field, and specify the action of SE(3) × SE(3) on the field. Since SE(3) × SE(3) can be decomposed as (T (3) × T (3)) (SO(3) × SO(3)) where T (3) is the 3-D translation group, the tensor field should be defined in the 6-D Euclidean space R 6 ∼ = T (3) × T (3), and the features attached to each location should be SO(3) × SO(3)-equivariant and T (3) × T (3)-invariant. Formally, we define the tensor field asEQUATIONwhere Z = {z u } L u=1 ⊆ R 6 is a 6-D PC, δ is the Dirac delta function, f u = ⊕ p,q f p,q u is the feature attached to z u , where f p,q u is the degree-(p, q) equivariant component. We then specify the action of SE(3) × SE(3) on the base space R 6 asEQUATION) where z = z 1 ⊕ z 2 , z 1 , z 2 ∈ R 3 are the first and last three components of z, ⊕ represents direct sum (concatenate), andg i = (r i , t i ) ∈ SE(3) for i = 1, 2. Thus, the action of SE(3) × SE(3) on the degree-(p, q) component of f is (g 1 × g 2 )f p,q (z) = ρ p,q (r 1 × r 2 ) f p,q (g 1 × g 2 ) -1 (z) .With the above preparations, we can now define a SE(3) × SE(3)-transformer layer in a message passing formulation similar to SE(3)-transformer:EQUATIONHere, we use notations o = (o 1 , o 2 ) and i = (i 1 , i 2 ) for simplicity. For example,f o represents the degree-o feature f o1,o2 . F o (z u ) ∈ R c×(2o1+1)(2o2+1)is the collection of all degree-o features at z u , where c is the number of channels of the degree-o features, and W o ∈ R 1×c is a learnable parameter for self-interaction. KNN(•) represents the k nearest neighborhood, and attention α uv is computed according toEQUATIONwhere Q, K and V are known as the query, key and value respectively. They are defined asEQUATIONwhere z vu = z v -z u , W i Q is a learnable parameter for Q, and the convolutional kernel W o,i (z) in V and K both take the form ofEQUATIONwhere the learnable radial component ϕ o,i J1,J2 : R × R → R is parametrized by a neural network, the non-learnable angular component is determined by the 2-nd order Clebsch-Gordan constant Q and the spherical harmonics Y J : R 3 → R 2J+1 , and vec(•) is the vectorize function reshaping a matrix to a vector. Formulation (8) is derived in Appx. B.Note that the kernel (8) is the main difference between a SE(3)-transformer layer and a SE(3) × SE(3)-transformer layer. In the special case where only SE(3)-equivariance is considered, i.e., all features are of degree (p, 0) (or (0, q)), a SE(3) × SE(3)-transformer layer becomes a SE(3)transformer layer.Finally, we adopt the equivariant Relu (Elu) layer similar to [9] as the point-wise non-linear layer in our network. Given an input degree-i feature F i with c channels, an Elu layer is defined asEQUATIONwhereF µ = W i µ F i and F ν = W i ν F i . W µ , W ν ∈ R c×c arelearnable weights and • is the channel-wise vector norm. Note that when i = (1, 0) or i = (0, 1), our definition (9) becomes the same as [9] . By interleaving transformer layers and Elu layers, we can finally build a complete SE(3) × SE(3)-equivariant transformer model.To utilize the transformer model defined in Sec. 4.2, we need to construct a 6-D PC as its input. To this end, we first extract key points from the raw 3-D PCs, and then concatenate them to a 6-D PC to merge their information. Thus, the resulting 6-D PC is not only small in size but also contains the key information of the raw PCs pairs. Formally, we extract L ordered key points X = {x u } L u=1 and Ỹ = {ỹ u } L u=1 from X and Y respectively, and then obtain Z = {x u ⊕ ỹu } L u=1 . Note that we do not require X ( Ỹ ) to be a subset of X (Y ). Specifically, we represent the coordinates of the key points as a convex combination of the raw PCs:EQUATIONwhere X ∈ R M ×3 and Y ∈ R N ×3 represent the coordinates of X and Y respectively, and SoftMax(•) represents the row-wise softmax. F 0 X ∈ R L×M and F 0 Y ∈ R L×N are the weights of each point in X and Y respectively, and they are degree-0, i.e., rotation-invariant, features computed by a shared SE(3)-transformer Φ E :EQUATIONFurthermore, inspired by [39] , we fuse the features of X and Y in Φ E before the last layer, so that their information is merged more effectively, i.e., the selection of X or Ỹ depends on both X and Y . Specifically, the fused features areEQUATIONwhere we only consider degree-0 and degree-1 features. f •,X and f •,Y represent the features of X and Y , and Pool is the average pooling over the PC.Note that X and Ỹ obtained in Eqn. 10 are permutation invariant. For example, according to Eqn. 10, the i-th key point of X is xi = j F ij X j , where F ij is the i-th channel of F 0 X at x j (after softmax normalization). When X is permutated by σ, then the i-th key point can be written as x i = j F ij X j , where j = σ(j). It is easy to see that x i = xi because both j and j iterate through {1, ..., M } in the summation.We now obtain the final output by projecting the feature extracted by the SE(3) × SE(3)-transformer to SE(3). Formally, let f be the output tensor field of the SE(3) × SE(3)-transformer. We compute the final output g = (r, t) ∈ SE(3) using an Arun-type projection as follows:EQUATIONwhere r = unvec(r) ∈ R 3×3 , t X ∈ R 3 and t Y ∈ R 3 are equivariant features computed asr = Pool u (f 1,1 u ), t X = Pool u (f 1,0 u ), t Y = Pool u (f 0,1 u ).We note that projection (13) extends Arun's projection (1) in two aspects. First, although r in (13) and r (2) are both degree-(1, 1) features, r is more flexible than r because r is a learned feature while r is handcrafted, and r is correspondence-free while r is correspondence-based. Second, projection (13) explicitly considers non-zero offsets t X and t Y , which allow solutions where the centers of PCs do not match.In summary, BITR computes the output g for PCs X and Y according toEQUATIONwhere Φ S : S × S → F is a SE(3) × SE(3)-transformer (with the PC merge step), Φ P : F → SE(3) represents projection (13) , and F is the set of tensor field. We finish this section with a proposition that BITR is indeed SE(3)-bi-equivariant. Proposition 4.1. Under a mild assumption (C.2), BITR (14) is SE(3)-bi-equivariant. 
swap-equivariance and scale-equivariance: 
 This section seeks to incorporate swap and scale equivariances into the proposed BITR model. These two equivariances are discussed in Sec. 5.1 and Sec. 5.2 respectively.This subsection seeks to incorporate swap-equivariance to BITR, i.e., to ensure that swapping X and Y has the correct influence on the output. To this end, we need to treat the group of swap as Z/2Z = {1, s} where s 2 = 1, i.e., s represents the swap of X and Y , and properly define the action of Z/2Z on the learned features.Formally, we define the action of Z/2Z on field f (3) as follows. We first define the action of s on the base space R 6 as swapping the coordinates of X and Ỹ : s(z) = z 2 ⊕ z 1 , where z = z 1 ⊕ z 2 , and z 1 , z 2 ∈ R 3 are the coordinates of X and Ỹ respectively. Then we define the action of s on feature f as s(f ) p,q (z) = f q,p (s(z))T , where we regard a degree-(p, q) feature f p,q u as a matrix of shape R (2p+1)×(2q+1) by abuse of notation, and (•) T represents matrix transpose. Intuitively, according to the above definition, degree-(1, 1), (1, 0) and (0, 1) features will become (the transpose of) degree-(1, 1), (0, 1) and (1, 0) features respectively under the action of s, i.e., r will be transposed, t X and t Y will be swapped. This is exactly the transformation needed to ensure swap-equivariant outputs. We formally state this observation in the following proposition. Proposition 5.1. For a tensor field f and a projection Φ P (13),Φ P (s(f )) = (Φ P (f )) -1 .Now the remaining problem is how to make a SE(3) × SE(3)-transformer Z/2Z-equivariant. A natural solution is to force all layers in the SE(3) × SE(3)-transformer to be Z/2Z-equivariant. The following proposition provides a concrete way to achieve this.Proposition 5.2. Let • represent the swap of index, e.g., if o = (o 1 , o 2 ), then õ = (o 2 , o 1 ). 1)For a transformer layer (5) , if the self-interaction weight satisfies W o = W õ, the weight of query (7) satisfiesW o Q = W õ Q , and the radial function satisfies ϕ i,o J1,J2 ( z 1 , z 2 ) = ϕ ĩ, õ J2,J1 ( z 2 , z 1) for all i, o, J 1 , J 2 , z 1 and z 2 , then the transformer layer is Z/2Z-equivariant.2) For an Elu layer (9) , if W i ν = W ĩ ν and W i µ = W ĩ µ for all i, then the Elu layer is Z/2Z-equivariant.More details, including the complete matching property (Prop. C.11), can be found in Appx. C.3.1.This subsection seeks to incorporate scale equivariance to BITR, i.e., to ensure that when X and Y are multiplied by a scale constant c ∈ R + , the output result transforms correctly. To this end, we need to consider the scale group (R + , ×), i.e., the multiplicative group of R + , and properly define the (R + , ×)-equivariance of the learned feature. For simplicity, we abbreviate group (R + , ×) as R + .We now consider the action ofR + on field f (3). We call f a degree-p R + -equivariant field (p ∈ N) if it transforms as (c(f )) (z) = c p f (c -1 z) under the action of R + , where z ∈ R 6 and c ∈ R + .We immediately observe that degree-1 R + -equivariant features lead to scale-equivariant output. Intuitively, if r, t X and t Y are degree-1 R + -equivariant features, then they will become cr, ct X and ct Y under the action of c, and the projection step will cancel the scale of r while keeping the scale of t X and t Y , which is exactly the desirable results. Formally, we have the following proposition.Proposition 5.3. Let Φ P be projection (13) , f be a degree-1 R + -equivariant tensor field, and (r, t) = Φ P (f ). We haveΦ P (c(f )) = (r, ct) ∀c ∈ R + .The remaining problem is how to ensure that a SE(3) × SE(3)-transformer is R + -equivariant and its output is of degree-1, so that scaling the input can lead to the proper scaling of output. Here we provide a solution based on the following proposition. Definition 5.4. ϕ : R × R → R is a degree-p function if ϕ(cx, cy) = c p ϕ(x, y) for all c ∈ R + .Proposition 5.5. 1) Denote ϕ K and ϕ V the radial functions used in K and V respectively. Let ϕ K be a degree-0 function, f in be a degree-0 R + -equivariant input field. For transformer layer (5) , if ϕ V is a degree-1 function and the self-interaction weight W = 0, then the output field f out is degree-1 R + -equivariant; If ϕ V is a degree-0 function, then the output field f out is degree-0 R + -equivariant.2) For Elu layer (9) , if the input field is degree-p R + -equivariant, then the output field is also degree-p R + -equivariant.More discussions can be found in Appx. C.4. 
experiments and analysis: 
 This section experimentally evaluates the proposed BITR. After describing the experiment settings in Sec. 6.1, we first present a simple example in Sec. 6.2 to highlight the equivariance of BITR. Then we evaluate BITR on assembling the shapes in ShapeNet [6] , BB dataset [31] , 7Scenes [32] and ASL [22] from Sec. 6.3.1 to Sec. 6.4. We finally apply BITR to visual manipulation tasks in Sec. 6.6.We extract L = 32 key points for each PC. The SE(3)-transformer and the SE(3) × SE(3)transformer both contain 2 layers with c = 4 channels. We consider k = 24 nearest neighborhoods for message passing. We only consider low degree equivariant features, i.e., p, q ∈ {0, 1} for efficiency. We train BITR using Adam optimizer [15] with learning rate 1e -4 . We use the loss functionL = r T r gt -I 2 2 + t gt -t 2 2, where (r, t) are the output transformation, (r gt , t gt ) are the corresponding ground truth. We evaluate all methods by isotropic rotation and translation errors: ∆r = (180/π)accos 1/2 tr(rr T gt ) -1) , and ∆t = t gt -t where tr(•) is the trace of a matrix. We do not use random rotation and translation augmentations as [23] . More details are in Appx. D.1.To demonstrate the equivariance property of BITR, we train BITR on the bunny shape [33] . We prepare the dataset similar to [41] : In each training iteration, we first construct the raw PC S by uniformly sampling 2048 points from the bunny shape and adding 200 random outliers from [-1, 1] 3 , then we obtain PCs {X P , Y P } by dividing S into two parts of ratio (30%, 70%) using a random plane P . We train BITR to reconstruct S using randomly rotated and translated {X P , Y P }. To construct the test set, we generate a new sample {X P , Y P }, and additionally construct 3 test samples by 1) swapping, 2) scaling (factor 2) and 3) randomly rigidly perturbing {X P , Y P }.The assembly results of BITR on these four test samples are shown in Fig. 3 . We observe that BITR performs equally well in all cases. Specifically, the differences between the rotation errors in these four cases are small (less than 1e -3 ). The results suggest that BITR is indeed robust against these three perturbations, which verifies its swap-equivariance, scale-equivariance and SE(3)-biequivariance. More experiments can be found in the appendix: a numerical verification of Def. In this experiment, we evaluate BITR on assembling PCs sampled from a single shape. When the inputs PCs are overlapped, this setting is generally known as PC registration. We construct a dataset similar to [41] : for a shape in the airplane class of ShapeNet [6] , we obtain each of the input PCs by uniformly sampling 1024 points from the shape, and keep ratio s of the raw PC by cropping it using a random plane. We vary s from 0.7 to 0.3. Note the PCs may be non-overlapped when s < 0.5.We compare BITR against the state-of-the-art registration methods GEO [23] and ROI [43] , and the state-of-the-art fragment reassembly methods NSM [7] and LEV [38] . For NSM and LEV, we additionally provide the canonical pose for each shape. Note that LEV and ROI are SE(3)-equivariant methods. For s ≥ 0.5, we report the results of BITR fine-tuned by ICP [45] (BITR+ICP). Note that BITR+ICP is SE(3)-bi-equivariant and scale-equivariant, but not swap-equivariant. We present the results in Fig. 4 . We observe that the performance of all methods decrease as s decreases.Meanwhile, BITR outperforms all baseline methods when s is small (s ≤ 0.5). On the other hand, when s is large (s > 0.5), BITR performs worse than GEO, but it still outperforms other baselines. Nevertheless, since the results of BITR are sufficiently close to optimum (∆r ≤ 20), the ICP refinement can lead to improved results that are close to GEO. More details can be found in Appx. D.5. To evaluate BITR on non-overlapped PCs, we extend the experiment in Sec. 6.3.1 to inter-class assembly. We train BITR to place a car shape on the right of motorbike shape, so that their directions are the same and their distance is 1. We consider s = 1.0 and 0.7. Note that this task is beyond the scope of registration methods, since the input PCs are non-overlapped.A result of BITR is shown in Fig. 5 . More details can be found in Appx. D.6.This subsection evaluates BITR on a fragment reassembly task. We compare BITR against NSM [7] , LEV [38] and DGL [44] on the 2-fragment WineBottle class of the BB dataset [31] . The data preprocessing step is described in Appx. D.7. We test the trained BITR 3 times using different random samples, and report the mean and standard deviation of (∆r, ∆t) in Tab. 1. We observe that BITR outperforms all baseline methods: BITR achieves the lowest rotation errors, and its translation error is comparable to DGL, which is lower than other baselines by a large margin. We provide some qualitative comparisons in Appx. D.7.This subsection evaluates BITR on an indoor dataset 7Scenes [32] and the outdoor scenes in ASL dataset [22] . We present the results on 7Scenes in this section, and leave the results on ASL and some qualitative results to Appx. D.8.For the 7Scenes dataset, we arbitrarily rotate and translate all frames, and train BITR to align all adjacent frames. We use the data from the first 6 scenes as the training set, and the data from 7-th scene as the test set. To train BITR, we use a random clipping augmentation similar to Sec. 6.3.1: we keep ratio s of each PCs by clipping them using a random plane, where s is uniformly distributed in [0.5, 1.0]. We compare BITR against GEO [23] , ROI [43] , ICP [45] and OMN [39] , where OMN is a recently proposed correspondence-free registration method. The results on 7Scenes are reported in Tab. 2. We observe that BITR can produce results that are close to the optimum (∆r ≈ 25) from a random initialization (∆r ∈ U [0, 180]), and extra refinements like ICP and OT can further improve the results (∆r ≈ 10). This observation is consistent with that in Sec. 6.3.1. In particular, BITR with the OT refinement is comparable with GEO and ROI, which use highly complicated features specifically designed for registration tasks and an OT-like refinement process. On the other hand, ICP and OMN fails in this task due to their sensitiveness to initial positions.This subsection investigates the potential of BITR in manipulation tasks. Following [26] , we consider two tasks: mug-hanging and bowl-placing. For both tasks, X represents an object grasped by a robotic arm, i.e., a cup or a bowl, Y represents the fixed environment with a target, i.e., a stand or a plate, and we train BITR to align X to Y , so that the cup can be hung to the stand, or the bowl can be placed on the plate.Fig. 6 presents the results of BITR on bowl-placing. We observe that although BITR is not originally designed for manipulation tasks, it can place the bowl in a reasonable position relative to the plate. However, we also notice that BITR may produce unrealistic results, e.g., the PCs may collide. Thus, post-processing steps [30] or extra regularizers [13] may be necessary in practical applications. More results and discussions can be found in Appx. D.9.  
conclusion: 
 This work proposed a PC assembly method, called BITR. The most distinguished feature of BITR is that it is correspondence-free, SE(3)-bi-equivariant, scale-equivariant and swap-equivariant. We experimentally demonstrated the effectiveness of BITR.BITR in its current form has two main limitations. First, BITR is computationally inefficient because each degree of feature is computed independently without parallel. This issue was also observed in SE(3)-equivariant networks, and was recently addressed by [21] . A promising future research direction is to develope similar acceleration techniques for BITR. Second, since BITR is deterministic, i.e., it only predicts one result for a given input, it cannot handle symmetric PCs. Although this feature does not cause any difficulty in this work (there is no strictly symmetric PCs in this work due to noise, random sampling, etc), it may be problematic in future applications such as molecule modelling where symmetric PCs exist, e.g., benzene rings. To address this issue, we plan to generalize BITR to a generative model in the future. More discussions can be found in Appx. E. 
experiments compute resources: 
 Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?Answer: [Yes] Justification: The information regarding the computation resource is provided in Appx. D.1Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?Answer: [NA] Justification: This work focus on the equivalence of the point cloud assembly task. We do not see any social impact that needs to be specifically addressed.Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.• The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?Answer: [NA] Justification: Our work is about assembly 3D point cloud. It does not generate new image/language or other types of data of high risk.Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?Answer: [Yes]Justification: We cited all dataset we used: bunny [33] , BB [31] , 7scenes [32] , shapenet [6] , and the data generated by PyBullet [8] .Guidelines:• The answer NA means that the paper does not use existing assets.• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. 
 
 Appendix: A well-known SE(3)-equivariant network is SE(3)-transformer [11] , which adapts the powerful transformer structure [35] to SO(3)-equivariant settings. In this model, the feature map f of each layer is defined as a tensor field supported on a 3-D PC:EQUATION)where δ is the Dirac function, X = {x u } M u=1 ⊆ R 3 is a point set, and f u is the feature attached to x u . Here, feature f u takes the form of f u = ⊕ p f p u , where the component f p u ∈ V p is the degree-p feature, i.e., it transforms according to ρ p under the action of SO(3). For example, when f u represents the norm vector of a point cloud, thenf u = f 1 u ∈ R 3 .We also write the collection of all degree-p features at x u as F p (x u ) ∈ R c×(2p+1) , where c is the number of channels.For each transformer layer, the degree-k output feature at point x i is computed by performing message passing:EQUATIONwhere N (u) represents the neighborhood of u, W l ∈ R 1×c is the learnable weight for self-interaction, c represents the number of channels, andEQUATIONis the attention from v to u. Here, key K, value V and query Q areQ u = l W l Q F l in (x u ), K uv = l k W lk K (x v -x u ) f k in (x v ), V lk uv = W lk V (x v -x u ) f k in (x v ) (18) where W l Q ∈ R 1×(2l+1) is a learnable weight, and the kernel W lk (x) ∈ R (2l+1)×(2k+1) is defined asEQUATIONwhere vec(•) is the vectorize function, the learnable radial component ϕ lk J : R + → R is parametrized by a neural network, and the non-learnable angular component is determined by Clebsch-Gordan constant Q and the spherical harmonic Y J : R 3 → R 2J+1 .To derive the kernel (8) for a SE(3) × SE(3)-transformer layer, we consider the equivariant convolution as a simplified version of the SE(3) × SE(3)-transformer layer:EQUATIONi.e., we only consider the message V o,i uv while fixing the self-interaction weight W o = 0 and attention α uv = 1 in (5). To ensure the SE(3) × SE(3)-equivariance of convolution (20) , i.e.,EQUATIONthe kernel W must satisfy a constraint:EQUATIONwhere we abbreviate r 1 × r 2 as r 12 , abbreviate vec(W o,i ) ∈ R (2i1+1)(2i2+1)(2o1+1)(2o2+1) as W, and assume z 1 = z 2 = 1 for simpler notations. Equation ( 22) is generally known as the kernel constraint, and its necessity and sufficiency can be proved in a verbatim way as Theorem 2 in [37] , so we omit the proof here. Now we can obtain the kernel (8) by solving this constraint. Note that a direct formulation is given in Theorem 2.1 in [5] , but here we derive it in a less abstract way.We first observe thatW * (z) = Y i1 (z 1 ) ⊗ Y i2 (z 2 ) ⊗ Y o1 (z 1 ) ⊗ Y o2 (z 2) is a special solution to equation (22) , where Y J (•) ∈ R 2J+1 is the column vector consisting of degree-J harmonics with order m = -J, ...0, ..., J as each row element, becauseρ i (r 12 ) ⊗ ρ o (r 12 )W * (z) = ρ i1 (r 1 ) ⊗ ρ i2 (r 2 ) ⊗ ρ o1 (r 1 ) ⊗ ρ o2 (r 2 ) Y i1 (z 1 ) ⊗ Y i2 (z 2 ) ⊗ Y o1 (z 1 ) ⊗ Y o2 (z 2 ) = ρ i1 (r 1 )Y i1 (z 1 ) ⊗ ρ i2 (r 2 )Y i2 (z 2 ) ⊗ ρ o2 (r 1 )Y o2 (z 1 ) ⊗ ρ o2 (r 2 )Y o2 (z 2 ) =Y i1 (r 1 z 1 ) ⊗ Y i2 (r 2 z 2 ) ⊗ Y o1 (r 1 z 1 ) ⊗ Y o2 (r 2 z 2 ) =W * (r 12 z). Then we point out that {Y m1 J1 (z 1 )Y m2 J2 (z 2 )} m1,m2,J2,J1is an orthogonal and complete basis for two variable square-integrable functions L 2 (X, Y ) : S 2 × S 2 → R, where S 2 is the 2-D sphere. Thus, we can write each component of W * (z) as a linear combination of this basis. Specifically, letV(z) = ⊕ J1,J2 Y J1 (z 1 ) ⊗ Y J2 (z 2 ), and let (m 1 , m 2 , m 3 , m 4 ) be the index of the ele- ment Y m1 i1 (z 1 )Y m2 i2 (z 2 )Y m3 o1 (z 1 )Y m4 o2 (z 2 ) in W * , and (J 1 , m 5 , J 2 , m 6 ) be the index of the element Y m5 J1 (z 1 )Y m6 J2 (z 2 ) in V.We have W * = P V, where P m1,m2,m3,m4,J1,m5,J2,m6EQUATIONHere, c(J 1 ) and c(J 2 ) are coefficients related to J 1 and J 2 respectively.i, m 1 , o, m 3 |J, m 5 is known as the Clebsch-Gordan coefficient, the producti 1 , m 1 , o 1 , m 3 |J 1 , m 5 i 2 , m 2 , o 2 , m 4 |J 2 ,m 6 is known as the 2-nd order Clebsch-Gordan coefficient, and we represent it as Q. In other words, we have P = Qc, where c is a block diagonal matrix, and each block is c(J 1 )c(J 2 )I. Therefore, we haveEQUATIONfor arbitrary z, we further haveEQUATIONby equating the above two equations. Finally, we stack this decomposition back to the kernel constraint (22) , and obtainEQUATIONwhich suggests thatEQUATIONEQUATIONwhere c is the coefficient matrix of the same shape as c, and each coefficient is arbitrary. Specifically,EQUATIONWe note that c J1,J2 is parametrized by a neural network in our model (8) .Now we prove that the last term in Eqn. (33) is equal. We have Ũ3 Ṽ T 3 = ±U 3 V T 3 , and we have shown thatEQUATIONBy multiplying this term withEQUATIONwhich suggests that last term in Eqn. ( 33) is equal. In summary, we have proved (33) .Finally, we can prove the uniqueness of Arun's method under the same assumption.Proposition C.4 (Uniqueness of Arun's method). Under assumption C.2, Arun's method (1) is unique.Proof. According to Prop. C.3, r = SVD(r) is unique. As a result, t = m(Y ) -rm(X) is also unique.Throughout this work, we assume that assumption C.2 holds, so that the uniqueness of Arun's method and BITR can be guaranteed.For the rest of this appendix, we use a simpler notation of SVD projection, where we simply absorb the sign matrix into Û and V :EQUATIONEQUATIONand Σ = diag( σ1 , σ2 , σ3 sign(det( Û V T ))),thus obtain the following equivalent definition. Definition C.5. The SVD projection of matrix A is defined asEQUATIONwhere A = U ΣV T and U, V ∈ SO(3).Finally, we can discuss the equivariance of Arun's method. Lemma C.6. Let SVD(A) = U V T (42) be the SVD projection of A, i.e., A can be decomposed as A = U ΣV T , Σ is a diagonal matrix and U, V ∈ SO(3). We haveEQUATIONEQUATIONEQUATIONfor arbitrary A ∈ GL(3), r 1 , r 2 ∈ SO(3), and c ∈ R + .The Proof of Lemma. C.6. We haveEQUATIONwhere r 2 U ∈ SO(3), r 1 V ∈ SO(3) and Σ is a diagonal matrix. In other words, ( 46) is a SVD decomposition of matrix r 2 Ar T 1 , thus the SVD projection can be computed asEQUATIONwhich proves the first part of this lemma. We omit the other two statements of this lemma since they can be proved similarly.The Proof of Prop. 3.2. Let Φ A be Arun's method and Φ A (X, Y ) = (r(X, Y ), t(X, Y )) = g. We directly verify these three equivariances according to Def. 3.1. We only prove the SE(3)bi-equivariance and swap-equivariance and omit the proof of scale-equivariance, because it can be proved similarity.We compute Φ A (g 1 X, g 2 Y ) for the perturbed PCs g 1 X = {r 1 x i + t 1 } N i=1 and g 2 Y = {r 2 y i + t 2 } N i=1 as follows. We first compute r(g 1 X, g 2 Y ) asEQUATIONThen we compute r via SVD:EQUATIONwhere the 3-rd equality holds due to Lemma. C.6. Since the mean values are also perturbed asEQUATION)In summary,EQUATIONwhich proves the SE(3)-bi-equivariance of Arun's method.2) swap-equivariance We compute Φ A (Y, X) as follows. We first compute r(Y, X) asEQUATION)Then we compute r(Y, X) via SVD:EQUATIONWe can finally compute t(Y, X) asEQUATIONwhich proves the swap-equivariance of Arun's method.Before proving the SE(3)-bi-equivariance of BITR, we first verify that each layer in a SE(3) × SE(3)-transformer is indeed SE(3) × SE(3)-equivariant. Intuitively, the equivariance of a transformer layer ( 5) is a result of the kernel design, i.e., the kernel constraint (22) , and the invariance of the attention. We state this result in the following lemma for completeness, and we note that similar techniques are also used in the construction of SE(3)-transformer [11] .Lemma C.7. The transformer layer (5) is SE(3) × SE(3)-equivariant.Proof. We abbreviate r 1 × r 2 as r 12 , and g 1 × g 2 as g 12 . Let L be a transformer layer (5) , we seek to prove (g 12 )(L(f )) = L((g 12 )(f )) for the input tensor field f and g 1 , g 2 ∈ SE(3). We compute the RHS of the equation asL(g 12 (f )) o (z u ) =W o g 12 (F ) o (z u ) + i zv∈supp(g12(f )) α(g 12 (f ), z u , z v )W o,i (z v -z u ) g 12 (f ) i (z v ) =W o F o (g -1 12 z u ) ρ o (r 12 ) T + i zv∈supp(g12(f )) α(g 12 (f ), z u , z v )W o,i (z v -z u )ρ i (r 12 )f i (g -1 12 z v ) = W o F o (g -1 12 z u ) ρ o (r 12 ) T A + i zv∈supp(f ) α(g 12 (f ), z u , g 12 z v ) B W o,i (g 12 z v -z u )ρ i (r 12 )f i (z v ) C .Note that here we haveEQUATION)) T because we write F in the channel-first form, i.e., the shape of F o is c × (2o 1 + 1)(2o 2 + 1). The LHS of the equation is computed asg 12 L(f ) o (z u ) = W o F o (g -1 12 z u ) ρ o (r 12 ) T A + i zv∈supp(f ) α(f, g -1 12 z u , z v ) B ρ o (r 12 )W o,i (z v -g -1 12 z u )f i (z v ) C .We now verify that these 3 terms are equal respectively. C = C : By design, the kernel W satisfies the kernel constraint ( 22)EQUATIONIn addition, let z u = g -1 12 z u . We haveEQUATIONThus, we can obtain the equality by combining the above two equations.B = B : We compute Q and K for B and B respectively. We haveEQUATIONK(f, g -1 12 z u , z v ) = o i W o,i K (z v -g -1 12 z u )f i (z v ) = o i ρ o (r 12 ) T W o,i K (g 12 z v -z u )ρ i (r 12 )f i (z v ),where the last equation holds due to the kernel constraint (22) and the orthogonality of ρ o (r 12 ). Specifically, ρ o (r 12 ) = ρ o1 (r 1 ) ⊗ ρ o2 (r 2 ) is an orthogonal matrix because ρ o1 and ρ o2 are orthogonal representations, i.e., ρ o1 (r 1 ) and ρ o2 (r 2 ) are orthogonal matrices.and the (m4 , m 3 , m 2 , m 1 )-th element of W õ, ĩ (s(z)) is o2+i2 J2=|o2-i2| o1+i1 J1=|o1-i1| J2 m6=-J2 J1 m5=-J1 ϕ ĩ, õ J2,J1 ( z 2 , z 1 ) i 2 , m 2 , o 2 , m 4 |J 2 , m 6 i 1 , m 1 , o 1 , m 3 |J 1 , m 5 Y m6 J2 (z 2 / z 2 )Y m5 J1 (z 1 / z 1). (70) Since the angular components in (69) and (70) are the same, and the radial components are equal by assumption, we immediately conclude that (69) and (70) are equal. In other words,EQUATIONwhich proves this lemma.Then we proceed to the proof of Prop. 5.2.The Proof of Prop. 5.2. Let L be a transformer layer (5) , we seek to prove s(L(f )) = L(s(f )) for the input tensor field f .To this end, we expand the RHS of the equation asL(s(f )) o (z u ) =W o s(F ) o (z u ) + i∈I(s(f )) zv∈supp(s(f )) α(s(f ), z u , z v )W o,i (z v -z u ) s(f ) i (z v ) =W o F õ(s(z u )) T23 + i∈I(s(f )) zv∈supp(s(f )) α(s(f ), z u , z v )W o,i (z v -z u ) f ĩ(s(z v )) T = W o F õ(s(z u )) T23 A + i∈I(f ) zv∈supp(f ) α(s(f ), z u , s(z v )) B W o, ĩ(s(z v ) -z u ) f i (z v ) T C, where I(f ) is the set of all degrees of field f , and supp(f ) is the support of field f . The last equation holds because we replace z v by s(z v ), and replace i by ĩ. We expand the LHS of the equation ass L(f ) o (z u ) = L(f ) õ(s(z u )) T = W õF õ(s(z u )) T A + i∈I(f ) zv∈supp(f ) α(f, s(z u ), z v ) B W õ,i (z v -s(z u ))f i (z v ) T C .We now verify that these three terms are equal respectively. A = A : By assumption, W o = W õ, thus we immediately haveW õF õ(s(z u )) T = W o F õ(s(z u )) T = W o F õ(s(z u )) T23 .(72) B = B : Since α = Q, K , we compute Q and K for B and B respectively. As for Q, we haveQ(s(f ), z u , s(z v )) = o∈I(s(f )) W o Q (s(F )) o (z u ) = o∈I(s(f )) W o Q F õ(s(z u )) T = o∈I(s(f )) W õ Q F õ(s(z u )) T ,where the last equation holds becauseW o Q = W õ Q by assumption. In addition, Q(f, s(z u ), z v ) = o∈I(f ) W o Q F o (s(z u )).As for K, we haveK(s(f ), z u , s(z v )) = o∈I(s(f )) i∈I(s(f )) W o,i K (s(z v ) -z u ) s(f ) i (s(z v )) = o∈I(s(f )) i∈I(f ) W õ,i K (z v -s(z u ))f i (z v ) T ,where the last equation holds due to the constraint of radial function and Lemma. C.9, and we replace i by ĩ. In addition,K(f, s(z u ), z v ) = o∈I(f ) i∈I(f ) W o,i K (z v -s(z u ))f i (z v ).We observe that Q(s(f ), z u , s(z v )) is just the transpose and re-ordering of each component of Q(f, s(z u ), z v ), and this is also true for K(s(f ), z u , s(z v )) and K(f, s(z u ), z v ), which suggests that their inner products are the same. Specifically, letQ o = W o Q F o (s(z u )) and K o = i∈I(f ) W o,i K (z v -s(z u ))f i (z v ). We have α(s(f ), z u , s(z v )) = o∈I(s(f )) Q õ T , o∈I(s(f )) K õ T = o∈I(s(f )) Q õ, K õ = o∈I(f ) Q o , K o and α(f, s(z u ), z v ) = o∈I(f ) Q o , o∈I(f ) K o = o∈I(f ) Q o , K o , which suggests that α(f, s(z u ), z v ) = α(s(f ), z u , s(z v )).C = C : By assumption, the radial function satisfies ϕ i,o J1,J2 ( z 1 , z 2 ) = ϕ ĩ, õ J2,J1 ( z 2 , z 1 ) for all i, o, J 1 , J 2 , z 1 and z 2 , thus W o,i (z) = W õ, ĩ (s(z)) T12T34 according to Lemma. C.9. As a result,EQUATIONIn summary, we have shown the Z/2Z-equivariance of a transformer layer (5) , which is the first part of Prop. 5.2. We omit the proof of the second part of this proposition, i.e., the equivariance of an Elu layer, as it is a simple extension of the proof of A = A .Remark C.10. The weight sharing technique in Prop. 5.2 reduces the number of learnable parameters of a SE(3) × SE(3)-transformer by about half, thus it makes the model more efficient. In practice, the weight sharing technique can be achieved by sharing weights between different o, i and J directly, except for the symmetric case, i.e., o = õ, i = ĩ and J 1 = J 2 , where the requirement of the radial function becomesEQUATION)We represent the radial function in this case as ϕ(x, y) = 1 2 (φ(x, y) + φ(y, x)), where φ is a neural network, which guarantees the symmetry of ϕ.Consider the following complete matching problem.The complete matching problem Given a pair of PCs X and Y , where Y is generated by a unknown rigidly transformation of X, i.e., Y = gX and g ∈ SE(3) is unknown, how to infer g?The complete matching problem is a prototype of the registration problem, i.e., in the simplest case (no outlier, no noise, no partial visibility..), how to derive the relative transformation between two fully overlapped PCs? Despite its simplicity, this problem is non-trivial, because we do not know the correspondence between X and Y , i.e., Arun's method does not apply.A surprising fact is that g can be exactly recovered using a SE(3)-bi-equivariant and swap-equivariant assembly method. We state this fact formally as follows. Proposition C.11 (Complete-matching property). Let Φ be a swap-equivariant and SE(3)-biequivariant assembly method. ThenEQUATIONfor arbitrary PC X and g 1 ∈ SE(3).Proof. First, since Φ is swap-equivariant, then Φ(X, X)Φ(X, X) = I. Then, since Φ is SE(3)-bi- equivariant, we have Φ(X, g 1 X)Φ(X, X) = g 1 Φ(X, X)Φ(X, X) = g 1 .According to the main text, BITR is swap-equivariance (Prop. 5.2), thus, we can derive a concrete algorithm, called untrained BITR (U-BITR), for the complete matching problem:EQUATIONwhere Φ B is BITR model. Prop. C.11 suggests that Φ U solves the complete matching problem without training. We numerically evaluate this property in Appx. D.4, and leave the theoretical analysis of U-BITR to future research.C. 4 The proof of Prop. 5.5First, we have the following straightforward lemma for the kernel. Lemma C.12. If the radial function ϕ : R × R → R is a degree-p function, then kernel W (5) satisfiesEQUATION)Then we directly verify Prop. 5.5.The proof of Prop. 5.5. Let L be a transformer layer (5) , f be a degree-0 R + -equivariant input field, ϕ K be a degree-0 function. We seek to prove: 1) If ϕ V is a degree-0 function, then c(L(f)) = L(c(f )) and L(f ) is degree-0 R + -equivariant; 2) If ϕ V is a degree-1 function and W = 0, then c(L(f )) = L(c(f )) and L(f ) is degree-1 R + -equivariant.To begin with, we expand L(c(f )) asL c(f ) o (z u ) = W o (c(F )) o (z u ) + i zv∈supp(c(f )) α(c(f ), z u , z v )W o,i V (z v -z u ) c(f ) i (z v ) = W o F o (c -1 z u ) + i zv∈supp(c(f )) α(c(f ), z u , z v )W o,i V (z v -z u )f i (c -1 z v ) = W o F o (c -1 z u ) A + i zv∈supp(f ) α(c(f ), z u , c(z v )) B W o,i V (c(z v ) -z u )f i (z v ) C , and expand c(L(f )) as c L(f ) o (z u ) = c p L(f ) o (c -1 z u ) = c p W o F o (c -1 z u ) A + i zv∈supp(f ) α(f, c -1 z u , z v ) B W o,i V (z v -c -1 z u )f i (z v ) C .We first point out that B = B , becauseK(c(f ), z u , c(z v )) = o i W o,i K (c(z v ) -z u ) c(f ) i (z v ) = o i W o,i K (z v -c -1 (z u ))f i (c -1 z v ) = K(f, c -1 z u , z v ),where the second equation holds because ϕ K is a degree-0 function by assumption, and Lemma. C.12 claims that the corresponding kernel W K is scale-invariant. In addition,Q(c(f ), z u , c(z v )) = o W o Q (c(F )) o (z u ) = o W o Q F o (c -1 z u ) = Q(f, c -1 z u , z v ).In other words, both Q and K are scale-invariant. Thus, we conclude that the attention is also scale-invariant:α(c(f ), z u , c(z v )) = Q(c(f ), z u , c(z v )), K(c(f ), z u , c(z v )) = K(f, c -1 z u , z v ), Q(f, c -1 z u , z v ) = α(f, c -1 z u , z v ).Now we discuss these two situations:1) If ϕ V is a degree-1 function and W = 0, thencW o,i V (z v -c -1 z u ) = W o,i V (c(z v ) -z u ) according to Lemma. C.A = A = 0. In other words, L(c(f ))(z) = c 1 L(f )(c -1 z), where the RHS is the action of c on the degree-1 R + -equivariant field, thus we haveL(c(f )) = c(L(f )), and L(f ) is degree-1 R + -equivariant. 2) If ϕ V is a degree-0 function, then W o,i V (z v -c -1 z u ) = W o,i V (c(z v ) -z u ) according to Lemma. C.12. Thus, L(c(f ))(z) = L(f )(c -1 z), where the RHS is the action of c on the degree-0 R + -equivariant field, thus we have L(c(f )) = c(L(f )), and L(f ) is degree-0 R + -equivariant. In summary, we have shown that the equivariance of the transformer layer. We omit the proof of the equivariance of the Elu layer as it can be proved similarly.We can now construct a special structure of BITR for producing degree-1 R + -equivariant output:EQUATIONIn other words, all tensor fields are degree-0 R + -equivariant, except for the final output which is degree-1 R + -equivariant. More specifically, we consider two types of layers, i.e., deg-0 → deg-0 and deg-0 → deg-1.Prop. 5.5 suggests that a transformer layer with degree-p radial function can generate a degree-p R + -equivariant field when the input field is degree-0 R + -equivariant (Elu layers do not influence the R + -equivariance). Therefore, we only need to consider degree-0 and degree-1 radial functions for our purpose. In practice, we represent degree-1 functions using neural networks consisting of linear and relu [1] layers, and we represent degree-0 functions using linear, relu and layer normalization [3] layers. Specifically, We represent degree-0 functions as neural network φ 0 :EQUATIONand represent degree 1-functions neural network φ 1 :EQUATIONRemark C.13. There are two major difficulties in developing a general scale-equivariance theory for BITR. First, the attention term is not scale-invariant due to the existence of the soft-max operation, i.e., SoftMax(A) = SoftMax(cA) for vector A and c ∈ R + . That is the reason why we only accept degree-0 R + -equivariant input in our current theory. Second, when p / ∈ {0, 1}, we are not aware of any general way to represent degree-p functions using neural networks.We run all experiments using a Nvidia T4 GPU card with 16G memory. The batch size is set to the largest possible value that can be fitted into the GPU memory. We set bs = 16 for the airplane dataset, and bs = 4 for the wine bottle dataset. We train BITR until the validation loss does not decrease. For the airplane dataset, we train BITR 10000 epochs, and the training time is about 8 days when s = 0.7. For the wine bottle dataset, we train BITR 1000 epochs, and the training time is about 12 hours. The FLOPS is 14.5G in a forward pass (including the computation of harmonic functions), and the model contains 0.17M parameters.For Sec. 6.3.1 and 6.3.2, we use the normal vector computed by Open3D [46] as the input feature of BITR. The airplane dataset used in Sec. 6. We quantitatively verify the equivariance of BITR according to Def. 3.1. Specifically, we computeEQUATIONEQUATIONEQUATIONto verify the SE(3)-bi-equivariance, swap-equivariance and scale-equivariance of BITR, whereΦ B represent the BITR model, (r B (•), t B (•)) = Φ B (•)are the output of BITR, and all g = (r, t) ∈ SE(3) are written asEQUATIONNote that if BITR is perfectly equivariant, these three errors should always be 0.The quantitative results of the experiment are summarized in Tab. 3, where we can see that all errors are below the numerical precision of float numbers, i.e., less than 1e -5 . The results suggest that BITR is indeed SE(3)-bi-equivariant, swap-equivariant and scale-equivariant.Table 3 : Verification of the equivariance of BITR.∆ bi ∆ swap ∆ scale 5e -6 2e -7 5e -7To show the practical effectiveness of our theory on scale and swap equivariances, we consider an ablation study. We use the same data as in Sec. 6.2, and remove the weight sharing technique in Sec. 5.1 to break swap-equivariance, and force ϕ V in all layers to be a degree-1 function to break the scale-equivariance.We evaluate the trained model on 100 test samples, and report the mean and standard deviation of ∆r in Tab. 4, where we can see that removing an equivariance of the model leads to the failure in the corresponding test case, which is consistent with our theory.Table 4 : Ablation study of scale and swap equivariances. We report mean and std of ∆r This experiment numerically evaluates the robustness of the complete-matching property (Prop. C.11) against resampling, noise and partial visibility. We first sample X and Y of size 1024 from the bunny shape, and a random g ∈ SE(3), then we use a random initialized U-BITR to match X to gY . We consider different settings: 1) X and Y are exactly the same; 2) X and Y are different random samples; 3) Gaussian noise of std 0.01 is added to X and Y ; 4) Ratio s of X and Y is kept by cropping using a random plane.We repeat the experiment 3 times, and report the results of U-BITR in Tab. 5. We observe that the transformation is perfectly recovered when X = Y , which is consistent with the complete-matching property. Meanwhile, cropping the PCs leads to large decrease of the accuracy, while noise and resampling have less effect. This is consistent with our expectation because cropping the PCs has larger effect on the shape of PCs. We provide qualitative results in Fig. 7 .Table 5 : Results of complete matching using U-BITR. We report the training process of BITR in Fig. 8 , where we can see that the loss value, ∆r and ∆t gradually decrease during training as expected.Some qualitative results of BITR are presented in Fig. 9 . We represent the input PCs using light colors, and represent the 32 learned key points using dark colors and large points. As we explained in Sec. 4.3, the key points are in the convex hull of the input PCs, and they are NOT a subset of the input PC. In addition, as can be seen, the key points of the inputs do not overlap. We repeat the test process 3 times, and report the results in Fig. 10 . We observe that BITR achieves lower rotation error than LEV, and their translation errors are comparable. Meanwhile, NSM fails in this experiment. Note that we do not report the results of registration methods, because their loss functions are undefined due to the lack of correspondence. For BITR, we first obtain raw PCs by applying grid sampling with grid size 0.005 to the shape, and then randomly sample 5% points from the raw PCs as the training and test samples. The sizes of the resulting PCs are around 1000, which is close to the data used in the baseline methods. The data is pre-processed following [38] for the methods.The random sampling process in our method causes the randomness of test error. We quantify the randomness by evaluating on the test set 3 times, and report the mean and std of the errors in Tab. 1. We preprocess the 7Scenes dataset by applying grid sampling with grid size 0.1. The training and test sets contain 278 and 59 samples.We consider the 5 outdoor scenes in the ASL dataset: mountain, winter, summer, wood autumn, wood summer. We preprocess all data by applying grid sampling with grid size 0.9. We arbitrarily rotate and translate all data, and train BITR to align the i-th frame to the (i + 2)-th frame. We use the first Figure 11 : Results of reassembling wine bottle fragments. We compare the proposed BITR with DGL [44] , NSM [7] and LEV [38] . Zoom in to see the details. We report the results in Tab. 6. Our observation is consistent of that in the 7Scenes dataset: the result BITR is close to the optimum, thus a refinement can lead to improved results. Note that BITR+ICP outperforms all baselines in this task. In addition, GEO causes the out-of-memory error on our 16G GPU. An assembly result (wood summer) is presented in Fig. 12 .Table 6 : Results on the outdoor scenes of ASL. We report mean and std of ∆r and ∆t. All data in this experiment is generated using PyBullet [8] , where the objects in training and test sets are different in shape and position.Note that we do not report the quantitative results for this experiment because the metric (∆r, ∆t) is ambiguous due to the non-uniqueness of the correct solution. For example, for a correct bowl-placing result, the bowl is allowed to rotate horizontally in the plate, so the metric ∆r can be very large even for this correct assembly. A suitable metric is the success rate of the manipulation in physical hardware experiments as in [27] , but measuring this metric is beyond the scope of this work. In addition, the assembly methods such as NSM [7] and LEV [38] are not applicable because the canonical pose is not known, and we do not report the result of any registration method because the correspondence does not exist. We present a result of BITR on mug-hanging in Fig. 13 .In our current implement, we have accelerated most of the layers of BITR using the "scatter" function [24] . However, BITR is still relatively slow due to the independent computation of convolutional kernels, i.e., the harmonic function and the independent multiplication of radial and angular component in each degree. This is reflected by a low GPU utility ratio (about 20%). We expect to see a large speed gain (about ×5) if the above-mentioned computation is implemented using CUDA kernel, i.e., GPU utility ratio can be close to 100%. On the other hand, a rotation-based technique was recently introduced to reduce the computation cost of SE(3)-equivariant networks [21] . A promising future direction is to extend this technique to BITR.To explain the limitation of BITR in handling symmetric PCs, we consider the following example. For a pair of PCs X and Y , if there exists a non-identity rigid transformation g 2 such that g 2 Y = Y , and g ∈ SE(3) is a proper transformation that assembles gX to Y , then g 2 g is an equally good transformation that gives the same assembly result. In other words, the optimal transformation g is non-unique, which cannot be modelled by BITR. Actually, we notice that the result of BITR is undefined in this case. Specifically, let g = Φ B (X, Y ), we haveEQUATIONdue to SE(3)-bi-equivariance. Then we have g 2 = 1, which contradicts the assumption.To address this issue, we plan to extend BITR to a generative model in future research, i.e., it should assign a likelihood value to each prediction. For the example considered above, it should assign equal probability to g and g 2 g.Apart from the above-mentioned two limitations, there are several directions for future investigation. First, it is important to generalize BITR to multi-PC assembly tasks where more than 2 PCs are considered [14, 38] . Second, we expect the U-BITR model to be useful in self-supervised 3D shape retrieval or detection models, such as those in image object detection/retrieval [19] .NeurIPS Paper Checklist • The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper. • The authors are encouraged to create a separate "Limitations" section in their paper.• The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be. • The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated. • The authors should reflect on the factors that influence the performance of the approach.For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon. • The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size. • If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness. • While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations. • The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper. • The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions). • The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.) • The assumptions made should be given (e.g., Normally distributed errors). • It should be clear whether the error bar is the standard deviation or the standard error of the mean.• It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified. • For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates). • If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.Code is available at: https://github.com/wzm2256/BiTr 


Decision: The paper introduces a bi-equivariant transformer model to solve the task of pointcloud assembly, where two pointclouds with no overlapping regions need to be registered. Although most of the reviewers provided an initial positive assessment of the work, one reviewer recommended rejecting the paper on the grounds that the method is not equivariant. After a careful read of the paper and authors' response, we concluded that this is due to a misunderstanding of Eq.10 of the paper by the reviewer and the method is in fact equivariant. Therefore, although several reviewers raised the concern of an evaluation focused only on small datasets or missing baselines, we will follow the recommendation of the other reviewers and recommend the paper for acceptance. However, in the final version of the paper, the authors should include the feedback received from reviewers and improve clarity in the method's description (especially regarding the equivariant nature of the method and why Eq.10 makes the method equivariant) and the experiments section.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel architecture that extends the SE(3) transformers to become bi-equivariant for the task of rigid point cloud assembly. That is the authors enforce a single assembly for all rigid transformations of the source or target point clouds that is learnable by the network. They also propose how to extend the method for scale and swap consistent assembly of the point clouds. The authors aim to design a method that is correspondence-free to deal with the cases where the point clouds have zero overlap. They first learn (the same number of) keypoints on the original point clouds and concatenate them channel wise. After extracting the bi-equivariant features they utilize the Arun's algorithm to align the point clouds. Some experiments in small-scale datasets of zero overlap showcase benefits of the method.
strength_weakness: strength_weakness: - **On the assumptions/setup** : The setup of zero overlap is not addressed properly when equivariant constraints are enfroced. In the zero overlap case multiple correct poses is inherent. For example a chair next to a table can be placed in many "correct" ways. Some of them will be in the training data. But when two different configurations are presented in the training data that breaks the equivariant assumption. Or symmetrically, enforcing equivariance from the first configuration can never reduce the error of the second configuration. I believe the problem of zero overlap has to be formulated more carefully and these consequences should be discussed. 
- **On the method**:
    1. **Correspondence-free methods should take care of the permutations of the points**: Concatenating the two point clouds after processing them independently does not properly take care of the permutation of points. In fact an individual permutation of the points in X,Y would result in permuted keypoints in Eq.(10). When the keypoints are concatenated each keypoint will match a different keypoint which means that the bi-equivariant transformer will view the permuted case as completely different features.    
    2. **Assumptions on the sizes of the parts**: Extracting the same amount of keypoints from both point clouds might be restrictive when sizes differ by a lot. 

- **On the experiments**:  Fewer experiments are performed in this paper than what is standard in the literature. Also the scale of the datasets is much smaller in number and sizes.
   - Low-overlap settings (<10%) exist in KITTI, 3DMatch  that the authors do not consider. This also raises the question of the scalability of the method. 
   - The sizes of the parts also answer the question regarding extracting fixed number of keypoints from the two point clouds. E.g. for large point clouds as in 3DMatch how many keypoints are enough?   

- **References**:  Missing literature. A lot of point cloud processing papers utilize the SE(3) transformer (some are even bi-equivariant) for registration, reconstruction, docking etc. Others address equivariant registration with other networks. Some suggestions:
    - On Point Cloud Registration:
       - M.Zhu et al. "Correspondence-Free Point Cloud Registration with SO(3)-Equivariant Implicit Shape Representations." 
       - C.Lin et al  "Coarse-to-Fine Point Cloud Registration with SE(3)-Equivariant Representations."
    - On PC Processing using SE(3)-transformer and/or bi-equivariance:
        - E. Chatzipantazis et al.   "SE(3)-Equivariant Attention Networks for Shape Reconstruction in Function Space".
        - Y.Peng et al "SE(3)-Diffusion: An Equivariant Diffusion Model for 3D Point Cloud Generation."
        - O. Ganea et al. "INDEPENDENT SE(3)-EQUIVARIANT MODELS FOR END-TO-END RIGID PROTEIN DOCKING."
        - C.Lin et al. "SE(3)-Equivariant Point Cloud-Based Place Recognition".
questions: questions: - Line 28: "Sensitive to the initial poses of PCs". Is there any reference in the literature for that?
- In practice we do not know id the point clouds have zero overlap or some overlap. How can you discriminate such cases?
- The zero overlap case could in principle involve many solutions. For example a chair next to a table. Enforcing the configuration that the training data suggest and restricting it with equivariant constraints can hurt performance as the dataset could have other possible configurations for other training samples that are inconsistent with the constraint.
- The setup of zero overlap has to be formulated more properly.  
- In the method the authors extract the same amount of keypoints for the X,Y point clouds. Is that restricitve in cases one point cloud is larger than the other?
- Shouldn't there be some sorting of the keypoints before concatenation? See Weaknesses too.
- Why is BITR+ICP bi-equivariant? 
- How does the method scale to large point clouds with complicated distributions as in 3DMatch? 
- Are normals given as features in this and the rest of the methods? Are the normals computed before or after the cut?
- Is the method trained per-class or for the whole dataset?
limitations: limitations: See the section in Weaknesses and then Questions.
paper_summary: paper_summary: The paper addresses the problem of assembling point clouds. Given two partial, potentially unmatched point clouds, the objective is to determine the rigid transformation that best aligns them in relation to the unknown complete shape. The proposed approach is a learning-based method, utilizing an architecture that adheres to symmetry constraints inspired by Arun's method. The effectiveness of the method is demonstrated through a toy experiment and synthetic data, with evaluations on real data included in the appendix.
strength_weakness: strength_weakness: The presentation quality could be improved. For example, figure 2 is hard to understand. The description of the toy experiment is not easy to follow as well.

Evaluating the benefits of bi-equivariance. An alternative to bi-equivariance, is to train a regular equivarince network on the task of shape completion and then align the two partial inputs according to the predicted completed shape (using one of the baselines that use correspondences).
questions: questions: Does the suggsted bi-equivariance architecture generalize better than a single-equivariance network trained on shape completion?
limitations: limitations: yes
paper_summary: paper_summary: This paper proposes an SE(3)-bi-equivariant approach for point cloud assembly, addressing the difficulty of assembling non-overlapping point clouds where traditional correspondence matching methods struggle. The proposed BITR (BI-equivariant TRansformer) solves this problem by exploiting the symmetry of the point cloud assembly task as a powerful inductive bias, instead of relying on point correspondence.

Specifically, BITR first merges two distinct 3D point clouds into a 6D point cloud with features that are the tensor product (also known as the Kronecker product or outer product) representation of two SE(3) irreps. These merged 6D point clouds are then processed through novel SE(3)xSE(3)-transformer layers derived from the Wigner-Eckart theorem of G-steerable kernels [5,17], and finally through an SE(3) projection layer that resembles Arun’s method.

The paper also includes theoretical analysis showing that BITR effectively inherits the symmetry properties of the classical Arun’s method in three aspects: 1) SE(3)-bi-equivariance, 2) Swap equivariance, and 3) Scale equivariance. This enables effective point cloud assembly without assuming known point correspondence. These benefits are also validated through extensive comparison with state-of-the-art baselines.
strength_weakness: strength_weakness: 1. The paper may be challenging for readers without a background in representation theory and SE(3)-equivariant neural networks. The proposed method is complicated and might be difficult to implement and extend.

2. ~~Experimental validation is rather confined to simple problems in which object parts are clipped by a random plane. It isn’t sufficiently verified for more general and realistic assembly tasks in which the parts are not cleanly cut by a plane.~~ Addressed.
questions: questions: 1. Why include type-1 features in the last layer before softmax in Eq (12)? Why not use only type-0 irreps? Why not use type-2 or higher?

2. How is locality (neighborhood) defined for 6D point clouds? Directly using KNN for 6D point clouds to construct a neighborhood graph does not make sense to me.

3. The key points of X and Y are ordered and thus not permutation-invariant. Could this have any negative consequences?
limitations: limitations: 1. The proposed method is not generative and thus cannot handle scenarios where multiple assembly poses are equally valid.

2. The $SE(3)\times SE(3)$-transformer layer in BITR relies on the Clebsch-Gordon tensor product, which is notorious for its computational complexity, scaling with $O(L^6)$ where $L$ is the maximum degree. The paper proposes a 2nd-order Clebsch-Gordon tensor product, which likely has similar or worse complexity. The lack of efficient CUDA kernels exacerbates this problem.

These limitations are appropriately discussed by the authors in Appendix E. The reviewer believes that these limitations are not fundamental and could be addressed in future research. For example, recently proposed eSCN Convolution [4] and Gaunt tensor product [5] have reduced the computational complexity of the C-G tensor product from $O(L^6)$ to $O(L^3)$. Extending these state-of-the-art $SE(3)$-equivariant mechanisms into a bi-equivariant form would be an interesting future direction.

[4] Passaro et al., "Reducing SO(3) Convolutions to SO(2) for Efficient Equivariant GNNs," ICML 2023

[5] Luo et al., "Enabling Efficient Equivariant Operations in the Fourier Basis via Gaunt Tensor Products," ICLR 2024
paper_summary: paper_summary: This paper introduces a new network to solve the point cloud assembly task, where a 3D transformation is predicted to “align” two point clouds. The proposed network, BTIR, is designed to enforce the symmetries of the point cloud assembly task into the network layers. Specifically, it enforces SE(3) bi equivariance, scale equivariance and swap equivariance through weight constraints. The network is evaluated on several point cloud assembly tasks. It significantly outperforms existing equivariant methods in some settings.
strength_weakness: strength_weakness: - There are several works from the robotic manipulation community that are related to BTIR. These should be discusses in RW and perhaps included as baselines, especially the bowl placing and mug hanging tasks. [1,2]
- The definition of point cloud assembly in the introduction is a bit imprecise. What does it mean for two non-overlapped point clouds to be “aligned”, when the points themselves will not overlap? Presumably what aligned means depends on who generates the dataset. 
- Some of the writing is inaccurate or unclear. On page 2, “[correspondence based methods] are often sensitive to initial positions of PCs”; most correspondence based approaches use Aruns method which is not sensitive to initial poses. On page 2, “SE(3)-bi-equivariance does not rely on correspondence”; do you mean your BTIR method does not? 
- The results on real data and visual manipulation are of interest and should be included in more detail in the main paper. They are more challenging and would better demonstrate that the proposed method is generally effective. Also, more discussion should be included about these results. For instance, why is it that BTIR without ICP is relatively bad on 7Scenes but better on outdoor scenes of ASL?
- The experimental results are somewhat limited. More baselines should be included (point cloud alignment is a very well studied problem) and more challenging or varied datasets used. For instance, compare performance across multiple shapenet classes rather than just the wine bottle. Very little space in the paper is currently dedicated to the experiments and discussion. Given BTIR is more computationally heavy than other methods, it should be clear from the experiments that it is robust and effective enough to motivate its use. 

[1] Pan, Chuer, et al. "Tax-pose: Task-specific cross-pose estimation for robot manipulation." Conference on Robot Learning. PMLR, 2023.
[2] Ryu, Hyunwoo, et al. "Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation." Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2024.
questions: questions: - In the limitations, it is mentioned that the method is deterministic. How does it solve the wine bottle task in Table 1? Is the loss computed up to the to next symmetry or is the symmetry broken in the loss?
- It is not clear whether the method is limited to type 0 and 1 features or if they were only mentioned for simplicity. Do the layers introduced in the paper work for higher order features ? Is there a reason other than computational effort that higher order features are not used?
- It should be mentioned that the enforcing scale equivariance on type 1 features was explored in another paper [1].
- Is this the first instance of biequivariant filters being used to process two inputs simultaneously? If so this should be stated more clearly as it has many interesting possibly applications even if it is more computationally costly. 

[1] Yang, Jingyun, et al. "Equivact: Sim (3)-equivariant visuomotor policies beyond rigid object manipulation." arXiv preprint arXiv:2310.16050 (2023).
limitations: limitations: Limitations are discussed in Appendix E. If possible these limitations should be included in the main paper if only briefly.
paper_summary: paper_summary: This paper presents an end-to-end framework for pointcloud correspondence and relative pose estimation with bi-equivariance to per-part poses. The framework is also equivariant to scaling and swapping part orders. It also designes a transformer architecture with $\mathrm{SE}(3)\times\mathrm{SE}(3)$-equivariance. The effectiveness of the framework is shown with experiments on the standard ShapeNet benchmark as well as some other applications.
strength_weakness: strength_weakness: - To my understanding, "shape assembly" usually refers to assembling parts of a shape that exactly matches their boundaries (like jigsaw), and usually with more than just two parts. I feel "shape registration" is a more proper description of the task in the paper.
- Following my previous comment, I think the method should also be compared to prior works on shape registration, including classic optimization-based methods like ICP, and more recent learning-based methods such as [1, 2] below.
	- A minor point: to compare to these works and show the advantages brought by equivariance, it may be a good setting to compare their data efficiency. I believe that with less training data or less pose augmentation, the proposed framework can have less performance drop than these non-equivariant prior works.
- [3] is also a related work, especially to the robot manipulation task in Sec. 6.6 (and Suppl. Sec. D.9).

[1] Wang, Y., & Solomon, J. M. (2019). Deep closest point: Learning representations for point cloud registration. In Proceedings of the IEEE/CVF international conference on computer vision (pp. 3523-3532).

[2] Wang, Y., & Solomon, J. M. (2019). Prnet: Self-supervised learning for partial-to-partial registration. Advances in neural information processing systems, 32.

[3] Ryu, H., Kim, J., An, H., Chang, J., Seo, J., Kim, T., ... & Horowitz, R. (2024). Diffusion-edfs: Bi-equivariant denoising generative modeling on se (3) for visual robotic manipulation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (pp. 18007-18018).
questions: questions: I don't have questions about the method. But I would really like to see experimental comparisons with the pointcloud registration works.
limitations: limitations: Limitations are well-discussed in the paper.

Comments: title: Opinion on the shape completion
Response to Authors
reply to authors
comment: I don't believe shape completion can replace bi-equivariance because there is always some uncertainty in shape completion. To show this, consider two parts, $X$ and $Y$. If one infers $g$ from the reconstruction $\hat{Y}\sim P(\hat{Y}|X)$, the marginal distribution of the inferred pose is $P(g|X,Y)=\int P(g|\hat{Y},Y)P(\hat{Y}|X) d\hat{Y}$, whose integral is intractable unless $P(\hat{Y}|X)$ is deterministic. In contrast, bi-equivariant models, which do not rely on shape completion, can directly infer $P(g|X,Y)$. Although BITR is not a generative model, the argument still holds that it does not require intractable marginalization over the uncertain shape completion $\hat{Y}$.
Thanks for the clarification. All concerns and questions have been addressed.
Thanks for your reply.  It is quite easy to see that the key points are permutation invariant, simply because the set of features of neiborhood points are always the same under permutation. (It has nothing to do with optimal transport) In other words, the whole network does not rely on the specific index of the node.
## Permutation Equivariance: 

I believe there is methodological error in this paper and the authors have not answered correctly the **permutation equivariance** question that has been raised by many reviewers. I will explain the problem next because I believe it will help the authors fix it in the future:

Concatenating the keypoints $\tilde{X}, \tilde{Y}$ that are extracted using equation (10) means that if the point clouds are presented with a different numbering of the points i.e. $P_1X, P_2Y$ for $P_1,P_2$ permutation matrices (which is **always the case** for point clouds), then the keypoints are also permuted with different permutations ie. $P_1 \tilde{X}, P_2 \tilde{Y}$. (They are not invariant as the authors suggest in their answer to reviewer fq5D because the index/node $j$ moves too to some $j'$ not only the index of the neighbors $i$; in other words, the **same feature** appears but **in different position in the vector**). Then, when the features are concatenated one gets $z_u =  x_{u'} \oplus y_{u''}$  for different $u',u''$ that depend on $P_1,P_2$. Thus, **the same input point clouds** will produce **different vectors $z$**! This breaks permutation equivariance and potentially needs many augmentations to recover. Moreover, it is not clear what this vector $z$ now consists of since all combinations are possible. 

This is not the same (as the authors suggested) for "INDEPENDENT SE(3)-EQUIVARIANT MODELS FOR END-TO-END RIGID PROTEIN DOCKING" because in that paper the keypoints are enforced to correspond by the optimal transport loss. The authors do not do such a thing (and maybe cannot, because it would turn the method from a correspondence-free to a correspondence based). 
 
This is probably why the method cannot generalize beyond per-class settings (like registration methods do) and even require many cuts from a single point cloud to do the assembly (overfitting).  

## Non-uniqueness and overlap:

This question I thing is not answered correct either. Of course, even in the large overlap setting there can be many solutions (mainly due to symmetry). The argument was not that though. In the no-overlap setting it is very common that there might be multiple solutions (even if no symmetric objects exist). For example, if one part is a table and another is a chair (that is we have no overlap) then the geometric relation turns into a functional one. You can put the chair near many places in the table. That is not due to the symmetry of the parts. That is why I am arguing that the setting need a better formulation regarding uniqueness. If in  the data there are tables and chairs in different configurations satisfying both the data and the equivariant loss is impossible. This has not been properly discussed in the limitations. 

## Same amount of keypoints in two parts:

The authors have not answered this question either which I think is important especially in cases that the two point cloud parts differ by a lot. In order to observe that fact, the paper needs to provide more strong experimental results than toy datasets like per-class ShapeNet. 

## Comment on the weak experimentation:

The authors after the questions admit that the non-overlap setting cannot be identified. Thus it is only reasonable that the method should perform equally well to a setting of small overlap (or when no-overlap and small overlap exist in the dataset). The authors also claim that the method is not "just a registration method" in the answer above which implies that it will not fail silently in such situations where small overlap exists. However, no experimentation in standard datasets with low overlap (<10%) like 3DLoMatch have been provided. I argued above why this might be a methodological issue. It is also probably a memory scalability issue.

Even in the 7Scenes  dataset, which I believe should be in the main text,  the method does not outperform GeoTransformer or ROI even when the authors replicate their steps to make their method as similar as possible to the other papers. Also it is not compared properly against other state-of the-art methods (like Lepard: Learning partial point cloud matching in rigid and deformable scenes, You Only Hypothesize Once: Point Cloud Registration with Rotation-equivariant Descriptors
).    

In total, based on the arguments above and the responses of the authors I believe that the paper has a good approach to the problem, however, its major methodological errors in the design and weak experimentation cannot reveal the benefits of the method in non-toy settings. I cannot in good conscience suggest acceptance of the paper as is, but I believe that if the authors take the reviews into consideration it could potentially improve the approach a lot.
Thanks for your reply.

As we mentioned in our first reply. No. We do not think the suggested method will work, nor do we know any assembly method like the suggested one. Because the generated complete shapes do not necessarily have corresponding points due to the non-uniqueness of shape completion (even for high quality completions). For example, for the outdoor scene (like the one in Fig 11), let's say a tree is observed in $Y$ but not in $X$. It is difficult to generate a tree to complete $X$ and guarantee that this tree is exactly the one observed in $Y$. (It would be difficult to guarantee that a tree is generated (the algorithm may generate a house instead of a tree) and this generated tree is exactly the one in $Y$ (the same position,  shape and orientation). )  This ambiguity also exists for other type of data like indoor scenes (a different table in a room), objects (a different tail of the airplane in Fig 1b), which makes the generated shape not suitable for the assembly task.
I thank the authors for their rebuttal. However, I remain concerned about the benefits of bi-equivariance compared to simpler baselines. To reiterate my suggestion: would it be possible to use an equivariant network trained on shape completion, and then apply a simple method like ICP to register the equivariant network outputs for the assembly task?
Thanks for your reply.

1. The fragments are not symmetric (If they were, there would be many possible solutions). These fragments are from the BB dataset, which are generated by simulating the physical process of breaking the objects. So they are not likely to be symmetric (the broken surfaces can be non-regular). This can be seen by zooming in Fig.10.

2. Agree. We have now added a sentence in the first bullet point in the introduction section. "...In addition,
    the $SE(3) \times SE(3)$-transformer used in BITR is the first $SE(3) \times SE(3)$-equivariant steerable network to the best of our knowledge."
Thank you for the response.

Re: 7, I am still confused.  In Figure 10, it appears that most of the fragments are symmetric about the up-axis, so doesn't that mean that many possible transformations produce good alignment?   

Re: 9, yes I see it now.  This is an important contribution to the field, and I think it should be restated clearly in the contribution bullets as well.