DeepSPF: Spherical SO(3)-Equivariant Patches for Scan-to-CAD Estimation
Authors: Driton Salihu
Adam Misik
Yuankai Wu
Constantin Patsch
Fabian Seguel
Eckehard Steinbach
Venue: ICLR 2024 poster (2024)
Keywords: 3D Point Cloud Representation
3D Point Cloud Registration
Scan-to-CAD
Spherical Gaussians
Equivariant
Primary area: representation learning for computer vision, audio, language, and other modalities
TL;DR: We present Learnable Spherical Patch Fields (DeepSPF), a versatile, SO(3)-equivariant, and easily integrable backbone suitable for instance-based point networks

Abstract: Recently, SO(3)-equivariant methods have been explored for 3D reconstruction via Scan-to-CAD.
Despite significant advancements attributed to the unique characteristics of 3D data, existing SO(3)-equivariant approaches often fall short in seamlessly integrating local and global contextual information in a widely generalizable manner.
Our contributions in this paper are threefold.
First, we introduce Spherical Patch Fields, a representation technique designed for patch-wise, SO(3)-equivariant 3D point clouds, anchored theoretically on the principles of Spherical Gaussians.
Second, we present the Patch Gaussian Layer, designed for the adaptive extraction of local and global contextual information from resizable point cloud patches.
Culminating our contributions, we present Learnable Spherical Patch Fields (DeepSPF) – a versatile and easily integrable backbone suitable for instance-based point networks.
Through rigorous evaluations, we demonstrate significant enhancements in Scan-to-CAD performance for point cloud registration, retrieval, and completion: a significant reduction in the rotation error of existing registration methods, an improvement of up to 17\% in the Top-1 error for retrieval tasks, and a notable reduction of up to 30\% in the Chamfer Distance for completion models, all attributable to the incorporation of DeepSPF.

Published as a conference paper at ICLR 2024
DEEPSPF: SPHERICAL S0(3)-EQIIVALLAN PATCHES FOR SCAN-TO-CAD ESTIMATION
Driton Salihu", Adam Misik*i, Yuankai Wu", Constantin Patsch",
Fabian Seguel", Eckehard Steinbach"
"Chair of Media Technology and Munich Institute of Robotics and Machine Intelligence Department of Computer Engineering
Technical University of Munich, School of Computation, Information and Technology
i Siemens Technology
[drrtton. salion,.aalhu, muankai.wu, constantin. / @r
cckehard.steinbaccheeee @tuma
adam.mis silisikkes emens.
ABSTRACT
Recently, SO(3) -Quuivaquv methods have been explored for 3D reconstruction via Scan-to-CAI -CDC Despite significant advancements attributed to the unique char- acteristics of 3D data, existing S0(3)-cquivariant approaches often fall short in seamlessly integrating local and global contextual information in a widely gen- eralizable manner. Our contributions in this paper threefold. First, intro- duce Spherical Patch Fields, a representation technique designed for patch-wise, SQuuivaquivarrant 3D point clouds, anchored theoretically on the principles of Spherical Gaussians. Second, we present the Patch Gaussian Layer, designed for the adaptive extraction of local and global contextual information from resizable point cloud patches. Culminating our contributions, we present Learnable Spher- ical Patch Fields (DeepSPF)- versatile and easily integrable backbone suitable for instance-based point networks. Through rigorous evaluations, we demonstrate significant enhancements in Scan-to-CAD performance for point cloud registra- tion, retrieval, and completion: significant reduction in the rotation error of ex- isting registration methods, an improvement of up to 17% in the Top-1 error for retrieval tasks, and notable reduction of up to 30% in the Chamfer Distance for completion models, all attributable to the incorporation of DeepSPF.
INTRODUCTION
Many 3D applications, such virtual reality (VR) and augmented reality (AR), rely manually designed 3D indoor environments. Recently, 3D point clouds have been considered as a replacement due to the large amount of work involved in manually designing 3D environments. With the prolifer- ation of 3D vision sensors (c.g., Stereolabs ZED, Azure Kinect DK), the quality of point clouds has increased significantly. However, due to the lack of explicit surface information, the improvements in scanning are not necessarily applicable to 3D content applications.
In this work, we target the subtask of reconstructing indoor environments through point clouds by Scan-to to CAD (S2C) methods (Avetisyan et al.) 2019a by 2020 Hampali et al. 2022 (alihu & Stein-
bach
2023). By retrieving existing CAD models from a public database, such as ShapeNet (Chang
et al. 2015) we can align those to scanned models to achieve a high-quality reconstruction of scanned objects or scenes. Therefore, the tasks we try to solve in the context of high-quality $2C reconstruction include registration, retrieval, and completion.
To address these tasks, networks such as PointNet (Qi et al. 2016) enable learning features from 3D data. However, we address three issues that hinder the learning capabilities in the current state of the art. First, most networks do not consider different poses of point clouds in the latent space. This is considered major problem, especially in registration and retrieval tasks (Deng al. 202I). To avoid extensive data augmentation, the recent introduction of equivariant networks (Deng al. 2021 Esteves al. 2018 Klee al. 2023 Fuchs et al. 2020) has enabled the extraction of

---

Published as a conference paper at ICLR 2024
S0(3) features from point clouds. These methods often equate local and global SO(3)- equivariance, which leads to an insufficient investigation of S0(3)-qquivarianc between global and local structures. Second, when local structures are defined, they usually based fixed, prede- fined patches that cannot be adjusted when learning from point clouds (Rao et al.) 2022). Third, consider zero point-to-point correspondence between point clouds acquired by different sens- ing devices. Representation methods such as Salihu & Steinbach| (2023). while neglecting local information, have shown that spherical representations can help improve registration tasks.
In this work, we address these three issues by introducing a novel point network for instance-based applications such as registration, retrieval, and completion. To extract local and global information, we propose Spherical Patch Fields (SPF), point cloud representation based on Spherical Gaussian (SG) to generate many spherical patches that each produce a cotation-equivarian representation. To improve the learnable representation, we introduce a Patch Gaussian Layer (PG-Layer) that can adapt the size of spherical patches to continuously obtain and correlate information from learnable local and global structures. Our contributions can be summarized as follows:
We introduce SPF, new representation that maps one 3D point cloud to many spherical patches represented by SGs. In addition, SPF learns inter- (spherical to point patch) and intra-relations (spherical to spherical patches) in graph- based manner.
We introduce PG-Layer, a convolutional layer based on Spherical Gaussians. PG-Layer retains the original mathematical description of SGs, allowing for deeper networks and thus improving learning capacity. Further, PG-Layer introduces low-frequency analysis to improve shape recognition. Finally, PG-L.ayer learns the size of each spherical patch, providing either local or global information depending the optimization goal.
We provide DeepSPF as an easily integrable backbone to demonstrate improvements in S2C and the corresponding subtasks of point cloud registration, retrieval, and completion through our extensive evaluation.
RELATED WORK
Spherical Gaussian Representations Salihu & Steinbach (0033
introduced the concept of SGs to the 3D point cloud domain with SGPCR and SGConv. SGPCR is a global rotation- -qqivarnnquu representation for point clouds, and SGConv is a rotation-invariant convolution for registration and retrieval. While SGPCR significantly improves registration between point clouds, local informa- tion lost due to global positional encoding. In contrast, we introduce SPF, spherical patch-wise graph-based representation. SPF enables extracting local and global information and exchanging information between different parts of the point cloud through a Icarnable graph. Further, we intro- duce PG-Layer, which relies differentiable radial component to adaptively control the size of SPF spherical patches. In addition, PG-Layer introduces low- cow-frequency analysis into the latent en- coding. Finally, compared to SGConv, PG-Layer retains the original form of SGs, allowing deeper network structures and resulting in higher accuracy for various S2C-related applications.
Eiqutiin-Eqqivaiiant Point Networks Recently, large body of rotation-equivariint methods has been introduced due to the impact rotation can have on the learning process. Similar to our approach, much previous work has aimed to construct an SO(3) representation (Esteves et al. 2018 Sun & Blu 2023 Chatzipantazis et al. 2023 Klee et al. 2023 Liu et al. 2023). In contrast, we introduce an adaptive patch-wise S0(3)-based representation. Deng et al. (021)) provides vector-based convo- lutions that allow the mapping of SO(3) actions to the latent space, enabling rotation equivariance in 3D point networks. In comparison, we show improvements over VN due acquiring local and global information. In addition, our rotation-equivatii method significantly improves the registra- tion performance compared to
approaches (Yuan et al.) 2020 Chen et al. 2019).
Scan-to-CAD Applications Many methods have been introduced for S2C (Avetisyan et al. 2019 9a b)
2020 Misik et al., 2023 Hampali et al.) 2021; Salihu & Steinbach, 2023] Zhao et al.) 2021). In
our method, we mainly focus on the registration and retrieval of CAD objects to real-world scans. As such, we are similar to the works introduced by Zhao et al. (2021) and [aalihu & Steinbach 2023).. In contrast, DeepSPF adaptively learns global and local information, allowing for tighter latent space between scan and CAD objects. We inspect three related methods for point cloud registration: DeepGMR (Yuan et al. 2020), DeepUME (Lang & Francos 2021), and SPGCR (Salihu
2

---

Published as a conference paper at ICLR 2024
& Steinbach 2023). DeepGMR considers PointNet-based network to extract features, which are used to obtain corresponding Gaussian Mixture Model (GMM). DeepUME is Transformer (Vaswani al., 2017) --ased registration network that improves registration between noisy point clouds. Compared to these approaches, our method is not only rotation-equivarian but also generalizes well previous works. Finally, many methods for completing point clouds exist, but to provide a fair comparison between encoder structures, we restrict ourselves PointNet (Qi et al. 2016)-based networks. Unlike DeepGMR, which uses PointNet, our encoder network significantly improves completion performance without further adaptations.
different data distributions and improves the results compared to
DeepSPF: Leamable Spherical Patch Fields
(b) processing
(c) SPF: Spherical Patch
(d) PG-Laer: Patch Gassian Layer
(b3)
Furthest (b.1) Point fno Sampling
(d.1)
(d2)
B(v.p)
Instance Point Cloud
2
Spherical Sampling
SPF(H)
((ro)
V(r))
Figure I: We propose DeepSPF, an encoder for the -qqui-equivariant representation of point clouds. Given an instance-based point cloud p (a), our preprocessing (b) obtains Jno samples using furthest point sampling (b. At each sampled point, we generate spheres (b.2) using the spherical sampling method presented in Vogel (1979). We visualize the spherical samples in (b.3) for fno 3 and
C = 32. (c) from v and p. we derive SPF using Eq.. (2) Finally, in (d), the PG-Layer introduced in Eq. (13) improves the latent encoding. Our PG-Layyer learns the size of the spherical patches and adjusts each one according to the information density of the point cloud.
DEFPSPF: LEARNABLE SPHERICAL PATCH FIELDS
In this section, we introduce the core components of SPF and convolutional PG-Layer, shown in Fig. ure In (eeciin 3.4, we provide DeepSPF, an example backbone using SPF and PG-Layer. First, provide an overview of SGs (G) that are based on the definition of Wang al. (2009):
(I) Salihu & Steinbach 2023) extended the definition of [Wang al. 2009) by assuming that the lobe axis E R is equivalent to the set of N unordered points of the point cloud X' = f In addition, was assumed that the lobe sharpness lobe amplitude a R3 is the distance from to the spherical direction parameter v S One global pre-defined sphere is used calculate a through the k-nearest neighbor algorithm. For more information, we refer to the original work done by Salihu & Steinbach (2023).
G(v; a, >, = A(v""
(0,) could be learnable parameter. The
In the following, we show how to adapt a SG to obtain the proposed SPF presentation and how to further improve the representation using PG-Layer.
3.1 SPF: SPHERICAL PATCH FIELDS
We consider the previously shown representation from Eq. (1) as a simplification and thus to be incomplete. This assumption is based on the definition of SGs first proposed by Narcowich & Ward
(1996). For more information on [arccowich & Ward (1996) we refer the reader to the appendix. In the context of representing point cloud p, propose encode the latent vector z
= SPF((;A,,v, = E(v, p)V(r)U(P(v) (eee(( Pu) p-1),
E N
2 E R|
(2)
where E(v.p) is the asymmetric edge function, first defined in Wang et al. (2018), and adapted to our problem following Eq. (3) with K as the max number of nearest neighbors.
B(v.p)
max [Ø(x, - xx) + 0(x,, - xx)) (v.p)
(3)
[J1,0EKK

---

Published as a conference paper at ICLR 2024
To define the edge function, which generates the graph, we consider two learnable parameters 0 and
6, which provide a trade-off between To (V p), and (v p). The term x; (v, p), shown Egg. (4)
represents the center of the patch generated by the sample spheres. 0, p). seen in [q. (3)
responds to the inter-relationships between the points and spheres that we establish by limiting the
distance through radius of each spherical patch. T. (v. p), defined in Eq. (6) corresponds to the intra-relationshis between the individual spherical patches.
[8]
(v. (2) pa (p) s.t. put)
2 [t|
t
(4)
()(up) = arg max || - pll s.t. ||V pll <r
(5)
(v = arg max ||V - pll
(6)
[e]
V(r) the volume of the sphere S obtained by the spherical sampling v proposed in Vogel 1979), extended by an adaptive differentiable radial component r. We consider the spherical volume due to Narcowich & Ward (1996), which states that SG constrained by spherical volume. For simplicity, we denote V(r) by the volume equation of a sphere:
V(r) ..
By simplifying (rr,, we obtain information about the radius of each sphere. By adjusting this parameter in differentiable manner, the size of the initial spherical sampling can be increased or decreased, moving from local global information depending on the information density of each area.
is a fully connected layer (fv : R² + IRA) that upscales the dimension of the Legendre Polyno- mial calculated through the Spherical Harmonics coefficients Y and its complex conjugate Y to a latent dimension A:
U(P(u,p) = (e(()u)))) P(u.P) = Y(u)
(8)
(9)
By applying Eq. (8) we add the Legendre polynomial to the estimation process. This allows the network to derive low-frequency information from the point cloud. For point cloud data, Hu et al. (2022) showed that now-frequency components contribute to the shape of the object, improving subsequent tasks depending the point cloud.
3.2 ROTATION-EQUIVARLANN
SPF
Given the inherent rotation- equivariant representation of SPF, show that for EorEE.. (2) random rotation Rp on the point cloud p, we obtain information respect to the inverse rotation R2 Rp on the sphere. For this purpose, parts of B... (2) examined individually. First, show that the low-frequency part of Eq. (9) is rotation-equivarian under the introduced assumptions:
assuming
P(Ree. Ryp) (RR,))
Y
R Y(v)
(10)
Since Spherical Harmonics are basis functions for SQiia-equivariant representations, we follow previous work such Esteves al. (2018).
Moreover, for the edge-based function E(v, p). we show oolation-quiivarianne by Eq. (11) As the mean is not affected by the rotation, for ((()) s p((), only show the rotation-equivariaace for
4

---

Published as a conference paper at ICLR 2024
Eq. (5)
Here we obtain:
3j arg max R Rppl12
|
arg max (RyV (pp) R
|
R
arg max IV -112
(11)
with fr denoting the trace operation.
3.3 PG-LAYER: PATCH GAUSSIAN LAYER
In addition to our proposed SPF, we also introduce a SG-based convolution layer. A major problem with previous work (Salihu & Steinbach 2023) is that the convolution of SGs deforms the original SG representation, which means that only one iteration of learnable encoding is commonly applied. This results in the inability learn relevant latent encoding from larger objects for tasks that do not rely solely on position-based coding. In contrast to conventional convolutions, we leverage the inherent benefit of SG and its parametric nature simplify the convolution limited number of multiplications.
The conventional convolution between SG-based point cloud representation G and SG kernel H results in GR. The convolution between two SGs is defined as:
=R = (G H) PH) 4тaçaи sinh (lAGPG + λиPн()) * (g
(12)
AA ([AcPa + >=PHIl)
here, in Eq. (12) previous work considered the simplification made by Tsai & Shih (006 which
results in the deformation of the original SG description. While this does not affect the rotation- equivariant representation, it does affect the possibility for multiple iterative convolutions. To retain the original mathematical description, we formulate the convolution as
(z SPF(()) PH, Ac,
=
B(v, Ee)E(, PH) PP: Do p) 1)) (13)
Notably, we introduce Vir), which allows to adjust the radius of the initial spheres in differen- tiable manner. We further assume that Ag ss An xe 2\n and obtain
a E(v, pR ((l Ac(""
(PGPs)-1) (p(r"(
SPF(pR),
(14)
where Eq. (14) matches the original definition of Eq. (2) and subsequently Eq. (1) allowing for mul-
tiple successive PG-Layer while learning and adjusting the size of the spherical patches depending on the objective goal.
3.4 ARCHITECTURE OF DEEPSPF
Although many different backbone structures could be created with the presented combination of SPF and PG-Layer, we focus one backbone structure that we throughout all experiments. In Figure 2, we show our presented architecture, which follows the structure of PointNet (Qi et al.) 2016). The DeepSPF backbone contains three Set Abstraction (SA) layers, each of which converts the point cloud into a SPF representation, with each SA layer containing m PG-Layer modules. For each SA layer, we set a different, decreasing radius for the initial patch generation. The final output 2, E IR XCxA, of the n-th layer is the latent vector, encoded from SPF and the PG-Layers. The latent dimension of each output is denoted as A,. Before the latent vector can be encoded, we sample the point cloud f.o times by Furthest Point Sampling (FPS). We sample the corresponding sampling spheres with C' points by the method presented in Vogel 1979). with |M| = ʃoo x C.
5

---

Published as a conference paper at ICLR 2024
4 EVALUATION
In this section, we show a qualitative and quan- titative evaluation of our architecture for vari- ous tasks related to S2C. First, we show results for point cloud registration and retrieval on real and synthetic data. In addition, we provide an ablation each component of SPF. We then show results for point cloud completion com- pared to comparable rotation- reuiivriiuivariinnt meth- ods.
5A Layer
Layer
Layer
PC LAONE
4.1 DATASETS
Example backbone of our proposed DeepSPF network. We consider a point cloud and sampled sphere input each abstraction layer. Each abstraction layer uses the latent vec- ån obtained by the previous layer but always re-uses the original inputs. Each sampled sphere initialized with a different decreasing radius at each layer.
Figure 2:
For registration, we evaluate on Model- Net40 (Wu et al. 2015), with 1024 points uni- formly sampled from each model in the dataset. The dataset consists of 40 object categories with 9843 point clouds in the training set and 2468 for the test set. Here, we consider two evaluation cases. First, improvements in registration compared to net- works trained with clean data and tested on noisy point clouds with random rotation be- tween 0 and 180 and random translation up to 5cm, evaluated Yuan al. (2020). we test ietro-ittersettion noise introduced in in|Lang & Francos (222)) For this, we uniformly sample 2048 points from each ModelNet object. Then, 1024 points are randomly selected for the source point cloud, and all remaining points are assigned to the target point cloud. This results in regis- tration between point clouds with zero point-to-point correspondences that simulate real-world data from different acquisition devices.
show significant
Second,
The ShapeNet (Chang et al. 2015) dataset consists of 30,974 models from eight varying categories. We use the ShapeNet models evaluate point cloud retrieval and completion. For retrieval, follow the approach of Zhao et al. (2021) and uniformly sample 2048 points from the surface of each object in the dataset. For point cloud completion, we the ShapeNet variation proposed by Yuan et al. (2018) with 2048 points for both partial and complete point clouds.
Finally, evaluate the real-world Scan2CAD (Avetisyan et al. 2019ab dataset. The Scan2CAD dataset consists RGB-D scans of indoor environments from the ScanNet (Dai et al.) (017) dataset manually aligned with CAD objects chosen from ShapeNet. Similar to Salihu Steinbach (002)). we initially use VoteNet (Qi et al.] 2019) to generate bounding boxes and crop the points of the scanned environment intersecting with the predicted boxes. We then use the predicted class and our retrieval and registration method on ShapeNet to reconstruct the 3D environment.
4.2 IMPLEMENTATION DETAILS
All results either taken directly from the respective work re-trained with the preferred con- figurations. We train DeepSPF using the ADAM optimizer with a learning rate of 0.001. For the scheduler and the number of training epochs, we guided by the configurations of the respective works. Each of our models is trained on one NVIDIA RTX A6000. For each state-of-the-arr ap- proach, we replace the bascline encoder with our proposed DeepSPF backbone. Each method evaluate uses the originally proposed decoder and loss functions.
4.3 METRICS
For registration, evaluate Chamfer Distance de, translation root square error (RMSE(t), and root rotation mean square error (RRMSE). The de, introduced in Barrow et al. (1977), denotes the average Fuclidean distance between the nearest points of two point clouds. The RRMSE that

---

Published as a conference paper at ICLR 2024
computes the rotation was first used in Lang & Francos (2021) for the case of point cloud registration under zero intersection noise. Additionally, we evaluate the inference time R.
For point cloud retrieval, evaluate using the Precision @ M and the Top-1 de following Zhao et al. (202I). Top-1 de is the de of the Top-1 retrieved model, and Precision @ M shows that in a dataset of models, the correct model is among the M 0.1n number of retrieved models.
In S2C, we evaluate on the Scan2CAD benchmark (Avetisyan et al. 2019a). which evaluates the
registered objects fulfill the condition 1R... $20" Atere <20em/Acz <20%
Finally, for point cloud completion, we consider the evaluation metrics shown in Yuan et al. (2018) and consider de and F-score. The F-score indicates the number of points reconstructed within
bounded distance dy. which we define following Yuan et al. (2018) as dy = 0.01 divided by all reconstructed points. Therefore, the F-score is expressed percentage.
4.4 REGISTRATION
For registration, we compare three different test cases. First, in Table we follow the evaluation scheme of Lang & Francos (202T) and Salihu & Steinbach (2023). Here, compare method with state-of of-the--rtrrr art methods for registration under the use case of acro-intersecctin noise. We fur- ther adopt evaluation in ablation style for this case by evaluating DeepSPF using the same decoder as (Salihu & Steinbach (2023) under three different conditions. The first condition
the registration using only the proposed patch-based graph network seen in Eq. (3) From this, can see that the combination of patch-wist SG representation and graph relations can already improve the rotation error by great margin. Condition U further considers the Legendre Polynomial, fol- lowing Eq. (8) and shows that the impact of low-frequency components is not only useful for shape recognition but also for the registration of point clouds. Lastly, V shows that our adaptive resizable patches further improve the results, especially for the rotation error. In addition, we also show re- sults by improving an already existing method, such as DeepGMR (Yuan et al., 2020). Here, we show that using backbone instead of PointNet more generalizable encoding. We append a qualitative evaluation in Figure 3
significantly improve results by creating
Second, in Table 2] we follow the evaluation method of Yuan et al. (2020) extended with the results shown in Lang & Francos (2021). this evaluation, we consider testing on point cloud data with additive Gaussian noise. We show that by using DeepSPF instead of PointNet, with the decoder of DeepGMR, we can significantly improve the results compared to the original DeepGMR approach. We assume that this is due to the combination of patch-wise and rotation-equivariant representation.
we evaluate on the Scan2CAD dataset following Avetisyan et al. (2020). Here, even though SceneCAD (Avetisyan al. 2020) learns object-to-object and object-to-layour relation- ships, our method results in significant improvements overall. We assume that due to the captured scene relationships of SceneCAD, our improvements are not significant as for the synthetic case.
Finally, in Table 3
Table 1: Registration results on ModelNet40 with zero-intersection noise show improvements for approaches using DeepSPF encoding. Our encoder can also be used to improve existing PointNet- based networks, such as DeepGMR, to boost registration results.
Model
dc L
RRMSE J RMSE()) 1
DCP Wang & Solomon
2019)
0.059
93.221
0.014
RGM (Fu et al. (021))
0.254
100.97
0.388
RolTr
Yu et al.
2023
81.14
0.1353
DeepGMI Yuan et al.
2020
0.026
67.282
0.010
DeepUME (Lang & Francos
2021
0.011
70.818
0.009
SGPCR Salihu & Steinbach
2023
0.0010
8.57
0.0034
Ours (E)
0.0007
7.45
0.0031
Ours (E+U)
0.0006
6.49
0.0029
Ours (E+U+V)
0.0004
5.17
0.0022
Ours + DeepGMR Yuan al. 2020)
0.0016
5.32
0.0210
7

---

Published as a conference paper at ICLR 2024
(a) Input
(b) DecpGMR
(c) DeepUME
(d) SGPCR
(c) Ours
Figure 3: Qualitative evaluation results for zeno-intersection-bee -ased registration ModelNet40. Red represents the target point cloud, while blue represents the source point cloud that transformed.
Table 2: Our encoder can improve existing methods for registration ModelNet40 data with addi- tive Gaussian noise.
Model
de 4
RRMSE 4
RMSE() +
R(ms)
Poi (Aoki al.
2019)
0.026
81.843
1.037
DCP Wang & Solomon
2019
0.055
90.715
0.014
RGM (Fu et al. 2021)
0.254
100.11
0.389
DeepGMR Yuan et al.
2020)
0.011
42.515
0.004
0.0148
DeepUME (Lang & Francos
2021
0.002
2.425
0.001
0.0617
Ours + DeepGMR Yuan et al.
2020)
0.0001
0.40
0.0068
0.0085
4.5 RETRIEVAL
In Table we have shown improvements in querying shapes implicitly for the Scan2CAD dataset by evaluating the Scan2CAD benchmark. To show these improvements explicitly, we perform distance- retrieval the synthetic ShapeNet dataset, similar to Zhao et al. (2021) and Salihu & Steinbach (2023). To retrieve objects from the dataset, we first train DeepSPF using point cloud registration between point clouds generated from ShapeNet. Then, we pre-compute all latent vectors for all point clouds the ShapeNet dataset. Using this, we can compute the cross-covariance matrix between the latent vector of the query point cloud and all computed latent vectors. We apply singular value decomposition to obtain the transformation matrix. We use the de to obtain the closest point cloud between the query and ShapeNet point clouds.
Table 3: Evaluation with the Scan2CAD benchmark on the Scan2CAD dataset shows overall im- provements. We follow training schematic of [Saiihu Steinbach (2023) by generating class- distinct cropped point clouds from VoteNet (Qi et al. 2019) bounding boxes.
Class
Avetisyan et al. (2019a)
Avetisyan et al. (2019b)
Avetisyan al. 2020
Ours
Bathtub 36.20
38.89
42.42
56.66
Cabinet
34.00
51.52
58.33
56.22
Chair
44.26
73.04
81.23
78.39
Sofa
30.66
76.92
82.86
73.56
Table
30.11
48.15
45.60
61.64
Bin
20.60
18.18
32.26
36,99
Avg.
38.17
51.11
57.11
60.57
8

---

Published as a conference paper at ICLR 2024
Table 4: Following the results and approach of Zhao al. (2021) and Salihu & Steinbach (2023).
we show improvements in the retrieval process on ShapeNet. We observe significant improvements for the Top-1 error.
Model
Zhao et al. 2021
Salihu & Steinbach (2023)
Ours
Top-1
60.23
70.84
Chair
P@M1
51.47
76.20
76.99
Top-1 de 4
0.115
0.014
0.008
Top-1 t
63.39
69.44
Table
P@ M f
57.77
75.15
78.32
Top-1 de 4
0.112
0.016
0.012
Table 5: Point cloud completion results compared to the original Yuan et (2018) PointNet-based approach. We see significant improvements for seen and unseen data. Seen data refers to test classes, which are also contained in the training set, while unseen data are classes not in the training set.
Average
Model
dc
F-score f
PCN (Yuan al.
2018
8.99
29.9
VN (Deng et al.
2021
PCN
Yuan et al.
2018
Seen
12.4
25.0
Ours PCN Yuan et al.
2018
6.34
39.4
PCN (Yuan et al.
2018
17.1
23.0
VN (Deng et
2021
PCN
Yuan al.
2018
Unseen
18.9
22.6
Ours + PCN Yuan et al.
2018
11.9
33.0
4.6 POINT CLOUD COMPLETION
In the final evaluation, show that our model improves point cloud completion the ShapeNet dataset compared to other models. In Figure 7 we provide a qualitative evaluation, which shows major improvements in point could completion quality, and quantitative evaluation in Table 5 We compare to PCN (Yuan et al. 2018) and VN-based version of PCN, where we replace the PointNet backbone with the official VN-PointNet. We see large improvements in the F-score compared the original PointNet and VN-based approach, as local details are better captured by our patch-witg ap- proach. This evaluation highlights the effectiveness of employing DeepSPF as inatiin-equivarian encoder in point cloud completion networks. Noteworthy advancements, like those presented by Yu ctt al. 2021), may benefit from integrating DeepSPF.
5 CONCLUSION
In this work, we introduced DeepSPF, a network that represents point cloud through a patch-wise rotation-equivarrian radial representation termed SPF. To encode relevant latent vector, we pro- posed PG-Layer to improve the learnable SPF representation using low-frequency information and adaptive resizable patches. We have shown that our approach can improve tasks such as registration, retrieval, and completion for sitaace-bassdd point clouds. We also show that DeepSPF can replace existing encoders in point networks to improve registration and completion results. Further, in an ablation study, we show that our model is capable of doing this without increasing the number of parameters compared to similar state-of- ofate-of-the-aaee -tthharrt art methods. While we improve on current methods, method is limited by the initial sampling through FPS. Thus, complexity increases for larger point clouds, similar to other encoder networks, Based on the improvements, we hope that SPF will serve basis for more complex point networks.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
The authors acknowledge the financial support by the Federal Ministry of Education and Research of Germany in the programme of "Souverän. Digital. Vernetzt.". Joint project 6G-life, project identification number: 16KISK002
REFERENCES
Yasuhiro Aoki, Hunter Goforth, Rangaprasad Arun Srivatsan, and Simon Lucey. Pointnetlk: Robust & efficient point cloud registration using pointnet. In Proceedings of the IEEE/CVE conference on computer vision and pattern recognition, pp. 7163-7172, 2019.
Armen Avetisyan, Manuel Dahnert, Angela Dai, Manolis Savva, Angel X. Chang, and Matthias Nießner. Scan2cad: Learning cad model alignment in rgb-d scans. 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2609-2618, 2019a.
Armen Avetisyan, Angela Dai, and Matthias Nießner. End-to-end cad model retrieval and 9dof alignment in 3d scans. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 2551-2560, 2019b.
Armen Avetisyan, Tatiana Khanova, Christopher Choy, Denver Dash, Angela Dai, and Matthias Nießner. Scenecad: Predicting object alignments and layouts in rgb-d scans. In Computer Vision- ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Proceedings, Part XXII 16, pp. 596-612. Springer. 2020.
Harry G. Barrow, Jay M. Tenenbaum, Robert C. Bolles, and Helen C. Wolf. Parametric correspon- dence and chamfer matching: Two new techniques for image matching. In IJCAI, 1977.
Angel X. Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva, Shuran Song, Hao Su, Jianxiong Xiao, Li Yi, and Fisher Yu. ShapeNet: An Information-Rich 3D Model Repository. Technical Report 1111v:1512.03012 [es.GR], Stanford University - Princeton University - Toyota Technological Institute at Chicago, 2015.
Evangelos Chatzipantazis, Stefanos Pertigkiozoglou, Edgar Dobriban, and Kostas Daniilidis. Se(3)- equivariant attention networks for shape reconstruction in function space. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Chao Chen, Guanbin Li, Ruijia Xu, Tianshui Chen, Mci Wang, and Liang Lin. Clusternet: Deep hierarchical cluster network with rigorously rotation-invariant representation for point cloud anal- ysis. 20/9 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp- 4989-4997, 2019,
Angela Dai, Angel X. Chang, Manolis Savva, Maciej Halber, Thomas A. Funkhouser, and Matthias Nießner. Scannet: Richly-annotated 3d reconstructions of indoor scenes. 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 2432 22232244, 2017.
Congyue Deng, Or Litany, Yueqi Duan, Adrien Poulenard, Andrea Tagliasacchi, and Leonidas J. Guibas. Vector neurons: A general framework for -quui-equiva networks. 2021 IEEE/CVF International Conference Computer Vision (ICCV), pp. 121 12180-1211 121891 2021.
Carlos Esteves, Christine Allen-Blanchette, Ameesh Makadia, and Kostas Daniilidis, Learning so(3) equivariant representations with spherical cnns. In ECCV, 2018.
Kexue Fu, Shaolei Liu, Xiaoyuan Luo, and Manning Wang. Robust point cloud registration frame- work based on deep graph matching. 2021 IEEE/CVF Conference Computer Vision and Pat- tern Recognition (CVPR), pp. 8889-8898, 2021.
Fabian Fuchs, Daniel Worrall, Volker Fischer, and Max Welling. Se trantrffffo:::::: 3d roto- translation equivariant attention networks. Advances in Neural Information Processing Systems, 33: 13:1970-1981, 2020.
10

---

Published as a conference paper at ICLR 2024
Andreas Geiger, Philip Lenz, Christoph Stiller, and Raquel Urtasun. Vision meets robotics: The kitti dataset. The International Journal of Robotics Research, 32:1231 1237, 2013.
Shreyas Hampali, Sinisa Stekovic, Sayan Deb Sarkar, C. Srinivasa Kumar, Friedrich Fraundorfer, and Vincent Lepetit. Monte carlo scene search for 3d scene understanding. 2021 IEEE/CVF Conference Computer Vision and Puttern Recognition (CVPR), pp. 13799-13808, 2021.
Qianjiang Hu, Daizong Liu, and Wei Hu. Exploring the devil in graph spectral domain for 3d point cloud attacks. In Computer Vision-ECCV 2022: 17th European Conference, Tel Aviv, Israel, October 23-27, 2022, Proceedings, Part III, pp. 229-248. Springer, 2022.
David M. Klee, Ondrej Biza, Robert Platt, and Robin Walters. Image to sphere:
Learning equivariant features for efficient pose prediction. In International Conference Learning Representations, 2023. URL
forem//oormmid 2bDpAtr7PI]
Arno Knapitsch, Jaesik Park, Qian- Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. ACM Transactions on Graphics, 36(4), 2017.
Natalie Lang and Joseph M. Francos. Deepume: Learning the universal manifold embedding for robust point cloud registration. In British Machine Vision Conference, 2021.
Xueyi Liu, Zhang, Ruizhen Hu, Haibin Huang, He Wang, and Yi. Self-supervised category- level articulated object pose estimation with part-level SE(3) equivariance. In The Eleventh Inter- national Conference on Learning Representations, 2023.
Adam Misik, Driton Salihu, Heike Brock, and Eckehard Steinbach. Cocca: Point cloud completion through cad cross-attention. 2023 IEEE International Conference on Image Processing (ICIP), pp. 580-584. IEEE, 2023.
Francis J. Narcowich and Joseph D. Ward. Nonstationary wavelets the m-sphere for scattered data send all correspondence to. 1996.
C. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning on point sets for 3d classification and segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 77-85, 2016
C. Qi, Hao Su, Kaichun Mo, and Leonidas J. Guibas. Pointnet: Deep learning point sets for 3d classification and segmentation. 2017 IEEE Conference on Computer Vision and Pattern Recog- nition (CVPR), pp. 77-85, 2017.
C. Qi, Or Litany, Kaiming He, and Leonidas Guibas. Deep hough voting for 3d object detection in point clouds. 2019 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 9276- 9285, 2019.
Yuchen Rao, Yinyu Nie, and Angela Dai. Patchcomplete: Learning multi- patch priors for 3d shape completion on unseen categories. Alice H. Oh, Alekh Agarwal, Danielle Belgrave, and Kyunghyun Cho (eds.), Advances in Neural Information Processing Systems, 2022. URL
net worumorrmmooo
Driton Salihu and Eckehard Steinbach, Sgper: Spherical gaussian point cloud representation and its application to object registration and retrieval. In The 2023 IEEF/CVF Winter Conference on Applications of Computer Vision (WACV 2023), Waikoloa, Hawaii USA, Jan 2023.
Zikai Sun and Thierry Blu. Empowering networks with scale and rotation equivariance using similarity convolution. In The Eleventh International Conference on Learning Representationss 2023.
Yu-Ting Ting Tsai and Zen-Chung Shih. All-frequency precomputed radiance transfer using spherical radial basis functions and clustered tensor approximation. ACM Trans. Graph., 25:967-976, 2006.
Shinji Umeyama. Least- estimation of transformation parameters between two point patterns. IEEE Trans. Pattern Anal. Mach. Intell., 13:376-380, 1991.

---

Published as a conference paper at ICLR 2024
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention all you need. Advances in neural informa- tion processing systems, 30, 2017.
Helmut Vogel. better way construct the sunflower head. Bellman Prize in Mathematical Biosciences, 44:179-189 1979.
Jiaping Wang, Peiran Ren, Minmin Gong, John M. Snyder, and Baining Guo. All-frequency render- ing of dynamic, opatially-varying reflectance. ACM SIGGRAPH Asia 2009 papers, 2009.
Yue Wang and Justin M. Solomon. Deep closest point: Learning representations for point cloud registration. 2019 IEEE/CVF International Conference Computer Vision (ICCV), pp. 3522- 3531,2019
Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. Dynamic graph enn for learning on point clouds. ACM Transactions on Graphics (TOG), 38:1 - 12, 2018.
Zhirong Wu, Shuran Song, Aditya Khosla, Fisher Yu, Linguang Zhang, Xiaoou Tang, and Jianxiong Xiao. 3d shapenets: deep representation for volumetric shapes. 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920, 2015.
Zi Jian Yew and Gim Hee Lee. Rpm-net: Robust point matching using learned features. 2020 IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 11821-11830, 2020.
Hao Yu, Zheng Qin, Ji Hou, Mahdi Saleh, Dongsheng Li, Benjamin Busam, and Slobodan Ilic. Rotation invariant transformer for point cloud matching. In Proceedings of the IEEE/CVF Con- ference on Computer Vision and Patlern Recognition, pp. 5384 5393, 2023.
Hong-Xing Yu, Jiajun Wu, and Li Yi. Rotationally equivariant 3d object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 1456-1440 2022.
Xumin Yu, Yongming Rao, Ziyi Wang. Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE/CVF interna- tional conference on computer vision, pp. 12498-12507, 2021.
Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. 2018 International Conference on 3D Vision (3DV), pp. 728-737, 2018.
Wentao Yuan, Benjamin Eckart, Kihwan Kim, V. Jampani, Dieter Fox. and Jan Kautz. Deepgmr: Learning latent gaussian mixture models for registration. In ECCV, 2020.
Tianyu Zhao, Qiaojun Feng, Sai Jadhav, and Nikolay A. Atanasov. Corsair: Convolutional object retrieval and sydde registration. 2021 IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS), pp. 47-54, 2021.
12

---

Published as a conference paper at ICLR 2024
A APPENDIX
A.1 OVERVIEW
Fine
JuftoreliSumppe CAD Cljecs
Patial Foint Cladi
Poin Cloud Comp ettionn
Recievll
Layel
SPF
Sean-to-AAD
Indoor Exioomnnt
Coarse
PG-Layed
Moderate
Point Cloud Imput
DeepSPF: Learnable Spherical Patch Fields
Multi- Task Estimation
Figure 4: Our proposed DeepSPF learns a patch-wise rotation-equiiarian representation from instance-based 3D point cloud data. Based on PG-Layer, DeepSPF adapts the size the patches depending the information density in the respective area. This enables improvements for 3D point cloud registration, retrieval, and completion.
We want to start the appendix with a small overview of the whole structure, for which we visualize the structure in in The inputs to our DeepSPF architecture are commonly instance- based point clouds. Instance point clouds are either sampled from meshes, scanned from real-world objects, or obtained by 3D object detection or instance segmentation techniques from large 3D scans. Then DeepSPF converts the instance point cloud from 3D points to the SPF representation from [Eq. (2), resulting in latent z. Then multiple PG-Layer are used to improve the latent encoding to obtain the latent vector & by adapting the size of the spherical patches and introducing low- cowffrequency components in the representation of SPF.
DeepSPF acts as encoder and is combined with existing decoder models to achieve higher ac- curacy in point cloud registration, retrieval and completion. Two hyperparameters must be con- sidered for the initialization of DeepSPF. Those two choose the number of initial patches fao and the number of spherical points C. While the radius r considerably affects the representa- tion learning through V(r), seen in Table the initialization for small models does not im- pact the final result significantly. We choose to be decreasing for each of the three SA of DeepSPF, as such [1.0, 0.5, 0.25]. Notably similar results with minor deviations are achieved for r = [1.0, 1.0, 1.0]. We find that the radius converges to similar values for both initializations.
A.2 RELATED WORK
In this section of the appendix, we would like to summarize the main elements of the work already carried out and used in the main manuscript.
A.3 SPHERICAL GAUSSIANS
In contrast to Salihu & Steinbach 2023 we define our point cloud representation based on the non-simplified version of SG from Narcowich & Ward (199)).
Narcowich Ward (1996) defined SG
G(v;a, = Σ ac x
Wm Pi(v p),
(15)
dm
1=0
13

---

Published as a conference paper at ICLR 2024
with Pt defined the normalized Legendre polynomial of degree Wm defines the volume of sphere on Sth and dm.i denotes the dimension of order-l Spherical Harmonics. Based on each component develop representation in Eq. (2)
In addition, we employ the defnition of the convolution originally discretized by Eq. (16)
GR = (G HU: PH)
age MaaeMelen
Wm Pi(po PH)
(16)
dm,1
Using our representation from Eq. (2) we obtain the definition the PG-Layer as defined by Eq. (13). Notably, simplified den.! by Eq. (7) to V(r), which allows us to adjust the radius
the intial spheres differentiable manner.
A.3.1 SPHERICAL SAMPLING
For the generation spheres, we follow the approach of Vogel (1979). To sample a sphere, Vogel (1979) first samples point sunflower structure in cartesian coordinate space and then convert these points into the spherical coordinates. For the notation of the sampling procedure, we follow the work Salihu & Steinbach (2023).
sample point v is generated by
Vi = (cos (i 8) sin Φ1 sin (1- sin o cos σ)
(17)
with 73 (5).
A.3.2 SPHERICAL GAUSSIAN POINT CLOUD REPRESENTATION
Spherical Gaussian Point Cloud Representation (SGPCR) is a method developed by Salihu & Stein- bach (2023) for precisely the task of point cloud registration. SGPCR is the first method to apply concepts of Spherical Gaussian (SG) into the point cloud domain.
As defined in Section 3.4 SGPCR represents a point cloud p by SG representation Zg
gg = G(u;;, a, >. p) = M(v=
(18)
Noticeably, it consists of three parameters: an amplitude a, which is defined by the distance the spherical sample v, and learnable parameter X.
Due to its limitations, the representation introduced by Salihu & Steinbach (0033) incorporates a novel convolution to enhance the representation via a trainable process This convolution, based on the previous work by Tsai & Shih (2006) and Wang et al. (2009). is defined as
4 sinh (dan).
= (z (v)
(19)
tA
dzh
Eq. (19) shows the convolution between the representation 2" and random kernel z/. Here, Salihu & Steinbach (2023) denotes digh \m + ApPAll
Despite the substantial reduction in computational complexity achieved through the convolution, characterized by constrained number of multiplications and additions, this approach distorts the original representation of the latent vector zg. Consequently, it deviates from the original represen- tation by SG, making consecutive convolutions using Eq. (19) impossible.
Although SGPCR demonstrates efficacy with limited parameter set, its applicability is constrained, rendering it unsuitable for numerous point cloud analysis tasks, including completion, classification, and segmentation. It performs exceptionally well in registering instance point clouds afflicted by zero-intersection noise. However, its performance is noticeably compromised as it relies on applying only one sphere across the entire object. Moreover, as indicated in the orig- inal manuscript of [aalihu & Steinbach (2023), the proposed method does not achieve performance comparable to state-of-the-arr approaches (Yuan et al. (2020): Lang & Francos (202T)) in situations involving point-to-poine correspondences.
larger 3D scenes,
14

---

Published as a conference paper at ICLR 2024
A.3.3 DYNAMIC GRAPH CONVOLUTIONAL NEURAL NETWORK
We apply the work of Wang et al. (2018) (DGCNN) for the estimation of our edge function E(v, p). DGCNN comprises multiple convolutions augmented by a dynamic graph akin to its 2D equivalent, facilitating the transmission of messages in the form of edges. The edges of DGCNN are not fixed to R², which is an advantage for general point cloud analysis tasks. We use this advantage to improve our spherical application, as we can apply it seamlessly to our spherical display. In the case of DGCNN, an asymmetric edge e(i,j) between two features Si and ʃj. which are estimated by convolutional layer, is described as:
e(i.)) h(o(f: f2) + +fz),
(20)
with h denoting the activation function, with & and being set of learnable parameters. DGCNN is particularly useful for obtaining local geometric information without losing permutation invariance.
A.4 REGISTRATION
A.4.1 DECODER HEAD
To obtain the rotation matrix Rp, translation vector Ep. and scaling vector & we follow the approach of Salihu & Steinbach (2023). Following their work, we estimate the cross-covariance matrix M R3x6 between the estimation of the target point cloud 2n and source point cloud žş using DeP
M
27
all 2T
xc
2/ .5....2 ".....g5x fmxx
(21)
We estimate Rs from M by the method of Umeyama (1991), using this estimation we can obtain tp from source and target point cloud ps Ps/pr
1p p(pr) Rp
(22)
Finally, scaling sp is estimated by estimating the difference between the oriented bounding box of the target Or E R883 and source Og E R88x3, multiplied with a differentiable component &:
Sp = (u(OT)
(23)
[u(os)
A.4.2 REGISTRATION LOss
Four components give the 9 degrees of freedom registration loss from Salihu & Steinbach (2023). First, the loss between the estimated rotation matrix R.p and the ground truth rotation matrix Rg
LR MIM RRg. R2p)
R RS
Taxs|1
(24)
where the mean isotropic (MIE) of Yew & Lee 2020 is added. With the addition of the chamfer distance dz, we summarize the losses of (alihuu & Steinbach| (2023) to:
L= TRR + (Pr. RpSpPS + fp)) + ((t,, 1) + u(|s, spl)
(25)
where Sp is the diagonal matrix converted from the estimated scaling vector Sp-
For the registration loss of 6 degrees of freedom, we omit the scaling loss but keep rotation, distance, and translation loss the same.
A.5 POINT CLOUD COMPLETION Loss
We train the two baselines, PCN (Yuan et al. 2018) and the rotation-equivariant VN-based (Deng) ct al. (02T), and our DeepSPF-based version using the chamfer distance dç. As we obtain a coarse result Y. and a detailed result Ya, the resulting loss function is compared with the ground truth Y, by
L= dc Yg) + add Yg).
(26)
where a is a hyperparameter that we kept the same for all evaluations. For decoding, we use the exact method introduced by PCN without any adaptation made by us.
15

---

Published as a conference paper at ICLR 2024
A.6 EVALUATION METRICS
To give a holistic overview of our evaluations, we show the definitions of the metrics used in our work.
For two point clouds ps E RNx3 and pr RNx3 we estimate de through the notation of Barrow al. 1977):
de PP.
Σ
min
vll
min
||
v112
(27)
Ps
УEPт
Pr
zEPS
zEPs
The translation root mean square emror is denoted as between prediction tp E R and ground truth trasnlation tg E IR3:
(MSE()((p; t.g)
Σ tg. tp,i)
(28)
3
For root rotation mean square error, we first compute the Euler angles ep E R² and eg R form the rotation matrices Rg and R.y Then calculate the root rotation mean square error follows:
g8
180e y2
RRMSE Eps Eg)
(29)
For S2C methods, we evaluated based on the Scan2CAD benchmark. This benchmark considers the rotation error Rerr, translation error terr, and scaling error Serrs generate the Scan2CAD bench-
mark following the condition 1 A200Ator Aforr <20cmAsuu <20% The rotation error Rşrr is estimated by
180 min
1, max
- cos
U(R R-
2
Rerr
(30)
with tr denoting the trace operation. The translation error terr is expressed by:
lerr
||4p fgll2
(31)
and the scaling error Serr between predicted scaling sp E R3 and ground truth scaling sy E RR3 by:
3
8p
Serr 100
(32)
3
8g
The F-score calculation follows Knapitsch et al. (2017), and as such, we need to first calculate the precision and recall for a certain threshold d, = 0.01. The precision Prec(ps, pr) follows:
Prec(ps,PT)
min | wll <dg dg) #EPr
(33)
pps
xEPS
while the recall (pcall(ps,,))) is:
Recall(ps.Pr)
min
|| 2/1 ds
(34)
|rr
xEps
#EPr
In cumulation, precision and recall are used to calculate the F-score F(ps, Pr))
Prec(ps, ( Recall(ps. Pr)
F(ps. pP) = Prec(ps. pP) + Recall(ps.Pr)
(35)
16

---

Published as a conference paper at ICLR 2024
Model
Rotation Error
Translation Error
#Flops
DeepGMR Yuan et DeepGMR Ours
2020
113.88
45.92
3.133G
Instances
5.64
3.97
0.232G
DeepGMR Yuan et al. DeepGMR Ours
2020
63.77
4.6132
Odometry
1.10
2.34
Table Registration evaluation on Kitti Geiger et al. 2013) with training on ModelNet40 data.
A.7 KITTI & FLOPS ANALYSIS
We evaluate our work on the Kitti (Geiger et al. 2013) dataset. In Table 6, we show results for DeepGMR (Yuan et al. 2020) with rotation-invariant features against our rotation-equivariant latent representation. We take the decoder head and losses from DeepGMR and replace the encoder from DeepGMR with our DeepSPF model. To obtain Table we trained on synthetic ModelNet40 data and tested it on Kitti.
A.8 ABLATION STUDY
We ablate, in Table and Table 8, different numbers spherical samples |o| = x fuo with the corresponding number of furthest point samples fno with number of spherical samples per patch C. For all results, we initialize the radius for each layer of DeepSPF with r = [1.0, 0.5, 0.25| and use K 20 for the Edge function E(v.p) of SPF.
Table 7: Ablation study on ModelNer40 using zero intersection noise for | = 1024.
V
fno
C
de 4
RRMSE 4
#Params
R(ms) +
64
16
0.00052
5.63
906435
0.0117
32
32
0.00052
5.61
906339
0.0096
16
64
0.00048
5.17
906291
0.0085
1024
8
128
0.00049
5.28
906267
0,0083
4
256
0.00051
5,43
906255
0.0082
2
512
0.00053
5.68
906249
0,0079
1
1024
0.00053
5.67
906246
0.0077
Table 8: Ablation study on ModelNet40 using zero intersection noise for | = 512.
|
fno
C
dc +
RRMSE
#Params +
R(ms)
64
8
0.00056
6.23
505539
0.0099
32
16
0.00052
5.69
505443
0,0085
16
32
0.0005
5.36
505395
0.0076
512
8
64
0.0005
5.37
505371
0.0071
4
128
0.00051
5,44
505359
0.0069
2
256
0.00052
5.68
505353
0.0068
1
512
0.00053
5.79
505350
0.0069
Lastly, we evaluate the number parameters compared to all the works we have addressed in main part of the publication. In Table 11] we evaluate each model we used in our evaluation and the number of parameters. In Table 11, we refer to the model with no publicly available code as "Replica" and try to estimate the minimum number of parameters. Noticeably, only SGPCR has significantly fewer parameters, but is not usable for different applications outside registration compared to method.
17

---

Published as a conference paper at ICLR 2024
Table Ablation study on ModelNet40 using zero intersection noise for different | with fro 16.
|
fno
C
de 4
RRMSE
#Params 4
R(ms)
128
8
0,00064
7.02
204723
0.0071
256
16
0.00055
6.00
304947
0.0071
512
16
32
0.0005
5.36
505395
0,0076
1024
0.00048
5.17
906291
0.0085
2048
128
0.00049
5.19
1708083
0.0111
Table 10: Ablation study ModelNet40 using zero intersection noise for different | with C 64.
nn
C
de 4
RRMSE +
#Params +
R(ms)
128
0.00070
7.26
204681
0.0062
256
4
0.00068
7.09
304911
0.0065
512
64
0.00056
6.23
505371
0.0071
1024
16
0.00048
5.17
906291
0.0085
2048
32
0.00049
5.22
1708131
0,0120
A.9
SCAN-TO-CAD
In the section, we further analyze the results in Table 3 of the results between the individual classes in our method and ii) the comparison with SceneCAD.
Specifically, we address the discrepancy
A.9.1 RESULT DISCREPANCY BETWEEN CLASSES
Two factors contribute significantly to the substantial variations in results among individual classes.
First, the misalignment and lower retrieval quality in Scan2CAD data can be attributed primarily to the point cloud density and representation. Larger objects are easily recognized during 3D scanning and thus generate scanning, leading to points with substantial occlusion. Although beyond the scope of our work, we believe that investigation into the Scan2CAD dataset could benefit the broader research commu- nity in the field of scan-to-CAD.
points. Conversely, smaller objects like the "bin" class pose challenges in
Second, use VoteNet proposed by Qi al. (0199) as object detection method. As such, every error created by the detection of VoteNet is propagated in our approach. When comparing the results of VoteNet directly ours, as shown in Table 12 it becomes evident that this constitutes notable source of eгror.
A.9.2 COMPARISON TO SCENECAD
Additionally, the varying degree to which our method surpasses the previous work (Avetisyan al., 2020) in specific classes is intricately linked to the primary contribution of Avetisyan et al. (2020). Avetisyan et al. (2020) presents an end-to-end pipeline that considers both object-to-object and object-to-layout relationships.
The notable reduction the quality our work is particularly evident in the chair and sofa classes. An intuitive explanation for Avreiissyann al. (2020) superiority in these classes is its ability to lever- age information across different chair objects, thereby enhancing the registration quality for the chair class. We assume that, especially for the sofa class, knowing the layout is erucial. This is particularly significant because, in the ScanNet dataset, the backside of most sofas is absent, often due to sofas being aligned against walls.
In Table 13]
we show our method compared to Avetisyan et al. (2020) without object and layout re- lations (I) and Avetisyan al. (2020) with object and layout relations (IT). Now, it becomes apparent
18

---

Published as a conference paper at ICLR 2024
Table 11: Summary all models used in the evaluation and the number of parameters each model uses. Replica refers to all models that do not have public implementation and for which we estimate the minimum number of parameters following the original work
Model
#Params
Replica
PointNet Qi ct al.
2017
3471473
DGCNN Wang
2018
5385432
E2E (Aveusyan et al.
2019b
2498972
yes
PointNetLK [Aoki et al.
2019
151680
no
DCP Wang & Solomon,
2019
5343040
RGM (Fu et al. (02T
17360074
DeepGMR Yuan al. 2020
1522512
DcepUME (Lang & Francos 2021
310720
no
VN-PointNet (Deng el al. (202T)
1972745
no
CORSAIR (Zhao el al. 2021 FCGF Backbone
8749232
yes
SGPCR (Salihu & Stembbac 2023
8195
yes
RolTr Yu al. 2023)
40259075
Ours ( v = 1024)
906291
no
Ours [ = 128)
204681
Direct comparison with VoteNet results, evaluated through mean average precision (mAP) and obtained from the manuscript of Yu et al. (2022).
Table 12:
Method
Bathtub
Cabinet
Chair
Sofa
Table
Bin
Avg
VoteNet [6] - mAP 40.1
47,5
84.1
67.4
72.3
27.4
56,46
that our model outperforms SceneCAD in every class except for the sofa class. The error in the sofa class presumed to come from the 3D object detection results obtained from VoteNet.
Table 13:
Comparison with Avetisyan et al. 2020 using not object and layout relations I) and with object and layout relations II).
Class
I) Avetisyan et al. 2020
II)
Avetisyan et al. (2020)
Ours
Bathtub
42.42
42.42
56.66
Cabinet
51.67
58.33
56.22
Chair
77.28
81.23
78.39
Sofa
77.14
82.86
73.56
Table
37.91
45.60
61.64
Bin
25.81
32.26
36.99
Avg. t
52.03
57.11
60.57
19

---

Published as a conference paper at ICLR 2024
A.10 EXTENDED QUALITATIVE RESULTS
Figure 5: Qualitative results for retrieval on ShapeNet.
(a) Query
(b) SGPCR
(c) DeepSPF
(d) GT
Figure 6: Extended qualitative results for registration on ModelNet40 with zero-intersection noise, such in Figure 3
(a) Input
(b) DecpUME
(c) SGPCR
(d) DecpSPF
20

---

Published as a conference paper at ICLR 2024
(a) Partial Inputs
(b) PCN
(c) VN+PCN
(d) DeepSPF
GT
Figure 7: We show a comparison of point cloud completion methods for networks trained with 2048 points from data of Yuan et al. (2018) derived from ShapeNet. Our model improves the encoding of fine local structures due to the patch-based approach.
21


Reviews:
[{"review_id": "aY444zDASU", "reviewer": "Reviewer_bCvD", "summary": "The paper suggests a network design aiming at providing SE(3) equivariant features consisting of local and global spatial neighborhoods. The neighborhoods are modeled using spherical Gaussian representations.  The size of the patches is adjustable by learnable elements in the proposed network.\nThe method is mainly evaluated on 3D scan2cad tasks: registration, retrieval, and completion.", "questions": "No specific questions. I would appreciate a response with respect to the weaknesses stated above.", "limitations": "", "rating": 5, "confidence": 2, "soundness": 2, "presentation": 1, "contribution": 2, "strengths": "The work tackles the challenging goal of learning the correct spatial receptive field for features.\n\nI appreciate the effort to evaluate the method on three different tasks. In addition qualitative results are also provided.", "weaknesses": "The main weaknesses of the paper are related to its exposition and readability qualities.\nIn particular, the authors should strive to reduce the level of technical detail in the main text and aim for a clearer presentation of the key concepts. Furthermore, the paper could benefit from the inclusion of more intuitive explanations. \n\nSome examples:\ni)  “ conventional SO(3)-equivariant methods do not sufficiently investigate the correlation between global and local structures”. Is it specifically true for equivariant methods? SO(3) symmetries? \nii) Figure 1 is unclear. It is unclear how the caption relates to elements in the figure itself.\niii) There is no clear analysis provided (proposition, theorem, etc.) that supports the claim that the proposed network is equivariant. \niv) Some equations are not clear, e.g. it is not clear what is z in eq. (3).\n\nIn summary, it seems the writing quality is such that the paper is not ready yet for publication.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJyBQX8g6m", "reviewer": "Reviewer_73te", "summary": "The authors propose Spherical Patch Fields (SPF), a point cloud representation based on Spherical Gaussian (SG) to generate many spherical patches to obtain local and global information, and proposed PG-Layer to improve the learnable SPF representation using low-frequency information and adaptive patches. Besides, the authors verified its effectiveness on three public datasets and their proposed method achieved competitive results.", "questions": "See above", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "The idea is somehow novel and interesting. The writing of the entire paper is fine in general, and the paper is easy to read.", "weaknesses": "1. More recent related works should be compared, such as Section 4.4, which does not compare with the latest rotation-invariant method: Rotation-invariant transformer for point cloud matching[1]. Section 4.6 does not compare the state-of-the-art point cloud completion methods: Diverse point cloud completion with Geometry-Aware Transformers[2].\n\n[1] Yu, Hao, et al. \"Rotation-invariant transformer for point cloud matching.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2023.\n[2] Yu, Xumin, et al. \"Pointr: Diverse point cloud completion with geometry-aware transformers.\" Proceedings of the IEEE/CVF international conference on computer vision. 2021.\n\n2. In Table 3, the proposed method does not perform well in some classes and reasons should be analyzed and discussed.\n \n3. Some sentences are not clear. For example, the red arrow in Figure 1 is ambiguous, and the lower-right circle in the SPF part lacks an arrow. In Section 5, typo: ``resizeable'' ⇒ ``resizable''.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "aMbJEPyd5S", "reviewer": "Reviewer_T7zb", "summary": "This paper presents an integrable backbone for SO(3)-equivariant point-cloud learning, called Learnable Spherical Patch Fields (DeepSPF).\nThe proposed method is based on a patch-wise representation obtained using spheres, introduced as Spherical Patch Fields. \nThe method aims at seamlessly integrating local and global contextual information, adaptively extracted by the presented Patch Gaussian Layer. The experimental validation of the proposed model focuses on Scan-to-CAD (S2C) — a sub-task of reconstructing indoor environments, including point cloud registration, retrieval, and completion — and provides a considerable improvement over the baseline models.", "questions": "Q1. Section 3\n\n•\tafter equation 1: Was $v \\in \\mathcal{S}^3$ meant to be $v \\in \\mathcal{R}^3$ or $v \\in \\mathcal{S}^2$? \n\n•\tafter equation 7: since we are in 3D, why is the sphere $\\mathcal{S}^3$ and not $\\mathcal{S}^2$?\n\nQ2. Section 3.2: \n\n•\tWhich “previous work” is meant in the second line in the first paragraph and the first line of page 5?\n\nQ3. Is the proposed method SPF also equivariant to reflections? I.e., is the entire O(3) group covered?\n\nQ4. Can the proposed method generalize to dimensions higher than 3?\n\nQ5. How sensitive is the model with respect to the number of spheres |v|?\n\nQ6. It would be useful if the authors summarized all the learnable parameters of SPF.\n\nQ7. What is the total number of parameters of the DeepSPF models used in the experiments?\nHow does this compare to the baselines?\n\nQ8. What is the computational complexity of the DeepSPF compared to the baselines? \nCould the authors provide a speed comparison?\n\nMinor:\n\n•\tEquation 4: Should $l$ be $i$ in the {} under the max?\n\n•\tPunctuation between the equations is missing, e.g., 9 and 10. \n\n•\tEquation 12 is a part of equation 11.\n\n\nIn the rebuttal, my questions were addressed appropriately, thus the assessment is updated.", "limitations": "", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "S1. The provided illustrations are neat and facilitate the reader’s understanding of the method. \n\nS2. The proposed DeepSPF successfully tackles an important problem of equivariant local and global feature extraction, by enabling multiple-layer learning of the spherical representations. \n\nS3. The effectiveness of the proposed DeepSPF backbone is demonstrated on various point cloud tasks with clear improvement over the baselines, both quantitatively and qualitatively.", "weaknesses": "W1. Unclear presentation. \nThe manuscript relies heavily upon the related and prior work, which makes in particular Section 3 hard to follow for readers less familiar those works. The important components from the related work, e.g., Vogel (1979), Wang et al. (2018), and Salihu & Steinbach (2023), could be presented in more detail in the Appendix.\n\nW2. SO(3)-equivariance is at the heart of the paper, as the title, Abstract, and Introduction imply. However, the method section is missing the crucial part of the equivariance discussion, which is instead placed in the Appendix.\n\nW3. A more detailed complexity analysis is required (see the Questions), as the brief mention in the conclusion does not suffice.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)