SPoVT: Semantic-Prototype Variational Transformer for Dense Point Cloud Semantic Completion
Authors: Sheng Yu Huang
Hao-Yu Hsu
Yu-Chiang Frank Wang
Venue: NeurIPS 2022 Accept (2022)
Keywords: point cloud
3D vision
completion

Abstract: Point cloud completion is an active research topic for 3D vision and has been widely
studied in recent years. Instead of directly predicting missing point cloud from
the partial input, we introduce a Semantic-Prototype Variational Transformer
(SPoVT) in this work, which takes both partial point cloud and their semantic
labels as the inputs for semantic point cloud object completion. By observing
and attending at geometry and semantic information as input features, our SPoVT
would derive point cloud features and their semantic prototypes for completion
purposes. As a result, our SPoVT not only performs point cloud completion with
varying resolution, it also allows manipulation of different semantic parts of an
object. Experiments on benchmark datasets would quantitatively and qualitatively
verify the effectiveness and practicality of our proposed model.


SPoVT: Semantic-Prototype Variational Transformer for Dense Point Cloud Semantic Completion
Sheng-Yo Yu Huang!*
Hao-Y Yu Hsu
Yu-Chiang Frank Wang
Graduate Institute Communication Engineering, National Taiwan University
NNVIDI
(108942095, r10922195) @ntu. edu. tu, frankvangenvidiaaccmm com
Abstract
Point cloud completion is an active research topic for 3D vision and has been widely studied in recent years. Instead of directly predicting the missing point cloud from the partial input, we introduce Semantic-Prototyy (SPoVT) in this work, which takes both partial point cloud and their semantic labels as the inputs for semantic point cloud object completion. By observing and attending to geometry and semantic information as input features, our SPoVT VT would derive point cloud features and their semantic prototypes for completion purposes. As result, our SPoVT not only performs point cloud completion with varying resolution, it also allows manipulation of different semantic parts of an object. Experiments on benchmark datasets would quantitatively and qualitatively verify the effectiveness and practicality of our proposed model.
Variational Transformer
Introduction
3D computer vision has been a popular research topic throughout recent years, related various extensive applications such as autonomous vehicles, augmented reality, and graphical rendering. As one of the most commonly used data representations, point clouds can be easily acquired by 3D sensors. However, these 3D scans are often incomplete and sparse due to self-occlusion or far distance from sensors, leading undesirable results for further applications (e.g.. g. 3D object detection for LiDAR point clouds often fails when target objects are far from the sensor). Therefore, recovering full point cloud data from partial observations becomes an important task.
As pioneers of object-level point cloud completion, Yuan et al. [1 propose PCN, combining PointNet encoder and folding-based decoder K] Since PCN and the subsequent folding-based methods 4
8 only adopt shared multi-layer perceptron (MLP) for point-wise decoding, detailed geometry information might not be properly recovered. To alleviate this problem, 9 [0] voxelize the partial point cloud so that 3D CNN can be directly used for local feature propagation. use Transformer [[S]-based frameworks and apply attention between neighboring points. Nevertheless, these approaches simply rely on 3D coordinates as input features for the point cloud, and thus how to preserve the geometry and the associated semantic information for each part of the output point cloud would be challenge.
5
6,
7
To better refine local geometry, [2 13 L4
Different from object-level completion approaches, recent scene-level completion methods [1 18 19 20 21 show that point cloud completion and semantic segmentation can be jointly performed by sharing semantic and geometrical information. With additional ground truth for 3D segmentation
"Equal Contribution
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Table 1: Comparisons between different point cloud completion methods.
Inpur
Capability
Methods
Setting
Sem Label
Completion
Sem. Label
Global / Part manipulation
Varying resolution
PoinTrr 14
Object
+
VRC-Net [2]
Object
41-
PCSSC-Net 20
Scene
+
Ours
Object
JI
labels as supervision, these approaches enforce 3D input data to capture geometries of individual objects a scene. However, it is not clear how such techniques can be easily extended to object-level point cloud completion with semantics properly preserved or manipulated.
In this paper, we propose Semantic-Prototype Variational Transformer (SPoVT) for semantics- preserving point cloud completion. Given a partial point cloud of an object with semantic labels observed for each point, our SPoVT able derive point cloud features and their semantic prototypes. With feature distributions for each semantic part properly observed, point cloud completion and semantic segmentation can be jointly achieved. Our SPoVT can be viewed as an pncoder-decodee based transformer. whose encoding process is to derive the semantic prototypes for each object part and their point-wise geometry features, while the decoding part is to recover the complete point cloud via sampling from the derived semantic-specific point cloud distribution. The proposed framework, as we discuss later, is not limited to the completion of point clouds at particular resolution (i.e., varying numbers of point cloud outputs). In addition, we show that the semantic prototypes allow us not only to recover but also to manipulate object parts for diverse completion outputs.
We now summarize the contributions of this work as follows:
We propose Transformer-based network, Semantic- Pemantic-Prototype Variational Transformer (SPoVT), taking partial point cloud and the associated part labels for point cloud completion.
By taking the above inputs, our proposed network learns point cloud distributions for each semantic part, allowing us to resample point features for decoding and generate point clouds with varying resolutions.
A ratio predictor is deployed in SPoVT for predicting point number distributions for each segment part, which serves and guidance for point cloud completion and alleviates potentially dense or sparse completion for particular object parts.
By learning prototypes and feature distributions for object parts, our model able perform point cloud completion and manipulate at instance or part levels.
2 Related work
2.1 Object completion
With the recent development of point cloud analysis 2
point cloud shape completion becomes a popular topic. As a pioneer, Yuan et al. [ propose PCN, a simple encoder-decoder network based on FoldingNet [3. By concatenating global features obtained from PointNet [2] with 2D points and passing through MLPs, PCN simulates the process of "folding" 2D plane into the 3D surface of an object. Tchapmi al. 4 further introduce a tree-structured folding approach, while Wang el al.repeat the folding process multiple times to iteratively grow and refine their prediction in CRN [5] and SCRN (6). On the other hand, ASFM-Net [7] adapts feature matching between latent codes obtained from the complete and the partial point clouds for improved point cloud completion. Since these folding-bass -aaee approaches directly apply shared MLPs to the concatenations of global features and predefined 2D points during decoding, interactions between neighboring points not considered, leading to possible coarse outputs without preserving local geometries.
23
To tackle the above issue, recent methods 100 9 12 13 24, 14 25 develop different approaches to achieve geometry-aware point cloud completion. VE-PCN IO] and GRNet 9 voxelize point cloud features via aggregating point features within voxel grids, directly apply 3D CNN for encoding and decoding and then regenerate point cloud by inverse gridding. VRCNet 12, PMP-Net++ [TT]
and PoinTr [14], the other hand, introduce variants of Transformer [15] for point cloud completion.

---

X
Xout
pouu
Semantic VAE
N x
Nour
OP
f
fM
fp
Ratio Predictor
Label-wise Transformer Decoder Layer
Lahel-wise Transformer Encoder Layer
Decoder O
Encoder OE
Cross A Actention Transformr Dvcoder Layer
Refiner OR
Mencatenate
Transformer Encoder Layer
Transformer Decoder Layer
Pyy
Cpx
Z1
in X
(noin)
Ph
P2
PN
Denormalize
M
Sy
Part Tokens
N XT)
N r
N
Figure I: Architecture of our proposed Semantic- Semantic-Prototppe Variational Transformer (SPoVT), which contains Encoder 0= Ratio Predictor 0",, Decoder 0º, and Refiner 0 Note that Semantic VAE is introduced for learning prototypes and feature distributions for each object parts during completion.
In order to focus on local geometry and to reduce computation, they only consider features from neighboring points when implementing attention modules. Although very promising results have been presented, using only 3D coordinates of partial inputs and relying on derived global features might not be sufficient due to the lack of semantic cues during completion.
2.2 Scene completion
proposes to complete both scene and the corresponding semantic segmentation from single 3D scan via 3D CNN. To perform more realistic completion by taking semantics as inputs, recent approaches I7 19 20 21 make use of additional semantic related input and show that 3D partial observations impressive results. For example, AIC-Net [16] combines multi-scale 2D and 3D features to make sure objects of different sizes are completed correctly. And, SPAwN [17 directly fuses voxelized 3D features with 2D priors obtained from segmentation map to generate detailed indoor scene.
SSCNet 26
semantic information are complementary to each other by exhibiting
Nevertheless, the above methods clouds. This is the reason why our work focuses on taking both point cloud geometry and semantic information as inputs for deriving the associated part prototypes and distributions. In Table [ we compare the characteristics of recent point cloud completion models with our SPoVT.
easily applied to complete and manipulate 3D object point
3 Proposed method
3.1 Problem formulation and model overview
For the sake of completeness, we first determine the setting and notations used in this paper. We have an input partial point cloud i containing total of N points 3D along with the associated semantic label. That is, we have x = {(... 8,) n-th point with attributes describing coordinates (Ir, Vn: zn ), and 8,, the corresponding semantic label (i.e., predict complete point cloud out with Nout points, given the ground truth GT with S points during training.
2, .22...,,T N), where P., E R denotes
{1,22 21,2,.... 2,.., M). where M is the total number of part classes). Our goal is to

---

To achieve this goal, we propose a Semantic-Prototyye Variational Transformer (SPoVT) in this paper. As depicted in Fig. our SPoVT has an Encoder 0º that learns the point fea- tures f P. = (f P.
Py for input point cloud and the prototypes of each semantic part SI:M (J....., f...... M) In particular, a learning scheme of Semantic VAE is presented during encoding, which learns point cloud distribution for each semantic part given the above features. Another unique characteristic is that a Ratio Predictor O is introduced for predicting the point number distribution across semantic parts, with output rout as --dimensionall vector. As for the decoding process, we deploy coarse Decoder 8² and Refiner @ aiming to recover coarse point cloud X and the refined output Xout, respectively.
It is worth repeating that, by taking both partial point cloud and the associate part labels, our SPoVT learned to jointly perform point cloud completion and semantic segmentation. Due to the ability to derive semantic prototypes for each part and to describe their feature distributions, instance/part-baseee point cloud synthesis and manipulation can be further achieved. We now detail our SPoVT in the following subsections.
3.2 Semantic-Prutotype Variational Transformer
3.2.1
Encoding
Given the partial input X our Encoder 8 in Figure is designed to derive features for each point f pp and semantic prototypes for each part f 1IMM In order to preserve both semantic and geometrical information during encoding, 8 is designed to take sequential input with length M M.N. The first M inputs are viewed as the tokens for semantic prototypes, i.e., G1:M = fei...., eM }. each e. contains the 3D coordinates (initialized by (0, 0, 0)) and the semantic label m. The latter N inputs denote the partial point cloud, i.e., Pp {epp................. ..... Epy ). with each ép,, described by the 4D representation of (P.: 8,). And, the encoded outputs would be f 1:MM and f p.y.. describing the emheddings derived for each prototype and input point cloud. It is worth noting that, instead of performing standard self-attention during encoding, we replace the last encoder layer with a Label-wise Transformer Layer, attending point cloud and the semantic parts with the same labels.
Semantic VAE. In order to better describe each semantic part and its feature distribution so that the output point cloud can be properly completed, we introduce the learning scheme of Semantic VAE in our SPoVT. Given the point features f f. PP-Py and the semantic prototypes f J:MM- we take the point features associated with label m and map its posterior distribution qe... (Sp. X to a predefined prior distribution (0, 1). Note that each of these distributions is parameterized by 0m via the reparameterization trick introduced in (27), and A( I) denotes the standard normal distribution.
Thus, the objective function for this process is defined as:
M
N
CKL
Σ m==1
Σ
KL Igom.. (fp. I [I"')| (o),))) I)],
(I)
Nm
where N is the number of points with label m, and K [-] denotes the KL-divergence. Note that the point feature f in Eqn. i normalized by fm. and we keep the same notation for simplicity. With this Semantic VAE scheme, our SPoVT is able to perform instance/part-baa based point cloud synthesis and manipulation, verified later in experiments.
Ratio predictor. Without the guidance of semantic labels, point cloud completion might produce dense or sparse point clouds for particular parts, which would be undesirable. Given both partial point cloud and their semantic labels, we additionally introduce a Ratio Predictor gP, which takes both the derived semantic prototypes f fi:Ms and partial point cloud X as inputs to predict r put which as an M -dmimensional vector indicating the point number distribution under the supervision of XGT Thus, the ratio loss for this ratio predictor is determined as Lratio GT , poot), where r GT denotes the ground truth point ratio and D(.) is a distance function. For simplicity, here we consider I.2 distance D-)). With this additional guidance, the decoding process can be performed accordingly.

---

3.2.2 Decoding
The decoding process of SPoVT is composed coarse Decoder 0² and Refiner 0² The former outputs coarse completion with N points from the complete point features sampled from the learned point distributions, while the latter predicts Nout point-wise displacements by considering both the partial input and the coarse output refine the results.
In order to select proper inputs to 02. our encoder design allows us to sample point features for each semantic part with predicted point number distribution. In other words, we simply sample N D-dimensional random noise vectors 2 N1:N= {z,,
1,2, .... No) from the Normal distribution for each part. The part labels ii--= = {8, n 1, 2....... ]o) are assigned to each sample according to the predicted your leading to No points for each label m, where N Th x No and r', denotes the m-th entry of pout, Finally, we recover these input noise vectors 1 into the complete point features N:: by denormalizing them according to the corresponding semantic prototypes.
Coarse completion. The coarse Decoder 0D takes the recovered point features 21:N= as the inputs and predicts a coarse point cloud output X where X {(p 8% n .,2,...,Au N.). Inspired by [1]]. we use cross-attention layer as the second layer of 0 to preserve point-wise structure from the partial input, with the last layer replaced by a Label-wise Transformer Layer for preserving semantics. The objective for coarse completion is the 1.2-Chamfer Distance:
N
NCT
Led(X xG)
Σ CDip,, GT Nc
1
Σ
CDiP, GT ,x),
(2)
NGT
where the point-to-set Chamfer Distance CD(p,, X of the i-th coarse predict point p to the ground truth point cloud XGT is defined as:
CD(p,,
min Ope
|l p - -pppi -pGT
(3)
Jexor
and the point-to-set Chamfer Distance CD X²) of the j-th ground truth point PT to the coarse prediction X defined similarly.
Refinement. For the task of point cloud completion, is necessary to preserve geometrical details for the given partial point cloud. Thus, deploy Refiner @ which takes the concatenation of partial input X and coarse output X as the inputs X for refined decoding. To be more precise, have X {ppaer , s(at)
= 1, (p..> Vn NN). and (prat, g takes concatenation of the coordinates of each p and the corresponding f as the inputs for predicting the coordinate displacement d,,, As result, our final completion result Xout obtained as X = =((p s.ttt)
Nout), where Noout = N + No, (p . sgaat) (p sf, E (N L...., Nout), Our 0² then
.. Nowu), where p?. p. + d,, Vn E (1...., Nowt).
However, despite the above refiner, the of standard L2 Chamfer Distance might still lead noisy results [8] Thus, we propose Gated Chamfer Distance as a novel objective, which regularizes outlier output points and favors the inliers with small displacements. More specifically, the Gated Chamfer Distance is defined as:
NoWE
NGT Σ
LgeD(Xouu, (
Nout Σ Cpred(pput XGT
1
CD CD(p.T CT. .XOu),,
NGT
(4)
Cyrea(p out xGT) [CD(pfue XGT), if CDDp"t XGT) 2 T,
(1| dill 2
otherwise,
Nout CD(p."e ,XCT). As shown in (4). if the where the threshold T is calculated as: T Nour Ei=
coordinate of a point pe before refinement sufficiently close to the ground truth, we consider as an inlier and suppress the magnitude of its displacement d, Oppositely, for an outlier point, we apply the standard Chamfer Distance term to penalize it. Finally, the overall loss for the decoding process is Ldecode LcD LgeD-
5

---

Table 2: Quantitative evaluation PCN in terms of L2- 12-Chamfer Distance (CDx10*) and mloU
( (%). Note that NGT 16384 for all methods across different categories.
Airplane
Car
Chair
Lamp
Table
Avg.
Method
mloU
CD
mloU
mloU
mloU
CD
mloU
CD
mloU
PCN
1.26
67.4
10.8
38.1
5.77
79.3
11.4
62.1
5.22
76.6
6.88
64.7
PMP-Net++
1.80
70.3
3.82
48.6
3.42
75.3
7.93
66.3
7.87
59.3
4.97
64.0
VRC-Net [2]
0.84
69.7
3.15
60.6
3,50
4.90
75.5
4.76
74.1
3.43
72.4
PoinTi [14]
1.88
53.6
3.73
50.8
4.55
60.5
2.97
76.1
3.23
64.0
SPoVT"
0.75
82.1
2.99
76.9
2.97
77.0
4.50
86.1
3.04
84.1
2.85
81.2
SPoVT (Ours)
0.73
82.6
2.86
82.5
2.36
85.2
4.12
91.5
2.50
86.5
2.51
3.3 Training and Inference
Training. Since the part labels si.ve for the coarse output are assigned during decoding according to out without being predicted by another segmentation model, we not need any additional objective for the segmentation results. Instead, make sure 0 able preserve the correct semantics, we pre- train 0 and 0 with the reconstruction branch before starting the completion task, i.c., taking SDAPN as inputs of 0 and predicting the reconstructed point cloud X {p", se) n = 1,2, NJ, where s = 8n: Vn (1,...,N) N). The loss function for the reconstruction is defined as:
Crecon = [mmeee" x + Axc
(5)
where myse (X", x ) is the Mean Square Error between the coordinates of X and X and Aice indicates the regularization weight of KL-divergence. Finally, we jointly optimize the reconstruction branch and completion branch by summing up the reconstruction loss Crecon the ratio loss Cration and the decoding loss Ldecode.
Inference with varying resolution. One unique feature of our SPoVT for point cloud completion is its ability to produce output point clouds with varying resolutions. Due to the design of Semantic VAE, our SPoVT learns complete point distribution for each semantic part, allowing us achieve varying resolution output by repeating the decoding process multiple times depending on the target resolution. Thus, if a total number of Noutt & points is required, we conduct the decoding process times with noises 1::0 resampled and Nowt points generated during each time. We then concatenate all the & outputs together obtain Noouu points in total.
Another feature our SPoVT is its ability to manipulate point cloud at instance and part levels. Due to Semantic VAE, the produced prototype and point features across different parts or objects can be manipulated with different point number distributions. We will verify this our experiments and confirm its effectiveness.
4 Experiments
4.1 Dataset and Implementation Details
Dataset. We conduct our experiments on the PCN dataset [1], following the training and testing split provided. To obtain semantic part labels for each object, we combine the PCN dataset and a part segmentation dataset ShapeNetPart (28), both subsets of ShapeNet [29]. As result, we have total number of 1262 different objects with five categories (i.e., airplane, car, chair, lamp, and table), where up to 4 part categories are available for each object. Each object is sampled with 16384 points as point cloud ground truth and has corresponding partial inputs rendered from 8 different virtual camera views. Please refer to Supplementary for more details and examples.
Implementation details. We set both the input point number N and the coarse output point number No to 512, leading to the total output points Nout 1024. In all our experiments, we train one model for each object category, using a single NVIDIA TESLA V100 GPU (32G) or NVIDIA RTX 3090 GPU (24G) for training, with batch size = 16, and learning rate = 10 -4 using the Adam [30] optimizer with an untuned linear warmup strategy (3I] We apply cyelical annealing
to Axc.
to avoid the potential collapse problem. After two cycles, Axc is set to 10-5 We implement our
model with Pyrorc 33 and Py PyTorch3I 34 libraries. We also use the official implementation of
12 14
and the PyTorch implementation of [ provided by [4 for comparison.

---

4.2 Semantic Point Cloud Completion
We now compare our SPoVT with a number of state-of-the-and approaches, including PCN M PMPNe+++ [T]] PoinTr [4. and VRCNet [12] in both the completion and the part segmentation tasks. We train the model for each category separately for fair comparisons. Note that since the previous methods are all designed for completion tasks only, we follow prior semantic instance completion methods such as RFD-Net [35] and RevealNet [66] when performing comparisons. That is, we adopt DGCNN [23] as the part segmentation model, which is pre-trained on the above dataset to predict the part labels for the above methods.
Quantitative evaluation. We evaluate the completion and part segmentation results with the L2- Chamfer Distance and mean intersection over union (mloU), respectively. For the I.2-Chamfer Distance, we follow VRCNet [12] and set NGT = 16384. As for the part segmentation, the mloU evaluates the average IoU of each part type in that object category. Note that the average scores are calculated by averaging over the score of five categories. The quantitative comparisons are summarized in Table 2 It shows that the proposed SPoVT achieves the best score on both the Chamfer Distance and mIoU among all methods. This verifies the design of encoding semantic information and learning point distribution of each part, where the sampled point features indeed represent the geometry of complete parts. To verify that our SPoVT is applicable for cases where ground truth segmentation labels are not observed, we conduct an additional experiment in which segmentation labels predicted by pre-trained DGCNN are used for training SPoVT, denoted as SPoVT" in T From this table, we see that while SPoVT" is not able to achieve comparable results the full-version does, it still performs against SOTA methods for both completion and segmentation tasks. This suggests that our proposed model is able to utilize pre-trained segmenters to assign point cloud labels for completion and segmentation purposes. Thus, the effectiveness and practicality of our proposed model can be verified.
Qualitative evaluation. The qualitative completion results are shown in Figure 2 From this figure, we observe that although existing methods complete global shapes of the desired ground truth objects, such methods do not sufficiently recover geometrical details which are not presented in the partial inputs. On the other hand, our SPoVT not only preserves details from the input but also recovers the fine geometry of the missing parts. This verifies that the design of our Semantic VAE really captures the feature distribution of each semantic part, and the sampled point features are recovered properly by the semantic prototypes. Moreover, our completion results are also the most visually uniformed among others, which shows that our Ratio Predictor indeed arranges the suitable sparsity of each part by predicting out precisely.
4.3 Further Analysis and Ablation Study
Manipulation. With the design of encoding semantic prototypes, we are able to conduct part-wise manipulation to create objects by interpolating between specific semantic prototypes and the corresponding point ratios. Selected results are shown in Figure For each row, we have a pair of the source and the target objects. We then choose one part of the source object interpolate with the same part of the target, while the rest remain the same as the source. We can observe that the selected part of each source object gradually deforms into that of the target object. Note that if the selected part is not contained in the target, as shown in the second row in Figure 3 then the corresponding part in the source object would disappear accordingly. This verifies the design of encoding semantic prototypes, which successfully preserve both semantic and geometric information of each part.
Similarly, demonstrate the results of instance-wiss interpolation in Figure 4 which shows that our SPoVT is also capable of instance-level deformation.
Varying resolution. We demonstrate the capability of varying resolution in Sec. where our SPo is capable of producing varying numbers of point cloud output by repeating the decoding process multiple times. To show the benefit of this property, we compare the surface reconstruc- tion results of our SPoVT with VRCNet [2), PoinTr [4]. and the ground truth point cloud in Figure 5 While VRCNet and PoinTr are restricted to the resolution of 16384 points due to relatively deterministic predictions, our ST easily achieves about 300k points and shows more details.
7

---

Partial
PCN
PMPNet++
PoinTr
VRCNet
Ours
GT
Figure 2: Qualitative evaluation. We compare the results produced by PCN M1. PMPNet++
PoinTr [14 and VRCNet [1]. Note GT represents the ground truth point cloud with NGT 16384.
Source
0.2
0.4
0.6
0.8
1.0
Target
Figure 3: Part-level point cloud manipulation. For each row, we select one semantic part (in red) of the source object interpolate with the same part of the target, with the interpolation step 0.2.
Ablation study. To further analyze the effectiveness of our designed modules, learning schemes, and loss functions, we now conduct ablation studies on chair objects in Table The baseline model A is composed only of the F 8 and 8 in Figure without applying the reconstruction pre-train introduced in Sec. 3.3 We then conduct the reconstruction pre-train (B) help preserve semantic information. Next, by adding back 0º (C), we correctly arrange the point number of each part to further raise the mloU by 18.5%. In model D. benefitting from finding the point distribution of each part, our completion result improves with 32% lower Chamfer Distance. Finally, our full model (model E) using LGeD achieves the best result. Thus, the proposed modules, learning schemes, and loss function can be successfully verified.
Sec.
Point cloud completion across object categories. In the above experiments, we train SPo to perform point cloud completion for each object category. worth noting that, one can also train SPo VT across multiple object categories by increasing the total number of semantic parts M. As

---

Source 0.2
0.4
0.6
0.8
Target
Figure 4: Instance-level interpolation. Given the source and target objects, we perform instance- level interpolation by interpolating between the corresponding semantic prototypes of the two objects, with the interpolation step as 0.2.
VRCNet (16k)
PoinTr (16k) PARA
Ours (16k)
Ours (300k)
GT (16k)
Figure 5: Surface reconstruction with varying resolution. The completed point cloud outputs (with numbers shown in each parenthesis) converted mesh outputs using the Alpha Shape method in Open3l) 377. Note that GT denotes the mesh obtained from the ground truth point cloud.
shown in Table 4 conduct new experiment, in which train single SPoVT model on both "Airplane" and "Car" categories. From the results listed in this table, we observe that a unified SPoVT exhibits slightly degraded completion and segmentation performances when compared to the original SPoVT, which is expected. Nevertheless, both SPoVT models still performs favorably against SOTA methods listed in Table 2 in which SOTA methods trained for each object category.
5 Limitations
We need to point out the potential concern of applying SPoVT when performing high-resolution point cloud completion. During the inference stage, the computation time grows linearly with the output point cloud resolution, since the completion output is produced by repeating the inference process

---

Table 3: Ablation studies on SPoVT. Note the the category of Chair is utilized for evaluation.
Model
Reconstruction Pretrain
0
Semantic VAE
LoCD
CD
mloU
5.23
43.07
B
5,01
62.83
C
S/
4.40
81.33
2.99
84.81
B
2.36
85.22
Table 4: Comparison of unified trained model and original results. We show the results that our SToVTi is trained on both "Airplane" and "Car" categories together in unified model.
Airplane
Car
Method
CD
mloU
CD
mloU
Ours (Original)
0.73
82.6
2.86
82.5
Ours (Unified)
0.84
84.6
3.39
80.6
Table Inference time and memory usage with different output point cloud resolution. Memory usage (GB)
Number of points
Inference time (ms)
2048
50.0
1.923
8196
145.6
1.923
16384
277.2
1.923
multiple times. In Table 5 list the inference time and memory usage for different point cloud resolutions. Another potential issue is the slight degradation when training one single SPoVT for performing completion across multiple categories, discussed in Sect. 4 The above are potential challenges, which are among the research topics for us to pursue the future.
6 Conclusion
In this paper, we proposed Transformer-basee network Semantic- Prottice Variational Trans- former (SPoVT), which can be applied for semantic point cloud completion. By taking partial point cloud and their semantic part labels as the inputs, our SPoVT is able to derive point cloud features and the associated semantic prototypes. With the deployed Semantic VAE scheme, the point feature distributions for each semantic part can be jointly observed, which allows resampling of point features from each part for completion and manipulation. With the proposed architecture, our SPoVT is able to perform completion across resolution, with the ability to preserve the point number distribution across different semantic parts, which alleviates possible dense sparse completion for each object part. From the experiments, we quantitatively and qualitatively confirm that our SPoVT performs favorably against tate-off-the-ar models in point cloud completion. We also verify its for point cloud interpolation and manipulation between different object instances or their parts, which cannot be easily handled by most existing methods.
Acknowledgement This work is supported in part by the Tron Future Tech Inc. and National Science and Technology Council via NSTC- 10-2634-F-002-05- We also thank National Center for high-peeffrrance Computing (NCHC) for providing computational and storage resources.
References
[1] Wentao Yuan, Tejas Khot, David Held, Christoph Mertz, and Martial Hebert. Pen: Point completion network. International Conference on 3D Vision (3DV). JEEE, 2018.
Charles Qi, Hao Su, Kaichun Mo, and Leonidas Guibas. Pointnet: Deep learning point for 3d classification and segmentation. In Proceedings of IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[3] Yaoqing Yang, Chen Feng. Yiru Shen, and Dong Tian. Foldingnet: Point cloud auto unco-en via deep grid deformation. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR). 2018
10

---

[4] Lyne P P'Tehapmi, Vineet Kosaraju, Hamid Rezatofighi, Ian Reid, and Silvio Savarese. Topnet: Structural point cloud decoder. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2019.
[5] Xiaogang Wang. Marcelo H Ang Jr. and Gim Hee Lee. Cascaded refinement network for point cloud completion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition (CVPR), 2020.
[6] Xiaogang Wang, Marcelo H Ang, and Gimhee Lec. Cascaded refinement network for point cloud completion with self-supervision. IEEE Transactions on Pattern Analysis and Machine Intelligence (TPAMI), 2021.
[7] Yaqi Xia, Yan Xia, Wei Li, Rui Song, Kailang Cao, and Uwe Stilla. Asfm-net: asymmetrical siamese feature matching network for point completion. In Proceedings of the ACM Conference Multimedia (MM), 2021.
[8] Minghua Liu, Lu Sheng. Sheng Yang. Jing Shao, and Shi-Min Hu. Morphing and sampling network for dense point cloud completion. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI), 2020.
[9] Haozhe Xie, Hongxun Yao, Shangchen Zhou, Jiageng Mao, Shengping Zhang, and Wenxiu Sun. Grnet: Gridding residual network for dense point cloud completion. In Proceedings of the European Conference on Computer Vision (ECCV). Springer, 2020.
[10] Xiaogang Wang, Marcelo H Ang, and Gim Hee Lee. Voxel-based network for shape completion by leveraging edge generation. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[11] Xin Wen, Peng Xiang. Zhizhong Han, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Yu-Shen Liu. Pmp- net++; Point cloud completion by caasformer-enhhance multi-step point moving paths. IEEE Transactions Puttern Analysis and Machine Intelligence (TPAMI), 2022.
[12] Liang Pan, Xinyi Chen, Zhongang Cai, Junzhe Zhang, Haiyu Zhao, Shuai Yi, and Ziwei Liu. Variational relational point completion network. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[13] Peng Xiang. Xin Wen, Shen Liu, Yan-Pei Cao, Pengfei Wan, Wen Zheng, and Zhizhong Han. Snowflak- enet: Point cloud completion by snowflake point deconvolution with skip-transformer. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[14] Xumin Yu, Yongming Rao, Ziyi Wang, Zuyan Liu, Jiwen Lu, and Jie Zhou. Pointr: Diverse point cloud completion with geometry-aware transformers. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), 2021.
[15] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[16] Jie Li, Kai Han, Peng Wang, Yu Liu, and Xia Yuan. Anisotropic convolutional networks for 3d semantic scene completion. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[17] Aloisio Dourado, Frederico Guth, and Teofilo de Campos, Data augmented 3d semantic scene completion with 2d segmentation priors. In Proceedings of the IEEE Winter Conference Applications of Computer Vision (WACV), 2022.
[18] Xuemeng Yang, Hao Zou, Xin Kong, Tianxin Huang, Yong Liu, Wanlong Li, Feng Wen, and Hongbo Zhang. Semantic segmentation-assin asssitee scene completion for lidar point clouds. In Proceedings of the IEEE/RSJ International Conference on Intelligent Robots and Systems (IROS). IEEE, 2021.
[19] Yingjie Cai, Xuesong Chen, Chao Zhang, Yee Lin, Xiaogang Wang, and Hongsheng Li. Semantic scene completion via integrating instances and scene in in-the-loop. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2021.
[20] Shoulong Zhang, Shuai Li. Aimin Hao, and Hong Qin. Point cloud semantic scene completion from rgb-d images. In Proceedings of the AAAI Conference on Artificial Intelligence (AAAI). 2021.
11

---

[21] Martin Garbade, Yueh-Tung Chen, Johann Sawatzky, and Juergen Gall. Two stream 3d semantic scene completion. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition Workshops (CVPR Workshops). 2019.
[22] Charles Ruizhongtai Qi, Li Yi, Hao Su, and Leonidas J Guibas. Pointnet++: deep hierarchical feature learning on point sets in metric space. Advances in Neural Information Processing Systems (NeurIPS), 2017.
[23] Yue Wang, Yonghin Sun, Ziwei Liu, Sanjay E Sarma, Michael M Bronstein, and Justin M Solomon. Dynamic graph can for learning on point clouds. ACM Transactions on Graphics (TOG), 2019.
[24] Bingchen Gong, Yinyu Nie, Yiqun Lin, Xiaoguang Han, and Yizhou Yu. Me-pen: point completion conditioned mask emptiness. In Proceedings the IEEE International Conference on Computer Vision (ICCV), 2021.
[25] Xin Wen, Tianyang Li. Zhizhong Han, and Yu-Shen Liu. Point cloud completion by skip-attention network with hierarchical folding. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2020.
[26] Shuran Song, Fisher Yu, Andy Zeng, Angel X Chang, Manolis Savva, and Thomas Funkhouser. Semantic scene completion from a single depth image. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
[27] Diederik Kingma and Max Welling. Auto-encoding variational bayes. arXiv preprint arXiv: 1312.1111. 2013.
[28] Li Yi, Vladimir G Kim, Duygu Ceylan, I-Chao Shen, Mengyan Yan, Hao Su, Cewu Lu, Qixing Huang, Alla Sheffer, and Leonidas Guibas. A scalable active framework for region annotation in 3d shape collections. ACM Transactions Graphics (TOG), 2016.
[29] Angel X Chang, Thomas Funkhouser, Leonidas Guibas, Pat Hanrahan, Qixing Huang, Zimo Li, Silvio Savarese, Manolis Savva. Shuran Song, Hao Su, et al. Shapenet: An information-rich 3d model repository. arXiv preprint ar:ivv:512.0000
2015.
[30] Diederik P Kingma and Jimmy Ba. Adam: method for stochastic optimization. arXiv preprint arXiv: 1412.6980, 2014.
[31] Jemry Ma and Denis Yarats. On the adequacy of untuned warmup for adaptive optimization. Proceedings of the AAAI Conference Artificial Intelligence (AAAI), 2019.
[32] Hao Fu, Chunyuan Li, Xiaodong Liu, Jianfeng Gao, Asli Celikyilmaz, and Lawrence Carin. Cyclical annealing schedule: A simple approach to mitigating kl vanishing. Annual Conference of the North American Chapter of the Association for Computational Linguistics (NAACL), 2019.
[33] Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, et al. Pytorch: An imperative style, high-performance deep leamnigg library. Advances in neural information processing systems, 32, 2019.
[34] Nikhila Ravi, Jeremy Reizenstein, David Novotny, Taylor Gordon, Wan- Yen Lo, Justin Johnson, and Georgia Gkioxari. Accelerating 3d deep learning with pytorch3d. 20Xiv:2007.08501, 2020.
[35] Yinyu Nie, Hou, Xiaoguang Han, and Matthias Nießner. Rfd-net: Point scene understanding by semantic instance reconstruction. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition (CVPR), 2021.
[36] Ji Hou, Angela Dai, and Matthias Nießner. Revealnet: Seeing behind objects in rgb-d seans. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2020.
[37] Qian-Yi Yi Zhou, Jaesik Park, and Vladlen Koltun. Open3d: A modern library for 3d data processing. arXiv preprint arv: 1 801009847, 2018.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's
contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes]
(c) Did you discuss any potential negative societal impacts of your work? [No]
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main exper- imental results (either in the supplemental material or as a URL)? [No] remain anonymous during the review stage, we will release our code and data after the paper published.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] We specify the dataset and model configurations in Section 4.1
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No]
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [Yes] See Section 4.1
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [Yes] We have cited the creators of the dataset/codes used in our experiments.
(b) Did you mention the license of the assets? [No] All assets mentioned in our paper (codes, data) are public for research, and their licenses can found their GitHub repositories or project pages. As result, since we have already cited the creators properly, we did not mention the licenses additionally.
(c) Did you include any new assets either in the supplemental material or as a URL? [No] To remain anonymous during the review stage, we will release our code and dataset after the paper is published.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [Yes] See Section 4.1 our dataset can be obtained by combining two public datasets.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] The point cloud dataset we used contains only human-designed 3DD models which do not include personally identifiable information and offensive content.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A] We didn't use any crowdsourcing or human subjects while conducting this research.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals. if applicable? [N/A] We didn't 't use any crowdsoureing or human subjects while conducting this research.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A] We didn't any crowdsourcing human subjects while conducting this research.
13


Reviews:
[{"review_id": "YXkTr588ypS", "reviewer": "Reviewer_fQ4M", "summary": "This paper proposes the Semantic-Prototype Variational Transformer model for the 3D point cloud semantic completion problem.  Experiments on benchmark datasets show that the proposed method outperforms existing methods.", "questions": "From Line 259 on Page 8, the authors perform an evaluation on the setting of varying resolution, I’m curious about the memory usage and runtime comparisons.", "limitations": "It seems there’s a lack of evaluation on the model runtime in the paper and supplemental. It’s suggested to provide it for objective evaluation.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\nThe proposed approach applies transformer-like structures with a semantic VAE module for 3D point cloud semantic completion problem. The authors also apply a coarse-to-fine strategy for improving the semantic completion, with the learning prototypes and feature distributions for each object part. The ablation analysis validates the efficacy of the proposed network structure.\n\nWeaknesses: \nThe motivation of using Transformer-structure for the encoding part for point cloud semantic completion seems not sufficient. Although the authors mentioned in the Introduction section that the reason is to better refine local geometry, in fact, point-based convolution methods such as PointConv [1] provide such utility and graph-based methods such as DGCNN [2] are also suitable for such task. A detailed discussion and comparisons in Related Works section are desired.\n\n[1] Wu, Wenxuan, Zhongang Qi, and Li Fuxin. \"Pointconv: Deep convolutional networks on 3d point clouds.\" Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition. 2019.\n[2] Wang, Yue, et al. \"Dynamic graph cnn for learning on point clouds.\" Acm Transactions On Graphics (tog) 38.5 (2019): 1-12.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Pfoz2XPoKJS", "reviewer": "Reviewer_Bq58", "summary": "A Transformer-based network for point cloud completion called Semantic-Prototype Variational Transformer (SPoVT) is proposed. Firstly, SPoVT learns point cloud distributions for each semantic part by taking partial point cloud and additional associated part labels as inputs, allowing resampling point features for decoding and generation point clouds with varying resolutions. Secondly, A ratio predictor is deployed in SPoVT for predicting point number distributions for each segment part, which serves and guidance for point cloud completion and alleviates potentially dense or sparse completion for particular object parts. Finally, by learning prototypes and feature distributions for object parts, SPoVT is able to perform point cloud completion and manipulate at instance or part levels.", "questions": "Questions:\n1.\tPlease address the above Weakness 1.\n2.\tIs the dataset the complete PCN dataset, or the intersection of PCN and ShapeNetPart? Does the intersection contain the annotations of all GT objects in the PCN? \n3.\tI noticed that you reproduced other methods for fair comparisons. I wonder if these methods are trained under the default configuration of the official implementation? Besides, both PMP-Net and PoinTr provide pre-trained models on PCN, how do these compare to your trained models?\n4.\tJust to be sure, like other completion methods, SPoVT needs to train the model separately for each category of objects, and you can't mix parts of different categories of objects, right?\n\nSuggestions:\n1.\tSome suggestions for figures.\na)\tFigure 1 is too cluttered, lacks annotations, and has a different style for each block. This is confusing.\nb)\tIn Figure 3, It might be better to mark the number of points in red.\n2.\tThe writing of the paper needs to be improved a little bit. \na)\tSome paragraphs are a little cluttered, like the Semantic VAE part, ablation studies part, etc.\nb)\tTypos, like “Tranformer-based” in the Introduction, “SWiVT” on Table 2, “mIOU”s , etc.\n", "limitations": "No.\nSuggestions: Please consider adding a discussion section to the paper to describe the limitations of the method as possible, such as the need for additional semantic annotations, inference speed, complex training process, etc.\n", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1.\tThe authors found an efficient way to introduce semantic label information in point cloud completion. Unsurprisingly, the performance is better than methods that only use partial point clouds as input.\n2.\tI like the idea of Semantic VAE, which can manipulate parts or objects with different point count distributions, making point cloud generation more flexible.\n3.\tThe experimental content is sufficient.\n\nWeaknesses:\n1.\tIn Table 2, pre-trained DGCNN is used for semantic segmentation for each completion method for comparison. Under this condition, the comparison of the segmentation metric is not too meaningful, it just means that the segmentation of those methods + DGCNN is worse than the proposed method.\n2.\tSome writing problems. Please see the suggestions below.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "6FX9lkwtct5", "reviewer": "Reviewer_QsJ3", "summary": "The authors propose Semantic-Prototype Variational Transformer for semantic point cloud completion, which combines the tasks of point cloud completion and part segmentation. The experimental results on the five categories of the PCN dataset demonstrate that the proposed method outperforms the state-of-the-arts methods.", "questions": "Please refer to the weaknesses.", "limitations": "Compared to the STOA completion methods (e.g. VRC-Net), the improvement of completion is small but requires more labor for data labeling. Moreover, the proposed method requires a model for each category. Do we really need to introduce the semantic information in point cloud completion?", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly written and well organized.\n\n**Weaknesses**\n\n- The motivation for introducing semantic labels in the point cloud completion is not clear. In other words, the proposed method can be regarded as point cloud completion + part segmentation.\n- The experiments are only conducted on the five categories of the PCN dataset, which is not enough. PCN is a very old dataset. I suggest conducting experiments on the ShapeNet34/55 and MVP datasets.\n- All competitive methods use DGCNN for part segmentation for a fair comparison. However, it is still not fair because the proposed method does not use DGCNN. Moreover, DGCNN is a very old baseline.\n- Why the mIoU of PointTr are much worse than the rest methods? All of these methods adopt DGCNN for part segmentation.\n- The experimental results are only compared on the synthetic dataset, how about the results on real-world scenarios?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "1vf2858IZIr", "reviewer": "Reviewer_Z3zB", "summary": "The authors proposed a Semantic-Prototype Variational Transformer approach for the dense point cloud semantic completion problem. In addition to the partial input, the paper also used the semantic labels as input for the completion task. An encoder-decoder Transformer architecture is used to leverage both the pixel level input and the semantic label input, so that the semantic label can guide the completion task. The Variational AutoEncoder (VAE) is used for model training and inference. Empirical results showed that the semantic label helped to improve the completion quality, outperforming other baselines.", "questions": "It has been reported in the literature that the VAE inference is prone to posterior collapse, which greatly degrades the quality of model inference. Have the authors encountered such problem in the VAE inference in this paper? Any procedures taken to alleviate this problem (e.g., through weight annealing)?\n\n", "limitations": "The model uses semantic labels as additional input for point cloud completion. A natural question is, what if such additional input is not available in practice? Is it practical to run a separate semantic labeler on the point cloud?", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1) The paper proposed to use semantic label from other sources as additional input to the point cloud completion problem. With this extra source of information, the model is able to infer the ground truth point cloud with better accuracy.\n2) The paper leveraged an encoder-decoder Transformer architecture to consume both the pixel level input and the semantic label input, so that the semantic label information can effectively guide the point cloud completion process. \n\nWeaknesses:\n1) The comparisons in Table 2 showed the advantages of the proposed method. However, it is not clear to me whether the comparison is apples-to-apples. For example, is the semantic label information used in other baselines (such as PCN)? It would be nice to explicitly mention which methods used such additional input, and which methods did not. That way, we would know where the improvement comes from (from the semantic label, or from the model architecture).", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept