Cross-modal Learning for Image-Guided Point Cloud Shape Completion
Authors: Emanuele Aiello
Diego Valsesia
Enrico Magli
Venue: NeurIPS 2022 Accept (2022)
Keywords: Point Cloud Completion
View-guided completion
Self-supervised completion
Multimodal Learning

Abstract: In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the information from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel self-supervised setting where the auxiliary image provides a supervisory signal to the training process by using a differentiable renderer on the completed point cloud to measure fidelity in the image space. Experiments show significant improvements over state-of-the-art supervised methods for both unimodal and multimodal completion. We also show the effectiveness of the self-supervised approach which outperforms a number of supervised methods and is competitive with the latest supervised models only exploiting point cloud information.

Cross-modal Learning for Image-Guided Point Cloud Shape Completion
Emanuele Aiello*
Diego Valsesia
Politecnico di Torino, Italy emanuele. aiello@polito.
Politecnico di Torino, Italy diego. Eoogovvalsese watpoovvaalessipooooooooooo it
Enrico Magli
Politecnico di Torino, Italy enrico wwg.mallili @polito.
Abstract
In this paper we explore the recent topic of point cloud completion, guided by an auxiliary image. We show how it is possible to effectively combine the informa- tion from the two modalities in a localized latent space, thus avoiding the need for complex point cloud reconstruction methods from single views used by the state-of-the-art. We also investigate a novel weakly-supervised setting where the auxiliary image provides supervisory signal to the training process by using differentiable renderer on the completed point cloud to measure fidelity the image space. Experiments show significant improvements over saate-of-the-ree art supervised methods for both unimodal and multimodal completion. We also show the effectiveness of the weakly-supervised approach which outperforms number of supervised methods and is competitive with the latest supervised models only exploiting point cloud information.
1 Introduction
The rise in popularity of 3D sensing technologies such as depth cameras, laser scanners, LiDARs, etc. is making the processing of point cloud data ever more important. The acquisitions produced by those instruments are often incomplete due to occlusions by objects in the environment, reflections, and viewing angles. This limits the exploitability of those data in tasks like scene understanding [1], robotic vision [2], autonomous driving [3] and many more. Completing a point cloud from partial observations is challenging iol-posed inverse problem that requires strong prior knowledge about shapes to be effectively regularized.
At the same time, we know that humans are very proficient mapping the visual concepts learnt from 2D images to understand the 3D world, and are able to successfully infer the shape of partial 3D objects from their 2D experiences. It is thus sensible to expect that point cloud completion techniques can benefit from 2D images to better characterize the 3D shape to be completed. Indeed, several applications of interest in robotic vision acquisitions of a depth-sensing instrument are paired with images from an RGB camera. It is also worth noting that the two modalities may be acquired from different vantage points, either thanks to disparities in the acquisition geometry or because the vantage point has changed with the passing of time. This makes it clear that the two modalities may carry complementary information and effectively fusing it key to unlock better completion performance. Nevertheless, the literature on the topic of point cloud completion [4, 5, 6, 7, 8, 9, 10, 11] has largely focused on the single-modality
take advantage of multimodal data where the 3D
"Code of the project: https://github. com hwt.a//..oooiiigallllleeeee htmm//ddegovalsssaa/MMeeee
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

problem, where only priors about 3D shapes are exploited. Only recently, image image-guided completion has started to receive attention [12].
In this paper, we study how the side information offered by a single image can be used in addition to shape priors to complete partial point cloud. While following the setting of ViPC [12], we extend the multimodal completion methodology in several different ways. First, ViPC [12] is bottlenecked by the need to estimate a coarse point cloud from the image via single-view reconstruction techniques to fuse the information. We avoid this task by proposing a novel architecture that performs fusion in a latent domain via cross-attention operations on fine-grainn graegiadddd, localized representations of the two modalities, coupled with a flexible decoder that allows to complete areas of varied size. Moreover, the multimodal setting is uniquely poised for -sueersse learning. In fact, the input image, especially when captured from a different vantage point, may offer a supervisory signal to guide the completion of those areas occluded in the partial point cloud but visible in the image. This is especially interesting for practical applications where it could be difficult to have access to complete shapes, but significantly easier to have images from a different viewing angle. Therefore, we propose to augment the known 3D seff-reconstruction losses with the exploitation of differentiable renderer to measure the fidelity of the completed point cloud in the image space.
Our experiments show that the proposed model significantly outperforms the state-of-the-art on both the supervised and weakly- seakly-supervised settings. In particular, the addition of the rendering loss allows the Seakly-supervisee image-gun image-guided model to outperform several supervised baselines and to be competitive with the latest supervised models only exploiting point cloud information.
2 Related work
Point cloud completion 3D shape completion is long-standing problem in computer vision. Early works devised explicit geometric descriptors or relied on shape retrieval from large datasets [13] [14]
[15] [16]. Since the advent of neural networks operating on raw point cloud data, several models for the completion problem have been studied [17]. They are mostly based on the encoder-decoder
architecture, pioneered by PCN [4], which was the first model that did not require any assumption of
structure or annotation information about the underlying shape. TopNet [5] presents hierarchical rooted tree structure that generates structured point clouds as collection of its subsets. AtlasNet [6] and MSN [18], the other hand, recreate the point cloud by assessing set of parametric surface elements. Convolutionall based approaches ([19, 20)) use a voxclixed representation of shapes as input to 3D CNNs; nevertheless, this representation introduces undesirable approximations in the shape due to coordinate quantization effects. GRNet [21] uses techniques to represent point cloud onto a 3D grid, that CNNs be exploited, without losing structural information. Recently, VRCNet [8] has proposed a dual path architecture and a VAE-based relation enhancement module for probabilistic modeling. Architectures based transformers have also been proposed. PointTr [9] changes the transformer block to take advantage of the inductive bias of 3D geometries, creating a geometry-aware block that models local geometry relations. Moreover, SnowflakeNet [22] generates child points by gradually splitting parent points by means of a Skip- Transfrrasorrmer that learns the appropriate splitting modes for particular regions. As a result, the network is able to predict highly detailed shape geometries. Finally, it is worth mentioning that the point cloud completion literature split between two settings: one, as in the aforementioned works, where the partial input has the same number of points as the completed point cloud and another where the completed point cloud has more points than the input such as in [23, 24]. In this paper, we will consider the former setting.
View-guided completion Recently, the usage of auxiliary data to complement point cloud comple- tion has been introduced by ViPC [12]. The idea is to help the reconstruction objective using side information available as a different imaging modality. In particular, ViPC assumes that an image corresponding to a view of the same object is also available for the completion task, and it exploits the image to retrieve the global shape information that is lacking in the incomplete point cloud. The image is processed by pre-trained single-view reconstruction model, which estimates coarse point cloud from the image, representing the entire shape. The key challenge in this setting is how to effectively combine features extracted from the two modalities. Unlike ViPC, our approach leverages direct fusion at a feature level, avoiding the need to explicitly reconstruct coarse point cloud from a single image, a generally hard inverse problem in itself and full of pitfalls.

---

PPS
PC Recommmttioo
Logs
ENCODER
DECODER
IMAGE ENCODER
DIFF RENDERER
HIHHUURTE
Rencering Loss
FRONTOLOUD
(N:3)
(N/6,F)
(N/18.F)
(N:3)
EC
EC
EC
P
EC
EC
0AN MMF
EC
EdgeConw dynamic graph convolution
SAGPoot graph pooling (4x downsempling)
Linear
BN
BatchNorm
R
ReLUU
Softmax
Transpose
CAT
Concatenation
Figure 1: Architecture overview. Localized features from the partial point cloud and the input image are jointly processed via cross- and self-attentions. A decoder reconstructs the target number of points from the feature space with iitention-based upsampling. The input partial point cloud downsampled with farthest point sampling (FPS) and concatenated. Supervised training only uses the training
point cloud reconstruction loss with respect to the complete point cloud. Weakly-superve has point cloud reconstruction loss with respect to less partial point cloud and rendering loss. N the number of points of the input point cloud; is the number points generated by each branch of the decoder, while F' represents the feature dimension.
Self-supervised strategies for completion All the previously mentioned approaches rely on complete ground truth as supervisory training signal. This may be difficult to obtain in real-word scenarios. Self-supervised training strategies avoid the need for retrieving such expensive ground truths. However, the amount of work on this topic is rather limited. Wang et al. [10] use resampling that removes further points from an already partial point cloud and mixup among shapes to train their completion network in self- self-supervised manner. Similarly, Mittal et al. [11] also propose an inpainting procedure that leverages further partializations of partial point clouds. To the best of our knowledge, there is no work exploring the availability of a different modality, namely an image to provide a weak supervisory signal the point cloud completion task.
Proposed Methods
We address the setting in which partial point cloud needs be completed with the assistance of an image of the object taken from a certain viewpoint. Our goal is to study how to leverage this side information in the most effective manner. To this purpose we study i) supervised learning setting, for which we show that the image features can be effectively fused with those of the partial point cloud latent space: ii) a weakly-supervised setting based on the idea that the image may contain clues about the missing part and can thus serve as supervisory signal. Fig. I shows an overview of the proposed method, named XMFnet SCross-Modaa Fusion network) which will be detailed in the next sections.
3.1 Architecture and Supervised Setting
At a high level, the architecture of XMFnet is composed by two modality-specin feature extractors that capture localized features of the input point cloud and image, summarized at small number of points/pixels, followed by a sequence of cross-attention and self-attention operations that progressively merge the two feature spaces. Finally, a decoder upsamples this localized information to estimate predefined number of points of the missing component. More formally, we denote the partial point cloud as X E R x3 the input view an image E IRPxxP, xP, x3 and the complete point cloud as Y E R xx3 The task of our model is to predict complete shape Y E Nx3 given X and I as inputs.

---

Notice we follow the conventional setting where the partial and complete point clouds have the same number points, meaning that there is resampling of the known part.
3.1.1 Point Cloud and Image Encoder
The point cloud encoder should extract localized features from the partial shape X. It is important to keep a degree of locality, i.c., associating features to a small number of points Nx < N rather than a single global embedding because the information about the missing part that needs to be estimated by the entire model is also mostly localized. However, is also important to have a sufficiently large receptive field to infer some global information about the object. For this reason, a graph-convolutional architecture with graph pooling operations. The architecture is a sequence of graph-convolutional layers (EdgeConv [25]) interleaved by pooling operations (Self-Attention Graph Pooling (26)) to reduce the cardinality of the point cloud. Pooling has the double purpose of expanding the receptive field to also include more global information and reduce the complexity of the subsequent cross- -tossnttt operations fusing the two modalities.
adopt
Any network that extracts features from an image can be utilized as encoder for view I. The design principles follow those of the point-cloud encoder, i.e., features localized at a subset N2 < Pa Py of the image pixels obtained from a sufficiently large receptive field are produced as output.
We will refer to the features produced by the point cloud encoder as Hx E Rxxxxxx and by the image encoder as H, R
3.1.2 Modality Fusion
Once we have collected localized information from the two modalities, we need to combine them effectively capture their complementary information, despite the obvious domain gap. The attention mechanism is particularly suited to find correspondences between the features of a region of the point cloud and region of the image. The cross-attention layer in architecture uses the Transformer's multihead attention mechanism [27]. The point cloud features are projected to form the query tensor, while the image features are projected form the key and value tensors, and then attention mechanism aggregates the features from different image regions according to the weights determined by the coreeccornl between the two modalities. More formally:
Qx W
K2 W,WK, V, H,W,
QK VF
H fused softmax
V
being Wo E FFXF WK, Wy e Frx the projection weights. The fused features Hiseed xF produced by the cross-attention mechanism can be regarded as the original point cloud features enriched by the image features.
The XMFnet architecture depicted in Fig. 1 shows a self-attention layer after the cross-attention fusion. The goal of this operation is to have a permutation-invariiat transformation of the features with a global receptive field so that any information from the image not properly integrated can be rectified. Self-attention works exactly like Eq. (2) except for the fact that Q, K, V are all different projections of the same features. Furthermore, a sequence of multiple cross- and self-attention layers can be used to more effectively integrate the information from the two modalities via "slow" fusion. We remark that at the end of this sequence we use a special cross-attention layer that merges information from the end and the beginning of the sequence allowing better flexibility in the decision of the desired abstraction level (higher- features cross-attend lower-level features).
3.1.3 Decoder and Supervised Loss
The decoder is a crucial component of our architecture as it should take the joint feature embedding and learn to reconstruct complete point cloud preserving both global and local structure. To be precise, the decoder seeks to estimate the positions of a number of points that upper bounds the size of the missing part, so that they can be concatenated to version of the input partial point cloud subsampled by means of farthest point sampling (FPS). This mechanism reminiscent of what done in the setting with variable number of points where only the missing part is estimated [23, 24).

---

For example, in our experiments, we upper bounded the size of the missing part to 50% of the total number of points. thus having N' points estimated by the decoder concatenated N' points from the subsampled input. However, our method allows to be flexible and handle more incomplete inputs by simply tuning the desired ratio of points to be estimated and points provided from the partial input. Typically, the latent space where we perform feature fusion is much more localized to constrain complexity and allow higher-level features so that Nx < NN, thus requiring the decoder to upsample the feature field. We perform this operation by using number of attention-baset operations, inspired by the work in [28], that convert features to points in parallel with the idea that each branch specializes on the reconstruction of sub-region of the missing part. The structure is depicted in Fig.1. More formally, calling H E R the features provided the decoder, the output Y, /KKx3 of each the K branches is computed as:
Z: ML (H) i= 1,,,..., K
(3)
Y, (enimax(MMLPP)) (Z.)) (Z.))
W out.
i K
(4)
where MLP RF R MLP F - RNN /K are multilayer perceptrons with different weights for each branch, projecting features to K subspaces and generating attention weights for the resampling process, respectively. Wou.s E F='x x3 is projection matrix to 3D space. Finally, the completed point cloud is generated by concatenation of the outputs of all decoder branches and the partial input subsampled by FPS as:
Y Y1, Y2........ YK: FPS(X))
(5)
Supervised training is performed using the 1. Chamfer Distance (CD) between the generated shapes and the ground truth shapes, defined as follows:
LCD Co, r)
Σ
min
y
min وý yeY JEÝ
Σ
y|l-
(6)
2N yEY YEY
2N
3.2 Weakly-supervised Setting
The multimodal completion problem addressed in this paper is uniquely poised for learning a setting where the full ground truth point cloud not available. In fact, the image available as input may contain complementary information with respect to the point cloud and, crucially, cues about the missing part. This especially true the image is collected from a different viewpoint or at different time with respect to the point cloud, resulting in different kinds of occlusions.
Existing architectures for self-supervised completion [10] [11] rely solely on point cloud supervision, due to their unimodal nature. The key insight of the proposed method (Fig. 2) is to supplement completion losses on points with a loss measuring a reprojection error in the image space, In particular, we measure whether the reconstructed point cloud produced by the architecture described in Sec. 3.1 leads to an image similar to the input one, when captured from the correct viewpoint.
Partialized Partial PC
Partial PC
Point cloud
MODEL
Image
DIFF. RENDER
Render
SILHOUETTE
Figure 2: supervie training.
In order to measure this information and use it in the training process, we include differentiable rendering module based on alpha compositing [29] which generates rasterized version of the object, employing provided camera parameters. In order to ensure consistency with the input image, intrinsic and extrinsic camera parameters may be estimated with a number of well-known methods [30] [31] [32]. In order to minimize the domain shift between the input image and the result of the rendering process, we work on silhouettes, i.e., binary masks of objects. The differentiable renderer produces a soft silhouette with continuous values, while the input image is directly binarized. Inevitable inaccuracies in the camera parameters and rendering process will typically yield unreliable borders of the silhouette. Therefore, we also compute a border mask with simple edge detector (Laplacian of
5

---

Gaussian) and discount the loss function by a factor E < 1 for the pixels in the mask. In summary, our rendering loss defined as:
render = M R -
M = {e
if (i. E edge otherwise
being R the differentiable silhouette renderer and S the silhouette binarizer. We remark that [33] proposes to use rendering to improve performance in point cloud completion, aiding the learning process through an image domain supervision. However, their approach is supervised and is based on rendering the ground truth and generated point cloud depth-maps with different view-points.
In addition to the rendering loss, our weakly-supervinn the point cloud domain. In particular, we use a combination of the resampling and mixup approaches proposed in [10, 11J. Resampling consists in removing random portions of the original partial input point cloud, yielding even more partial shapes. As a result, the original partial input employed as a pseudo-ground truth. Mixup combines a pair of partial shapes weighed according to Beta distribution an attempt at increasing the complexity of the shapes processed by the network. Differently from [10), we also have images associated with a partial shape in our setting. Hence, we also mix the images such a way that the mixup technique carried out symmetrically for the two modalities.
framework also adopts self-supervision in
We remark that the rendering loss is comparatively weaker than the point cloud loss and, by itself, has number of ambiguities due to the lack of depth information and the use silhouettes. For this reason, it is important to combine it with the point cloud loss. We found that the density-aware Chamfer Distance (DCD) [34], version of CD that more sensitive to non-uniform point distributions is scenario to regularize the ambiguities of the rendering loss. Our
superior in this weakly-supery overall weakly-supervisee training procedure is therefore as follows. We alternate between step that optimizes the point cloud loss consisting in weighted CD:
Cpc =
1- 3
Σ
min y - yl+ VEY
Σ SEY
min 18 yl, yeY
(8)
2N
2N
and step optimizing combination of DCD and rendering loss:
==
Σ yEY
1 N
wlo
Σ YEY
1
1 N
alls-all
+ XCrender
2N
2N
where w arg mingey y f112 and z arg minye y Notice that CPC and the DCD part of Li use resampling and mixup for point clouds, while Crender does not (Ci is computed with full minibatch for the point-cloud part and half minibatch with the original partials for rendering).
4 Experimental results
4.1 Experimental Settings and Implementation Details
All the experiments are conducted on the MhapeNettViPPCI11 dataset. The dataset contains 38,328 objects from 13 categories; for each object it comprises 24 partial point clouds with occlusions generated under 24 viewpoints, using the same settings as ShapeNetRendering [35]. The input and ground truth point clouds contains N 2048 points each. Each 3D shape is rotated to the pose corresponding to certain view point after being normalized within the bounding sphere with radius of 1. Images are generated from the 24 view points ShapeNetRendering and have a resolution of 224 x 224 pixels. For all the experiments in this paper, employ the same selection used in [12]: we used 31,650 objects from eight categories, with 80% of them for training and 20% for testing.
The partial point cloud is downsampled by farthest point sampling to N' = 1024 points and concatenated to the output of the decoder that produces N' 1024 points leading to a completed point cloud with 2048 points. The decoder has K = 8 branches, each of them producing M = 128 points. The point cloud encoder employs EdgeConv and SAGPooling layers; the EdgeConv layers selects & 20 nearest neighbors, while the two pooling layers use k 16 and & 6 nearest neighbors, respectively. The original point cloud overall downsampled by a factor of 16, resulting in Nx = 128 points with Fx 256 features. The image encoder is built with a ResNet18 8 (36) as

---

Table 1: Mean Chamfer Distance per point (x 10 ShapeNet-ViPe dataset, supervised.
Methods
Avg
Airplane
Cabinet
Car
Chair
Lamp
Sofa
Table
Watercraft
AtlasNet [6]
6.062
5.032
6.414
4.868
8.161
7.182
6.023
6.561
4.261
FoldingNet [39]
6.271
5.242
6.958
5.307
8.823
6.504
6.368
7.080
3.882
PCN [4]
5.619
4.246
6.409
4.840
7.441
6.331
5.668
6.508
3,510
TopNet [5]
4.976
3.710
5.629
4.530
6.391
5.547
5.281
5.381
3.350
ECG [7]
4.957
2.952
6.721
5.243
5.867
4.602
6.813
4.332
3,127
VRC-Net [8]
4.598
2.813
6.108
4.932
5.342
4.103
6.614
3.953
2.925
ViPC [12]
3.308
1.760
4.558
3.183
2.476
2.867
4.481
4.990
2.197
XMFnne
1.443
0.572
1.980
1.754
1.403
1.810
1.702
1.386
0.945
Table 2:
Mean F-Score @ 0.001.
ShapeNet-Vin ViPC dataset, supervised
Methods
Avg
Airplane
Cabinet
Car
Chair
Lamp
Sofa
Table
Watercraft
AtlasNet [6]
0,410
0.509
0.304
0.379
0.326
0.426
0.318
0469
0.551
FoldingNet [39]
0.331
0.432
0.237
0.300
0.204
0.360
0.249
0.351
0.518
PCN [4]
0,407
0.578
0.270
0.331
0.323
0.456
0.293
0.431
0.577
TopNet [5]
0.467
0.593
0.358
0.405
0.388
0.491
0.361
0.528
0.615
ECG [7]
0.704
0.880
0.542
0.713
0.671
0.689
0.534
0.792
0.810
VRC-Net [8]
0.764
0.902
0.621
0.753
0.722
0.823
0.654
0.810
0.832
ViPC [12]
0.591
0.803
0.451
0.512
0.529
0.706
0.434
0.594
0.730
XMFnet
0.796
0.961
0.662
0.691
0.809
0.792
0.723
0.830
0.901
backbone, it extracts N2 14 x = 196 pixels with F2 256 features. The multihead attention has attention heads, with embedding size = 256. In the Ci loss we use À = 0.15. The mask factor for the edge detector has been set to 0.4.
The differentiable renderer has been implemented with PyTorch3D[37]. The rendered silhouettes H x W has size 224 x 224 that is the same size of input views in our experiments. We adopt radius p 0.025 in point rasterization. The proposed framework is implemented in PyTorch and trained on an Nvidia V100 GPU. Class-specific training is performed for all models, using the Adam optimizer [38] for roughly 200 epochs with batch size of 128. The learning rate is initialized to 0.001 and reduced by factor of 10 at epoch 25 and 125.
4.2
Main Results
4.2.1
Supervised Learning
We first compare XMFnet against several baselines for the point cloud completion under supervised learning. Since the multimodal setting with auxiliary image has been introduced only recently. ViPC [12] represents the only method fully comparable to ours. However, we also report the results of number of state-of-the-art architectures for completion with only point cloud input, when retrained on the ViPC dataset. AtlasNet[6] reconstructs a point cloud by estimating parametric surface elements. [o]dingNet[[9] is 2-D grid based auto-encoder. PCN[4] is an encoder-decoder framework that reconstructs the point cloud in a coarse-to-fin manner. TopNet[5) has a rooted tree structure in the decoder. ECG[7] is an edge-aware completion method based on Graph Convolutions. VRC-Net[8] is the most recent method adopting a VAE-based model with a dual path architecture and probabilistic modeling. In line with the previous evaluation protocols, we use CD and F-score [40] as metrics for the reconstruction quality. Before evaluating the CD, we normalize the output of the models to fit into the unit sphere. Table and Table 2 report the experimental results and show XMFnet outperforming the other techniques by a significant margin. While part of this gain with respect to state-of-the ofite--f-the-eee of-the-art models for point cloud completion can be attributed the use the input image, worth noting that we also report significant improvements over the multimodal ViPC. This highlights the sub-optimality of performing modality fusion by resorting to estimating coarse point cloud from the single input image, as in ViPC, rather than working in a latent feature space. Qualitative
7

---

Partial
PCN
ECG
VRCnet
ViPC
XMFnet
GT
1
Figure 3: Qualitative comparison of completed point clouds for different classes.
comparisons? are shown Fig. 3. Our method is capable of producing cleaner completions than the other baselines, with fewer outliers and more uniform point distribution.
4.2.2 Weakly-supervised
We remark that we are the first to propose a weakly-supervised training strategy for multimodal completion, the setting in which supervisory information be gathered from input image is unexplored. For this reason, we compare to a number of unimodal and multimodal supervised methods as well as a self-supervised version using resampling and mixup of a unimodal Mate-of-the art model?. The results are reported in Table 3 for the CD and Table 4 for the F-Score. We notice that our weakly-supervised method outperforms a number of supervised baselines and it close the performance of the most recent unimodal supervised models.
4.3 Ablation Studies
In order verify the effectiveness of the proposed design, study the impact of the auxiliary image input on the completion performance. To ensure comparison as fair as possible, the version of XMFnet that uses only point cloud information has the image encoder removed and the cross-attention blocks replaced with self self-attention ones.
VVisualizations for ViPC [12] have been kindly provided by the original authors. "W remark the difficulty reproducing several published methods in the superviee setting. The code for [10, 11] was not available. The code to reproduce ViPC [12] is also incomplete so we cannot retrain the most sensible self-supervised haseline of ViPC Resampling + Mixup.

---

Table 3: Mean Chamfer Distance per point
(x10-3) )). ShapeNet- ViPC dataset.
Table Unimodal vs. Multimodal comple- tion (supervised)
Methods
Airplane Lamp Watercraft
AtlasNet [6]
5.032
7.182
4.261
FoldingNet [39]
6,504
6.368
3.882
Method
Avg
Airplane Cabinet Lamp
PCN [4]
4,246
6.331
3.510
Unimodal
1.570
0.626
2.114
1.980
TopNet [5]
3.710
5.547
3.350
Multimodal
1.470
0.572
1.973
1.810
ECG [7]
2.952
4.602
3.127
4, best view
1.223
0.545
1,426
1.697
VRC-Net [8]
2.813
4.103
2.925
L, worst view
1.819
0.722
2.621
2.115
VRC-Net(self-sup.)
4.315
8.023
7.259
ViPC [12]
1.760
2.867
2.197
XMFnet (sup.)
0.572
1.810
0.945
Table 6: Ablation study for the Weakly- Supervised method (airplane)
XMFnet (weakly-sup.)
2,426
6.269
3.423
Table 4: Mean F-Score 0.001. ShapeNet-ViPCC
Resampling Mixup Rendering CD(10
dataset.
x
4.568
4.239
Methods
Airplane Lamp Watercraft
/
/
2.426
AtlasNet [6]
0.509
0.426
0.551
FoldingNet [39]
0.432
0.360
0.518
PCN [4]
0.578
0.456
0.577
Table 7:
Ablation study for the Weakly- Supervised method DCD (cabinet)
TopNet [5]
0.593
0.491
0.615
ECG [7]
0.880
0.689
0.810
VRC-Net [8]
0,902
0.823
0.832
DCD CD(10-3)
VRC-Net(self-sup.)
0.689
0.710
0.673
ViPC [12]
0.803
0.706
0.730
3.012
XMFnet (sup.)
0.961
0.792
0.901
2.426
XMFnet (weakly-sup.)
0.742
0.542 0.704
The unimodal architecture then trained with the same settings as the multimodal one, and the results reported in Table 5. The results show that the addition of the image input pro- vides a significant improvement in performance. Notice that besides the result averaged over all the possible 24 views, we also report the performance with the worst view and the best view. Indeed, we are interested in investigating how the viewpoint of the image affects completion. Fig. 4 reports the average CD for different views, ordered from worst to best, for the cabinet category. It clear that some views provide complementary information due to their different vantage point and allow to substantially improve over the average result. A small number of "bad" views leads to results comparable to the unimodal case.
2:15
2.1
8)
CD (x
Furthermore, interesting to study the impact of our novel rendering loss in the wcakly-supervised setting. We noticed that allows the training process to have a faster and smoother convergence and that the overall completion performance in- creased from both a qualitative and quanti- tative point of view. A qualitative compar- ison between the weakly- weakly-supervised supervised strat- egy with and without the rendering module can be visualized in Fig. 5 and quantitative
1.9
Mutimodal
1.05
Moliindal Average
1.8
5
10
15
20
Figure 4: Impact of image contribution function of point of view, sorted by reconstruction CD (from worst best) averaged cabinet category, supervised ting.

---

Partial
Without Rendering
With Rendering
GT
Figure 5: Qualitative visualization of the effect of the proposed weakly-supervised rendering loss.
The sample without rendering has CD 7.812, the one with rendering has CD= 3.743.
Partial
Normal CD
DCD
GT
Figure 6: Qualitative visualization of the effect of the DCD for the weakly-supervised setting.
results reported in Table 6. The mixup loss provides only small improvement from the perspec- tive the CD. However, it substantially improves the completed shape from a qualitative point of view, helping the network generate more complete shapes. Moreover, the computational overhead due to creating the mixed input shape is very small, so we decided to keep the method as it offered a very favorable cost-perfen perfformrom trade-off. We also include the ablation for the DCD component of the -speevssuee loss, we found helpful from both a qualitative and quantitative point of view. Table 7 reports the impact of the DCD on the cabine! category. while Fig. 6 provides qualitative example, where the shape completed with the DCD presents a more uniform distribution of points and better overall quality.
Conclusions
In this paper, explored the topic of point cloud completion guided by auxiliary image, discover- ing that effective fusion can be achieved in latent space via cross-attention. Our method achieves state-of-ther of-the-art art results the ShapeNetViPC--attae itself to weakly-supervised learning where the image can be used for supervision via a differentiable rendering approach, when the full ground truth point cloud is not available. The major limitation of our work the lack of study work, will focus on extending the work to real acquisitions, thus dealing with complex effects like acquisition noise, background or additional occlusions in the auxiliary images, and many more. This paper has provided proof of concept that effective multimodal completion is possible but more in-depth study of such issues on real scenes is needed, along with suitable improvements to our design towards increased robustness.
Morcover, we showed how this setting lends
real-world scenario for the proposed framework. In future
Acknowledgments and Disclosure of Funding
Computational resources were provided by HPC@ POLITO, project of Academic Comput- ing within the Department of Control and Computer Engineering at the Politecnico di Torino (ttp://wwww.hccpooiio...) This research received no external funding.
10

---

References
[I] J. Hou, A. Dai, and M. NieBner, "3D-SIS: 3D semantic instance segmentation of RGB-D scans," in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, 2019, pp. 4421-4430.
[2] Varley, C. DeChant, A. Richardson, Ruales, and Allen, "Shape completion enabled robotic grasping," in 2017 JEEE/RSJ international conference on intelligent robots and systems (IROS). IEEE, 2017, pp. 1442-2447,
[3] Y. Li, L. Ma, Z. Zhong, F. Liu, M. A. Chapman, D. Cao, and J. Li, "Deep learning for lidar point clouds autonomous driving: A review," IEEE Transactions on Neural Networks and Learning Systems, vol. 32, no. 8, pp. 3412-3432, 2020.
[4] W. Yuan, T. Khot, D. Held, C. Mertz, and M. Hebert, "PCN: Point completion network," in 2018 International Conference 3D Vision (3DV). IEEE, 2018, pp. 728-737,
[5] L. P. Tchapmi, V. Kosaraju, H. Rezatofighi, I. Reid, and S. Savarese, "Topnet: Structural point cloud decoder," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2019, pp. 383-392.
[6] T. Groueix, M. Fisher, V. G. Kim, C. Russell, and M. Aubry, "A papier-mäse måchéé approach to learning 3d surface generation," in Proceedings of the IEEE conference on computer vision and pattern recognition, 2018. pp. 216-224.
[7] I.. Pan, "ECG: Edge-aware Point Cloud Completion with Graph Convolution," IEEE Robotics and Automation Letters, vol. 5, no. 3, pp. 4392-4398, 2020.
[8] L. Pan, X. Chen, Z. Cai, J. Zhang. H. Zhao, S. Yi. and Z. Liu, "Variational relational point completion network," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp. 8524-8533.
[9] X. Yu, Y. Rao, Z. Wang, Z. Liu, J. Lu, and J. Zhou, "PoinTr: Diverse point cloud completion with geometry-aware transformers," in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 12 498--125577
[10] X. Wang, M. H. Ang, and G. Lee, "Cascaded refinement network for point cloud completion with self-supervision" IEEE Transactions on Pattern Analysis and Machine Intelligence, 2021.
[11] H. Mittal, B. Okorn, A. Jangid, and D. Held, Suefr-Sueee Point Cloud Completion via Inpainting," in British Machine Vision Conference, 2021.
[12] X. Zhang, Y. Feng. S. Li, Zou, H. Wan, X. Zhao, Y. Guo, and Gao, "View- point cloud completion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2021, pp. 15 890-15 899.
[13] M. Sung, V. G. Kim, R. Angst, and L. Guibas, "Data-driven structural priors for shape comple- tion." ACM Transactions on Graphics (TOG), vol. 34, no. 6. pp. 1-11,,015.
[14] N. J. Mitra, L. Guibas, and M. Pauly, "Partial and approximate symmetry detection for 3d geometry," ACM Transactions on Graphics (TOG), vol. 25, no. 3, pp. 560-568, 2006.
[15] M. Pauly, N. J. Mitra, Wallner, H. Pottmann, and L. J. Guibas, "Discovering structural regularity in 3d geometry," ACM Transactions on Graphics (TOG), vol. 27, no. 3, pp. 1-11, 2008.
[16] R. Schnabel, P. Degener, and R. Klein, "Completion and reconstruction with primitive shapes." Computer Graphics Forum, vol. 28, no. Wiley Online Library, 2009, pp. 503-512.
[17] B. Fei, W. Yang, W. Chen, Z. Li, Y. Li, T. Ma, X. Hu, and L. Ma, "Comprehensive review of deep learning-based 3d point clouds completion processing and analysis," arXiv preprint arXiv:2203.033: 03311, 2022.

---

[18] M. Liu, L. Sheng, S. Yang, J. Shao, and S.-M. Hu, "Morphing and sampling network for dense point cloud completion." " in Proceedings of the AAAl conference on artificial intelligence. vol. 34, no. 07, 2020, pp. 11 596-11 1 603.
[19] A. Dai, C. Ruizhongtai Qi, and M. Nießner, "Shape completion using 3D-encoder-predictor CNNs and shape synthesis," " Proceedings of the IEEE conference on computer vision and pultern recognition, 2017, pp. 5868-5877.
[20] X. Han, Z. Li, H. Huang, E. Kalogerakis, and Y. Yu, "High-resolution shape completion using deep neural networks for global structure and local geometry inference," Proceedings of the IEEE international conference on computer vision, 2017, pp. 85-93.
[21] H. Xie, H. Yao, S. Zhou, J. Mao, S. Zhang, and W. Sun, "GRNet: Gridding residual network for dense point cloud completion," in European Conference on Computer Vision. Springer, 2020, pp. 365-381.
[22] Xiang. X. Wen, Y.-S. Liu, Y.-P. Cao, P. Wan, W. Zheng, and Z. Han, "Snowflakeeet:: Point cloud completion by snowflake point deconvolution with skip-transformert in Proceedings of the IEEE/CVF International Conference on Computer Vision, 2021, pp. 5499-5509.
[23] Z. Huang. Y. Yu, J. Xu, F. Ni, and X. Le, "Pf-net: Point fractal network for 3d point cloud completion," in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2020, pp. 7662-7670.
[24] A. Alliegro, Valsesia, G. Fracastoro, E. Magli, and T. Tommasi, "Denoise and contrast for category agnostic shape completion," in Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, 2021, pp. 4629-4633
[25] Y. Wang, Y. Sun, Z. Liu, S. E. Sarma, M. M. Bronstein, and J. M. Solomon, "Dynamic graph enn for learning point clouds," ACM Transactions on Graphics (TOG), 2019.
[26] J. Lee, I. Lee, and J. Kang, "Self-attention graph pooling," in International conference on machine learning. PMLR, 2019, pp. 9734-3743.
[27] A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L. Jones, A. N. Gomez, L. Kaiser, and I. Polosukhin, "Attention all you need," Advances in neural information processing systems, vol. 30, 2017.
[28] K. Zhang, X. Yang, Y. Wu, and C. Jin, "Attention- based transformation from latent features to point clouds," arXiv preprint arXiv:2112.05333 2021.
[29] O. Wiles, G. Gkioxari, R. Szeliski, and J. Johnson, "Synsin: End-to-end view synthesis from a single image." in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, 2020, pp. 7467-7477.
[30] E. Insafutdinov and A. Dosovitskiy, "Unsupervised learning of shape and pose with differ- entiable point clouds," in Advances in Neural Information Processing Systems (NeurIPS), 2018.
[31] A. Grabner, Y. Wang, P. Zhang, P. Guo, T. Xiao, P. Vajda, P. M. Roth, and V. Lepetit, "Geometric correspondence fields: Leamed differentiable rendering for 3d pose refinement in the wild," in European Conference on Computer Vision. Springer, 2020, pp. 102-119.
[32] A. Palazzi, L. Bergamini, S. Calderara, and R. Cucchiara, "End-to-end 6-dof object pose estimation through differentiable rasterization." " in Proceedings of the European Conference on Computer Vision (ECCV) Workshops, 2018, pp. 0-0.
[33] C. Xie, C. Wang, B. Zhang, H. Yang, D. Chen, and F. Wen, "Style-based point generator with adversarial rendering for point cloud completion," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2021, pp.
[34] T. Wu, L.. Pan, J. Zhang, T. Wang, Z. Liu, and D. Lin, "Balanced Chamfer Distance as a Com- prehensive Metric for Point Cloud Completion," " Advances in Neural Information Processing Systems, vol. 34, 2021.
12

---

[35] A. X. Chang, T. Funkhouser, L. Guibas, P. Hanrahan, Q. Huang, Z. Li, S. Savarese, M. Savva, S. Song. H. Su, J. Xiao, L. Yi. and Yu, "ShapeNet: An Information- Rich 3D Model Reposi- tory." Stanford University - Princeton University- - Toyota Technological Institute at Chicago, Tech. Rep. arXiv:1512.03012 [es.GR], 2015.
[36] K. He, X. Zhang, S. Ren, and J. Sun, "Deep residual learning for image recognition," " in Proceedings of the IEEE conference on computer vision and pattern recognition, 2016, pp. 770-778.
[37] N. Ravi, J. Reizenstein, D. Novotný, T. Gordon, W.-Y. Lo, Johnson, and G. Gkioxari, "Accelerating 3d deep learning with pytorch3d," SIGGRAPII Asia 2020 Courses, 2020.
[38] D. P. Kingma and J. Ba, "Adam: A method for stochastic optimization." in Internation Confer- ence on Learning Representations, 2015.
[39] Y. Yang, C. Feng. Y. Shen, and D. Tian, "FoldingNet: Point Cloud Grid Deformation," in Proceedings of the IEEE conference computer vision and pattern recognition, 2018, pp. 206-215.
via Deep
[40] A. Knapitsch, J. Park, Q.-Y. Zhou, and V. Koltun, "Tanks and temples: Benchmarking large- scene reconstruction," ACM Transactions on Graphics (TOG), vol. 36, pp. 13, 2017.
Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations your work? [Yes] We discussed the main limitations in Section 5.
(c) Did you discuss any potential negative societal impacts of your work? [No] Our work foundational research, we do not see direct connection towards malicious usage of shape completion method presented in the paper.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [N/A]
(b) Did you include complete proofs of all theoretical results? [N/A]
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main experi- mental results (either in the supplemental material or as a URL)? [Yes]. . We included a preliminary version of the code in the supplementary material, we will release it upon the paper accepted.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [Yes] See Section 4
(c) Did you report error bars (e.g., with respect the random seed after running experi- ments multiple times)? [No] We followed the typical evaluation setting of point cloud completion papers: we reported the best results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, cloud provider)? [Yes] See Section 4
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) If your work uses existing assets, did you cite the creators? [Yes]
(b) Did you mention the license of the assets? [Yes]
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A]
13

---

(c) Did you discuss whether the data you are using/curating contains personally identifiable information or offensive content? [N/A]
5. If you used crowdsoureing conducted research with human subjects...
(a) Did you include the full text of instructions given to participants and screenshots, if applicable? [N/A]
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A]
(c) Did you include the estimated hourly wage paid to participants and the total amount spent on participant compensation? [N/A]
14


Reviews:
[{"review_id": "rIZHcz7ztY3", "reviewer": "Reviewer_eDcw", "summary": "The paper proposes a point cloud completion approach with images as auxiliary information. The main challenge while utilizing the information from image for point cloud competition is effectively fusing local context between image and partial point cloud. This paper proposes using cross attention modules to fuse the local information from the image and point cloud. Furthermore, to supervised the neural network, two losses are used: 1) supervised loss based on chamfer distance between predicted and ground truth point cloud and 2) self-supervised silhouette reconstruction loss with the help of a differentiable rendering of predicted point cloud.\nExperiments are done on ShapeNet-ViPC and the proposed approach outperforms the baselines.\n", "questions": "1. Explain M, N, and F in Figure 1. The figure and its captions should be self-contained. Try to have consistent notations in Eq 3,4 and the Figure 1.\n2. What is \\beta in the Eq. 8?\n3. How are hyper parameters selected? Is there is a validation set?\n4. To make a more convincing argument, the Figure 3 can be modified to show an average over several shapes.\n\n", "limitations": "I think more qualitative results indicating failure cases should be provided.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths:**\n1. The problem statement is clear, contributions are clearly highlighted and the method section is well written.\n2. Fusion of local information between two modalities with the help of cross attention is a good idea considering that the point cloud is a partial representation of the original shape and direct one-to-one mapping between two modalities doesn't exist.\n3. Interestingly the differentiable rendering of the silhouette gives rather a large improvement even though the silhouette is a rather weak signal.\n\n**Weakness**\n1. The main limitation of the work is its benchmarking, which is done on ShapeNet shapes. Though mentioned by the authors are the limitation, it still makes the paper rather weak because we don't know how well it works in complicated scenes (ScanNet).\n2. The loss function is not fully ablated for example density aware Chamfer Distance.\n3. Error bars are not shown in the experiments. I understand that the benchmark shows the best performing result, still, error bars can be shown in a separate table or in the Supplementary material.\n4. Why do experiments in Table 1 only show 8 categories whereas the original dataset consists of 13 categories?\n5. The mixup loss only provides a tiny improvement, is it worth including it in the training? How much computational overhead this does loss contribute to the training?\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "YyfHHX8C2sy", "reviewer": "Reviewer_WuT7", "summary": "The authors propose a new method for point cloud completion from the depth and RGB images. With the proposed cross attention and self-attention, the RGB and depth information can be better leveraged. Experimental results indicate that the proposed method outperforms the state-of-the-art methods. The self-supervised approach also outperforms a number of supervised methods.", "questions": "Please refer to the weaknesses.", "limitations": "In the Introduction, the authors claim that the application of the proposed method is to complete the missing shape of an object due to the wide use of RGBD cameras. However, the manuscript does not provide any experiments on real-world scenarios.", "rating": 5, "confidence": 5, "soundness": 2, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "**Strengths**\n\n- The paper is clearly written.\n- The experimental results are solid in both supervised learning and self-supervised learning.\n\n**Weaknesses**\n\n- The running time and model size comparison should be included.\n- Most of the competitive methods only take the point cloud as input. Adding the RGB information will definitely improve the completion results. The proposed method outperforms ViPC significantly, but ViPC is not well compared to the other methods. PSGN (Fan et al., CVPR'17) should also be regarded as a simple baseline.\n- Maybe it is harsh, but the manuscript does not bring new technical contributions.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "NkP8w3Jt_Ij", "reviewer": "Reviewer_dnf9", "summary": "This paper tackles the problem of point cloud completion through using an incomplete point cloud and an auxiliary view-image as input. Moreover, this paper proposes using view-image as supervised signal and conducting self-supervised point cloud completion.", "questions": "See the weakness.", "limitations": "The authors have not addressed the limitations and potential negative societal impact of this work.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths: \n(1) This paper achieves state-of-the-art results on point cloud completion. (2) Furthermore, it is the first proposing self-supervised setting for point cloud completion task, and I think it is the core contribution of this paper. Also, by only using the self-supervised setting, this paper achieves a comparable result with previous fully-supervised manners. (3) The writing for this paper is relatively straightforward and well-organized.\n\nWeakness:\n (1) This paper is heavily based on the previous work ViPC. The core differences in the fully-supervised setting are only cross-modal fusion (attention), which is widely used for other multi-modalities tasks. \n(2) The Unimodal result in Table 3 is already higher than that of previous art (multi-modal), however, there is no clear illustration of why baseline performs pretty well. In addition, the performance improvement through introducing an auxiliary view-image seems marginal. \n(3) I think the self-supervised method in this paper seems overclaimed. Since auxiliary view-image is introduced in the supervision which contains partial information for the point cloud ground truth, it seems more like a weakly-supervised setting.\n (4) There is no ablation for the DCD loss.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "InZkK0-wInM", "reviewer": "Reviewer_jHpu", "summary": "This method proposes a corss-modal learning for image-guided point cloud shape completion. In this work, they employ a mixed strategy to train the network in a supervised and self-supervised manner. ", "questions": "I have several conerns for this work:\n1. In the introduction, their novel method is proposed to address one limitation in ViPC. However, the strategy used in ViPC is explicit, while the proposed method is implicit. Actually, once the fusion strategy happens in the feature domain, it is difficult to say what happens during the fusion process. If the authors want to say that their proposed method significantly outperforms ViPC, they should better clarify the limitations of ViPC.\n2. In the checklist, the authors say that they have disccused the limitations in their manuscript, while it is actually just a hand-wavy sentence in the conclusion. I do not think this is a serious discussion for the limitation of this work. \n3. What is the main advantage of the proposed self-supervised loss. It seems that it is a very important component in this framework. However, I cannot find the main contribution of this module, considering the existence of the reconstruction loss. Besides, in the ablation study, the authors also say that this module just allows the training process to have a faster and smoother convergence. This description makes the module less important. Is is possible to provide more results like Figure 5? \n4. In the third column of Figure 4, the results obtained by XMFNet cannot outperforms PCN and VRCnet. ", "limitations": "I have shown all my concerns in above sections.  I have read the rebuttal and keep my original rating. Thanks.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The image-guided framework seems to be an interesting setting for this problem.  On the other hand, the use of self-supervised stratgey also increases the generalization and robustness of this framework. ", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}]

Author rebuttal:

Decision: Accept