MVP-Net: Multi-View Depth Image Guided Cross-Modal Distillation Network for Point Cloud Upsampling

Authors: jiade  chen
jin  wang
nam  ling
yunhui  shi
baocai 2024 yin
  mvp-
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 15:20:53

Abstract: Point cloud upsampling concerns producing a dense and uniform point set from a sparse and irregular one. Current upsampling methods primarily encounter two challenges: (i) insufficient uni-modal representations of sparse point clouds, and (ii) inaccurate estimation of geometric details in dense point clouds, resulting in suboptimal upsampling results.
To tackle these challenges, we propose MVP-Net, a multi-view depth image guided cross-modal detail estimation distillation network for point cloud upsampling, in which the multi-view depth images of point clouds are fully explored to guide upsampling.
Firstly, we propose a cross-modal feature extraction module, consisting of two branches designed to extract point features and depth image features separately. This setup aims to produce sufficient cross-modal representations of sparse point clouds.
Subsequently, we design a Multi-View Depth Image to Point Feature Fusion (MVP) block to fuse the cross-modal features in a fine-grained and hierarchical manner. The MVP block is incorporated into the feature extraction module. 
Finally, we introduce a paradigm for multi-view depth image-guided detail estimation and distillation. 
The teacher network fully utilizes paired multi-view depth images of sparse point clouds and their dense counterparts to formulate multi-hierarchical representations of geometric details, thereby achieving high-fidelity reconstruction.
Meanwhile, the student network takes only sparse point clouds and their multi-view depth images as input, and it learns to predict the multi-hierarchical detail representations distilled from the teacher network.
Extensive qualitative and quantitative results on both synthetic and real-world datasets demonstrate that our method outperforms state-of-the-art point cloud upsampling methods.

Main paper: 
 introduction: 
 As a 3D data format, point clouds are widely used in many fields, including autonomous driving [26] and 3D reconstruction [25] , thanks to their effective representation capability and the advancements in 3D sensing technology. However, the inherent limitations of 3D sensing devices often result in point clouds being sparse, non-uniform, noisy, and containing outliers. These drawbacks pose a hindrance to the practical application of point clouds. Therefore, converting sparse point clouds into high-fidelity versions that accurately represent the underlying 3D shapes is crucial. To this end, various point cloud upsampling techniques such as [21-23, 28, 37] have been developed.The predominant methods for point cloud upsampling are those based on deep learning [21, 23, 28, 31, 40] . As illustrated in Figure 1 (a), most of these methods consist of two fundamental modules: feature extraction and upsampling tail. Initially, the feature extraction module is employed to extract point features. Subsequently, the upsampling tail is utilized to expand point features and regress the 3D coordinates of upsampled points, thereby achieving upsampling. However, these methods encounter two primary issues. The first issue is that these methods rely solely on the 3D coordinates of sparse point clouds to predict their dense counterparts. They do not fully utilize other available modality information, resulting in insufficient uni-modal representations of sparse point clouds. The second issue is the inaccurate estimation of geometric details in dense point clouds, resulting in a lack of fidelity in the fine geometric details in the upsampled results (refer to Figure 5 ). Existing methods utilize KNN [3, 27] or attention [34] mechanisms to capture the geometric details of point clouds. However, the former is limited by its receptive field and has restricted modeling capabilities. Meanwhile, the latter faces challenges in capturing geometric details due to the absence of inductive bias regarding them. To tackle these two issues, we introduce a Multi-View Depth Image Guided Cross-Modal Distillation Network for Point Cloud Upsampling (MVP-Net) as shown in Figure 1(b) . With MVP-Net, we explore the optimal utilization of multi-view depth images to effectively guide point cloud upsampling.Firstly, we argue that alongside the 3D coordinates, feature information from other modalities can also provide complementary information to assist the precise reconstruction of 3D point clouds. The 2D depth image of a 3D point cloud not only contains the depth value of each individual point but also provides information, reflecting the shape, contour, and details of the underlying 3D object. However, capturing such information using only 3D coordinates is usually challenging due to the sparsity, irregularity, non-uniformity, noise, and outliers in the point cloud. Therefore, we propose a crossmodal feature extraction module to comprehensively utilize the cross-modal feature information from both 3D point clouds and 2D depth images. It contains dual branches to extract both point features and depth image features. In the point cloud branch, each point cloud is treated as a graph and graph convolutions are utilized to capture local details. In the depth image branch, the multi-view depth images of the point cloud are taken as input, and 2D convolution is employed. We then introduce a Multi-View Depth Image to Point Feature Fusion (MVP) block, which is integrated into the feature extraction module and effectively fuses these cross-modal features in a fine-grained and hierarchical manner.Secondly, though the 2D depth image provides complementary information, this information is somewhat limited due to the sparsity of the sparse point cloud. In contrast, the dense point cloud includes rich detail information, as well as its depth image. Therefore, we introduce a detail estimation and distillation structure to make full use of detail information from dense point clouds. Specifically, we establish both a teacher network and a student network. The teacher network uses the multi-view depth images of dense point clouds as additional inputs. Therefore, a well-trained teacher network can effectively and accurately reconstruct fine geometric details and form multi-hierarchical detail representations, thereby demonstrating excellent performance (refer to Tables 1 and 2 ). Next, the student network is trained using only the sparse point cloud and its multi-view depth images. It learns to predict the multihierarchical detail representations from the teacher network. This enables the student network to produce finer geometric details in the upsampled results without relying on extra information. Performance evaluations across various datasets and settings demonstrate that MVP-Net achieves state-of-the-art performance.In summary, our contributions are as follows:• We first propose to use the multi-view depth images of a point cloud as guidance to point cloud upsampling, then we present a novel network, MVP-Net, which makes full use of cross-modal representations and thus can accurately estimate geometric details. • A cross-modal feature extraction module and a Multi-View Depth Image to Point Feature Fusion (MVP) block are designed. The former consists of dual branches to extract both point features and depth image features, while the latter is integrated into the former to fuse these cross-modal features in a fine-grained and hierarchical manner, thereby generating sufficient cross-modal representations. • A distilling structure is introduced to produce finer geometric details in upsampled point clouds, guided by the multi-view depth image. The rich detail information from the dense point cloud is derived by the teacher network and transferred to the student network, facilitating the generation of geometric details without relying on extra information. • Extensive experimental results show that our method outperforms state-of-the-art point cloud upsampling methods on both synthetic and real-world datasets. 
related works 2.1 point cloud upsampling: 
 With the significant success of deep learning technology in point cloud processing, deep learning-based methods have become mainstream in the field of point cloud upsampling [6, 13, 21, 23, 24, 28-30, 35, 40] . PU-Net [40] is a pioneering deep learning-based work, which expands a point set via a multi-branch convolution in the feature space. Then, MPU [37] proposes to progressively upsample point patches in multiple steps. PU-GAN [21] introduces to employ a generative adversarial network for upsampling. PUFA-GAN [23] adopts a frequency-aware discriminator to tackle the issue of noise in the generated shape. PU-GCN [28] uses graph convolutions and shuffle operations to expand features. In addition, PUGeo-Net [29] initially generates points on 2D the tangent plane and then evolves them into 3D space. MAFU [30] incorporates a similar concept to PUGeo-Net to achieve flexible upsampling. Furthermore, Dis-PU [22] leverages a dense generator and a spatial refiner to achieve upsampling by disentangled refinement. PUCRN [6] generates points in multiple steps and then adjusts the spatial positions of each point. PU-Flow [24] employs the normalizing flow model to achieve upsampling by interpolating in latent space. NeuralPoints [8] employs the neural implicit functions to represent the local neural fields to achieve arbitrary upsampling. Recently, SSPU-Net [35] introduces a structure-sensitive upsampling network to handle and reconstruct complex geometric structures. Grad-PU [13] utilizes the idea of gradient descent with learned distance functions to achieve arbitrary-scale upsampling. However, these above methods only utilize 3D coordinates to learn the mapping from sparse point clouds to their dense counterparts. To overcome this limitation, we propose MVP-Net, which possesses the capability to generate sufficient cross-modal representations of sparse point clouds and produce finer geometric details in upsampled point clouds.Point cloud completion and point cloud upsampling are both tasks related to point cloud reconstruction. The former involves predicting a complete point cloud from an incomplete one, and some crossmodal methods have been developed for this purpose. For example, ViPC [43] proposes to leverage a pre-trained single-view reconstruction model to infer complete shapes. Subsequently, XMFNet [1] leverages direct cross-modal information fusion at a feature level, avoiding explicit reconstruction. CSDN [46] proposes to transfer the intrinsic shape characteristics from single images to guide the geometry generation of the missing regions. Recently, SVD-Former [45] incorporates multi-view depth images of incomplete point clouds to predict global complete shape by a self-view fusion network. However, these cross-modal completion methods fuse cross-modal features in a coarse-grained manner and focus on leveraging information from other modalities to predict complete shapes, rather than generating geometric details. Therefore these cross-modal methods cannot be directly used for point cloud upsampling. Currently, there is a shortage of cross-modal methods tailored for point cloud upsampling. To this end, we propose a dedicated cross-modal feature extraction module for upsampling, along with a fine-grained cross-modal feature fusion block that integrates pixel-level depth image features into corresponding point features.A common application of knowledge distillation for point cloud learning, similar to its original concept introduced by [14] , is for model compression [11, 15, 20, 42] . For the point cloud reconstruction task, RaPD [7] proposes a reconstruction-aware prior distillation method for point cloud completion, where the student network learns the distribution of the latent space generated by the teacher network. Recently, CPU [44] , which introduces vector-quantization into upsampling, incorporates knowledge distillation to facilitate effective codebook learning and enhance the utilization of code entries in the student network. We have a different purpose, which is to utilize the teacher network to capture geometric details in dense point clouds and then transfer this knowledge to the student network, guided by multi-view depth images. 
method 3.1 overview: 
 Point cloud upsampling aims to produce a dense point cloud Q ∈ R 𝑟 𝑁 ×3 of 𝑟 𝑁 points from a sparse one P ∈ R 𝑁 ×3 of 𝑁 points, where 𝑟 denotes the upsampling ratio. Given a sparse point cloud P, we employ a basic renderer [10] to generate multi-view depth imagesI P ∈ R 𝑁 𝑣 ×𝐻 ×𝑊 ×1, where 𝑁 𝑣 is the number of viewpoints and 𝐻,𝑊 denote the resolution of each depth image. We first train a teacher network T using additional multi-view depth images I Q of ground truth Q to derive the geometric details. Then, we train a student network S to learn the knowledge from the teacher network. In the following sections, we first describe the network structure of both the teacher network and the student network, which include a cross-modal feature extraction module (see Sec. The cross-modal feature extraction module takes a sparse point cloud P and its multi-view depth images 𝐼 P as input. To fully extract the features of these two modalities, we introduce a dual-branch network. In the depth image branch, we input 𝐼 P and utilize the ResNet [12] to extract multi-hierarchical feature maps {𝐹 𝑙 𝐷 } 𝐿 𝑙=1 by hierarchically downsampling, where 𝐿 is the number of layers. In the point cloud branch, we treat each point cloud as a graph and employ the dynamic graph and edge convolution [36] to capture local details. Different from the depth image branch, we utilize intra-level (Figure 3 ) and inter-level (Figure 2 ) dense connections [16, 37] to organize point features at different hierarchies for extracting point cloud feature maps {𝐹 𝑙 P } 𝐿 𝑙=1 , instead of using residual connections [12] . Additionally, we refrain from downsampling the point cloud, as it is already sparse. To effectively fuse the aforementioned cross-modal features, we introduce the MVP block, which fuses cross-modal features at different hierarchical levels, as shown in Figure 2 . There are two reasons why we use two different strategies to process depth images and point clouds, respectively: (i) The depth images with a resolution of 128 × 128 of a sparse point cloud with 2048 points are extremely sparse with at most only 12.5% of the pixels having an actual meaning. As an auxiliary branch, the depth image branch should be made as lightweight as possible. Therefore, we employ ResNet and hierarchically downsampling in this branch to expand the receptive field. (ii) Graph convolution [19, 28, 36] has been demonstrated to have brilliant performance in processing irregular point clouds. Furthermore, dense connections can more effectively utilize shallow features, particularly when cross-modal features are fused.By using simple 2D convolutions on the depth image, capturing geometric detail information becomes straightforward. However, capturing such detail information directly in 3D space is challenging due to the sparsity, irregularity and non-uniformity present in the point cloud. Furthermore, the multi-hierarchical representations of depth images comprehensively provide multi-scale geometric details, ranging from coarse to fine.To effectively fuse the cross-modal features mentioned above, we propose the MVP block, as depicted in Figure 4 . Given the feature maps 𝐹 𝐷 = {𝑓 𝑖 } 𝑁 𝑣 𝑖=1 of multi-view depth images and the point features 𝐹 P at the same hierarchical level, our approach aims to align these cross-modal features within the feature space. Subsequently, we gather relevant pixel features in this aligned feature space. Finally, these features are projected back into the point feature space. The point features are first projected to queries by multiple parameter matrices {𝑊 𝑖 𝑄 } 𝑁 𝑣 𝑖=1 . Then, the features of multi-view depth images are transformed into keys and values by parameter matrices 𝑊 𝐾 and 𝑊 𝑉 , respectively. This process is formulated as:EQUATION)where 𝑖 is the index of viewpoints. Then, the cross-attention [34] is calculated between 𝑄 𝑖 and 𝐾 𝑖 to generate aggregated depth image features 𝑂 𝑖 :EQUATIONSince the calculation of attention mentioned above is conducted in the aligned feature space, an inverse projection is necessary for further feature fusion. This process can be expressed as:EQUATIONwhere Concat(•) denotes the concatenation operation, and 𝑊 and 𝑏 are the parameters of a Linear Layer. Note that until now, we have only obtained the aggregated features O of depth images andhave not yet fused it into 𝐹 P . A straightforward approach is to add them together, followed by a Feed Forward Network [34] . However, we find that this approach does not fuse them harmoniously. In practice, we modify the attentional feature fusion (AFF) [5] to fuse them:EQUATION)where A is a sub-network that starts with an additional Batch-Norm Layer [17] to eliminate differences in data distribution between different modalities to predict 𝛽, and ⊙ denotes element-wise multiplication.The upsampling tail takes the outcome of the cross-modal feature extraction module as input and expands it to achieve upsampling. We utilize the feature expansion module and the multi-scale spatial refinement module proposed in SSPU-Net [35] as our upsampling tail to generate upsampled point clouds in a coarse-to-fine manner, as shown in Figure 2 . We don't use its frequency-aware attention mechanism to show that our approach's advantage comes from our contribution.We introduce a distilling structure to make full use of detail information from dense point clouds, achieving the generation of finer geometric details in upsampled point clouds. As illustrated in Figure 2 , this distilling structure contains two training stages.3.5.1 Stage I: Teacher Network. We concatenate the multi-view depth images of a sparse point cloud and the ground truth from the same viewpoint together along the channel dimension and input them to the depth image branch of the teacher network.Utilizing 𝐼 Q , the teacher network can accurately reconstruct the geometric details and form multi-hierarchical detail representations{ F𝑙 𝐷 = { f 𝑙 𝑖 } 𝑁 𝑣 𝑖=1 } 𝐿 𝑙=1of a dense point cloud. 3.5.2 Stage II: Student Network. The student network only takes a sparse point cloud P along with its multi-view depth images 𝐼 P as input. During its training, the parameters of the well-trained teacher network are frozen, and the teacher network only performs inference to produce { F𝑙 𝐷 } 𝐿 𝑙=1 . To promote the student network to better learn the knowledge from the teacher network, we investigate two types of knowledge distillation methods: feature-based distillation [32] and response-based distillation [14] .For the feature-based distillation, the student network aims to mimic the multi-hierarchical detail representations { F𝑙 𝐷 } 𝐿 𝑙=1 :EQUATIONwhere{ F𝑙 𝐷 = { f 𝑙 𝑖 } 𝑁 𝑣 𝑖=1 } 𝐿 𝑙=1is the multi-hierarchical detail representations predicted by the student network. We also employ the spatial attention map-based distillation [41] to predict the spatial attention map of the { F𝑖 𝐷 } 𝐿 𝑖=1 , which enables the student network to predict more meaningful pixels. The spatial attention map-based distillation is as follows:EQUATIONwhere G(•) is an operator [41] to calculate the one-dimensional spatial attention map. For the response-based distillation, the student network learns to predict the upsampled point clouds generated by the teacher network. This can be summarized as:EQUATIONwhere Q ′ S and Q S are the coarse upsampled point cloud and fine upsampled point cloud generated by the student network respectively, and Q ′ T and Q T are the results predicted by the teacher network. Hence, the distillation loss of the student network is as follows:EQUATIONThe distillation constraints guide the student network in absorbing the knowledge necessary to reconstruct geometric details in dense point clouds. This knowledge is distilled from the teacher network and covers both feature and response perspectives. This approach helps the student network to reconstruct finer geometric details.The training objectives of the teacher network are only employed to ensure the quality of upsampled point clouds. We utilize the Chamfer Distance (CD), a prevalent choice in recent studies [6, 35] , as one part of the reconstruction loss for the teacher network. We also employ the depth image matching loss [39] to measure the reconstruction error. Thus the training objectives of the teacher network can be summarized as follows:EQUATION)where 𝛼 is a hyper-parameter to balance the relative importance of each term, and Φ(•) is the differentiable point renderer [39] . We set 𝛼 = 0.25 for the PU-GAN dataset [21] and 𝛼 = 0.5 for the PUGeo-Net dataset [29] . The training objectives of the student network comprise the reconstruction term L 𝑟𝑒𝑐 for its upsampled results, and the knowledge distillation term L 𝑑𝑖𝑠𝑡𝑖𝑙𝑙 :EQUATIONwhere 𝜆 𝑑𝑖𝑠𝑡𝑖𝑙𝑙 is a hyper-parameter. We set 𝜆 𝑑𝑖𝑠𝑡𝑖𝑙𝑙 = 0.15 for both PU-GAN [21] and PUGeo-Net [29] dataset.4 Experimental Results 
experimental settings: 
 4.1.1 Datasets. We conduct quantitative and qualitative experiments on two synthetic datasets: the PU-GAN dataset [21] and the PUGeo-Net dataset [29] . The latter contains more complex geometry and high-frequency details, compared to the former. We adhere to the same experimental settings as SSPU-Net [35] , utilizing Poisson disk sampling [4] to generate patches for training and point clouds for testing. We also conduct qualitative experiments on the real-world ScanObjectNN [33] and KITTI [9] datasets.Following previous works [13, 21, 28, 35] , we adopt four evaluation metrics: (i) Density-aware Chamfer Distance (DCD) [38] , (ii) Chamfer Distance (CD), (iii) Hausdorff Distance (HD), and (iv) Point-to-Surface (P2F) Distance. P2F metric include the mean (P2FM) and standard deviation (P2FS). For all metrics, the smaller the metric, the better the performance. The units of CD, HD, and P2F metrics are 10 -3 .Comparison. We compare our method with a series of state-of-the-art methods, including PU-Net [40] , MPU [37] , PU-GAN [21] , Dis-PU [22] , PU-GCN [28] , PUGeo-Net [29] , MAFU [30] , PUFA-GAN [23] , PU-Flow [24] , PUCRN [6] , Grad-PU [13] and SSPU-Net [35] . We utilize three orthogonal viewpoints to generate multi-view depth images with a resolution of 128 × 128. The hyper-parameters 𝜆 𝑚𝑖𝑚𝑖𝑐 , 𝜆 𝐴𝑇 and 𝜆 𝑟𝑒𝑠 are set to 0.1, 1 and 500, respectively. During training, we set 𝑁 = 256 for each sparse patch, which is randomly downsampled from the dense patch. We train both the teacher network and the student network with a batch size of 32. The former is trained for 200 epochs, while the latter is trained for 240 epochs. The Adam algorithm [18] is adopted to optimize our network. The initial learning rate is set to 0.001 with a decay rate of 0.7. 1 presents the quantitative comparison results on the PU-GAN dataset. Our method outperforms previous approaches in terms of DCD and CD metrics, indicating that our method achieves minimal reconstruction errors at different upsampling ratios. Since DCD can measure the density difference between two point clouds, our upsampled results exhibit the closest density distribution to the ground truth, suggesting that our results are more uniform. For the HD metric, our method outperforms all other methods at 2× and 4× upsampling ratios and achieves the second-best performance at 8×. For the P2F metric, our method exhibits the most competitive performance with SSPU-Net [35] on P2FM and the best results on P2FS. This demonstrates that our results are stably distributed on the surface of the underlying object. As depicted in the upper section of Figure 5 , previous methods tend to generate more noise and outlier points when handling complex geometry. In contrast, our method can accurately reconstruct complex geometric details with minimal noise and outliers.Results on the PUGeo-Net dataset. Table 2 shows the 4× quantitative results on the PUGeo-Net dataset. Even when faced with more complex geometric details, our method consistently achieves the best performance in terms of DCD and CD metrics. Furthermore, for the P2F metric, our method also demonstrates comparable results to recent state-of-the-art methods, namely SSPU-Net [35] and Grad-PU [13] . However, for the HD metric, Grad-PU [13] achieves better performance, and we believe this phenomenon arises from the differences in the upsampling tail. Grad-PU [13] refines the positions of interpolated points by a point-to-point (p2p) iterative optimization process. This process enables it to achieve the smallest maximum p2p error, resulting in the lowest HD metric. However, it lacks overall consideration and thus cannot surpass us in DCD and CD metrics. This is evidenced in the lower section of Figure 5 , where our method can produce the most realistic results.We also conduct experiments on two real-world datasets, the ScanOb-jectNN dataset [33] and the KITTI dataset [9] , as shown in Figure 6 and Figure 7 , respectively. Since there is no ground truth available, we only perform qualitative comparisons. Figure 6 illustrates the upsampled results (bottom row) and surface reconstruction results (top row), where two consecutive 4× upsamplings are conducted using models trained on the PU-GAN 4× dataset. As shown in the first column, the input point cloud is extremely sparse and non-uniformly distributed. There are noticeable gaps in the upsampled results generated by some methods, such as PUCRN [6] , PU-Flow [24] and SSPU-Net [35] . Grad-PU [13] GT PUGAN [21] PUGCN [28] DisPU [22] PUCRN [6] PUFA [23] PUFlow [24] SSPU [35] Grad [13] Ours plausible results. The surface reconstruction results of PU-GAN [21] and Dis-PU [22] reveal holes. While our method can reconstruct dense and uniformly distributed upsampled results, as well as achieve smooth surface reconstruction results. Figure 7 shows our method can accurately preserve the original structure (orange box) and fill in the gaps caused by occlusion (red box). Since point clouds usually contain noise and outliers in real scenarios, we conduct experiments to assess the noise robustness of our proposed method. Specifically, we add Gaussian noise at different levels from 0 to 2.0% to the PU-GAN 4× testing dataset. Table 3 demonstrates that our method achieves the best performance at different noise levels. DCD is used as the metric instead of CD because the upsampling results of Grad-PU tend to exhibit an aggregation of points around the original point locations under higher noise levels. This density imbalance can even result in a lower CD, as explained in DCD [38] . The results of the model complexity analysis are shown in Table 4 . Since we only use the student network in practice, the number of parameters utilized in the inference phase of PUFA-GAN [23] is approximately 2.5 times that of our method. Additionally, our method is comparable to Dis-PU [22] . The inference time of our method is slightly longer than previous methods, such as MAFU [30] , but less than PUFA-GAN. This is because our method needs to render multi-view depth images of each point cloud and extract PUGCN [28] DisPU [22] PUCRN [6] PUFA [23] PUFlow [24] SSPU [35] Grad [13] OursFigure 6 : 16× upsampled results on the ScanObjectNN dataset [33] . The first row and the second row showcase the meshes reconstructed by [2] and the upsampled point clouds, respectively. Ours denotes the result of the student network. We initially carry out ablation studies focusing on the detail estimation and distillation, as illustrated in the upper section of Table 5 . Based on models A2, A3 and A4, we conclude that the depth image branch of the teacher network effectively captures the fine geometric details in dense point clouds. This inference is drawn from the observation that removing L 𝑚𝑖𝑚𝑖𝑐 or L 𝐴𝑇 results in suboptimal performance of the student network. From model A5, the response-based knowledge distillation further improves the upsampling performance. We then conduct ablation studies on the cross-modal feature extraction without the guidance of the teacher network, as illustrated in the lower section of Table 5 . Compared to model A1, model B1, which utilizes only 3D coordinates for upsampling, demonstrates that leveraging cross-modal information can significantly improve point cloud upsampling. Model B2, which fuses O and 𝐹 P by direct addition, achieves worse results than the uni-modal model B1, highlighting the importance of a suitable feature fusion method for cross-modal features. Models B3 and B4 confirm the effectiveness of employing multiple 𝑊 𝑄 in the MVP block and the depth image matching loss, respectively. 
conclusion: 
 We propose a novel multi-view depth image guided cross-modal distillation network for point cloud upsampling. Specifically, we first introduce the cross-modal feature extraction module integrated with the MVP blocks. The former consists of two branches to extract both depth image features and point features. The latter fuses these cross-modal features in a fine-grained and hierarchical manner. Additionally, we introduce the detail estimation and distillation structure to produce more realistic geometric details. Extensive experiments demonstrate that MVP-Net achieves state-of-the-art performance. 
 
 Appendix:  


Decision: This paper initially got mixed reviews. After the rebuttal phase, the reviewer gWFG also upgraded his/her rating. Although he/she still had some concerns about the motivation, which should be easily addressed in the revised manuscript. By taking reviews into consideration, ACs recommend the acceptance of the paper.

Reviews: review: paper_summary: paper_summary: This paper introduces a method called MVP-Net, which addresses the challenges of point cloud upsampling by using multi-view depth images to guide the upsampling process. It aims to produce a dense and uniform point set from a sparse and irregular one, overcoming issues such as insufficient representations of sparse point clouds and inaccurate estimation of geometric details in dense point clouds. The proposed method consists of a cross-modal feature extraction module, a Multi-View Depth Image to Point Feature Fusion (MVP) block, and a multi-view depth image-guided detail estimation and distillation paradigm. The method has shown superior performance compared to state-of-the-art point cloud upsampling methods in both synthetic and real-world datasets.
strength_weakness: strength_weakness: 1. The authors propose a novel network (MVP-Net) for upsampling point cloud data into a dense cloud. They introduce a cross-modal feature extraction module, a Multi-View Depth Image to Point Feature Fusion (MVP) block, and a distillation paradigm for point cloud upsampling.

2. The paper (MVP-Net) compares results with state-of-the-art methods to demonstrate the effectiveness of the proposed technique. Standard evaluation metrics such as density-aware chamfer distance and chamfer distance are utilized to make the comparison. Additionally, the complexity of MVP-Net is compared with other models. Also, they show the qualitative comparisons in the paper.

3. The paper clearly explains the modules of the proposed MVP-Net in achieving point cloud upsampling. It also details some limitations associated with MVP-Net.
limitations: limitations: 1. The paper does not study different values for the hyper-parameters ($\lambda_{mimic}$, $\lambda_{AT}$, $\lambda_{res}$) in the distillation loss but sets fixed values. However, these hyper-parameters influence MVP-Net's performance. 


2. Also, the authors do not study different values for $\alpha$ but set a value (0.25 for the PU-GAN dataset and 0.5 for the PUGeo-Net dataset) for the reconstruction term (Equation 9). Similarly, $\lambda_{distill}$ is not studied for different values in Equation 10. Experimental details on these above parameters are required to understand how the authors arrived at these hyper-parameters values.
paper_summary: paper_summary: This manuscript presents a point cloud upsampling method by combining multi-view depth images rendering and knowledge distillation technique. The innovation is limited. Firstly, the depth images are rendered from point clouds, so it is doubtful whether they can provide additional information. Secondly, the teacher-student paradigm is a widely-applied framework, which does not need any modification to extend to point cloud upsampling in this paper.
strength_weakness: strength_weakness: 1.This paper is well-written, and the main idea of this work is clearly presented. 

2.The introduction provides a thorough background and the methodology is presented in a clear and accessible manner.
limitations: limitations: 1.The motivation is unclear. The authors claimed that previous methods suffer from two shortcomings: (i) insufficient uni-modal representations of sparse point clouds, and (ii) inaccurate estimation of geometric details in dense point clouds, resulting in suboptimal upsampling results.
Correspondingly, (i) the depth images are rendered from sparse point clouds. How can they provide sufficient representations for upsampling? (ii) The key of point cloud upsampling is to estimate the geometric details of dense point clouds from sparse ones. How can we claim that previous works failed to extract the geometric details of dense point clouds?

2.The teacher-student paradigm is a widely-applied framework. Does this paper do anything to make it more suitable for a point cloud upsampling task?

3.More training programs need to be studied. Refer to the [1] for more details.
[1] Wang X, Cui W, Xiong R, et al. FCNet: Learning Noise-Free Features for Point Cloud Denoising[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2023.
paper_summary: paper_summary: This paper introduces MVP-Net, a novel approach for point cloud upsampling that utilizes multi-view depth images to guide the upsampling process. The network addresses challenges related to sparse point cloud representations and inaccurate geometric detail estimation in dense point clouds. Experimental results demonstrate that MVP-Net outperforms existing point cloud upsampling methods.
strength_weakness: strength_weakness: 1. The authors propose to use multi-view depth images as guidance to point cloud upsampling and design a cross-modal distillation network named MVP-Net for point cloud upsampling.
2. A distilling structure and cross-model feature fusion block are introduced to produce finer geometric details, the authors claim that this MVP-Net achieves the state-of-the-art performance.
limitations: limitations: 1. The authors have only utilized depth maps from three orthogonal views. Can incorporating depth maps from additional views enhance the model's accuracy? Are there any relevant experiments conducted to investigate this?
2. This paper mentions that the concept of incorporating depth maps is inspired by SVDFormer. However, the depth map is view-dependent and lacks multi-view consistency. Additionally, SVDFormer utilizes camera poses as positional embeddings for differentiation. Is it appropriate to directly apply depth map features to enhance point cloud features in this paper?
3. The experimental results presented in the paper, particularly on the PU-GAN dataset in Table 1 and the PUGeo-Net dataset in Table 2, do not show a significant improvement in accuracy.
4. In Table 4, the MVP-Net proposed in this paper exhibits long training and inference times, along with a large number of parameters. Have the authors considered strategies to enhance efficiency in light of these observations?
paper_summary: paper_summary: The paper proposes a novel cross-modal approach named MVP-Net for point cloud upsampling, utilizing multi-view depth images to guide the upsampling process. It addresses two main challenges encountered by current upsampling methods: insufficient uni-modal representations of sparse point clouds and inaccurate estimation of geometric details in dense point clouds.
strength_weakness: strength_weakness: The strengths of this paper are as follows:
1. This paper introduces a novel method for cross-modal point cloud upsampling.
2. To accommodate this cross-modal method, the paper proposes new modules for feature extraction and feature fusion.
3. The paper conducted numerous experiments to validate the entire method.
limitations: limitations: The limitations of this paper are as follows:
1. Some aspects of the paper lack novelty. For example, the cross-modal feature extraction module mentioned in the contributions section seems to essentially apply DGCNN[1] to point clouds and ResNet[2] to depth images, respectively. I would like the authors to elaborate further on the novelty.
2. Some descriptions in the paper are confusing, such as in the ablation experiments, where the authors do not explicitly explain the abbreviations in the table.
3. The authors do not provide a thorough explanation for the modifications of the feature fusion module.

Further comments:
1. Whether adding the dense point cloud‘s multi-view depth images to the teacher network is a fair approach compared to other baselines? Does it lead to overfitting?

Reference:
[1]Yue Wang, Yongbin Sun, Ziwei Liu, Sanjay E. Sarma, Michael M. Bronstein, and Justin M. Solomon. 2019. Dynamic Graph CNN for Learning on Point Clouds. ACM Trans. Graph. 38, 5 (2019), 146:1–146:12
[2]Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. 2016. Deep Residual Learning for Image Recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2016, Las Vegas, NV, USA, June 27-30, 2016. IEEE Computer Society, 770–778