Crossmodal Few-shot 3D Point Cloud Semantic Segmentation via View Synthesis

Authors: l  ce
adaptive  weights
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 05:51:01

Abstract: Cross-modal 2D–3D point cloud semantic segmentation on few-shot-based learning provides a practical approach for borrowing matured 2D domain knowledge into the 3D segmentation model, which reduces the reliance on laborious 3D annotation work and improves generalization to new categories. However, previous methods use single-view point cloud generation algorithms to bridge the gap between 2D images and 3D point clouds, leaving the incomplete geometry of an object or scene due to occlusions. To address this issue, we propose a novel view synthesis cross-modal few-shot point cloud semantic segmentation network. It introduces the color and depth inpainting to generate multi-view images and masks, which compensate for the absent depth information of generated point clouds. Additionally, we propose a cross-modal embedding network to bridge the domain features between synthesized and original, collected 3D data, and a weighted prototype network is employed to balance the impact of multi-view images and enhance the segmentation performance. Extensive experiments on two benchmarks show the superiority of our method by outperforming the existing cross-modal few-shot 3D segmentation methods.

Main paper: 
 introduction: 
 The broad spectrum of applications of 3D point clouds has recently attracted the interest of researchers, leading to an increased focus on this area of study, such as robotics [9, 13, 21] , autonomous driving [4] [5] [6] , and virtual and augmented reality [17, 19] . Among them, the 3D point cloud semantic segmentation is the most fundamental and vital task. However, designating a correct semantic label for each point is challenging due to point clouds' sparsity, irregularity, and variable density. Furthermore, point clouds often contain noise, missing data, and occlusions, further complicating the segmentation procedure.Existing deep-learning-based methods [10, 14, 22, 26, 27, 29, 33 ] rely on large-scale labeled 3D point cloud datasets to achieve good performance. However, acquiring labeled 3D point cloud datasets can be a complex and expensive process, often involving specialized hardware and software. Thus, cross-modal few-shot learning [34] is introduced to mitigate this limitation, where it back-projects singleview 2D images into 3D pseudo point clouds as support exemplars to help segment the corresponding categories in the query point clouds. However, our in-depth investigation of this framework has uncovered several performance hindrances. The fundamental drawback is that the generated 3D pseudo-point clouds are incomplete, due to the occlusions in the single-view image. This can lead to inaccuracies in the prototypes extracted by the Embedding Network, resulting in a notable shape discrepancy between the real query point clouds and pseudo-incomplete point clouds. To address this challenge, a potential solution is to leverage point cloud completion algorithms to infer the missing shapes of pseudo-point clouds [3, 28, 31] . Nevertheless, these point completion algorithms are currently designed for simple foreground point clouds without complex backgrounds or simple background point clouds that overlook the small object details, making them impractical for intricate scenes. Another bottleneck arises from the gap in the latent feature space between the support and query features. Although [34] utilizes the statistical domain generalization method, "MixStyle" [35] , to bridge the gap, this method was originally proposed to capture style (domain) information for images and it overlooked the modality difference, i.e., 2D to 3D, that existed between the support and query data.Hence, in this paper, we aim to solve these drawbacks and propose a novel cross-modal few-shot 3D point cloud segmentation method. Figure 1 shows the key differences between the proposed method and the previous method. Especially, to address the occlusions in the single-view image, we employ a novel view synthesis module to generate multi-view RGB and depth images from a singleview image and back-project them into 3D point clouds. In other words, we generate several point clouds from only one single-view RGB image to potentially maximize the coverage of the 3D object and improve the representation ability of support sets. Then, to bridge the domain gap between the generated support point clouds and the real query point clouds, we further employ a cross-attention layer for the cross-modal embedding module and a weighted prototype network that can adaptively calculate the weights when extracting prototypes of 3D embedding space from the support branch. These approaches aim to enhance the alignment between the generated and real point clouds, ultimately improving the accuracy and effectiveness of the model.Our methods are evaluated on two standard 3D datasets, S3DIS [2] , and ScanNet [7] . Similarly to [34] , we take the standard 3D point cloud dataset as a query set and create a small collection of 2D images with semantic labels as the support set that covers all semantic categories in the query set. Experimental results demonstrate the effectiveness of the proposed method, achieving superior performance compared to previous methods. Once more, the extensive ablation studies demonstrate that all designed components in our network ultimately provide performance enhancements to the overall system.Our contributions can be summarized as follows:• We propose a novel cross-modal few-shot 3D point cloud semantic segmentation method based on multi-view synthesis over 2D images. To our knowledge, we are the first to introduce view synthesis to compensate for the information loss in the few-shot 3D semantic segmentation task. • We employ a novel cross-attention module in the feature embedding network and a weighted prototype module to align the heterogeneous gap of input spaces. • We conduct exhaustive experiments to demonstrate the superior performance of the proposed method in the cross-modal 3D point cloud semantic segmentation task on unseen categories given a few or even one example(s). 
related work 2.1 3d point cloud semantic segmentation: 
 3D point cloud semantic segmentation is the task of assigning semantic labels to each point in a point cloud. One of the pioneering approaches is PointNet [22] , which introduced a neural network architecture that directly takes point clouds as input and can learn discriminative features from point clouds. However, PointNet lacks steps to capture the local geometry from point cloud and thus limits its performance. As a result, PointNet++ [23] is proposed to improve the performance of point cloud processing by introducing a hierarchical feature learning architecture and improving the network's ability to capture local features. Another commonly used approach for point cloud semantic segmentation is to project the 3D point cloud into a 2D image and then apply 2D semantic segmentation techniques. For example, Jaritz et al. [11, 12] utilizes a 2D-3D crossmodal adaptation method for 3D point cloud segmentation. They project a 3D point cloud to a 2D image and sample the 2D features at the corresponding pixel location. However, these approaches are implemented by lifting 2D features to 3D spaces, which requires two streams to extract 2D and 3D features respectively. Different from previous approaches, we focus on RGB-Depth fusion to produce a dense point cloud that utilizes a single feature extractor for obtaining 3D features.Few-shot learning has been extensively utilized in the semantic segmentation of 2D images. Vinyals et al. [26] presented a framework for one-shot learning on image segmentation which involved using a weighted nearest-neighbor approach to convert the support set and testing samples into a shared embedding space via a matching mechanism. Zhang et al. [32] developed a Pyramid Graph Network that treats the support branch as a graph, where each element in the support latent space functions as a node. This technique allows for the efficient propagation of label information from support to query.In addition to 2D images, few-shot semantic segmentation methods have recently expanded to 3D space. The Prototypical Network (PN) is a metric learning-based strategy widely used in few-shot learning. In the standard PN approach [25] , the prototype for each class is computed by taking the mean of the support samples belonging to that class in the latent feature space. This approach is simple and effective and reduces the risk of over-fitting for few-shot learning. With these advantages, many variants of PN are proposed [1, 8, 15, 16, 18, 36] for employment in diverse domains and modalities applications. For example, [18] proposed integrating Group Equivariant Convolutional Networks into PN to address the lack of canonical structure in dermatological images. Zhu [36] employed a weighting mechanism to reduce noisy samples in feature embeddings. However, these methods do not explicitly weigh the support samples based on their distinguishing characteristics. We propose a novel weighted prototypical network tailored for the few-shot segmentation task. Our approach involves setting up an encoder function to estimate weights for the masked embedding features obtained by mapping input images from different views. By doing so, we can identify which views are more informative for our segmentation task and weigh them accordingly. This allows us to give more importance to the support samples that are most relevant and distinguishing for the task at hand. 
method 3.1 definition: 
 Our network is designed to effectively align heterogeneous input spaces between 2D images and genuine query 3D data to benefit few-shot 3D point cloud semantic segmentation. Following [34] , we initially set up a concise 2D image dataset covering all categories included in the point cloud dataset. Each category is explicitly annotated on these images. Then, each such 2D image, along with its annotated mask (I 𝑠 , I 𝑀 ), serves as a support sample. For every training episode, we define a 𝐺-way 𝐾-shot learning task, comprising 𝐾 pairs of (I 𝑠 , I 𝑀 ) for a total of 𝐺 different categories. Each 3D point cloud P q serves as a query sample, with its associated ground truth L q providing supervision throughout the training process. Each point cloud P 𝑞 ∈ R 𝑁 × (3+𝑐 0 ) contains 𝑁 points associated with 3 + 𝑐 0 channel features, i.e. coordinate and extra features.The support (2D) and query (3D) data are partitioned into separate training and testing sets, ensuring that categories in the testing set are distinct and not encountered during training. For clarity, we describe our network's architecture and methods under a 1-way 1-shot setup for training and testing.Given single-view support images and their depth estimation, we first synthesize multi-view images and masks to compensate for the occluded information. Next, we back-project them to 3D point clouds via 2D-3D transformations as detailed in Section 3.3. Then, the generated and real point clouds in the support and query set are fed into a newly designed cross-modal embedding network to bridge the domain gap, see Section 3.4. To adjust the contribution of each view in segmentation, we propose a weighted prototype network to aptly integrate multi-view support features in Section 3.5. Segmentation outcomes are achieved by computing the Euclidean distance between prototypes and query feature vectors based on the object category, as detailed in Section 3.6. Figure 2(a) shows the architecture of our network under a 1-way 1-shot setting.To bridge the gap between 2D and 3D modals, previous works typically extract two modality feature representations from two distinct encoders [11, 12] and then integrate them via converting the 2D features into 3D pseudo features. However, employing different encoders will result in a substantial domain discrepancy between the two modalities, posing a challenge to their integration. Therefore, similar to [34] , our approach involves initially converting 2D images into 3D point clouds, allowing us to subsequently apply one shared encoder to mitigate domain discrepancies. But, in contrast to [34] , we synthesize multi-view images with depths to compensate for the potential information loss. Specifically, for each 2D image with mask in the support set (I 𝑠 , I 𝑀 ), we initially estimate its depth value. Subsequently, we employ a pre-trained 3D photo generation module [24] to generate multi-view RGB-D images and their masks (I 𝑣 𝑠 , I 𝑣 𝑀 ), where 𝑣 ∈ 𝑉 and 𝑉 represents the number of generated views. This module leverages Layered Depth Image (LDI) as a foundational representation and incorporates a learning-based inpainting model to generate new color and depth in occluded regions. Then given the RGB and Depth information at each pixel, we can convert the whole image to the corresponding 3D point could (P 𝑣 𝑠 , M 𝑣 ) with following back-projects:EQUATIONwhere (𝑢, 𝑣) is the pixel coordinate, (𝑐 𝑥 , 𝑐 𝑦 ) is the principal point coordinate, (𝑓 𝑥 , 𝑓 𝑦 ) is the focal lengths, and 𝑧 is the depth value.Then, a shared cross-modal embedding network is introduced to extract shape features from both pseudo point clouds and the real query point clouds. It is built on top of the point cloud embedding network in [33] , as illustrated in Figure 2 (b). The proposed cross-modal point cloud embedding network is composed of three modules: 1) EdgeConv (Econv) layers [20] to produce local geometric features and semantic features; 2) a self-learning module on the generated semantic features to further explore the semantic interaction among points ; 3) a metric learner to adapt embedding features to well perform on few-shot tasks. The final output features of the embedding network are obtained by concatenating the outputs of the first EdgeConv layer, the self-learning module, and the metric learner.Then, to further alleviate the domain gap between features extracted from the generated support point clouds and the real query point clouds, we introduce a cross-attention module. This module leverages both intra-modality relationships within each modality and inter-modality relationships between generated point cloud features and real point cloud features to complement and enhance each other for features of support and query matching. To be specific, for each view, the support feature map represented by f 𝑠 ∈ R 𝑁 ×𝑐 is the output of embedding function 𝑓 𝜃 (P 𝑠 ) for each pseudo point cloud, where 𝑁 and 𝑐 denote the number of input points, and the channel of features, respectively. Similarly, let f 𝑞 be the query features from the embedding function 𝑓 𝜃 (P 𝑞 ). We can obtain the aggregated support feature f 𝑠 * for each view via a cross-attention module:EQUATIONwhere 𝑄, 𝐾, and 𝑉 are linear embedding functions with trainable parameters from support or query features, √ 𝑑 𝑐 is a scaling factor.The prototype vectors are the fundamental components in the fewshot learning, as they are used to infer the semantic labels of the query point cloud. To establish a correct connection between support and query features, we propose a novel weighted learning approach to adaptively merge support features and create prototypes for few-shot segmentations.To be specific, the weighted prototype network, denoted as 𝑤 Φ ∈ R 𝑉 ×𝐻 ×𝑊 ×4 → R 𝑉 ×𝑁 ×𝑐 , first stacks all the synthetic views together and then concatenates the RGB channels of images with their binary masks for stacked synthetic views. Here, 𝑉 stands for the number of views, while 𝐻 and 𝑊 denote the image size, and 𝑐 represents the number of feature channels. This network, parameterized by Φ, aims to concurrently learn a metric space for all-view images, producing predicted weights for the adaptive support feature aggregations:EQUATIONwhere we scale the value of the weight into [0,1) by the 𝑆𝑖𝑔𝑚𝑜𝑖𝑑 function. Then, the weighted prototypical representation can be computed as:EQUATIONwhere F 𝑛,𝑣 𝑆 denotes the extracted point-wise feature of 𝑣-th view from the cross-modal embedding network, 𝑠 𝑖 represents the target class and 1[.] is an indicator function to indicate the foreground object if the mask matching the target category.The background prototype can be computed in a similar fashion:EQUATIONwhere O = {𝑠 1 , 𝑠 2 , 𝑠 3 , ..., 𝑠 𝑘 } is a set containing all target classes.With extracted query features and support prototypes, the label of a new point cloud sample X 𝑖 from the query set 𝑄 (𝑒.𝑔. 𝑘-th class in O) is obtained by calculating the Euclidean distance of the query feature to each class prototype vector plus the background and applying softmax on the distances as:EQUATIONwhere 𝑓 * 𝜃 represents our cross-modal embedding function, p stands for the prototypes of all classes 𝑃 = {𝑃 𝑓 𝑔 ∪ 𝑃 𝑏𝑔 } and 𝑑 (.) denotes the Euclidean distance function between the query sample and prototype vector. The proposed network are optimized using crossentropy loss between L 𝑖 and 𝑝 X 𝑖 , given as:EQUATIONwhere 𝐿 𝑖 is the ground truth of sample 𝑋 𝑖 .  
experiments 4.1 data preparation: 
 Datasets. We evaluate the proposed method on two datasets, S3DIS [2] , which contains 272 point clouds with 12 different indoor scene categories plus one clutter class, and ScanNet [7] , which consists of 1,513 indoor scene scans fell into 20 semantic categories plus one unannotated background class. To collect a support 2D image set that covers all categories in both 3D datasets is challenging. Therefore, we adhere to the dataset setup in [34] , where typically gather annotated images of 11 semantic classes, which are "𝑐𝑜𝑙𝑢𝑚𝑛", "𝑡𝑎𝑏𝑙𝑒", "𝑤𝑎𝑙𝑙", "𝑓 𝑙𝑜𝑜𝑟 ", "𝑐ℎ𝑎𝑖𝑟 ", "𝑑𝑜𝑜𝑟 ", "𝑐𝑒𝑖𝑙𝑖𝑛𝑔", "𝑠𝑜 𝑓 𝑎", "𝑤𝑖𝑛𝑑𝑜𝑤", "𝑏𝑜𝑎𝑟𝑑", and "𝑏𝑜𝑜𝑘𝑐𝑎𝑠𝑒" to cover the S3DIS dataset. Then, for the ScanNet dataset, we can utilize 10 categories of them, excluding "𝑐𝑜𝑙𝑢𝑚𝑛". Furthermore, each category comprises 3-shot images that capture different indoor scenes. To classify each pixel in all the collected images according to its corresponding category, we utilize binary masks. Setup. To align with the few-shot network design and balance the outcomes of our 2D dataset for semantic segmentation, we divide the 11 (or 10) semantic categories of the 2D image support set into two separate subsets. These subsets were non-overlapping and equally distributed. To be specific, we randomly choose 6 (or 5) semantic categories P0 and designate them as training sets and take the remaining 5 (or 6) categories P1 for testing. Take S3DIS for instance, given the large scale of indoor scene point clouds, we utilize the data pre-processing methodology explained in [22] to split the total 272 point clouds into 7,547 blocks. This is achieved by sliding a non-overlapping window of size 1m×1m on the 𝑥𝑦 plane. We randomly select 4,096 points for each block and ensure that at least 100 points belong to the target category for segmentation when the block is used as the query set. Similarly, for the generated support point clouds, we randomly pick 4,096 points, of which 2,048 points are evenly selected from the target category region, and the remaining 2,048 points are from the background.To enhance the precision of experiments, we carry out the 2-Fold Cross-Validation, which means the model trained on P1 is tested on P0, while the model trained on P0 is tested on P1. In detail, P0 consists of "𝑐𝑜𝑙𝑢𝑚𝑛", "𝑡𝑎𝑏𝑙𝑒", "𝑤𝑎𝑙𝑙", "𝑓 𝑙𝑜𝑜𝑟 ", "𝑐ℎ𝑎𝑖𝑟 " and "𝑑𝑜𝑜𝑟 "; P1 consists of "𝑐𝑒𝑖𝑙𝑖𝑛𝑔", "𝑠𝑜 𝑓 𝑎", "𝑤𝑖𝑛𝑑𝑜𝑤", "𝑏𝑜𝑎𝑟𝑑" and "𝑏𝑜𝑜𝑘𝑐𝑎𝑠𝑒" ("𝑐𝑜𝑙𝑢𝑚𝑛" is removed when used for ScanNet dataset).Training. We follow a similar training setup in [33] to pre-train the feature embedding network, self-learning, and metric learner module. Typically, we pre-train the cross-modal embedding backbone (without cross-attention module) on training set through appending three MLP layers at the end to serve as the segmentor. During pre-training, the batch size is set to 32 in a total of 100 epochs. We employ the Adam optimizer with a learning rate of 0.001. For training our few-shot network, the learning rate is initialized with 0.001 to optimize the entire model and decayed by half after 5,000 iterations. The focal length is set to 525 to produce the depth estimation.Evaluation Metrics. We utilize the mean Intersection over Union (mean-IoU) metric as our evaluation method, which is commonly employed in point cloud semantic segmentation. We perform the mean-IoU by calculating the average of each testing class.As mentioned above, we are the first to introduce view synthesis in cross-modal few-shot 3D semantic segmentation. To comprehensively evaluate the proposed method, we design and perform three baselines for comparisons.DepProto. In this baseline, we introduce DepProto in [34] , which uses only one labeled image for each shot setup. Then the generated support point cloud is fed to the embedding network (without the cross-attention module), and each class is given one prototype by calculating the mean feature of its support points. The segmentation prediction is produced by ProtoNet [33] .3ViewDepProto. This baseline builds upon the DepProto, where we introduce multi-view synthesis to produce multi-view RGB images and masks from a single labeled RGB image. In this baseline, we set it to 3 frames for each shot. Each generated point cloud is extracted by the embedding network. The prototype is solely determined by computing the average feature of the support points. The predictions for query points are based on their squared Euclidean distance from the prototypes.CFS. CFS is introduced in [34] , which utilizes the Co-embedding network to extract and integrate the features of the support and query branches, respectively.The quantitative results of different methods are shown in Table 1 and Table 2 . From the first two baselines, we can see DepProto, with only 𝐾 point clouds generated for 𝐾-shot indoor scenes for each class, is inferior to 3ViewDepProto, which employs view synthesis to create multi-view point clouds for each shot. This is reasonable since the feature embedding network is able to access more information about the object from different views, allowing it to learn more valid shape information for each shot. Compared with CFS, our method gains around 10% and over 15% on S3DIS and ScanNet in all settings, respectively. This performance underscores the significance of the cross-attention module and weighted prototypical network for effectively combining the support and query features. Overall, In all four settings (1/2-way 1/3 shot) across both datasets, our method exhibits consistent and significant superiority over the baselines and CFS.We also demonstrate the performance of our method by visual comparison in Figure 3 and Figure 4 under 1-way 1-shot for each class in both datasets. In Figure 3 , we perform 5 categories of segmentation results, which are "𝑠𝑜 𝑓 𝑎", "𝑡𝑎𝑏𝑙𝑒", "𝑐𝑒𝑖𝑙𝑖𝑛𝑔", "𝑐𝑜𝑙𝑢𝑚𝑛" and "𝑤𝑖𝑛𝑑𝑜𝑤". Despite some instances of misclassification, it is evident that our method produces segmentation results that are more similar to the ground truth. For example, it is difficult for CFS to distinguish the dissimilarity between "𝑐𝑜𝑙𝑢𝑚𝑛" and wall (the 4th row of Figure 3 ) due to their analogous architectural features, while our technique is able to correctly identify the fundamental structure of "𝑐𝑜𝑙𝑢𝑚𝑛" by acquiring several valid depth information. Even though the support information is limited in the 1-shot scenario, we enhance the segmentation against CFS, highlighting the significance of multiview generation and 2D inpainting in our cross-modal 3D semantic segmentation.In contrast to the S3DIS dataset, we exhibit another 5 classes, "𝑏𝑜𝑜𝑘𝑐𝑎𝑠𝑒", "𝑤𝑎𝑙𝑙", "𝑐ℎ𝑎𝑖𝑟 ", "𝑑𝑜𝑜𝑟 " and "𝑓 𝑙𝑜𝑜𝑟 " in the ScanNet dataset. Our proposed method can accurately segment most of the semantic classes within these categories, whereas CFS produces poor segmentation results that blend the other semantic class or backgrounds. For instance, it is hard to clearly segment the "𝑏𝑜𝑜𝑘𝑐𝑎𝑠𝑒" (the 1st row of Figure 4 ) from the "𝑤𝑎𝑙𝑙" in the results of CFS, while our method, with the aid of multi-view depth information, can successfully separate the most parts of this category. We attribute our accurate segmentation results to the collaborative efforts of multiview point cloud generation, cross-modal embedding network, and weighted prototypical network, which helps to enrich the information contained in the generated point clouds and effectively bridges the modality gap between the support and query.To gain further insights and evaluate our design choices, unless stated otherwise, we perform ablation studies using our primary network architecture on the S3DIS dataset under the 1-way 1-shot setup. The resulting segmentation outcomes are reported as the Mean-IoU (mIoU).We first verify the contribution of the number of views to 3D semantic segmentation. Specifically, we compare 1-view, 3-view, and 5-view frames generated from 1 shot 2D image, as presented in Table 3 . When the number of views increased, the segmentation results gradually improved by approximately 3.7% and 6.8%.This suggests that incorporating more views can indeed offer valuable additional information for segmentation. However, this comes at the expense of increased computational demands, as illustrated in the fifth column of Table 3 . As the number of views increases, the inference time exponentially rises by approximately 185% and 11 times compared to using only one view per shot. Furthermore, adding more views would likely lead to even greater time consumption. Therefore, taking into account both computational resources and performance results, we choose to utilize three views as the default option in this trade-off. 4.6.2 Cross-modal feature aggregation. Then, to prove the effectiveness of our proposed cross-attention module, we compare it with a feature transfer method "Mixstyle" introduced in [34] , which is a statistical method via computing the mean and standard deviation to normalize support and query feature representations. To ensure a fair comparison, we include the "Mixstyle" at the same 4 : Additionally, we compare the time for each training iteration regarding the cross-attention module and the "MixStyle" module, respectively. The results are listed in Table 5 : We have found that our approach yields better segmentation results than the "Mixstyle" mechanism accordingly by approximately 8.0%, 8.7%, and significantly more than 18.4%, 16.3% when no feature aggregation module is adopted for 1-view and 3-view. While it is worth noting that the "Mixstyle" mechanism has a shorter training time of around 6.1% due to its non-learnable nature, our cross-attention module is crucial for achieving effective cross-modal semantic segmentation. 4.6.3 Weighted prototype network. Finally, we study the effect of our proposed weighted prototype network under a 1-shot setup on the S3DIS dataset. In table 6, we report the quantitative results between our method and conventional prototypical method [25] , which are calculated by average pooling of the support feature in the same category. We can roughly infer the weighted prototypical network produces superior segmentation results compared to the average pooling approach. Since the average pooling method cannot capture the salient features from various view images, segmentation results tend to remain relatively consistent even as the number of views increases. To gain a deeper understanding, we provide the qualitative comparison of 2-view weights for "𝑠𝑜 𝑓 𝑎" and "𝑡𝑎𝑏𝑙𝑒" categories in Figure 5 . The computed weights effectively distinguish between the original image's contribution and the poorly inpainted image for segmentation. The original image provides more accurate information on depth and shape compared to the other view image with poor inpainting performance in features during the calculation of support prototypes. Therefore, we can conclude that using a weighted prototypical network is more appropriate for our multi-view cross-modal segmentation task than using the average pooling method.We conduct experiments to evaluate the time consumption, FLOPs and model parameters of various baselines and a competitor method, as presented in Table 7 . For 3ViewCFS, we utiliz the CFS network and employ the view synthesis module to generate three-view images for each shot. Specifically, each generated point cloud is fed into the co-embedding network integrated with "MixStyle", followed by average pooling to extract prototypes. Our findings are as follows: ❶ When comparing different baselines (first two rows), segmentation without any domain gap balancing modules perform much poorly. ❷ After incorporating the view synthesis module to enrich the generated point cloud, there was a notable performance improvement of 8.4%, with only a corresponding drop in inference time of around 10ms. ❸ Comparing 3ViewCFS with our method, both under the three-view setting, we observe that our method outperforms by approximately 2%. This indicates that combining a view synthesis module with a weighted prototype network, which adaptively contributes to the segmentation results, can further enhance performance compared to solely relying on naively averaging the multi-view features. Importantly, this improvement is achieved while ensuring computational costs remain within an acceptable range.4.6.5 2D Inpainting VS 3D Completion. The introduction mentions a solution to overcome the limitation of previous methods, which involves utilizing point cloud completion algorithms to deduce the absent shapes of pseudo-point clouds. This study compares our results with previous point cloud completion methods, SnowflakeNet [28] and PoinTr [30] , in the inference stage under a 1-way 1-shot 3-view setup. Since the existing point cloud completion algorithms are implemented only on simple foreground objects and not on complex backgrounds, our experiment aims to complete only the foreground category object. The findings of the study are presented in Table 8 .The results demonstrate that our approach of enhancing scene information using 2D inpainting to generate multi-view images is clearly effective. This is logical as we only utilize 3D completion to complete the foreground objects, which may not accurately match their backgrounds that are produced from RGB and depth inpainting. As a result, the cross-modal embedding network is unable to correctly extract the appropriate representations based on the provided support samples. 
limitation: 
 Several limitations persist in our method. Despite our efforts to address the absence of semantic and depth information by introducing view synthesis and inpainting , as well as incorporating a crossattention module instead of a statistical method as used in [34] to bridge the feature space, our method still cannot seamlessly adapt to all genuine 3D features. Additionally, the generation of multiview synthesis incurs higher computational costs, as it necessitates a larger weighted prototype network to balance the contribution to segmentation. Furthermore, our method is constrained by the existing issue of limited customized 2D training data highlighted in [34] . Consequently, we may not comprehensively cover all types of 3D structures from real-world scans. 
conclusion: 
 In this paper, we further explore the few-shot 3D point cloud semantic segmentation task, construct a few-shot network and utilize the view synthesis and 2D inpainting to enhance the single-view depth estimation on point cloud generation. To further enhance the segmentation performance, we propose a cross-modal embedding network and a weighted prototypical module that integrate the multi-view generation approach. Comprehensive experiments are conducted on two datasets to verify the exceptional contribution of each module to 3D segmentation.  
 
 Appendix:  


Decision: This paper presents multi-view synthesis for 2D-3D 3D few-shot point cloud segmentation. The view synthesis method is novel and technical sound. The experiments are good and achieve sota performance.

The reviewers raised concerns regarding lack of comparisons with recent methods, lack of ablation studies on the proposed method, incorrect equations, etc. The AC checked the paper and the rebuttal and was convinced that the authors well addressed these concerns. The authors are suggested to include the necessary content in rebuttal to the final main paper, and add more comparisons/discussions with recent top conference/journal papers on few-shot 3D segmentation, such as Prototype Adaption and Projection for Few- and Zero-shot 3D Point Cloud Semantic Segmentation, which is highly related to the paper but is missing. The AC recommend this paper for acceptance.

Reviews: review: paper_summary: paper_summary: This paper proposes a novel view synthesis cross-modal few-shot point cloud semantic segmentation network. It uses color and depth inpainting to generate multi-view images and masks, compensating for the absent depth information of generated point clouds. It shows SOTA results on two benchmarks compared to the existing cross-modal few-shot 3D segmentation methods
strength_weakness: strength_weakness: The paper is the first to introduce view synthesis in cross-modal few-shot 3D semantic segmentation.

The obtained result outperforms the existing methods
limitations: limitations: 1. The paper lacks comparisons from works from 2024, such as

PDF: A Probability-Driven Framework for Open World 3D Point Cloud Semantic Segmentation, CVPR24

Rethinking Few-shot 3D Point Cloud Semantic Segmentation, CVPR24


2. The View Synthesis & Inpainting depends on the quality of the depth estimation models. It would be interesting to see the impact of the depth quality on the network performance: at which point the network is robust (or not) to the depth noise? 

3. Using pseudo depth has been widely used in previous works. These works should be referred to.

Source-free depth for object pop-out. ICCV 

CDNet: Complementary depth network for RGB-D salient object detection, TIP

4. While the authors build a novel benchmark by reproducing results from existing works, the paper lacks ablation studies on its own, such as the key component analysis
paper_summary: paper_summary: This paper introduces a few-shot point cloud segmentation method based on 2D-3D cross-model learning. The authors propose to use an off-the-shelf view synthesis method based on depth inpainting to generate multi-view images and corresponding masks from the support images. Then, they use a cross-model embedding network to lift 2D features to 3D and aggregate prototypes with adaptive weights learned from 2D to get the final prediction.
strength_weakness: strength_weakness: 1. The method employs view synthesis in 2D to enhance the segmentation of 3D point clouds.
2. The authors employing a weighted prototype network to balance the impact of multi-view images may enhance segmentation performance, ensuring that no single view disproportionately influences the outcome.
3. The authors performed several ablations studies to prove the effectiveness of each module.
limitations: limitations: 1. The parentheses in the softmax function presented in equation 6 are incorrect.
2. The authors conducted experiments using an N-way-K-shot framework, such as 2-way and 3-shot setups, but details regarding the implementation are lacking, and no visualizations are provided. Specifically, it is unclear whether the K-shot prototypes are simply stacked along the V dimension in these scenarios.
3. The quality of the visualizations in the paper could be enhanced to more effectively illustrate and communicate the results.
paper_summary: paper_summary: The paper addresses the challenges of 3D point cloud semantic segmentation using a cross-modal few-shot learning approach. The proposed method leverages matured 2D domain knowledge to enhance 3D segmentation models, reducing the dependency on extensive 3D annotations. Traditional methods suffer from incomplete geometry due to occlusions in single-view point cloud generation. This paper introduces a novel view synthesis network that uses color and depth inpainting to generate multi-view images and masks, thus compensating for missing depth information. A cross-modal embedding network bridges the domain features between synthesized and collected 3D data, and a weighted prototype network balances the impact of multi-view images to improve segmentation performance. Experiments on two benchmarks demonstrate the superiority of the proposed method over existing techniques.
strength_weakness: strength_weakness: 1. The view synthesis method to generate multi-view images from a single-view image is a novel approach in the context of 3D point cloud segmentation, and the combination of color and depth inpainting to address the issue of occlusions is innovative and enhances the completeness of the generated point cloud.
2. This paper proposes a robust framework that includes a cross-modal embedding network and a weighted prototype network, which are theoretically sound and technically well-founded. The use of cross-attention in the feature embedding network to align heterogeneous gaps in input spaces is a strong theoretical contribution.
3. This method is extensively evaluated on two standard datasets, S3DIS and ScanNet, demonstrating superior performance compared to existing methods. Comprehensive experiments, including ablation studies, validate the effectiveness of each component of the proposed network.
limitations: limitations: 1. The multi-view synthesis approach increases computational costs, requiring more resources for processing and potentially limiting the method's applicability in real-time scenarios.
2. Despite improvements, the method may not comprehensively cover all types of 3D structures, particularly in complex real-world scenes. The reliance on limited 2D training data constrains the method's ability to generalize to all 3D features.
3. While the method performs well on foreground objects, its performance on scenes with complex backgrounds is less evaluated. This limitation could affect its effectiveness in more intricate environments.
4. The aesthetics of Figure 2 still needs to be improved, especially subfigure a, which looks a bit confusing. Hope the author can draw a clearer and more concise method architecture diagram.
paper_summary: paper_summary: This paper tackles few-shot 3D point cloud semantic segmentation task by introducing multi-view synthesis 2D images. A cross-attention module and weighted prototype module are devised to mitigate the gap of input spaces. The method achieves good experimental results on S3DIS and ScanNet datasets.
strength_weakness: strength_weakness: 1. This paper introduces multi-view synthesis into few-shot point cloud semantic segmentation task for the first time, which provides more comprehensive and reliable support samples.
2. The method outperforms previous methods by a large margin across all metrics.
limitations: limitations: 1.	As the main contribution of this paper, the introduction of multi-view synthesis module is too short. What inpainting model is utilized? How to choose the view point and angle to generate novel images? How are the masks of novel views generated? What is the concrete pipeline of this module?
2.	The name of “Cross-modal Embedding Network” is confusing. This network encodes the support and query point clouds, which belong to the same modality. 
3.	There is an error in Figure 2. The $f_{\theta}$ in the lower right corner and the caption of (b) become a black square.
4.	The ablation studies are not concise and clear enough. 
(a) In Table 4, the performance of different feature aggregation methods is compared under 1-view and 3-view settings, while in Table 6, the weighted prototype network is ablated under 1-, 3- and 5-view settings. It is suggested to keep it consistent across the experiment part.
(b) The Table 5 which compares the training time is useless.
(c) The explanation of Table 7 is not clear. It is suggested to specify between which lines the comparison is carried out.