Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper presents a novel 3D representation called "Voint Cloud." The authors argue against existing heuristic-based aggregation, which could misleadingly incorporate representation from an arbitrary viewpoint. Then they attempted to address the research question of how to aggregate the per-view representation (in a multi-view setting) for a given 3D point cloud. In the process, they proposed their 3D representation, Voint, which is a view-dependent representation associated with each 3D point in a multi-view representation learning setting. The paper derived and described how their Voint Cloud (the proposed multi-view point cloud) could be used to learn representation end-to-end fashion to solve various 3D scene understanding tasks. Experimentally the authors demonstrated the usefulness of the proposed representation on three different 3D understanding tasks: i) 3D point cloud classification (on ScanObjectNN dataset), ii) 3D shape retrieval (on ScanNet Core55), and iii) 3D part segmentation (on ScanNet). The proposed method attained state-of-the-art performance in 3D point cloud classification and 3D object retrieval.
strength_and_weaknesses: Strengths:
+ New 3D representation which seamlessly integrates view-dependent representation for each 3D point
+ Proposed new pooling and convolution operation on the Voint Cloud leading to a new deep neural network called VointNet
+ state-of-the-art performance on 3D point cloud classification and 3D shape retrieval tasks using the proposed Voint Cloud representation

Weaknesses:
+ Generalizability to outdoor 3D scene understanding task: All experiments were done on indoor 3D scene understanding tasks (ScanObjectNN, ScanNet). To make a more generalizable claim for Voint Cloud representation, at least one experiment should have been done on an outdoor 3D scene understanding task, e.g., the KITTI dataset. Is there any specific reason for not doing that? 


+ 3D part segmentation: The proposed method achieved better results against other baselines on a more challenging setup (rotated); however, CurveNet still retained the best performance 84.9 % mIoU on the "Unrotated setup." The paper did not discuss what could be the possible reason for this slight discrepancy.  The ablation study in Table 6 shows different backbones and different variations of VointNet. It is unclear which result from that table corresponds to the one reported in Table 4. I also acknowledge the fact that the paper mentions that unless otherwise stated, the default method for VointNet is shared MLP. 

+ Backbone in 3D part segmentation: The backbones used in the 3D part segmentation task are widely used methods (FCN, DeepLabv3). The newer backbone might be more effective and possibly could improve the results. Please refer to the model "HRNet-W48" which was used in the following work: "MSeg: A Composite Dataset for Multi-domain Semantic Segmentation - Lambert, John, et al. CVPR'20"

+ Number of views: The authors showed an ablation study for the effect of the number of views on the performance of the Voint Cloud. However, it was shown only for the 3D part segmentation task. How about the 3D point cloud classification and 3D object retrieval tasks? 

+ 3D point cloud classification and 3D shape retrieval experiment: The proposed Voint Cloud-based representation achieved state-of-the-art results. It would be interesting to show the comparison against methods with deformable convolution on Point Cloud, such as 
"KPConv: Flexible and Deformable Convolution for Point Clouds - Hugues Thomas et al. ICCV'19"


+ VointNet variations: The paper shows three different variations of VointNet, the simplest of which is a shared MLP-based architecture. Figure 6 (in the appendix) could be included in the main paper to describe the method clearly. 



clarity,_quality,_novelty_and_reproducibility: The paper is well-written for the most part. I would recommend incorporating one item or two from the appendix to the main paper.
It brings a new idea -- new 3D representation VointCloud and also has methodological novelty. The experiments seem to be reproducible.


summary_of_the_review: The paper has some weaknesses that have been discussed in the limitation section; however, it introduced a new idea (novel 3D representation) and a new methodology (VointNet and its corresponding operations).
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: This paper proposes a multi-view 3D point cloud representation called Voint cloud. The multi-view features are extracted from 2D networks and then aggregated into 3D point clouds using correspondence. Visibility of the points in each view is also attached. The proposed method reaches state-of-the-art-performance on several 3D understanding tasks, including 3D shape classification, retrieval, and robust part segmentation.
strength_and_weaknesses: Strength:
- The idea of extracting low-level features from multi-view 2D images first and then aggregating back into 3D is novel. This takes advantage of mature 2D networks and still maintain the 3D geometry.
- The paper is well-written with extensive experiments and ablation studies.

Weakness:
I would like to see latency analysis of each component of proposed method and also in comparison with other methods. Looks like unprojection requires a lot of irregular memory access.
clarity,_quality,_novelty_and_reproducibility: The paper is well-written, idea is novel.

Typo: table 3 SahpeNet->ShapeNet
summary_of_the_review: The idea is somewhat novel and results show the effectiveness of proposed method. Incremental change compared to previous method. It'd be nice to see the latency analysis.
review_title: null

final_score: rating: 6: marginally above the acceptance threshold
review_content: summary_of_the_paper: The paper proposed a novel point cloud backbone. The backbone renders point clouds into 2D multi-view renderings and extracts 2D features with 2D backbones. These features are then unprojected back onto 3D point clouds where they are fused into final point features. 
The backbone is very compact and efficient.
The authors have performed extensive experiments including 3D point cloud classification, shape retrieval, and 3D part segmentation, and shown good performance on all benchmarks.
strength_and_weaknesses: Strength
- Though counter-intuitive, the idea of extracting features in 2D and fusing in 3D is interesting, clean, and neat. The method achieves very impressive performance on all tasks while keeping a compact model size and fast inference efficiency.
- Experiments have shown that the proposed backbone is less prone to partial observation and change of orientation.

Weaknesses
- The authors have not studied how noise will affect the backbone performance. It could be much harder to extract useful information in 2D than in 3D for noisy point clouds. This could potentially be a case where 3D methods take advantage.
- The authors seem to deliberately use different view numbers (8 vs. 12) and sampling methods (random vs. spherical) during training and testing. This is not explained and studied in the paper.
- The authors should include more details about the renderer e.g. speed, color?, point size, etc.


- If it is possible to render point clouds from different distances (say 0.5, 1.0, and 2.0) and use them as multi-scale features? Many 2D and 3D backbones employ similar ideas explicitly. Any discussion will be appreciated.

Minor
- Table 3: SahpeNet -> ShapeNet
- Figure 14: Distnace -> Distance
- Figure 12-15 appear to be very large and not nicely rendered (14,15)
clarity,_quality,_novelty_and_reproducibility: The paper is overall clear and well-written, with some typos to fix. The idea is interesting and novel. The authors have provided their code.
summary_of_the_review: I'm leaning toward acceptance due to the impressive performance and efficiency of the method. Adding more discussions and completing the details suggested above could further strengthen my recommendation.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: The authors propose the use of image based features coupled with 3D points to train a network to perform various 3D perception tasks, such as classification, retrieval or segmentation, on point clouds.  The authors demonstrate state of the art performance on synthetic several datasets.
strength_and_weaknesses: The paper is well written. The approach is well evaluated and the results are strong. The proposed idea is a well-executed derivation of existing art.

Weaknesses:

1) The novelty of the paper is limited. Essentially, the authors create am image based viewing track of a 3D point (common in SFM and SLAM) and modify pointNet to work in a higher dimensionality space.

2) There is too much effort from the authors to establish the term Voint (It is not clear if there is a fundamental difference between max pooling and convolution with VointMax and VointConv other than applying the operation on higher dimension. )

3) The method is assuming that the render is able to project 3D points to 2D with perfect occlusion detection. Have the authors considered how VointCloud can be extended to more complex point-clouds where mesh reconstruction from a renderer may not guarantee high occlusion detection accuracy?

4) How efficient is VointCloud and VointNet in terms of computational and space complexity? 
Minor comments 
1) Un-projection: The correct term is back-projection. 
2) Eq2 M,N are undifined symbols 
3) Table 3 ShapeNet 
clarity,_quality,_novelty_and_reproducibility: The paper is well written and all ideas are well explained 
The novelty of the paper is limited but the method is well executed 
The authors provide their code for reproducibility 
summary_of_the_review: Although I have some concerns about the overall terminology and limited novelty of the paper, I think that the method is well executed and the results support the direction. I recommend the paper for publication.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: This work proposes to use multi-view representations for 3D point clouds and recommends first computing 2D features on renderings of a given point cloud, and then (via a learned module), aggregating these to obtain per-point representations. Empirical results across datasets and task show the generality and efficacy of the approach, and in particular demonstrates the benefits over 3D point cloud processing as well as prior multi-view methods with adhoc aggregation. The reviewers unanimously recommend acceptance, and the AC agrees.

justification_for_why_not_higher_score: While the work presents strong empirical results, it scope is limited to simpler synthetic object-centric settings -- a wider set of applications would have made this work stronger. Moreover, unlike recent trends in 3D processing where approaches aim to guarantee equi-variances to certain classes of transformations, this work may not be easy to adapt for such purposes.
justification_for_why_not_lower_score: The paper presents an approach that is slightly unintuitive (and arguably not too elegant) but works surprisingly well, and is demonstrated across tasks. The community would benefit from seeing this approach for processing point clouds.
Decision: Accept: poster