Point Cloud Self-supervised Learning via 3D to Multi-view Masked Leaner

Authors: Zhimin Chen, Xiao Guo, Yingwei Li, Longlong Jing, Liang Yang, Bing Li
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords: Self-supervised learning, Multi-modality learning, 3D representation, Masked autoencoder

Abstract:
Recently, multi-modal masked autoencoders (MAE) has been introduced in 3D self-supervised learning, offering enhanced feature learning by leveraging both 2D and 3D data to capture richer cross-modal representations. However, these approaches have two limitations: (1) they inefficiently require both 2D and 3D modalities as inputs, even though the inherent multi-view properties of 3D point clouds already contain 2D modality.
(2) input 2D modality causes the reconstruction learning to unnecessarily rely on visible 2D information, hindering 3D geometric representation learning.
To address these challenges, we propose a 3D to Multi-View Learner (Multi-View ML) that only utilizes 3D modalities as inputs and effectively capture rich spatial information in 3D point clouds. 
Specifically, we first project 3D point clouds to multi-view 2D images at the feature level based on 3D-based pose.
Then, we introduce two components: (1) a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images from 3D and projected 2D features; 
(2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interactions in each decoder transformer block through attention heads at various scales. 
Additionally, a novel two-stage self-training strategy is proposed to align 2D and 3D representations.
Empirically, our method significantly outperforms state-of-the-art counterparts across various downstream tasks, including 3D classification, part segmentation, and object detection.
Such performance superiority showcases that Multi-View ML enriches the model's comprehension of geometric structures and inherent multi-modal properties of point clouds.

Decision: reject
Reviewer scores: 5, 3, 6, 8, 8

Reviews and discussion:
<thread>
creation date:
1729050342942

thread_type:
official_review

<thread_object>
creator: reviewer_et1w

<content>
summary:
This paper presents a 3D-to-multi-view learner (Multi-View ML) that uses only 3D modalities as input and efficiently captures the rich spatial information in 3D point clouds. Specifically, we first project the 3D point cloud to a feature-level multi-view 2D image based on 3D pose. Then, we introduce two components: (1) a 3D-to-multiview autoencoder that reconstructs point cloud and multiview images from 3D and projected 2D features; and (2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interaction in each decoder-converter block through different scales of attention heads. Furthermore, a novel two-stage self-training strategy is proposed to align 2D and 3D representations. The proposed method significantly outperforms state-of-the-art methods in a variety of downstream tasks, including 3D classification, part segmentation, and object detection.

strengths:
1. The methodology section of the paper is quite well written, easy to understand and clearly guided. The four consecutive subsections show the implementation of the proposed method in a somewhat innovative way. It is recommended to add more details, e.g. the questions that follow.
2. In particular, the two-stage training strategy mentioned in the paper, i.e., aligning 2D and 3D representations using a network of teachers and students, is feasible and, moreover, the technique is challenging. The ability to appropriately apply it to multimodal self-supervised learning is a significant contribution, and I do hope that the authors will soon open-source this project for community advancement.
3. The proposed method is experimentally quite adequate and clearly outperforms state-of-the-art methods in a variety of downstream tasks including 3D classification, partial segmentation and object detection.

weaknesses:
1. An obvious grammatical error, "leaner" in the title should be "learner". In addition, it is recommended to enlarge the fonts of the images in the paper to enhance the presentation quality, as they are even smaller than the font size of the main text.
2. The novelty of this work needs to be additionally and strongly illustrated. On the one hand, this paper only uses point clouds and as input, however, I2P-MAE (CVPR’2023) and TAP (ICCV’2023) also follow this paradigm. On the other hand, the multi-scale multi-head (MSMH) attention mechanism mentioned in this paper aims to mine more features of the network, however, this is also explored in Point-M2AE (NeurIPS’2023) and more unsupervised methods.
3. The implementation part of the paper has some advantages over contemporaneous methods. However, as point cloud self-supervised learning evolves, more excellent work such as PointGPT (NeurIPS’2023), ReCon (ICML’2023), and ACT (ICLR’2023) should be considered for inclusion in the comparative experiments.

questions:
1. How are the feature-level images in the abstract represented in the main text, please?
2. How does the number or size of multi-view images affect the proposed method, please? This may require further experimentation and analysis by the authors, and reference to CrossNet (TMM’2023) and Inter-MAE (TMM’2023), which use multi-view images for comparative learning between multiple modalities, is highly recommended.
3. Could the authors please provide appropriate qualitative visualisations such as complementary maps in their experiments? It is suggested to refer to Point-MAE (ECCV’2022), Point-M2AE (NeurIPS’2023) and TPM (arxiv’2024), which compare missing inputs and complementary outputs before and after self-supervision.

Overall, this is a relatively good work. If the author can take my comments into consideration and make explanations and revisions, I may further improve my score. AND vice versa.
</content>
</thread_object>

</thread>
<thread>
creation date:
1729545582492

thread_type:
official_review

<thread_object>
creator: reviewer_mcvh

<content>
summary:
This work proposed a 3D to multi-view autoencoder that reconstructs both point clouds and multi-view images. The proposed mutli-scale multi-head attention module provides broader local and global information. Besides, the two-stage training strategy ensures the student model learns well-aligned representations. The extensive experiments show the effectiveness of the proposed method.

strengths:
1. The manuscript is well-organized and easy to follow. 
2. The experiments are extensive. The proposed method is tested on four tasks and compared with multiple baselines.

weaknesses:
1. For the task part segmentation and few-shot learning, the proposed method achieved little increase. It would be better if more explanation and analysis can be given.

questions:
N/A
</content>
</thread_object>

</thread>
<thread>
creation date:
1730540790034

thread_type:
official_review

<thread_object>
creator: reviewer_5imv

<content>
summary:
The paper focuses on the problem of self-supervised point cloud representation learning and presents a method named Multiview Masked Learner. The method learns 3D representation by first training a 3D to multi-view autoencoder to create informative latent features. Then a student network is trained to predict the latent features from masked point cloud input. The autoencoder is carefully designed so that  it encodes 3D point cloud and decodes both 3D point clouds and the corresponding multi-view projections. Multi-Scale Multi-Head attention mechanism is integrated to increase the expressivity of the features. The resulting 3D representation shows promising results in various point cloud analysis benchmarks including object classification, part segmentation, and object detection.

strengths:
I would like to summarize the strengths of the submission from the following aspects.
1. The writing is clear and easy to follow. Though some claims are not quite intuitive to me which I will detail later but the overall flow is good.
2. The idea of unsymmetric encoder-decoder design is interesting. Enforcing the autoencoder to decode multiview representations from 3D point cloud only inputs sounds a reasonable way to encourage the multi-view geometric understanding in the learned representations.
3. The figures are quite helpful for presenting the architecture and training flow. 
4. The experiments cover a good range of tasks and the ablation studies also shows the effectiveness of the proposed MSMH attention, the choice of recovering token representations rather than the raw data, and the design of instance-level intra and inter-modality prediction.

weaknesses:
There are several weaknesses with the submission.
1. I am concerned with the technical novelty. Combining MAE with cross-modal distillation has been explored previously, e.g., as in [1]. Though in [1], images are used rather than projection of point clouds but I think the general framework is quite similar. It seems not quite challenging to replace the images used there with the projection of point clouds. The MSMH attention scheme also looks very similar to the Grouped Vector Attention operation in Point Transformer V2 [2]. The current pipeline seems more like an ensemble of existing techniques.
2. The motivation of the method is not strong enough. I do not see a particularly strong reason to avoid using images during the 3D representation learning stage if the multimodal-based pretraining gives better performance in downstream applications. Notice this line of work does not require images while using the pre-trained 3D backbone. The authors claim that incorporating both 2D and 3D during training is redundant and inefficient but do not provide concrete evidence to back up this claim. The claim from Line 49 to Line 53 also confuses me. I do not understand in what context this discussion happens and what is the key idea the authors want to deliver. In line 90, the authors say that their key insight is “the limited effectiveness of using 2D images as input for 3D geometric learning through MAE”. But doesn’t this suggest that we should develop better ways of using both 2D and 3D modalities instead of aborting the 2D modalities? In my humble opinion, previous works already discovered this and that’s why many works use MAE for the 3D modality and link 2D and 3D modalities through contrastive learning as done in [1].
3. The experiments conducted in the main paper are not comprehensive enough. The compared baselines are not very up-to-date and some more recent baselines are missing. For example, there is no comparison with [3]. Also, it feels very strange to leave important comparisons with ReCon [1] and I2P-MAE to the supplementary. The comparisons with [1] are incomplete in the supplementary with the few-shot classification experiments missing. In [1], experiments are also conducted in the 3D-only pretraining setup, and the results there seem comparable or even better than what is presented in the submission. When it comes to few-shot classification, the submission is not always winning either. In [3], a large collection of images helps further boost the representation quality to another level. This is to say, the presented results in the submission do not seem to be the state of the art as claimed.
4. Some ablation studies can be further improved. For example, what the performance would be if the input modality is 3D and the output modality is 2D? Also, a more detailed analysis of the MSMH design would be helpful for understanding its effectiveness and difference from previous works.

[1] Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining.
[2] Point Transformer V2: Grouped Vector Attention and Partition-based Pooling.
[3] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction.

questions:
1. Can authors better justify the motivation of the work? Especially given the fact that leveraging multimodal data for 3D representation learning is indeed achieving impressive results in many applications.
2. Can authors carefully compare their design differences with [1] and the MSMH design with the Grouped Vector Attention operation in [2]?
3. Experiments-wise, the authors need to provide a more comprehensive comparison with [1] and add comparisons with more recent methods such as [3]. Additional ablation studies would also be helpful.

[1] Contrast with reconstruct: Contrastive 3d representation learning guided by generative pretraining.
[2] Point Transformer V2: Grouped Vector Attention and Partition-based Pooling.
[3] ShapeLLM: Universal 3D Object Understanding for Embodied Interaction.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730716139590

thread_type:
official_review

<thread_object>
creator: reviewer_uyiy

<content>
summary:
They first project 3D point clouds to multi-view 2D images at the feature level based on 3D-based pose. Then, they introduce two components: (1) a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images from 3D and projected 2D features; (2) a multi-scale multi-head (MSMH) attention mechanism that facilitates local-global information interactions in each decoder transformer block through attention heads at various scales. Additionally, a two-stage self-training strategy is proposed to align 2D and 3D representations.
The contributions are summarized as follows:
(1) They propose a 3D to multi-view autoencoder that reconstructs point clouds and multi-view images solely from 3D point clouds
(2) They propose a Multi-Scale Multi-Head (MSMH) attention mechanism that integrates local and global contextual information by organizing distinct, non-overlapping local groups at multiple scales within the reconstructed features.
(3) They develop a two-stage training strategy for multi-modality masked feature prediction

strengths:
They propose a Multi-Scale Multi-Head (MSMH) attention mechanism that integrates local and global contextual information.
They employ a two-stage training strategy for multi-modality masked feature prediction.

weaknesses:
We think that the paper does not present its designs and motivations clearly

questions:
My major concerns are:
(1)The inputs of this model consists of both point clouds and 2D depth images. WWhat is the role of depth images within the model? How do they differ from the rendered images provided by the dataset?
(2)“incorporating both 2D and 3D modalities as input for training is redundant and inefficient.”However, the projection for 2D depth images is time-consuming. Additionally, two-stage training always need much time. 
(3)As stated in the abstract, "the input 2D modality causes the reconstruction learning to unnecessarily rely on visible 2D information, hindering 3D geometric representation learning." However, the proposed model also depends on 2D depth images: "These depth images then guide the reconstruction from 3D to 2D." The proposed method does not address or optimize the identified drawbacks.
(4) In the part segmentation experiment, there are no IoU (%) results of each category and visualization results, such as Point-MAE and Point-M2AE. 
(5) In the ablation experiments, there is no experiment conducted with other types of images (e.g., silhouettes, contours) as inputs.
(6)How about the training efficiency and parameters number? Introducing 3D to 2D projection, MSMH, and two-stage training strategy leads to additional training costs. This should also be discussed.
</content>
</thread_object>

</thread>
<thread>
creation date:
1730827015473

thread_type:
official_review

<thread_object>
creator: reviewer_k1ii

<content>
summary:
This paper proposes Multiview-ML, a novel 3D representation learning model that solely uses 3D point cloud data as input to reconstruct both the original point cloud and multiple depth images from different viewpoints. 

It leverages a two-stage training strategy with a teacher and student model, and outperforms existing approaches across various downstream tasks.

strengths:
1. The paper is well-written and easy to follow, presenting good-quality figures.


2. The experimental results look promising.

weaknesses:
1. The authors mention a limitation in prior work, stating that these methods *"inefficiently require both 2D and 3D modalities as inputs, even though 3D point clouds inherently contain 2D modality through their multi-view properties."* However, the authors provide insufficient evidence or ablation studies to substantiate this claim. Notably, previous works have often utilized only 3D inputs, projecting them into 2D during encoding without requiring both 2D and 3D modalities as explicit inputs.


2. The authors mention that the epoch number is 300, while do not specify how these are distributed across each stage. If both stages indeed run for 300 epochs, it raises the question of whether the observed improvement primarily results from an extended training period, which is computationally intensive.

3. It is better to demonstrate the individual effectiveness of each component in Table 5.

4. The ScanObjectNN and ModelNet40 datasets have reached saturation in point cloud understanding. Additional results on more complex and larger datasets, such as Objaverse, would be valuable.

5. In Supplementary Table 1, should "Ours (Point-M2AE)" actually be labeled as "Ours (Recon)"?

5. Typos: 'pertaining' on line 396.

questions:
Please kindly see the weaknesses above.
</content>
</thread_object>

</thread>
<thread>
creation date:
1732134601322

thread_type:
official_comment

<thread_object>
creator: authors

<content>
title:
Author Rebuttal by Authors

comment:
**General Response**

We sincerely thank each reviewer for their thoughtful feedback and detailed reviews. Below, we address the main concerns regarding novelty and comparisons with other peer research.

We have included a detailed discussion on the novelty of our approach and a comparison with ShapeLLM, ReCon, I2P-MAE, TAP, and other methods in the **revised supplementary material**, Sections A and B.


**Comparison with Related Works: I2P-MAE, ReCon, and ShapeLLM**

These methods focus primarily on two types of features:

1. **3D Geometric Features**: They employ MAE-based structures to reconstruct the original point clouds, thereby capturing detailed 3D geometric data.

2. **Semantic and Textual Representations**: They utilize techniques such as contrastive learning or knowledge distillation to extract semantic and textual features from 2D images and language models.

These methods directly adopt existing 3D MAE frameworks to learn geometric features. Specifically, I2P-MAE utilizes Point-M2AE, while ReCon and ShapeLLM leverage Point-MAE for geometric representation. **Their innovation lies in the novel use of foundation models for knowledge distillation.**

- **I2P-MAE** performs pixel-to-3D token knowledge distillation by adding additional layers after the M2AE encoder, calculating MSE loss between the point tokens and 2D pixel-level features derived from foundation models.
- **ReCon** uses Point-MAE as the base structure to reconstruct original point clouds from masked point cloud inputs. It also incorporates instance-level contrastive learning to distill knowledge from both text and image foundation models.
- **ShapeLLM** builds upon ReCon by using larger models with more parameters, leveraging large language models to enable advanced 3D reasoning.

In contrast, our approach is designed to specifically enhance the learning of **3D geometric features without relying on external foundation models**. As shown in Supplementary Table 1, replacing the 3D geometric learning structure in I2P-MAE and ReCon with our method results in substantial performance improvements. This clearly demonstrates that our approach enhances 3D geometric representation learning in a way that effectively **complements** the semantic and textual features derived from 2D images and language foundation models.




**Motivation and Novelty of the Proposed Method**

As mentioned in the previous part, this work focuses on **3D geometric representation learning.** The motivation behind our work is twofold:

1. To highlight the limitations of using 2D images as inputs for 3D geometric learning through MAE without leveraging foundation models.
2. To emphasize the importance of incorporating multi-view information for more effective 3D geometric representation learning.

A key insight from our study is the limited effectiveness of using 2D images as input for 3D geometric learning via MAE, and the necessity of fully utilizing the multi-view attributes of 3D point clouds within the MAE framework. Our ablation studies, presented in Table 6 of the main paper and Table 3 of the supplementary material, substantiate this claim. These findings highlight an opportunity to develop more effective strategies for 3D representation learning.

The novelty of our method can be summarized as follows:

1. To the best of our knowledge, this is the first work to reconstruct both 3D and 2D information solely from 3D inputs for 3D **geometric representation learning**. A key innovation of our approach is projecting 3D point clouds to multi-view 2D images at the feature level, allowing for multi-view 2D and 3D reconstruction from a single 3D modality (see Fig. 2 in the main paper). Our projection method directly maps 3D tokens to corresponding 2D tokens for each view, ensuring seamless alignment between 3D and 2D features, thereby enabling the reconstruction of original 2D depth images from point cloud data alone.


2. We introduce a multi-scale, multi-head attention (MSMH) mechanism that enhances the interplay between local and global perspectives within each decoder transformer block, improving the quality of learned representations.

3. We propose a two-stage self-training strategy with latent space prediction, integrating latent space prediction into the MAE task to ensure that student models learn well-aligned representations during training.

It is important to note that during the fine-tuning and inference stages, we remove these additional components like projection layers and MSMH decoder, retaining the **same architecture as the baseline** to ensure a fair comparison.
</content>
</thread_object>

</thread>


Metareview:
metareview:
The paper proposes a self-supervised learning method for 3D point clouds. The proposed method can be combined with current Masked Autoencoder (MAE) methods to improve their performance and can utilize multi-modal information for pre-training without relying on 2D encoders. Using a two-stage self-training strategy, the proposed method achieves strong results.
However, this work has several significant drawbacks: 
1. Lack of Comprehensive Comparison: the comparison with the latest methods is not comprehensive. Specifically, the paper does not fully evaluate against recent methods in 3D point cloud pre-training, such as I2P-MAE and ReCon.
2. Unclear Motivation: the rationale for not utilizing 2D foundation models is not well-justified. As demonstrated by methods such as I2P-MAE and ReCon, incorporating 2D foundation models can significantly enhance the performance of 3D point cloud pre-training.
3. Incremental Improvements: the proposed Multi-Scale Multi-Head (MSMH) module is an incremental improvement over multi-head attention mechanisms, rather than a substantial innovation. Additionally, the two-stage self-training strategy bears similarities to existing methods like iBot [1] and dBot [2].
4. Lack of Self-Containment: key explanations and experiments that support the motivations and claims are missing from the main paper, making it less self-contained.

Although the paper clearly has merit, the decision is not to recommend acceptance at this time. We encourage the authors to carefully consider the reviewers' comments when revising the paper for submission elsewhere, particularly in providing a more comprehensive comparison with the latest methods, clarifying the motivation and unique contributions, and ensuring the main paper is self-contained with all key explanations and supporting experiments.

[1] Zhou J, et al. ibot: Image bert pre-training with online tokenizer. ICLR 2022.
[2] Zhou J, et al. Exploring Target Representations for Masked Autoencoders. ICLR 2024.