Title: Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
Venue: NeurIPS 2025
Accepted: yes
Keywords: Point Cloud

3D Representation Learning

3D Semantic Segmentation

Self-supervised Learning
Abstract: Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, as demonstrated by zero-shot visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14.2\% and 4.8\%, respectively, as well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e.g., 80.7\% mIoU on ScanNet). We further present a variant of Concerto tailored for video-lifted point cloud spatial understanding, and a translator that linearly projects Concerto representations into CLIP’s language space, enabling open-world perception.
These results highlight that Concerto emerges spatial representations with superior fine-grained geometric and semantic consistency.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes the Concerto framework, which fuses image and point cloud modalities through 2D-3D joint self-supervised learning, and combines intra-modal self-distillation (3D point cloud feature refinement) with cross-modal joint embedding prediction (image-point cloud feature alignment) to achieve more consistent spatial representation learning. Experiments show that Concerto significantly outperforms single-modal methods on benchmarks such as ScanNet, with linear detection accuracy improved by 4.8%-14.2%, and achieves SOTA performance of 80.7% mIoU under full fine-tuning. In addition, its generalization and open-world perception potential are verified through video point cloud expansion and CLIP language space mapping.\n\nSTRENGTHS AND WEAKNESSES:\n**Advantages**:\n1. Multimodal collaborative innovation: simulate human multi-sensory learning mechanism, break through the limitations of single-modal representation through cross-modal alignment (image-point cloud) and intra-modal distillation (point cloud feature optimization), and fill the gap between 2D and 3D features.\n2. Comprehensive performance leadership: In tasks such as semantic segmentation and instance segmentation, Concerto significantly surpasses single-modal models such as DINOv2 and Sonata, and even outperforms feature splicing baselines, proving the effectiveness of multimodal collaboration.\n3. Strong technical universality: supports pure point cloud input, video sequence expansion and language space mapping, adapts to multiple scenarios such as autonomous driving and robots, and has industrial landing potential.\n\n**Disadvantages**:\n1. Modality dependency limitations: cross-modal alignment depends on camera parameters, and performance may degrade in scenarios without calibration or multi-view missing, and the extreme case of pure point cloud without image input has not been verified.\n2. High computational cost: 40k point clouds + 300k images are required for pre-training, and the video memory usage and training time are significantly higher than those of single-modal models. The ablation experiment did not quantify the difference in computational overhead.\n3. Shallow language alignment: CLIP space is only mapped by linear projection, and the zero-shot segmentation accuracy (44.56% mIoU) is far behind the supervised results, and deep semantic fusion needs to be strengthened.\n\nQUESTIONS:\n1. **Cross-modal generalization**: In the LiDAR-vision heterogeneous sensor fusion scenario, is an additional calibration step required?\n2. **Long-term tracking performance**: How to solve the drift problem of the video extension version in long sequences?\n3. **Lightweight potential**: Can a parameter compression version (such as distillation or quantization) be launched for mobile devices?\n\nLIMITATIONS:\n1. **Scene constraints**: Relying on high-quality image-point cloud pairing data, not robust enough for textureless or low-light scenes.\n2. **Semantic gap**: Language mapping only stays at linear projection, and does not model the complex semantic association between spatial concepts and natural language.\n3. **Ecological dependence**: Highly dependent on 2D models such as DINOv2. If other backbones (such as Swin Transformer or SAM) are used, will the performance be stable?", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-04"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Concerto, a minimalist yet powerful framework for learning spatial representations by combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. I found the central idea of connecting 2D and 3D feature spaces particularly novel and compelling. The intuition that humans can abstract and recall spatial concepts across modalities is well-motivated and convincingly translated into the model design.\n\n* The empirical results are somewhat strong: Concerto demonstrates fine improvements over both 2D and 3D SOTA baselines in fine-tuned settings\n---> while the zero shot evaluation is missing and lacking. \n\nOverall, the method is conceptually sound and has fine practical relevance.\n\nThat said, there are several comments that should be addressed prior to publication.\n\nSTRENGTHS AND WEAKNESSES:\n** Strengths **\n1. Novel and Well-Motivated Idea: The paper introduces a creative and compelling approach that bridges 2D and 3D modalities for spatial representation learning. The core intuition is both original and conceptually sound.\n\n2. Good Experimental Results and Ablation: The experimental evaluation is comprehensive, with solid performance across multiple benchmarks. The ablation studies effectively isolate the contributions of different components (I appriciate this part very mach)*\n\n3. Data Efficiency Results – A Strength: The results on data efficiency are excellent and, in my view, one of the paper’s key strengths. I suggest highlighting this more prominently in both the abstract and conclusion.\n\n** Weaknesses **\n\n1. Writing \n  * Redundancy: The manuscript suffers from a noticeable degree of textual redundancy. I recommend a thorough revision to reduce repetition and improve clarity throughout the paper.\n\n  * Lack of Formal Definitions: While the paper is generally well-written and conceptually clear, it would benefit from a more formal and mathematical presentation. Providing explicit formulas at each step of the pipeline would make the method easier to follow and reproduce.\n\n* Some Statements Are Confusing: Certain parts of the paper are unclear or potentially contradictory. See Questions 1 and 2 in my review for specific examples that should be clarified.\n\n2. Figure 3 – Needs Improvement: This figure is central to the paper, but currently it is unclear and potentially misleading. If it is meant to depict the architecture, it should be expanded with clearer annotations, supporting formulas, and step-by-step explanations. Consider reducing textual redundancy elsewhere in the paper to make room for a more informative and detailed illustration.\n\n3. Intra-Modal Self-Distillation Section: This section is underexplained, despite being a key preliminaries of the method. Please provide more details on how intra-modal self-distillation is implemented and how it contributes to the learned representations.\n\n4. Zero-Shot Segmentation – Untapped Potential: Have you considered evaluating your model on zero-shot segmentation tasks, similar to ConceptFusion [1] or FollowAnything [2], by providing an image or user click to specify the object of interest and not only text?\n\n5. Long-Tail Concepts – Missing Evaluation: It would be informative to test the method on long-tailed or rare object classes, as found in datasets like unCOCO. This would demonstrate robustness in more realistic, open-world scenarios.\n\n6. Comparison to ConceptFusion: ConceptFusion is a closely related work that also aligns 2D and 3D representations and achieves strong results on ScanNet and many other dtasets. A direct comparison would strengthen the paper, especially in light of similar goals. For zero short text, click, images as input.\n\n[1] https://concept-fusion.github.io/\n[2] https://arxiv.org/abs/2308.05737\n\nQUESTIONS:\nVery important questons related also to the Weaknesses:\n\n\n  1. Line 92–94 (Sentence Starting with \"Ideally\"): This sentence is confusing and seems to contradict the surrounding paragraph. Please clarify its intent and explain how it aligns with the rest of the section.\n\n   2. Lines 112–113 – Clarification Needed: It took time to uderstand what is meant by \"linear probing\" in this context. you are projecting point cloud features into the CLIP feature space, given that CLIP already aligns images and text during its pretraining -- the goal is to align the 3d representation with text. The question is \n * why CLIP if you dont use the image embeddings? or do you? \n * can you use any other text feature extractor? \n  * How to do align, what is the loss here?\n I think this pargraph need a better writing. \n\n3. DINOV2 as a Target – Clarification: If your model is trained to predict DINOv2 features, does this imply that DINOv2 defines an upper bound for your performance? If so, how is your method able to outperform it? I found this somewhat confusing. \n\n4. Lines 145–146 – Ambiguity: The phrasing “divide a scene into one point cloud” is unclear. Please revise this for clarity. Do you mean that each scene is represented as a single point cloud - then what does divide mean? If not, please specify the intended meaning.\n\nLIMITATIONS:\nSee my Weaknesses 4 5 and 6.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 4, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper presents Concerto, a framework motivated from multi-sensory learning in humans, that aims at achieving better spatial representations for self-supervised point cloud features. It first self-distills the 3D model through a teacher-student paradigm, then utilizes 2D-3D joint embedding prediction to align the 3D point cloud features with the 2D image features. Experimental results show the superiority of the proposed Concerto when applying to downstream tasks of semantic and instance segmentation under different tuning settings including linear probing, decoder probing, and full finetuning.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths**:\n\n++ The idea of using Joint Embedding Predictive Architecture (JEPA) to predict 3D features from pretrained 2D encoders is neat and effective solution to me. \n\n++ The experimental settings are comprehensive, including different tuning strategies of linear probing, decoder probing, and full finetuning, demonstrating the broad use case of the proposed Concerto under different circumstances and training budgets. Also, the model almost always outperforms the baseline Sonata, showing superior performance on downstream tasks. \n\n++ The ablation study on multiple aspects including criteria type and weight, the ratio of point clouds and images, *etc.* of the implementation can be useful for guiding towards the optimal configuration of the model. \n\n**Weaknesses:**\n\n-- The paper lacks the comparisons with other methods that also focus on spatial understanding with joint 2D-3D data, essentially the methods listed in the third paragraph of the related work section. More specifically, Large Spatial Model [1], FiT3D [2], PonderV2 [3], GGSD [4] are some open-source options, and also representing the different categories listed in the related work section. If the original works are on 2D segmentation, we can lift the 2D features to 3D using camera poses for comparison in 3D segmentation. This comparison is very important because they are actually the existing works for spatial understanding with joint 2D-3D data, while the current compared methods like MSC, Sonata are all pure 3D self-supervised methods. It will help demonstrate how the proposed solution for joint 2D-3D learning performs compared to alternative solutions in other works like lifting projections, differentiable rendering, direct distillation, *etc.*\n\n-- The paper claims to have synergy between the 2D and 3D models. However, from my understanding of the method, the 2D model is fixed, so the model is essentially getting knowledge from pretrained 2D model to predict a better 3D model, but there is no benefit the other way around. Therefore, I think the synergy from the joint learning paradigm is overclaiming.\n\n-- The 3D intra-model self-distillation part follows the identical approach of previous work Sonata. Therefore, I think it cannot be regarded as part of the proposed method. However, the paper gives a feeling that the proposed method has two steps \"(a) intra-modal self-distillation + (b) 2D-3D joint embedding prediction\". In fact, only (b) is the technique proposed in the paper. \n\n-- It sounds a bit far-fetched to me to connect the proposed method with the human concept learning in cognition. Basically the paper is proposing a paradigm for 2D-3D joint learning, which is just very loosely related to the multi-sensory learning concept of humans illustrated in the paper. It is definitely good to have some bio-inspired motivations or designs in the method, but I think the concept is very loosely related to the proposed method. Otherwise, any method that introduce additional modality in its framework can be related to human multi-sensory learning.\n\n-- In the supplementary the paper mentioned the term \"DINOv2.5\". However, I think this is not a standard term. I could only refer to the Sonata paper where they refer to [5] as DINOv2.5. I think a more standard term for this work is called DINOv2-Reg. Also, the submitted manuscript does not have reference about [5], which makes readers feel confusing.\n\n[1] Fan et al. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024.\n\n[2] Yue et al. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. ECCV 2024.\n\n[3] Zhu et al. PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. arXiv:2310.08586.\n\n[4] Wang et al. Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation. ECCV 2024.\n\n[5] Darcet et al. Vision Transformers Need Registers. ICLR 2024.\n\nQUESTIONS:\n-- How is the proposed method compared with other methods that also focus on spatial understanding with joint 2D-3D data like [1, 2, 3, 4]? The comparisons are important to understand which is a better solution for joint 2D-3D spatial understanding.\n\n-- Is the pretrained 2D model also tunable during training? If not, I think it cannot be called \"synergy emerged from joint self-supervised learning\". If so, will the 2D model gets improved from jointly training with the 3D model?\n\n-- Does there exist difference from the self-distillation stage presented in the paper and the self-distillation operation in the Sonata paper? If not, I think this stage cannot be an essential component of the proposed method, because the proposed method is built upon Sonata, while the self-distillation stage is just a part of the original Sonata paper.\n\n-- For the Concerto variant specifically for video, is the only difference of this model and the original model being the training data is constructed from video using VGGT instead of point cloud? If so, why this is treated as a separate variant, instead of adding the video data to the training dataset to enrich the training set, and get a more powerful model with the more diverse training set?\n\n[1] Fan et al. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024.\n\n[2] Yue et al. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. ECCV 2024.\n\n[3] Zhu et al. PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. arXiv:2310.08586.\n\n[4] Wang et al. Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation. ECCV 2024.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes Concerto, which extends a 3D self-supervised learning (SSL) method (Sonata) by cotraining with 2D image features via joint embedding predictive objective, and demonstrate consistent performance gain on various tasks, surpassing the single modality performance and naive concetenation of 2D and 3D features, providing a straightforward and effective way of enhancing 3D SSL encoders.\n\nSTRENGTHS AND WEAKNESSES:\n**Strengths:**\n1. Straightforward and effective method. SSL on 2D image features has witnessed a substantial progress. Combing their power to improve 3D features has sigfinicance to the community.\n2. Consistent performance gain. Concerto demonstrates consistent performance gain over baselines on various benchmarks, providing a strong empirical evidence.\n3. Ablation studies covered many aspects.\n\n**Weaknesses:**\n1. Clarity of writing can be improved if more important details can be included in the main paper. Specifically:\n - 1a) how are the image and point feature represented when computing JEPA? If would be nice to have some concise and clear math expressions and formulations.\n- 1b) In ablation study, what is the definition of \"image usage\"?\n- 1c) In ablation study, how do you specifically control the number of visible points and how is the specific number 65536=256^2 chosen? How do you get the number from the visibility check in the appendix?\n- 1d) Lack of implementation details on the learning probing and decoder probing.\n2. Performance seems to saturate when scaling up data and model, in Table 5h. Scaling is an important indicator but only two settings are ablated. It would be great if the authors can provide more results on this, perferablly a plot and some explanations.\n3. Lack of quantitative results of training on the VGGT lifted video data. It would be great to see the method can robustly leverage VGGT lifted data as training source and possibly showing some scale performance on the video data where data is abundant while annotation is scarce.\n\nQUESTIONS:\n1. Does batch size matter in the cross-modal training?\n2. Can the method work with noisy camera parameters? Will it be sensitive if depth, camera poses or intrinsics are inaccurate?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 2, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-01"}