Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
Authors: Yujia Zhang
Xiaoyang Wu
Yixing Lao
Chengyao Wang
Zhuotao Tian
Naiyan Wang
Hengshuang Zhao
Venue: NeurIPS 2025 poster ()
Keywords: Point Cloud
3D Representation Learning
3D Semantic Segmentation
Self-supervised Learning
Primary area: applications

Abstract: Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, as demonstrated by zero-shot visualizations. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14.2\% and 4.8\%, respectively, as well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e.g., 80.7\% mIoU on ScanNet). We further present a variant of Concerto tailored for video-lifted point cloud spatial understanding, and a translator that linearly projects Concerto representations into CLIP’s language space, enabling open-world perception.
These results highlight that Concerto emerges spatial representations with superior fine-grained geometric and semantic consistency.

Concerto: Joint 2D-3D Self-Supervised Learning Emerges Spatial Representations
Yujia Zhang' Xiaoyang Wu'
Yixing Lao' Chengyao Wang
Zhuotao Tian Naiyan Wang
Hengshuang Zhao'
The University of Hong Kong The Chinese University of Hong Kong
Harbin Institute of Technology (Shenzhen)
https:/ ptrpa://poiiiie incept.......... g CoCCararto
RGB
Raw
DINO2 2D Imagee SSL
3D Point SSL
Sonata
Concerto (Ours) 2D-3D Joint SSL
Figure We visualize the principal components of point features learned by 2D and 3D self- supervised models [28, 49], mapped to RGB colors. DINOv2 lacks geometric awareness, and Sonata struggles to capture fine textures. Concerto integrates intra-modal self-distillation with cross-modal joint embedding prediction, enabling a self-supervised point cloud transformer [47] to learn richer, emerging spatial representations with fine-grained geometric and semantic consistency across views.
Abstract
Humans learn abstract concepts through multisensory synergy, and once formed, such representations can often be recalled from a single modality. Inspired by this principle, we introduce Concerto, a minimalist simulation of human concept learning for spatial cognition, combining 3D intra-modal self-distillation with 2D- 3D cross-modal joint embedding. Despite its simplicity, Concerto learns more coherent and informative spatial features, demonstrated by zero-shot visualiza- tions. It outperforms both standalone SOTA 2D and 3D self-supervised models by 14.2% and 4.8%. respectively, well as their feature concatenation, in linear probing for 3D scene perception. With full fine-tuning, Concerto sets new SOTA results across multiple scene understanding benchmarks (e.g., 80.7% mloU
"Conresponding author
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

ScanNet). We further present variant of Concerto tailored for video-lifted point cloud spatial understanding, and translator that linearly projects Concerto rep- resentations into CLIPs language space, enabling open-world perception. These results highlight that Concerto emerges spatial representations with superior fine- grained geometric and semantic consistency.
Introduction
Learning strong spatial representations in a self-supervised manner is foundational for spatial cog- nition tasks, spanning from low-level machine perception to high-level reasoning in domains such autonomous driving [8, 38], mixed reality [7, 23, 37], and roboties [17]. Recent advances in self-supervised learning have significantly improved foundational representation models for the two dominant spatial modalities: 2D images [4, 12, 18. 51, 56] and 3D point clouds [29, 46, 49, 50, 53]. Without the need for human annotations, these models have demonstrated strong performance across various downstream tasks by enabling the learning of geometry, and semantics scale.
However, despite their individual successes within each data modality, our pilot study reveals that self-supervised representations learned independently from images and point clouds do not fully overlap. Specifically, concatenating features from self-supervised image models (e.g., DINOv2 [28]) and point cloud models (e.g.. Sonata [49]) leads improved linear probing performance, suggesting that each modality captures complementary, rather than redundant, aspects of spatial information. The observation hints at the existence of more robust and rich feature space that emerges from the interaction between 2D and 3D modalities, indicating the core aim of this research: to uncover superior spatial representations through multi-modal self-supervised learning.
Our inspiration toward this target is rooted in how humans learn abstract concepts: through multisensory synergy [6, 36]. Consider the example of an apple (as illustrated in Fig. 2)our understanding of it is not lim- ited to its visual appearance. Instead, the concept is formed through repeatedly seeing, touching, and tasting apples, allowing us to internal- ize its geometry, texture, and semantic meaning in a unified, predictive way. This cognitive process reflects a continuous, tion of sensory data into a grounded concept space (Fig. top). Yet once such a representation is formed, it can be evoked from just a single modality: sceing an image of an apple can vividly recall its weight and texture, just as holding one can bring its color and shape to mind (Fig. 2 bottom). This ability retrieve rich, structured knowledge from partial sensory input underscores the importance of learning modality-agnostie representations that are both unified and predictive.
Multi-Sensory Learning
See
Touch
moosl integra-
The concept of "Apple" geometry, texture, semantics, etc.
Imagino
Driven by this vision, our methodology aims to offer a simple yet effective imitation of human multisensory synergy. To this end, we compose Concerto of 2D-3D joint self-supervised learning, coupling intra-modal delf-distillation for point representations [28, 49] with cross-
Senloy-Sesoory Retrieval
Figure 2: The "Apple"
concept in cognition.
Ultimately, this training yields sulf-supervised PTv3 [47] model, pretrained on 40k raw point clouds and 300k im- ages [1, 5, 7, 15, 33, 52, 60]. In addition, we present a variation of Concerto, augmented with an additional set of 50k point clouds with 200k corresponding images lifted from scene videos [61] via feed-forward reconstruction [43], tailored for video-based spatial understanding. We also introduce an interlude Concerto: learned translator that linearly projects self-supervised representations into CL.PP's language space [27, 32), enabling open-world perception.
modal joint embedding prediction from images to point clouds [26].
The intersection of 2D and 3D self-supervised learning in Concerto yields a powerful synergy. enabling the emergence of superior spatial representations. PCA-colored visualizations reveal that Concerto captures more coherent and informative spatial features than SOTA 2D or 3D self- supervised models trained on single modality (see Fig. 1). Concerto exceeds its predecessor, Sonata, with 4.8% improvement in linear probing, achieving 77.3% mloU on ScanNet semantic segmentation using a single linear layer. Notably, this performance also surpasses the concatenation of Sonata and DINOv2 (1.4%), demonstrating that the multisensory synergy in Concerto exceeds the representational upper bound achievable by single-modality self-supervised learning. With full fine-tuning, Concerto achieves SOTA performance across a range of scene perception tasks. For example, reaching 80.7% mloU on ScanNet semantic segmentation.

---

Semantic Segmentation
ScanNet Val [15)
ScanNer200 Val [35]
Method
Type
Encoder
mloU
mAce
allAcc
mloU
mAcc
allAcc
DINOv2 [28]
2D Image SSL
ViT-G
63.09
75.50
82.42
27.42
37.59
72.80
Sonata [49]
3D Point SSL
PTv3-B
72.52
83.11
89.74
29.25
41.61
81.15
Sonata x DINOv2
3D SSLX2D SSL
Both
75.91
85,36
91.25
36.67
46,98
82.85
Concerto (ours)
2D-3D Joint SSL.
PTv3-B
77.32
86.58
91.74
37.41
49.49
83,29
Table 1: Linear probing results on 3D semantic segmentation. We compare self-supervised
features learned from 2D, 3D, their feature concatenation, and our 2D-3D joint SSL model, Concerto (as preview). Notably, the concatenation 2D and 3D features outperforms either modality alone, suggesting that the two modalities encode complementary information. Concerto achieves the best performance across all metrics, demonstrating its ability to learn superior spatial representations.
2 Beyond Single Modality: Toward a New World of Representations
This section presents pilot study to explore high-level questions surrounding self-supervised rep- resentations. These questions form the conceptual foundation of our research, and our methodology emerges a natural and simple response the insights gained here.
2.1 Is There a Superior Representation Space Beyond Single-Modality Learning?
Sule-supe learning on 2D images and 3D point clouds has achieved remarkable progress in visual representation learning. However, when trained independently, these models may capture only modality-specinf perspectives of the spatial world. Just person who has only seen an apple but never tasted one may lack a sense of its flavor or texture, single-modal learning inevitably misses critical dimensions of the world. This raises fundamental question: Is there a superior representation space that can emerge from the synergy between 2D and 3D modalities?
To probe this question, we begin with simple pilot experiment: fusing supervied features from image and point cloud models, prior to any explicit learning of cross-modal synergy. Specifically, select two representative self supervipe models trained independently on images (DINOv2 [28]) and point clouds (Sonata [49]). We lift image features into 3D space using depth and camera param- eters, and concatenate them with point cloud representations to enable feature-level fusion. We benchmark the 2D, 3D, and fused representations via linear probing on 3D scene-level semantic segmentation using the ScanNet [15] dataset, with results presented in Tab. 1. Notably, this naive combination outperforms both individual modalities, suggesting the presence of complementary in- formation and hinting richer representational space.
However, simply concatenating 2D and 3D self-supervised features, while yielding a stronger rep- resentation space, still falls short of uncovering the unexplored new world we are seeking. This approach lacks integration during learning and cannot fully capture the synergy that emerges when modalities learned together. The deeper insight lies in the potential of multi-modat joint represen- tation learning- not only to align complementary signals across modalities, but also to form coher- ent, predictive embeddings that generalize beyond their source. Ideally, such fused representations can be retrieved from a single modality, even if they were originally learned through multi-modal interaction. This form of joint 2D-3D representation learning intuitive, as it mirrors how humans form concepts, as discussed in Sec. and illustrated in Fig. 2.
This insight leads to our methodology: designing a unified framework that learns to embed spatial information through both intra-modal refinement and cross-modal prediction.
2.2 Can Multi-modal Self-Supervised Representations Speak the Language of Concepts?
Human language is often considered a compressed and symbolic interface to abstract concepts leamed through multisensory synergy [6]. If multi-modal self-supervised representation learning succeeds in forming unified abstract concepts, then such representations should, in principle, be able to align with human hanguage-perhaps even through simple linear projection. This perspective raises a natural question: Can multi-modal self-supervised representations, learned entirely without human language, speak the language of concepts?
We believe the answer to this question is ultimately yes. However, our current study is grounded in the spatial domain, leveraging only 2D images and 3D point clouds. This limited sensory scope
3

---

(a) Self-distillatin
44
(b) Joint Embedding Prodiction
P
Point
Predictor
Encoder
Js,
Encoder
Image
Image Patch Point Cloud 4 Camera (2)
Figure 3: Overview of the Concerto architecture. Concerto simulates human multisensory synergy by coupling (a) intra-modal self-distillation on 3D point clouds to progressively refine its internal spatial representations (see Sec. 3.1), and (b) cross-modal joint embedding prediction that aligns point features with corresponding image patch features using camera parameters (see Sec. 3.2). The self-distillation branch (a) employs a restricted online clustering objective, while the joint embed- ding prediction (b) applies looser cosine similarity constraint. This dual sulf-supe objective encourages the emergence coherent, modality-agnosti spatial representations.
makes challenging to fully align with human language, which emerges from far richer blend of modalities. Still, we propose that linearly probing self--upervised representations into a language embedding space, such as CLIPPs, offers a meaningful way to evaluate progress toward this goal. Beyond serving as diagnostic tool, this projection also extends the open-vocabulary capabilities of self-supervised spatial representations, offering a step toward broader concept grounding.
We propose linear probing into the CLIP feature space as a next-level evaluation criterion for self- supervised learning beyond single modality.
Concerto: Joint 2D-3D Self-Supervised Learning
This section introduces the joint 2D-3D self-supervised learning framework of Concerto. The pro- posed architecture is intentionally simple, designed to highlight the power of multisensory synergy through strong empirical performance. An overview of concerto architecture is present in Fig. 3.
3.1 Intra-Modal Self-Distillation
The primary duty of Concerto is to validate the superiority of multi-modal synergy, rather than seeking innovation in single-modality self-supervised learning architecture. For this reason, the intra-modal branch Concerto builds upon the recent Sonata framework [49], which applies self- distillation [10, 28] learn point cloud representations without supervision. We briefly revisit this architecture here and refer readers to the original Sonata paper for a detailed discussion on building reliable supervieee framework for sparse and unstructured point cloud data.
The intra-modal delf-distillation in Concerto focuses solely on the 3D domain, where a Point Transformer V3 [47] is trained to produce stable and predictive features through teacher- student paradigm. The student encoder is optimized to match the output of a momentum-updiee teacher [18), using clustering-based objective [9] that promotes consistency across augmented views of the same point cloud. A unique challenge in sparse point clouds is the geometric shortcut, where models collapse easily accessible low-level geometric cues. These cues not learned but introduced implicitly through the local kernel definitions of point cloud operators. Sonata mitigates this through several micro-designs that obscure explicit spatial signals and encourage learning from input features. This self-motivated refinement process allows the model to internalize geometric and structural priors from 3D data, forming the foundation for multi-modal learning in Concerto.
3.2
Cross-Modal Joint Embedding Prediction
We introduce an additional cross-modal self-supervised objective that continuously stimulates syn- ergy from the images self-supervised representation into the point cloud domain. This design aligns with the core vision of LeCuns Joint Embedding Predictive Architecture (JEPA) [26], which advo- cates learning by predicting latent representations across modalities using conditional predictor. The goal is to predict point cloud embeddings that match the associated pixel embeddings extracted from a self-supervised image encoder (e.g., DINOv2 [28]). Empirically, we find that cosine similar- ity provides the most effective criterion for training this predictive branch, and by applying strong point cloud data augmentations and exploring less aggressive image augmentations compared to DINOv2, Concerto learns more generalizable representations.

---

In our implementation, we divided scenes with a large number of images into several data pieces. Each one consists of one point cloud with 4 images. For scenes with fewer images (e.g.. 5), we retain the original dataset divisions. As shown in Fig. 3, we respectively obtain point representa- tions 8, and image representations *y from the point encoder and image encoder. In the context of 2D images and 3D point clouds, our predictor takes camera parameters as the condition 2 to estab- lish correspondences between image pixels and point cloud points. The details of this process are provided in the Appendix. Then, for each image patch, we compute the mean of the features of points falling within it to get predicted patch features 8y from point clouds. Finally, we compute the loss (sy Sy) using cosine similarity. This process introduces 2D self supervisd features into 3D supe-supervieed learning and stimulates the point intra- modal self- eeff-disilllation process.
3.3 Synergy Emerged from Joint Self-Supervised Learning
The combination of intra-modal self-distillation and cross-modal joint embedding in Concerto emerges as a strong synergy, surpassing single-modality learning and elevating the interaction be- tween self-suuer point cloud and image representations beyond naive fusion. The complemen- tary cognitive signals from image self-supervised learning, through cross-modal joint embedding prediction, encourage point intra-modal self-distillation extend capacity beyond single modality and emerge a superior spatial representation surpassing the naive combination of self-supervised features from images and point clouds. This process yields spatial representations that more expressive than those obtained by merely concatenating features from separate 2D and 3D models. We believe this chain reaction is central to the design of Concerto. Notably, the architecture also supports training on point clouds without paired images, enabling hybrid self-supervised learning without compromising scalability on large-scale 3D datasets.
Experiments
We evaluate Concerto's representations with various scene perception tasks using the same protocols as Sonata [49]. Specifically, linear probing, which keeps the encoder frozen and adapts the upcasted original scale features to downstream tasks; decoder probing, adding lightweight decoder after the frozen encoder to facilitate adaptation; and full fine-tuning, where the encoder and decoder are optimized for downstream tasks. We further analyze Concerto's 's key properties with these results.
4.1 Main Results
Semantic segmentation. In Tab. 2a, we compared Concerto with the previous 3D encoder Sonata [49] on semantic segmentation tasks across multiple datasets: ScanNet [15], Scan- Net200 [35], ScanNet++ [52], and S3DIS Area5 [1] with full fine-tuning, Across all datasets, Con- certo achieves SOTA performance with notable mloU results, including 80.7% on ScanNet, 39.2% on ScanNer200, and 50.7% on ScanNet++. The most significant improvement is seen in the Scan- Net200 dataset, which contains 200 class categories. This suggests that while detecting fine-grained objects in sparse point clouds remains challenging, joint 2D-3D cross-modal learning enables Con- certo to capture detailed semantic and geometry information, thus improving the models ability for such objects.
Instance segmentation. In Tab. 4, we further validate the robustness of Concerto across 4 widely recognized instance segmentation benchmarks. Concerto demonstrates the strongest performance in all evaluation methods. Notably, decoder probing on ScanNet outperforms full fine-tuning, suggest- ing that Concerto learns rich, generalizable representations during pretraining without task-specific adjustments. This demonstrates the advantage of leveraging general pretrained representations, which reduces the risk of distorting pretrained representations and overfitting in fine-tuning.
Parameter efficiency. In Tab. 2b, we demonstrate Concerto's 's parameter efficiency using the sim- plest linear probing and decoder probing across 4 semantic segmentation benchmarks. In partic- ular, Concerto outperforms supervised learning using the PTv3 backbone [47] on all benchmarks with decoder probing. Even with linear probing, Concerto surpasses the supervised PTv3 on Scan- Net200 [35] and S3DIS [1]. Compared to Sonata [49]. Concerto shows significant improvements on ScanNet200 [35] and ScanNet++ [52] with linear probing (+8.1% and +6.7% respectively). These results highlight a substantial improvement in scenes with larger numbers of classes.
Data efficiency. In Tab. 2c, we examine the data efficiency performance of Concerto on ScanNet Ef- ficient Datasets [19] with limited scenes and annotations. Concerto outperforms Sonata [49] across all evaluation protocols. Notably, linear probing results surpass decoder probing and even full fine-
5

---

Fine-Tuning
Purams
ScanNet Val
ScanNer200 Val [35]
ScanNet++ Val [52]
S3DIS Area
Methods
Learn.
Pet.
mloU
mAce allAce mloU mAce allAce mloU mAce allAce mloU mAce allAce
SparseUNel [14]
39.2M
100%
72.3
80.2
90.0
25.0
32.9
80.4
28.8
38.4
80.1
66.3
72.5
89.8
PC [50]
39.2M
100%
72.3
80.9
90.1
26.2
33.0
79.9
29.2
39.7
82.7
68.1
73.5
90.0
CSC [19]
39.2M
100%
72.8
81.0
90.7
26.9
33.7
80.6
32.5
41.1
83.7
70.7
76,4
90.8
MSC [46]
39.2M
100%
75.7
83.4
91.3
32.0
41.6
82.3
39.4
49.6
84.9
70.7
76.1
91.0
PTv3 [47]
124.8M
100%
77.6
85.0
92.0
35.3
46.0
83.4
48.2
61.6
87.0
73.4
78.9
91.7
MSC [46]
124.8M
100%
78.2
85.3
92.2
33.4
43.7
83.4
48.7
61.9
87.2
69.9
74.9
91.2
Sonata [49]
124.8M
100%
79.4
86.1
92.5
36.8
46.5
84,4
49.3
62.4
87.6
76.0
81.6
93.0
Concerto
124.8M
100%
80.7
87.4
93.1
39.2
50.2
85.0
50.7
63.3
87.9
77.4
85.0
93.2
(a) Full fine-tuning. evaluate Concerto using full fine-tuning, unlocking both encoder and decoder, and compare semantic segmentation mloU, mAcc, allAcc(96) (%) results across 4 benchmarks.
Param. Efficiency
Params
ScanNet [15]
ScanNei200 Val [35]
ScanNet++ Val [52]
S3DIS Area [1]
Methods
Leam.
Pet.
mioU mAce allAcc mIoU mAce allAcc mloU mAce allAcc mloU mAce allAcc
SparseUNet [14]
39.2M
100%
72.3
80.2
90.0
25.0
32.9
80.4
28.8
38.4
80.1
66.3
72,5
89.8
PC [50] (lin.)
<0.2M
<0.1%
5.6
9.7
50.0
0.5
0.9
40.3
1.8
3.1
46.4
11.4
18.6
52.3
CSC [19] (lin.)
<0.2M
<0.1%
12.6
18.1
64.2
1.3
2.1
53.0
2.8
4,5
53.6
24.4
32.0
66.4
MSC [46] (lin.)
<0.2M
<0.1%
14.1
20.3
62.9
1.5
2.5
53.6
6.6
61.3
27.9
35.5
71.1
PTv3 [47]
124.8M
100%
77.6
85.0
92.0
35.3
46.0
83,4
48.2
61.6
87.0
73.4
78.9
91.7
MSC [46] (lin.)
<0.2M
<0.2%
21.8
32.2
65.5
3.3
5.5
57.5
8.1
11.9
64.7
32.1
42.4
70.9
Sonata [49] (lin.)
<0.2M
<0.2%
72.5
83.1
89.7
29.3
41.6
81.2
38.9
52.8
84.3
72.3
81.2
90.9
Concerto (lin.)
<0.2M
<0.2%
77.3
86.6
91.7
37.4
49.5
83.3
45.6
60.5
86.5
73.5
81.3
90.9
Sonata [49] (dec.)
16.3M
13%
79.1
86.6
92.7
33.5
44.5
84.1
45.2
57.4
86.8
74.5
80.4
92.6
Concerto (dec.)
16.3M
13%
79.5
87.6
92.6
37.8
50.5
84.1
48.3
62.3
87.7
75.5
84.2
92.3
(b) Parameter efficiency. By using linear probing (lin.) and decoder probing (dec.), we compare semantic segmentation mloU, mAcc, allAcc (6) results across 4 benchmarks.
Data Efficiency
Limited Scenes (Pct.)
Limited Annotation (Pts.)
Methods
1%
5%
10%
20%
Full
20
50
100
200
Full
о PTv2 [45]
24.8
48.1
59.8
66.3
75.4
58.4
66.1
70.3
71.2
75.4
UparseUNet [14]
26.0
47.8
56.7
62.9
72.2
41.9
53.9
62.2
65.5
72.2
CSC [19]
28.9
49.8
59.4
64.6
73.8
55.5
60.5
65.9
68.2
73.8
MSC [46]
29.2
50.7
61.0
64.9
75.4
61.0
65.6
68.9
69.6
75.4
PTv3 [47]
25.8
48.9
61.0
67.0
77.2
60.1
67.9
71.4
72.7
77.2
PPT [48] (sup.)
31.1
52.6
63.3
68.2
78.2
62.4
69.1
74.3
75.5
78.2
Sonata [49] (lin.)
43.6
62.5
68.6
69.8
72.5
69.0
70.5
71.1
71.5
72.5
Sonata [49] (dec.)
44.5
64.1
69.8
72.5
79.1
69.8
75.0
76.3
79.1
Sonata [49] (f.t.)
45.3
65.7
72.4
72.8
79.4
70.5
73.6
76.0
77.0
79.4
Concerto (lin.)
48.2
69.1
73.6
75.0
77.3
73.9
75.2
76.2
76.3
77.3
Concerto (dec.)
44.6
67.9
73.7
74.6
79.5
72.6
74.6
76.7
77.6
79.5
Concerto (f.t.)
46.5
69.0
75.3
76.1
80.7
73.3
76.7
77.6
78.4
80.7
(c)
Data efficiency. We adopt the ScanNet Data Efficient [19] benchmark and compare the validation mloU((%
results of Concerto with previous methods three evaluation protocols.
Table 2: Semantic segmentation. We train Concerto ScanNet [15], ScanNet++ [52], Struc- tured3D [60], S3DIS [1]. ArkitScenes [7], and HM3D [33] datasets, utilizing ScanNet, ScanNer200, ScanNet++, and S3DIS to evaluate the model by linear probing, decoder probing, and full fine- tuning and ScanNet Data Efficient [19] to evaluate the data efficiency. The pre-training setting is the default, described in Tab. 6. More specific pre-training details are available in the Appendix.
tuning (SFT) in extreme data-lima limited scenarios (1%, 5% limited scenes, and 20-point annotation per scene). This observation aligns with findings in the image domain [57]. where linear probing out- performs full fine-tuning in the whole evaluation dataset becomes an out-of-disrribution situation. This significant emerging property reveals two key insights: more generalizable representations and more efficient adaptation potential. This could signal a potential shift toward Low-Rank Adaptation (LoRA) methods [20] for fine-tuning point cloud backbones. Detailed LoRA fine-tuning results provided in the Appendix.
situations. In our case, when training on limited data,

---

Scale
ScanNet Val
ScanNet200 Val
ScanNet++ Val
S3DIS Area
Model Size
mloU
mAcc
allAce
mloU
mAce
allAce
mloU
mAce
allAcc
mloU
mAce
allAce
16M(T)
67.7
78.5
87.4
24.9
34.4
79.3
33.7
45.9
82.7
65.2
73.6
88.6
39M(S)
76.6
86.6
91.5
34.4
46.3
83.1
43.1
57.6
86.2
71.3
80.4
90.1
108M(B)
77.3
86.6
91.7
37.4
49.5
83.3
45.6
60.5
86.5
73.5
81.3
90,9
207M(L)
77.5
86.6
92.1
38.6
49.8
83.9
46.3
59.9
86.7
73.7
81.4
91.1
Table Scaling Up. Model T, S, B is trained on point cloud datasets, while Model trained on point cloud datasets and an additional video dataset.
Lifted
Frame
DINOV2
Concerto
Figure 4: Video spatial perception. Concerto can be directly applied to video-lifted data (top row). The PCA visualizations (bottom two rows) illustrate that Concerto learns more fineggrained and semantically consistent features compared to DINOv2.
Video processing. As shown in the Fig. 4, the variation of Concerto demonstrates strong adaptability video-lifted data. We hypothesize that certain spatial-specific information is more effectively cap- tured in the lifted space. By leveraging the current feed-forward reconstruction method VGGT [43] reconstruct the point clouds from videos, we generate a diverse range of point cloud data. In- corporating these lifted point clouds into the pipeline allows Concerto to learn more generalizable representations, enhancing its ability for real-time video spatial perception. Moreover, by including video data into training datasets, we aim to extend the scaling ability of Concerto further. In Tab. 3, we provide linear probing results of different model sizes, where the large model variant trained with the additional video data demonstrates significant potential for further scaling, Details of this Concerto variation and the method of lifting are presented in the Appendix.
Language probing. Language develops from a funda- mental understanding of the physical world, which moti- vates many works on language alignment with 3D knowl- edge as [21, 24J. In Tab. 5, demonstrate Concerto's ability to formulate concepts similar to human language, paving the way for future exploration of alignment with text-based semantic spaces. With linear probing trained on the same datasets as the pretraining stage, we translate Concerto's 's representations to language space by aligning LSeg [27] image encoder output to our linear probing output. Without ground truth labels, Concerto achieves 44.56% mloU on ScanNet zero-shot segmentation. Although this lags behind the 77.3% mloU by supervised linear probing,
Language
ScanNet
Methods
mloU
mAce
allAcc
MSC [46]
12.42
18.55
59.89
Sonata [49]
41.71
61.29
78.86
Concerto
44.56
64.76
80.76
Table 5: Language probing.
7

---

Ins. Seg.
Params
ScanNet Val [15]
ScanNet200 Val [35]
ScanNet++ Val [52]
S3DIS Area 5
Methods
Learn.
Pet. PAP2mmAPPPa mAP
mAP2smAPso mAP
mAP2smmAPso mAP
mA P25mAPPP mAP
PTv3 [47]
124.8M
100%
77.5
61.7
40.9
40.1
33.2
23.1
46.3
39.6
25,4
55.7
49.4
37.8
MSC [46] (lin.)
<0.2M
<0.2%
13.3
5.3
2.3
2.3
1.0
0.4
4.8
2.6
1.3
19.0
13.0
9.7
Sonata [49] (lin.)
<0.2M
<0.2%
72.6
53.9
30.7
30.9
21.3
10,9
34,6
26.5
14.8
45.8
36,6
26.1
Concerto (lin.)
<0.2M
<0.2%
75.4
55.7
31.1
38.2
27.7
14.9
40.1
31.3
18.2
52.5
40.8
25.7
Sonata [49] (dec.)
16.3M
13%
76.8
62.8
40.8
40.8
33.3
22.8
39.4
33.5
22.7
63.7
57.1
45.1
Concerto (dec.)
16.3M
13%
81.1
64.2
42.7
41.8
34.4
24.0
42.2
35.3
23.4
66.8
58.1
45.1
MSC [46] (f.t))
124.8M
100%
78.4
62.9
41.1
40.5
33.8
23.4
56.3
50.5
38.1
PPT [48] (sup.)
124.8M
100%
78.9
63.5
42.1
40.8
34.1
24.0
57.5
51.2
39.7
Sonata [49] (f.t.)
124.8M
100%
79.2
63.9
42.4
42.1
35.6
25.4
43.3
36.5
24.6
63.8
57.4
45.5
Concerto (f.t.) 124.8M
100%
79,5
64.9
42.9
45.8
38.7
27.4
44.3
38.3
26.0
67.5
61.0
46.4
Table 4: Instance segmentation. Concerto demonstrates the strongest performance for instance segmentation across 4 datasets with all evaluation protocols.
cross
c.
c.
0.
image usage
0%
20%
50%
70%
100%
intra
0. c.
0. c.
c. s.
c. s.
lin.
70.9
73.5
75.2
75.4
75.6
lin.
60.7
75.6
31.6
74.7
dec.
72.0
78.6
66.0
78.3
dec.
77.4
77.4
77.8
78.4
78.6
(a) Criteria type. o.c.: online clustoring crite-
(b) Image usage. We control the image usage ratio to present the effectiveness of joint cross modal learning.
ria DINOv2 [28]] c.s.: cosine similarity.
criteria weight
1:2
2:2
3:2
img. aug.
w/o
vis. points
x1
x2
lin. 75.6
76.1
75.7
lin.
76.7
75.6
lin. 75.6
75.5
dec. 78.6
78.6
78.8
dec.
78.6
78.6
dec.
78.6
78.1
(c) Criteria weight. suitable weight ratio (cross:intra) needed balance intra- and cross-modal components.
Image augmentation. Weak image augmentation
(d)
Visible points. Fower visible points, better perfor-
shows a positive impact.
mance. x1: 65536 points.
upcust level
4
data scale
23k
model scale
lin. dec.
75.1
75.6
75.3
lin.
75.6
76.6
lin.
76.6
77.3
78.5
78.6
78.2
dec. 78.6
79,2
dec.
79.2
79.5
(i) Upeast level. Upcast level, processed as [49], refers to the concatenation level of features in cross-modal learning here.
(g) Data scale. 23k the de- fault ablation setting and the total dataset contains 40k.
(h) Model scale. Training on
40k. s: 39M backbone; b:
108M backbone.
Table 6: Ablation study. The default ablation setup trains on ScanNet [15] and Structured3d [60] with 39M PTv3 [47] model as in Sonata [49]. If not specified, other default settings are in the Appendix. For Tab. 6g and Tab. 6h, we scale the setup to match the model used in the main results. All of our designs are enabled by default. Default settings marked in blue
we expect that further language-conditione probing will yield comparable results, marking a signif- icant step toward bridging 3D spatial representations with text.
4.2 Ablation Study
In this section, we ablate intriguing properties for Concerto in Tab. 6 with the default setting in cap- tions, evaluating the ablation ScanNet semantic segmentation using linear and decoder probing. Joint cross-modal learning. In Tab. 1, we investigate the influence of joint cross-modal learning by comparing Concerto with strong baseline models DINOv2 and Sonata. Concerto outperforms both and surpasses their native feature concatenation. These results demonstrate that joint cross-modal learning does more than merely merge information from different modalities; it enables the model to learn richer emerging representations that were previously unattainable.
Criteria type. In Tab. 6a, we show that using cosine similarity as the loss function in the cross- modal joint embedding prediction component and cross-entropy-ii based online clustering loss from DINOv2 [28] in the self-distillation component facilitates joint 2D-3D self-supervised learning in latent space. This combination reduces strict constraints and minimizes conflicts between cross- modal and intra-modal learning, enabling smoother joint learning process for the two objectives. Image usage. Tab. 6b, we further investigate the effect of multisensory interactions by varying the input ratio of point clouds with images. The results show that even with a small ratio, such as 20%,
8

---

joint cross moossl learning is effective, leading to improvements in linear probing. When the image usage ratio reaches 50%, the linear probing result comparable to that with 100% image usage. while decoder probing continues to show potential for further improvement. These results suggest that shallow linear representations are easier to discover with a smaller proportion of images, while deeper representations, which require decoder probing, benefit from higher image usage ratio.
Criteria weight. In Tab. 6c, we observe that the criteria weight ratio between cross-modal and intra-modal components affects performance. Given the distinct objectives of intra-modaa and cross- modal learning, maintaining balanced loss-weight ratio is essential for optimal performance. The 2:2 ratio outperforms the others in linear probing.
Image augmentation. Data augmentations are crucial in self-supervised learning. As Sonata [49] already explores point cloud augmentations in self-distillation we focus on the cross-modal image augmentations here. Initially, we follow DINOv2 strong data augmentations. which results in a lower performance with linear probing mloU of 75.27%. However, when we apply less aggressive augmentations, the performance surpasses our default setting without image augmentations, as in Tab. 6d. The details of the image augmentations are provided in the Appendix. Since the image encoder here is frozen, does not benefit from data augmentations. Additionally, overly aggressive image augmentation may confuse the point encoder with excessive distortions. Thus, careful selec- tion of image augmentations is essential. Currently, our model does not apply image augmentations, and we plan to explore this in future updates.
Visible points. In Tab. 6e, we investigate the impact of visible points in the point cloud from image views. We hypothesize that while large amount of matched point-pixel pairs can offer more complete information, the smaller number of pairs forces the model to predict across modalities for the surrounding context. As the task becomes more challenging, the model is encouraged to dig deeper into semantics, leading to better performance. ablation, the performance is quite similar as the point numbers we selected are still quite small, compared with the number of image pixels and points in point clouds.
Upcast level. While Sonata [49] studies the influence of upcast levels on self-supervised learning representations, we present the performance of different cross-modal feature upcast levels in Tab. 6f. The model at upcast level 3 achieves the best performance, indicating that the upcast level 3 is close to the corresponding scales of the image and point cloud. Additionally, level 3 outperforms level 4 as level may retain too many low-level details that are not beneficial for joint embedding learning. Furthermore, level 3 surpasses level 2, as the use of Sonata's self-distillation technique at level 2 might introduce conflicts between intra-modal self-supervised learning and cross-modal joint learning of the same upcast level, ultimately leading to negative effects.
Data scale and model weight. Additionally, following the approach of Sonata [49]. we scale our datasets from 23k to 40k, resulting in a significant improvement shown in Tab. 6g. Likewise, align- ing our model size with that of Sonata (108M) further enhances the performance, as demonstrated in Tab. 6h. Larger datasets provide more diverse and comprehensive information, enabling the model to learn more general patterns. As the model size increases, it becomes better equipped to capture complex relationships. Combining both naturally leads to more generalized representation outputs.
5 Related Work
2D image self-supervised learning. Aimed utilizing oceans of unlabeled data, image self- supervised learning has seen significant progress [3, 12, 18, 28, 56]. These methods always focus on learning invariant representations through transformations or augmentations of the data. One of the most notable achievements in this field is DINOv2 [28], producing high-quality image represen- tations. Building on the success of DINOv2, Concerto extends its reliable 2D image representations to the cross-modal domain, incorporating both 2D image and 3D point cloud data for superior rep- resentations in the 3D domain.
3D point self-supervised learning. While self-supervise learning has made significant progress in the image domain, it is still in the starting stage of 3D point clouds. Building on the success of Sonata [49], we further extend the previous works [42, 46, 50] on unimodal self-supervised learning with scene-level data to joint 2D-3D self-supervised learning for superior representation extracting ability. Before Sonata [49]. most of the point self-super learning works suffer from geometry shortcuts due to the sparse and unordered nature of point clouds. Based on its predecessor, Con- certo includes 2D images in its pipeline: leveraging point clouds to predict image features from DI-

---

NOv2 [28] by cross modal joint embedding prediction and including 200k video-lifted point clouds by fred-forward reconstruction methods [43] into training datasets.
Spatial understanding with joint 2D-3D data. With the rapid advancements in 2D self-supervised learning and its remarkable performance, many methods for point cloud representation now incor- porate image features into their pipelines. Approaches such as lifting projections [16, 39], differ- entiable rendering [25, 54, 58], direct distillation [11, 13, 55, 62], attention-baa based feature fusion [59], and using text-aligned image encoders for open-vocabulay tasks [22, 30, 40, 44] aim to in- corporating image features in 3D learning. However, these methods primarily focus on imitating image features in point cloud representations, often overlooking the full potential of multi-modal in- teraction. Recently, Locate3D [2] seeks to develop generalizable representations beyond 2D image features but still relies on 2D image features during inference with complicated pipeline. In con- trast, Concerto utilizes joint 2D-3D embedding prediction during training, resulting unified and rich representations beyond individual 2D or 3D features and their simple combination, enabling superior representations in inference with only point clouds.
6 Conclusion and Discussion
In this work, we present Concerto, achieving SOTA performance across multiple benchmarks. Ad- ditionally, we present the variation of Concerto for video spatial perception and the interlude of Concerto, which explores potential future alignment with text spaces. Concerto holds great promise for joint multi-modal self-supervised learning. Currently, it excels as a joint 2D-3D self-supervised learning model in the 3D domain, delivering superior performance spatial representation learning. However, our goal extends beyond this. We discuss limitations and future works as follows:
representation learning. Our current training recipe freezes the image en- coder, treating it as a static feature extractor. A compelling future direction is to unfreeze both the image and point cloud encoders for joint native multi-modal representation pre-training. This approach would enable the two modalities to mutually enhance one another during learn- ing, fostering the development of a more robust shared representation space.
Native
Deep semantic grounding of language in point clouds. While our work Concerto employs linear probing as an effective metric for evaluating feature quality, this method deliberately promotes shallow alignment between point clouds and language to avoid the influence of post-training part on the evaluation performance of pretraining features. For real-world applications, this is insufficient just have shallow language alignment with specific key terms. A critical next step is to develop architectures and training objectives that move beyond simple feature align- ment towards deep semantic grounding. The goal is to enable the learned representations to comprehend and respond to nuanced, indirect, or compositional linguistic descriptions, which remains significant open challenge.
Unified self-supervised learning paradigm for diverse point cloud domains. Self-supervised learning for point clouds has historically been fragmented, with models tailored to specific do- mains (e.g., indoor, outdoor, object-level) to handle their distinct characteristics of scale and density. We believe that unified pre-training paradigm trained on the data from different domains can produce more powerful and generalizable representations.br incorporating var- ied data sources like lidar point clouds, video-lifted point clouds, object-centric point clouds, and dynamic egocentric point clouds, single self-supervised model can learn features that are robust domain shifts. This enhanced generalization is expected to sigmificantly boost performance on a wide array of downstream tasks, even those confined to a single domain.
Acknowledgments. The research presented in this paper was supported by the National Natural Science Foundation of China (No. 62422606, 62401484).
10

---

References
[1] Iro Armeni, Ozan Sener, Amir R. Zamir, Helen Jiang. Ioannis Brilakis, Martin Fischer, and Silvio Savarese. 3d semantic parsing of large-scale indoor spaces. In CVPR, 2016. 5, 6, 8, 22, 23
Sergio Arnaud, Paul McVay, Ada Martin, Arjun Majumdar, Krishna Murthy Jatavallabhula, Phillip Thomas, Ruslan Partsey, Daniel Dugas, Abha Gejji. Alexander Sax, et Locate 3d: Real-world ob- ject localization via supervsd learning in 3d. arXiv:2504. 2112250441411,, 2025. 10
[3] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Self-supervised learning from images with a joint-embedding predictive architecture, In CVPR, 2023. 9
[4] Mahmoud Assran, Quentin Duval, Ishan Misra, Piotr Bojanowski, Pascal Vincent, Michael Rabbat, Yann LeCun, and Nicolas Ballas. Sule--upe learning from images with a joint-embedding predictive architecture. In CVPR, 2023.
[5] Armen Avetisyan, Christopher Xie, Henry Howard-Jenkins, Tsun-Yi Yang. Samir Aroudj, Suvam Patra, Fuyang Zhang, Duncan Frost, Luke Holland, Camphell Orme, al. Scenescript: Reconstructing scenes with an autoregressive structured language model. In ECCV, 2024. 2
[6] Lawrence W Barsalou. Grounded cognition. Annual Review of Psychology, 2008. 3,3
[7] Gilad Baruch, Zhuoyuan Chen, Afshin Dehghan, Tal Dimry, Yuri Feigin, Peter Fu, Thomas Gebauer, Brandon Joffe, Daniel Kurz, Arik Schwartz, and Elad Shulman. ARKitscenes diverse real-world dataset for 3d indoor scene understanding using mobile RGB-d data. NewrlPSW, 2021. 2, 6, 22,
[8] Holger Caesar, Varun Bankiti, Alex Lang, Sourabh Vora, Venice Erin Liong, Qiang Xu, Anush Krish- nan, Yu Pan, Giancarlo Baldan, and Oscar Beijbom. nuscenes: multimodal dataset for autonomous driving. In CVPR, 2020. 2
[9] Mathilde Caron, Ishan Misra, Julien Mairal, Priya Goyal, Piotr Bojanowski, and Armand Joulin. Unsu- pervised learning of visual features by contrasting cluster assignments. In Neur/PS, 2020. 4
[10] Mathilde Caron, Hugo Touvron, Ishan Misra, Hervé Jégou, Julien Mairal, Piotr Bojanowski, and Armand Joulin. Emerging properties in self-supervised vision transformers. In CVPR, 2021. 4
[11] Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, and Wenping Wang. Clip2scene: Towards label-efficient 3d scene understanding by clip. In CVPR, 2023.
10
[12] Ting Chen, Simon Korbllihh, Mohammad Norouzi, and Geoffrey Hinton. A simple framework for con- trastive learning of visual representations. In ICMI,, 2020. 2,9
[13] Zhimin Chen, Longlong Jing, Yingwei Li, and Bing Li. Bridging the domain gap: Self-supervised 3d scene understanding with foundation models. In NeurlPS, 2023. 10
[14] Christopher Choy, Young Gwak. and Silvio Savarese. 4d spatio-temporal convnets: Minkowski con- volutional neural networks. In CVPR, 2019.
[15] Angela Dai, Angel X. Chang. Manolis Savva, Maciej Halber, Thomas Funkhouser, and Matthias Nießnnr. Scannet: Richly-annotated 3d reconstructions indoor scenes. CVPR, 2017. 3, 5, 8, 22, 23
[16] Zhiwen Fan, Jian Zhang, Wenyan Cong, Peihao Wang, Renjie Li, Kairun Wen, Shijie Zhou, Achuta Kadambi, Zhangyang Wang, Danfei Xu, et Large spatial model: End-to-end unposed images to semantic 3d. Advances in neural information processing systems, 37:40212-40229, 2024. 10
[17] Jiayuan Gu, Fanbo Xiang, Xuanlin Li, Zhan Ling, Xiqiang Liu, Tongzhou Mu, Yihe Tang, Stone Tao, Xinyue Wei, Yunchao Yao, Xiaodi Yuan, Pengwei Xie, Zhiao Huang, Rui Chen, and Hao Su. Maniskil12: A unified benchmark for generalizable manipulation skills. In ICLR. 2023. 2
[18] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsuper- vised visual representation learning. In CVPR, 2020. 2, 4,
[19] Ji Hou, Benjamin Graham, Matthias Nießner, and Saining Xie. Exploring data-cfficient 3d scene under- standing with contrastive seene contexts. In CVPR, 2021. 5, 6
[20] Edward J Hu, Yelong Shen, Phillip Wallis. Zeyuan Allen-Zhu, Yuanzhi Li, Shean Wang, Lu Wang. Weizhu Chen, et al. I Lora: I Low-rank adaptation of large language models. In ICLR, 2022. 6
11

---

[21] Zhening Huang, Xiaoyang Wu, Xi Chen, Hengshuang Zhao, Lei Zhu, and Joan Lasenby. Openins3d: Snap and lookup for 3d open-vocabulary instance segmentation. European Conference Computer Vision, 2024."
[22] Ayush Jain, Alexander Swerdlow, Yuzhou Wang. Sergio Arnaud, Ada Martin, Alexander Sax. Franziska Mcier, and Katerina Fragkiadaki. Unifying 2d and 3d vision-language understanding. arXiv:2503. 10745, 2025. 10
[23] Stephen James, Zicong Ma, David Rovick Arrojo, and Andrew J. Davison. Rlbench: The robot learning benchmark learning environment. RAI, 2020. 2
[24] Krishna Murthy Jatavallabhula, Alihusein Kuwajerwala, Qiao Gu, Mohd Omama, Tao Chen, Shuang Li, Ganesh Iyer, Soroush Saryazdi, Nikhil Keetha, Ayush Tewari, Joshua B. Tenenbaum. Celso Miguel de Melo, Madhava Krishna, Liam Paull, Florian Shkurti, and Antonio Torralba. Conceptfusion: Open-set multimodal 3d mapping. Roboties: Science and Systems (RSS), 2023. 7
[25] Sosuke Kobayashi, Eiichi Matsumoto, and Vincent Sitzmann. Decomposing nerf for editing via feature field distillation. In NeurlPS, 2022. 10
[26] Yann LeCun. A path towards autonomous machine intelligence version 0.9.2. OpenReview, 2022. 2. 4
[27] Boyi Li, Kilian Q Weinberger, Serge Belongie, Vladlen Koltun, and Rene Ranftl. Language-driven se- mantic segmentation. In ICLR, 2022. 2, 2.
[28] Maxime Oquah, Timothée Darcet, Theo Moutakanni, Huy V. Vo, Mare Szafraniec, Vasil Khalidov, Pierre Fernandez. Daniel Haziza, Francisco Massa, Alaaeldin El-Nouby. Russell Howes. Po- Yao Huang, Hu Xu, Vasu Sharma, Shang-Waat Wen Li, Wojciech Galuba, Mike Rabbat, Mido Assran, Nicolas Ballas, Gabriel Syn- naeve, Ishan Misra, Herve Jegou. Julien Mairal, Patrick Labalut. Armand Joulin, and Piotr Bojanowski. Learning robust visual features without supervision. TMLR, 2024. 1, 2, 3, 4, 8, 9, 10, 22, 24
Dinov2:
[29] Yatian Pang. Wenxiao Wang. Francis EH Tay, Wei Liu, Yonghong Tian, and Li Yuan. Masked autoen- coders for point cloud self-supervised learning. In ECCV, 2022. 2
[30] Songyou Peng, Kyle Genova, Chiyu Jiang, Andrea Tagliasacchi, Mare Pollefeys, Thomas Funkhouser, et al. Openscene: 3d scene understanding with open vocabularies. In CVPR, 2023. 10
[31] Xavier Puig, Eric Undersander, Andrew Szot, Mikacl Dallaire Cote, Tsung- Yen Yang, Ruslan Partsey, Ruta Desai, Alexander William Clegg, Michal Hlavac, So Yeon Min, et al. Habitat 3.0: A co-habitat for humans, avatars and robots. arXiv preprint arXiv:2310.13722 2023. 22
Sen
[32] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh. Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin. Jack Clark, al. Learning transferable visual models from natural language supervision. In JCML, 2021.
[33] Santhosh Kumar Ramakrishnan, Aaron Gokaslan, Erik Wijmans, Oleksandr Maksymets, Alexander Clegg, John M Turner, Eric Undersander. Wojeiech Galuba, Andrew Westbury, Angel X Chang, Manolis Savva, Yili Zhao, and Dhruv Batra. Habitat- 3d dataset (HM3d); 1000 large-scale 3d environ- ments for embodied AI. In NewrIPS, 2021. 2, 6, 22, 23, 24
[34] Mike Ranzinger, Greg Heinrich, Jan Kautz, and Pavlo Molchanov. Am-radio: Agglomerative vision foundation model reduce all domains into one. In Proceedings of the IEEESCVF Conference on Computer
Vision and Pattern Recognition, pages 12490-12500, 2024. 24
[35] David Rozenberszki, Or Litany, and Angela Dai. manguage-grounder indoor 3d semantic segmentation in the wild. In ECCV, 2022. 3, 5, 6, 8
[36] Ladan Shams and Aaron R Seitz. Benefits of multisensory learning. Trends in cognitive sciences, 2008.
[37] Julian Straub, Daniel DeTone, Tianwei Shen, Nan Yang, Chris Sweeney, and Richard Newcombe, Efm3d: A benchmark for measuring progress towards 3d egocentric foundation models. arXiv:2406. 10224, 2024.
[38] Pei Sun, Henrik Kretzschmar, Xerxes Dotiwalla, Aurelien Chouard, Vijaysai Patnaik, Paul Tsui, James Guo, Yin Zhou, Yuning Chai, Benjamin Caine, et al. Scalability in perception for autonomous driving: Waymo open dataset. In CVPR, 2020. 2
[39] Mukund Varma T., Peihao Wang. Zhiwen Fan, Zhangyang Wang, Hao Su, and Ravi Ramamoorthi. Lift3:: Zero-shot lifting of any 2d vision model to 3d. In CVPR, 2024. 10
12

---

[40] Ayça Takmaz, Elisabetta Fedele. Robert W. Sumner. Mare Pollefeys. Federico Tombari. and Francis Engelmann. Openmask3d: Open-vocabulary 3d instance segmentation. In NewrlPS, 2023. 10
[41] Michael Tschannen, Alexey Gritsenko, Xiao Wang. Muhammad Ferjad Naeem, Ibrahim Alabdulmohsin, Nikhil Parthasarathy, Talfan Evans, Lucas Beyer, Ye Xia, Basil Mustafa, al. Siglip 2: Multilingual vision-language encoders with improved semantic understanding. localization, and dense features. arXiv preprint 25v: 14786, 2025. 24
[42] Chengyao Wang. Li Jiang, Xiaoyang Wu, Zhuotao Tian, Bohao Peng, Hengshuang Zhao, and Jiaya Jia. Groupcontrast: Semantic-aware self-supervised representation learning for 3d understanding. In CVPR, 2024. 9
[43] Jianyuan Wang, Minghao Chen, Nikita Karaev, Andrea Vedaldi, Christian Rupprecht, and David Novotny,
Vggt Vggt: Visual geometry grounded transformer. In CVPR, 2025. 2, 7, 10, 22, 23
[44] Pengfei Wang, Yuxi Wang, Shuai Li, Zhaoxiang Zhang. Zhen Lei, and Lei Zhang. Open vocabulary 3d
scene understanding via geometry guided self-distillation. In ECCV, 2024. 10
[45] Xiaoyang Wu, Yixing Lao, Li Jiang, Xihui Liu. and Hengshuang Zhao. Point transformer v2: Grouped vector attention and partition- bassed pooling. In NeurIPS, 2022. 6
[46] Xiaoyang Wu, Xin Wen, Xihui Liu, and Hengshuang Zhao. Masked scene contrast: A scalable framework for unsupervised 3d representation learning. In CVPR, 2023. 2, 6, 7, 8,9
[47] Xiaoyang Wu, Li Jiang. Peng- Wang. Zhijian Liu, Xihui Liu, Yu Qiao, Wanli Ouyang. Tong He, and Hengshuang Zhao. Point transformer v3: Simpler, faster. stronger. In CVPR, 2024. 2. 5, 6. 8
[48] Xiaoyang Wu, Zhuotao Tian, Xin Wen, Bohao Peng, Xihui Liu, Kaicheng Yu, and Hengshuang Zhao. Towards large-scale 3d representation learning with multi-datasel point prompt training. In CVPR, 2024. 6,
[49] Xiaoyang Wu, Daniel DeTone, Duncan Frost, Tianwei Shen, Chris Xie, Nan Yang. Jakob Engel. Richard Newcombe, Hengshuang Zhao, and Julian Straub. Sonata: Self-supervised learning of reliable point representations. In CVPR, 2025. 1, 2, 3, 4, 5, 6, 7, 8, 21
[50] Saining Xie, Jiatao Gu, Demi Guo, Charles R Qi, Leonidas Guibas, and Or Litany. Pointcontrast: Unsu- pervised pre-training for 3d point cloud understanding. In ECCV, 2020. 2, 6,99
[51] Zhenda Xie, Zheng Zhang, Yue Cao. Yutong Lin, Jianmin Bao, Zhuliang Yao, Qi Dai, and Han Hu. Simmim: A simple framework for masked image modeling. In CVPR, 2022. 2
[52] Chandan Yeshwanth, Yueh-Cheng Liu, Matthias Nießner, and Angela Dai. Scannet++: A high-fidelity dataset of 3d indoor scenes. In ICCV, 2023. 2, 6. 22, 23
[53] Xumin Yu, Lulu Tang, Yongming Rao, Tiejun Huang, Jie Zhou, and Jiwen Lu. Point-bert: Pre-training 3d point cloud transformers with masked point modeling. In CVPR, 2022.
[54] Yuanwen Yue, Anurag Das, Francis Engelmann, Siyu Tang, and Jan Eric Lenssen. Improving 2d feature representations by 3d-aware fine- munitining. In ECCV, 2024. 10
[55] Karim Abou Zeid, Kadir Yilmaz, Daan de Geus, Alexander Hermans, David Adrian, Timm Lin- der, and Bastian Leibe. Dino in the room: Leveraging 2d foundation models for 3d segmentation. arXiv:2503. 18944, 2025. 10
[56] Hao Zhang, Feng Li, Shilong Liu, Lei Zhang, Hang Su, Jun Zhu, Lionel M. Ni, and Heung- Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. arXiv:2203.03605. 2022. 2.9
[57] Michael Zhang. Aditi Raghunathan Wang, Shiori Sagawa, Tatsunori B Hashimoto, and Percy Liang. Fine- tuning can distort pretrained features and underperform nut-of-distributee In ICLR, 2022. 6
[58] Xiaoshuai Zhang, Zhicheng Wang, Howard Zhou, Soham Ghosh, Danushen Gnanapragasam, Varun Jam- pani, Hao Su, and Leonidas J. Guibas. CONDENSE: consistent 2d/3d pre-training for dense and sparse features from multi-view images. In ECCV, 2024. 10
[59] Yifan Zhang, Junhui Hou, Siyu Ren, Jinjian Wu, Yixuan Yuan, and Guangming Shi. Self-supervised learning of lidar 3d point clouds via 2d-3d neural calibration. JEEE Transactions on Pattern Analysis and Machine Intelligence, page 117, 2025. 10
13

---

[60] Jia Zheng, Junfei Zhang, Jing Li, Rui Tang, Shenghua Gao, and Zihan Zhou. Structured3d: A large photo-realistic dataset for structured 3d modeling. In ECCV, 2020. 6, 8, 22, 23
[61] Tinghui Zhou, Richard Tucker, John Flynn, Graham Fyffe, and Noah Snavely. Stereo magnification: Learning view synthesis using multiplane images. In SIGGRAPH, 2018. 2, 22, 23
[62] Haoyi Zhu, Honghui Yang, Xiaoyang Wu, Di Huang, Sha Zhang, Xianglong He, Tong He, Hengshuang Zhao, Chunhua Shen, Yu Qiao, et al. Ponderv2: Pave the way for 3d foundataion model with universal pre braitraining paradigm, arXiv:2310.00 2023,
14

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? Answer: [Yes]
Justification: By imitating the human cognition learning process, Concerto combines joint
cross-modal embedding learning and intra-modal self-distillation achieve SOTA perfor-
mance
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made
the paper.
The abstract and/or introduction should clearly state the claims made, including the contri- butions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]
Justification: Although Concerto exceeds previous SOTA results, it does not fully leverage the joint 2D-3D self-supervised learning. In the future, we aim to bridge 2D-3D spatial space with text for more unified model.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to vi- olations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low images taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers grounds for rejection, worse outcome might be that reviewers discover lim- itations that aren't 1t acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play important role in developing norms that preserve the integrity of the community. Reviewers will be specifi- cally instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
Answer: [NA]
Justification: Concerto is based on large-scale training and designs on the joint 2D-3D self- supervised learning network, which doesn't contain theoretical results.
Guidelines:
15

---

The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced in the statement of any theorems. The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced. 4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Answer: [Yes]
Justification: Concerto will release the code and model for the public to use. And the detailed information is in the paper and its appendix.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
the contribution a dataset and/or model, the authors should describe the steps taken to make their results reproducible verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be nec- essary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (c.g.. g. in the case of a large lan- guage model), releasing of model checkpoint, other means that are appropriate the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (c.g., large language model), then there should either be way to access this model for reproducing the results or a way reproduce the model (e.g., with an open- spen-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way ge.g., to registered users), but should be possible for other researchers to have some path to reproducing verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material? Answer: [Yes]
Justification: Concerto is based on the current welcome codebase Pointcept, which contains code and required datasets in it. And we provide with the code and data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https:/ /ni http://iissscooo prips.cc. cc/
public/guides// wetpuiddeeeeeeeuuuuaeeeee www.uumisssioooooiiii... for more details.
16

---

While we encourage the release of code and data, we understand that this might not be possible, so No an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https:
/ /niis, oc cttppcccpppcciiccc publicclcc// Milbiic/ae /ttlc///iieeeeeeeee//// CoddSSumiis ssi ceiioinolyyyy for more details.
The authors should provide instructions on data access and preparation, including how to
access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts reproduce all experimental results for the new pro-
posed method and baselines. If only a subset of experiments are reproducible, they should state which ones omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized ver- sions (if applicable).
Providing as much information as possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyperparam- eters, how they were chosen, type of optimizer, etc.) necessary to understand the results? Answer: [Yes]
Justification: We provide the detailed version in the code, appendix, and supplemental material. Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should presented the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: The training of the model is computationally expensive. Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results accompanied by bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors). It should be clear whether the eтror bar is the standard deviation or the standard error of the mean.
It is OK to report I-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should careful not to show in tables or figures symmetric error bars that would yield results that are out of range (c.g. negative error rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the ex- periments?
Answer: [Yes]
Justification: The detailed information is listed in the appendix, supplemental material, and the code.
17

---

Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or
cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual ex- perimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper). 9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS c/ /Eth caGu i de ines?
Code of Ethics https:/ /neur pss cc/ publ cc/publ
Answer: [Yes]
Justification: the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there is a special considera- tion due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper diseuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the positive and negative impacts of society in the conclusion and the Appendix.
Guidelines:
The answer NA means that there is no societal impact of the work performed. If the authors answer NA or No, they should explain why their work has societal impact or why the paper does not address socictal impact.
Examples of negative societal impacts include potential malicious unintended uses (e.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g.. de- ployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point out. For example, is legitimate point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, not needed to point that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mecha- nisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Question: Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Answer: [NA]
Justification: Our paper has no such risks.
Guidelines:
The answer NA means that the paper poses no such risks.
18

---

Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We correctly cite the paper and other assets. And we properly respect the license.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset is used and, if possible, include a URL.
The name of the license (e.g.. CC-BY 4.0) should be included for each asset.
For scraped data from particular source (e.g., website), the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswi thoode. com da tasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that re-packaged, both the original license and the license of the derived asset (if has changed) should be provided. If this information is not available online, the authors are encouraged to reach out to the asset's creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer:
[NA]
Justification: the paper does not release assets. Our study solely relies existing, publicly available resources, and thus, no additional documentation for new assets is necessary.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/model part of their sub- missions via structured templates. This includes details about training, license, limitations,
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsoureing experiments and research with human subjects, does the paper include the full text of instructions given participants and screenshots, if applicable, well as details about compensation (if any)?
Answer: [NA]
Justification: There are no crowdsourcing experiments in our paper.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Including this information in the supplemental material fine, but the main contribution of the paper involves human subjects, then much detail possible should be included in the main paper.
According to the NeurlPS Code of Ethies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
19

---

15. Institutional review board (IRB) approvals or equivalent for research with human sub- jects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: the paper does not involve crowdsourcing nor research with human subjects. Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with
human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent)
may be required for any human subjects research. If you obtained IRB approval, you should clearly state this the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guide- lines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if is an important, original, non- standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.
Answer: [NA]
Justification: The core method development in this research does not involve LLMs any important, original, or non-standard components.
Guidelines:
The answer NA means that the core method development this research does not involve LLMs any important, original, non-standard components.
Please refer to our LLM policy (attpa: //n/urirssp.... ww//onneeeencs///000000 LLM) for what should should not be described.
20

---

Figure 5: Qualitative visualization. Concerto performs well across different point cloud inputs: complete scene (top two rows) and an incomplete scene (bottom two rows).
Appendix
Concerto is superior spatial representation point encoder capable of handling wide range of scene types, including those with varying completeness Fig. 5, video-lifted point clouds in Fig. 6, and the large scene in Fig. 7. Here, we further present the detailed implementation and results.
A Additional Implementation
We adopt the detailed parameters from Sonata [49] for intra- moora- ceff-distillation and refer readers to the original Sonata paper for an in-depth description of its implementation. In this section, provide a thorough explanation of the implementation for cross-modal joint embedding prediction.
A.1 Combination of Intra-Modal and Cross-Modal Learning
As Sonata, we use local views, masked views, and global views, with the first global view serv- ing the principal view. For cross- cross-modal joint embedding prediction, utilize the representations from the first masked view (based on the principal view) to predict the corresponding image repre- sentations. The cross-modal cosine similarity loss is computed upeast level 3, while the online clustering cross-entropy loss for intra-modal self-distillation is calculated at upcast level 2.
A.2 Correspondence Between Pixels and Points
To establish reliable 3D point to 2D pixel correspondences across camera views, employ two- step approach: 3D-to-2D projection followed by depth-based visibility verification.
Let p- (X,Y, Z) denote a 3D point in world coordinates. Each camera c is defined by intrinsic matrix K and extrinsic matrix R t). The standard pinhole camera model projects the 3D point p to 2D pixel coordinates (x, y) and projected depth dproj:
Y Z
dproj
y
K
R
To account for occlusions, we perform visibility check comparing dproj with the depth value d. = De(a, retrieved from camera c's depth map D. at the projected pixel coordinate (æ, v).
21

---

Dutaset
Source
Train
Val
Test
All
ScanNet [15]
real
26,428
7.354
2.877
36,659
ScanNel++ [52]
49,315
1,583
1,208
12,
S3DIS [1]
10,977
3,668
14,645
ArkitSeencs [7]
72,481
9,786
82,267
HM3D [33]
real
64,936
8,240
73,176
Structured3D [60]
synthesis
66,160
6,722
6,396
78,278
REIOK [61]
real
166,680
0
18,464
185,144
Concerto (ours)
mixed
455,972
37,353
28,945
522,270
Table 7: Image Data Source Collection.
Dataset
Source
Train
Test
ScanNet [15]
1,201
312
100
1,613
ScanNet++ [52)
SADIS [1]
real
204
68
ArkitScenes [7]
real
4,498
549
5,047
HM3D [33]
real
8,117
1,030
0
9,147
Structure33D [60]
synthesis
16,635
1,722
1,648
20,005
REIOK [61]
real 41,670
0
4,612
46,282
Concerto (ours)
mixed
74,894
3,785
6,459
85,138
Table 8: Point Cloud Data Source Collection.
The point is considered visible if:
d. dprojl Cdepth:
where Edepth is 0.01 in our experiments. Additionally, the correspondence rejected if (æx, y) (z,
falls outside image bounds or (z y) contains invalid depth. This visibility check establishes mapping between 3D points and corresponding 2D pixels, enabling direct correspondence between 3D points and ViT patches for cross cross-model joint embedding prediction mechanisms. Depending on the dataset, the depth map D is obtained in different ways:
RGBD datasets. Depth maps are directly available as the depth channel of RGBD images, such as Structured3D [60].
Known ground truth mesh. For datasets like ScanNet [15], ScanNet++ [52], S3DIS [1], and ARKitScenes [7], depth maps are rendered from the ground truth 3D mesh using camera parameters.
Pixel-aligned point clouds. For video-lifted point clouds (c.g.. using VGGT [43] on RealEstate 10K [61]), per-view depth maps D; are generated alongside point clouds P,, A point p E P, from camera can be visible from camera
passes the visibility check.
For HM3D [33], which does not provide the raw images, we leverage Habitat-Sin [31] to simulate the scenes. For each navigatable room, we capture four images around the room with random initial camera orientations. The angular difference between consecutive images is 90 degrees. We record the camera parameters compute the correspondence between points and pixels, described pre- viously. The total collections of our training data are shown in Tab. 7 and Tab. 8.
A.3 Image Augmentations
We implement the same point cloud augmentations Sonata. For image augmentations, we initially adopt the process from DINOv2 [28], excluding geometric augmentations to simplify the alignment between pixels and points. Specifically, we apply color jittering, random grayscale, and Gaussian blur to the images, consistent with the settings used in DINOv2. This results in slight drop in the mloU on ScanNet semantic segmentation to 75.27%, compared to using the original images. Consequently, we continue to explore more suitable image augmentations. In the ablation study, we apply random color jittering, with the same intensity as the point cloud augmentations, along with Gaussian blur. This weaker augmentation improves Concerto's 's performance, which is expected since the image encoder is currently frozen. Stronger augmentations may yield better results once both the image and point branches are unlocked for joint learning.
22

---

Single View
Multi Views
Lifted Point Cloud
RGB
DINOv2
Concerto
Figure 6: Video perception. Concerto can be applied to single-view (top row) and multi-view video- lifted data (bottom three rows). We visualize the PCA of one video in RE10K [61]. In the multi-view setting, the representations from all the frames are computed together for consistency.
A.4 Experimental Setting
Software and hardware environment.
CUDA version: 12.4
PyTorch version: 2.4.1
Python version: 3.10.15
GPU: Nvidia H20 x 16 for pretraining: Nvidia H20 x for evaluation.
CPU: x 360 for pretraining; x 180 for evaluation.
Memory: 3600GB for pretraining: 1800GB for evaluation.
Time: 85h for pretraining of base model without video data.
Data license. We use the open-soures datasets ScanNet [15], ScanNet++ [52]. S3DIS [1], Struc- tured3D [60]. ARKitScenes [7], Habitat Matterport3D [33] and RealEstate 10K [61] latest ver- sions. S3DIS, Structured3D, ScanNet, and ScanNet++ have custom licenses. RealFistate 10K is licensed by Google L.L under a Creative Commons Attribution 4.0 International License. ARK- itScenes is licensed by Apple Inc. HM3D is licensed by Matterport.
Training details. For pretraining, we we leverage all train, val, and test splits train the self-supervised model. For evaluation with linear probing, decoder probing, and full fine-tuning, we train on the train split and test on the val split of ScanNet, ScanNet++, ScanNet200 and Area 5 of S3DIS. We use AdamW V as the optimizer, and cosine annealing policy the scheduler. The learning rate is adjusted with the encoder depth. and the max one 0.004. The pretraining epoch is 100. For cross cross-modal joint embedding prediction, we set DINOv2 image encoder input resolution 518x518.
B Additional Results
Concerto with Video-Lifted Point Clouds
We utilize the current feed-forward reconstruction model VGGT [43] to lift RealEstate [61] video data point clouds. Based on camera poses, we heuristically select video clips with larger camera pose transforms in comparison and abandon those with smaller camera pose transforms. With these video clips, we can build video dataset with more completed scenes. In Fig. 6, we utilize Concerto deal with single-view lifted data and multi-view lifted data. The visualizations
23

---

Raw
Chair
Sink
PCA
Sofa
Bed
Figure 7: Language locate. We visualize the PCA of a large house scene from HM3D [33] along with the heatmap of zero-shot language-based object localization results. The upper-left part of the scene shows detailed local information. Given specific words, Concerto with text-aligned linear probing successfully locates objects zero-shot setting.
Model
ScanNet Val
ScanNec200 Val
ScamNet++ Val
S3DIS Area 5
img. enc.
mloU
mAcc
allAcc
mloU
mAcc
allAcc
mloU
mAce
allAcc
mloU
mAce
allAcc
DINOv2 (lin.)
77.3
86.6
91.7
37.4
49.5
83.3
45.7
60.5
86.5
73.5
81.3
90.9
SigLIP2 (lin.)
76.3
86.0
91.4
36.7
48.9
82.7
45,8
61.4
86.8
72.3
79.4
91.0
RADIO (lin.)
73.5
84.0
31.0
42.3
81.8
42.7
$7.2
85.3
72.9
80.5
90.9
DINOv2 (dec.)
79.5
87.6
92.6
37.8
50.5
84.1
48.3
62.3
87.7
75.5
84.2
92.3
SigLIP2 (dec.)
78.8
87.0
92.4
37.5
47.7
83.7
46.8
58.1
87.0
73.6
79.8
91.3
RADIO (dec.)
77.9
85.7
92.3
33.9
44.6
83.4
44.9
56.5
86.2
74.8
81.2
92.2
DINOv2 (full.)
80.7
87.4
93.1
39.2
50.2
85.0
50.7
63.3
87.9
77.4
85.0
93.2
SigLIP2 (full.)
79.7
86.9
92.7
38.4
49.9
83,9
50.0
62.0
88.2
75.0
80.2
92.5
RADIO (full.)
79.6
86.6
92.7
36.1
46.9
83.8
48.4
60.6
88.2
75.1
80.5
92.8
Table 9: Segmentic segmentation of Concerto with different image encoders.
Concerto with
DINOv2 based on self-distillation has the best performance in general.
show that Concerto adapts well to these two situations, suggesting that Concerto cannot only be applied to the offline video reconstruction but also to the single-view forward situation.
B.2 Concerto with Language Probing
We leverage simple linear layer to translate the representations from Concerto to CLIP's text space. During training, force the linear probing output to align with the LSeg [27] image encoder's output, which does not need the ground truth labels to supervise. In the aligning process, we do not use masks and crop augmentations. The visualization results are shown in Fig. 7.
B.3 Results with Different 2D Encoder
In this section, we compare the performance of different strong image encoders: DINOv2 [28], SigLIPv2 [41], and RADIO [34]. We adopt DINOv2 L. version with resolution of 518 x188, IigLIPv2 So400m version with a patch size of 16 and resolution 512x512, and RADIOv2.5 L ver- sion with a resolution of 768 x 768. For each model, we pretrain variant of Concerto 40k data,
24

---

Data Efficiency
Params
Limited Scenes (Pct.)
Limited Annotation (Pts.)
Methods
Learn.
Pet.
1%
5%
10%
20%
100%
20
50
100
200
Full
Concerto (lin.)
0.02M
0.02%
48.2
69.1
73.6
75.0
77.3
73.9
75.2
76.2
76.3
77.3
Concerto (dce.)
16,3M
13.1%
44.6
67.9
73.7
74.6
79.5
72.6
74.6
76.7
77.6
79.5
Concerto (full.)
124,8M
100.0%
46.5
69.0
75,3
76,1
80.7
73.3
76.7
77.6
78.4
80.7
Concerto (lora)
0.3M
0.2%
48.4
70.2
74.9
76.8
79.8
75.1
77.2
78.3
78.7
79.8
Table 10: Parameter Efficiency with LoRA. Concerto with LoRA significantly improves the per- formance with a minimal number of learnable parameters, highlighting the reliability of pretrained Concerto representations and the effectiveness of LoRA fine-tuning.
LoRA
Params
ScanNet Val
ScanNet200 Val
ScanNet++ Val
S3DIS Area
Methods
Leamn
Pet.
mIoU
mAce allAce
mIoU mAce allAce mIoU mAcc allAce
mloU
mAce allAcc
Concerto (lin.)
<0.2M
<0.2%
77.3
86.6
91.7
37.4
49.5
83,3
45.6
60.5
86.5
73.5
81.3
90.9
Concerto (dec.)
16.3M
13.1%
79.5
87.6
92.6
37.8
50.5
84.1
48.3
62.3
87.7
75.5
84.2
92.3
Concerto (full.)
124.8M
100.0%
80.7
87.4
93.1
39.2
50.2
85.0
50.7
63.3
87,9
77.4
85.0
93.2
Concerto (lora)
<0.5M
<0.5%
79.8
87.9
92.7
38.4
51.9
84.1
47.3
60.8
87.7
75.5
81.4
92.6
Table 11: Semantic segmentation with LoRA. We compare the LoRA fine-tuning method on Con- certo across four semantic segmentation benchmarks, demonstrating LoRA's remarkable capacity in general and the reliability of Concerto's 's original pretrained representations.
excluding lideo-lifted data. We evaluate these models across four datasets on semantic segmen- tation, as shown in Tab. The results reveal that the Concerto model based on DINOv2, using self-distillation, achieves the highest mloU in general. This suggests that in our joint self-supervised learning framework, the optimal synergy achieved when representations from different domains are derived through intra-modal self-distillation. RADIO, which incorporates distilled information from multiple models, may damage the original self-distillation features from DINOv2, thus leading to decrease in performance.
B.4 Results with LoRA Finetuning
From the main results, we observe that linear probing outperforms full-finetuning in extreme data- scarce scenarios. This suggests that training methods may benefit from shifting toward LoRA-based fine-tuning. In this section, we present the results of LoRA fine-tuning Specifically, we adapt LoRA to the point encoder and evaluate it with linear probing. We set the LoRA rank to 8, the LoRA alpha to 16, and the dropout rate to 0.1.
The results of ScanNet Data Efficiency are shown in Tab. 10. These results demonstrate that the LoRA-based method outperforms both linear probing and full fine-tuning in terms of mloU across most scenarios, despite a small increase in learnable parameters compared to the original linear probing. This suggests that LoRA is an effective fine-tuning approach, particularly when data is limited. Notably, linear probing with LoRA achieves performance comparable to decoder probing in the full evaluation and only a 0.9% performance drop compared full fine-tuni fnn-tuning on mIoU, while offering significant improvements in training efficiency.
We also evaluate the LoRA fine-tuning on Concerto across four benchmarks, as shown in Tab. 11. The results demonstrate that LoRA fine-tuning shows performance comparable to decoder probing, even with relatively small learnable parameters. Overall, the LoRA fine-tuning demonstrates strong efficiency and performance across various benchmarks, highlighting two key insights: Concerto already yields reliable and generalizable representations, and leveraging pretrained representations combined with LoRA fine-tuning is both efficient and effective for further task adaptation.
C Broader Impact
In this work, we introduce Concerto, a powerful model for spatial representation learning, achieving SOTA performance in full fine-tuning and linear probing. Looking ahead, Concerto holds great promise for extending multi-modal learning beyond 2D-3D, benefiting downstream tasks such as autonomous driving, robotics, and mixed reality. However, if not properly trained, point cloud encoders may learn biases from the data, reinforcing societal stereotypes. Researchers should be aware of such potential negative social impacts and continuously monitor the training data for bias.
25


Reviews:
[{"review_id": "HPHPWcjksQ", "reviewer": "Reviewer_T3Xs", "summary": "This paper proposes Concerto, which extends a 3D self-supervised learning (SSL) method (Sonata) by cotraining with 2D image features via joint embedding predictive objective, and demonstrate consistent performance gain on various tasks, surpassing the single modality performance and naive concetenation of 2D and 3D features, providing a straightforward and effective way of enhancing 3D SSL encoders.", "questions": "1. Does batch size matter in the cross-modal training?\n2. Can the method work with noisy camera parameters? Will it be sensitive if depth, camera poses or intrinsics are inaccurate?", "limitations": "yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 2, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Strengths:**\n1. Straightforward and effective method. SSL on 2D image features has witnessed a substantial progress. Combing their power to improve 3D features has sigfinicance to the community.\n2. Consistent performance gain. Concerto demonstrates consistent performance gain over baselines on various benchmarks, providing a strong empirical evidence.\n3. Ablation studies covered many aspects.\n\n**Weaknesses:**\n1. Clarity of writing can be improved if more important details can be included in the main paper. Specifically:\n - 1a) how are the image and point feature represented when computing JEPA? If would be nice to have some concise and clear math expressions and formulations.\n- 1b) In ablation study, what is the definition of \"image usage\"?\n- 1c) In ablation study, how do you specifically control the number of visible points and how is the specific number 65536=256^2 chosen? How do you get the number from the visibility check in the appendix?\n- 1d) Lack of implementation details on the learning probing and decoder probing.\n2. Performance seems to saturate when scaling up data and model, in Table 5h. Scaling is an important indicator but only two settings are ablated. It would be great if the authors can provide more results on this, perferablly a plot and some explanations.\n3. Lack of quantitative results of training on the VGGT lifted video data. It would be great to see the method can robustly leverage VGGT lifted data as training source and possibly showing some scale performance on the video data where data is abundant while annotation is scarce.", "was_revised": true, "final_justification": "Concerns are addressed. Decide to stay at the original rating \"accept\". Detailed justification has been replied to the authors.", "rebuttal": "Thanks for your acknowledgment of our work! Below, we provide detailed responses to your questions and comments. Hopefully, they will further clarify our approach.\n### **W1: Clarity of Writing**\n\n- Clarity of Writing\n\n  Thanks for your suggestion. We agree that adding math formulas will make the process clearer for readers, and we will revise this section in the final version.\n\n  As shown in Fig. 3 of the paper, we respectively obtain point representations $s_x$ and image representations $s_y$ from the point encoder and image encoder, as\n\n  $$s_x = \\Psi_{\\text{pcd}}(x),$$ \n  $$s_y = \\Phi_{\\text{img}}(y).$$\n\n  With camera parameters $z$ as conditions, the correspondence between the points and pixels can be captured, which is further explained in Appendix A.2. First, the intrinsic matrix $K$ and extrinsic matrix $[R|t]$, derived from $z$, are used to project 3D point coordinates $p_x$ to the 2D pixel coordinates $p_y$: \n\n  $$p_y = K[R|t] p_x.$$\n\n  Then, for each image patch $j$, we compute the mean of the features of points falling within it, as\n\n  $$\\hat{s}\\_{y,j} = \\frac{1}{|\\mathcal{N}\\_j|} \\sum_{i \\in \\mathcal{N}\\_j} s_{x,i},$$\n\n  where $\\mathcal{N}\\_j$ is the set of indices of points whose projected coordinates $p_y$ fall within the $j$-th image patch, and $|\\mathcal{N}\\_j|$ is the number of such points. $s\\_{x,i}$ is the representation of the $i$-th point. Then, we compute the loss $D(s_y, \\hat{s}_y)$ using cosine similarity\n\n  $$D(s_y, \\hat{s}_y) = \\left\\langle \\frac{s_y}{\\||s_y\\||_2}, \\frac{\\hat{s}_y}{\\||\\hat{s}_y\\||_2} \\right\\rangle$$\n\n  between representations of point clouds and image patches. This process introduces 2D self-supervised features into 3D self-supervised learning and stimulates the point intra-modal self-distillation process.\n\n- Definition of \"Image Usage\"\n\n  The term \"Image Usage\" refers to the ratio of the total number of images used during the pretraining process. If the Image Usage is 50\\%, it means we randomly selected 50\\% of the data with images, while the remaining is pure point cloud data. We will clarify this definition in the final version of our paper.\n\n- Visible Points\n\n  The control of visible points is crucial in the self-supervised learning process. The model needs to capture local cues to predict surrounding representations. Too many visible points can diminish this exploration: as training progresses, random views of the same data are similar, which reduces diversity in the model’s learning. Conversely, too few visible points make the task unmanageable. The value of 65536 is chosen experimentally by visualizing randomly selected views from the point clouds.\n  \n  Fully controlling the visible points for each data instance is impractical, as points in the point cloud do not have a one-to-one correspondence with pixels in the image. The chosen number of visible points (65536) is the maximum that ensures the model focuses on different regions of the data at an appropriate scale. Typically, the number of visible points falls within a user-defined proportional range of the total point count. Any count exceeding 65536 is capped at this maximum value.\n  \n  We will update the related section in the final version for better clarity.\n\n- Linear Probing and Decoder Probing\n\n  **Linear Probing**: In this approach, the pretrained encoder remains frozen. We upcast the features to their original resolution and adapt them to downstream tasks using a single linear layer, which accounts for less than 0.2\\% of the model’s total parameters.\n  \n  **Decoder Probing**: For this method, we use a lightweight, hierarchical decoder that comprises 13\\% of the total parameters. The encoder is frozen, and we train only the decoder, allowing it to learn task-specific representations.\n  \n  We will add this explanation to the final version. As these methods are described in detail in the Sonata paper, we will also refer readers to the original Sonata [1] work for further clarification.\n\n### **W2&3: Data Scaling and Video-Lifted Data Potential**\n\nThanks for your suggestion. Currently, 3D self-supervised learning faces the challenge of limited data, which restricts further development. To address this, we explore leveraging large-scale video data through feed-forward methods like VGGT to expand the dataset. Below, we provide evaluation results (mIoU) on ScanNet Val using linear and decoder probing for different data scales:\n\n| Data Scale | 13k | 23k | 35k | 40k | 87k |\n| --- | --- | --- | --- | --- | --- |\n| lin. | 76.5 | 76.6 | 77.4 | 77.3 | 77.8 |\n| dec. | 78.5 | 78.9 | 79.3 | 79.5 | 79.8 |\n\n\nHere, 87k means 40k original point clouds and 47k VGGT-lifted point clouds. For quantitative results of VGGT-lifted video data, we provide the mIoU results with base model size on ScanNet and ScanNet++:\n\n| Data Scale | ScanNet Val | ScanNet++ Val |\n| --- | :---: | :---: |\n| Concerto(lin.) | 77.3 | 45.7 |\n| +video training data(lin.) | **77.8** | **46.3** |\n\nThe table above shows that, with additional video data, Concerto can further improve the performance. As for model size, it should be increased with the data scale. Currently, the model is still in the stage of being data hungry. Therefore, we don't scale up our model size to a giant level but provide a tiny version for robotic downstream tasks and a large model trained with additional video data here. All the models are evaluated using linear probing with mIoU metric.\n\n| Model Size | ScanNet Val | ScanNet200 Val | ScanNet++ Val | S3DIS Area5 |\n| --- | :---: | :---: | :---: | :---: |\n| 16M(T) | 67.7 | 25.0 | 33.7 | 65.2 |\n| 39M(S) | 76.6 | 34.4 | 43.1 | 71.3 |\n| 108M(B) | 77.3 | 37.4 | 45.7 | 73.5 |\n| 207M(L) | 77.5 | 38.6 | 46.3 | 73.7 |\n\n\nWe believe that, as feed-forward methods continue to improve, point cloud training will scale more effectively, leading to more robust and reliable spatial representations. We will append these scaling results in the appendix of the final version and release a HuggingFace demo and website with more visualizations showcasing performance on VGGT-lifted point clouds.\n\n### **Q1: Does batch size matter in the cross-modal training?**\n\nWhile batch size is an important parameter in self-supervised learning, Concerto shows strong adaptability to different batch sizes during pretraining. The following table presents results across various batch sizes on different benchmarks:\n\n| Batch Size | ScanNet Val | ScanNet200 Val | ScanNet++ Val | S3DIS Area5 |\n| --- | :---: | :---: | :---: | :---: |\n| 48 | 78.0 | 38.2 | 46.9 | 73.9 |\n| 96 | 77.3 | 37.4 | 45.7 | 73.5 |\n| 192 | 77.9 | 36.4 | 45.3 | 73.8 |\n\n\n### **Q2: Can the method work with noisy camera parameters? Will it be sensitive if depth, camera poses, or intrinsics are inaccurate?**\n\nYes, our method is robust to noisy camera parameters, which is one of its advantages. In our approach, we do fusion first and then learning. This allows minor misalignments between points and pixels without significantly impacting the loss calculations. Just as humans can understand noisy point clouds, Concerto shows a similar capability. Current feed-forward reconstruction methods may not always produce accurate point clouds and camera parameters, but Concerto shows strong adaptability to such noisy point clouds. Notably, we avoid cherry-picking video examples in our ongoing demos and paper, further proving the robustness.\n\n[1] Wu et al. Sonata: Self-Supervised Learning of Reliable Point Representations. CVPR 2025."}, {"review_id": "OOgAzD7it4", "reviewer": "Reviewer_NTix", "summary": "This paper presents Concerto, a framework motivated from multi-sensory learning in humans, that aims at achieving better spatial representations for self-supervised point cloud features. It first self-distills the 3D model through a teacher-student paradigm, then utilizes 2D-3D joint embedding prediction to align the 3D point cloud features with the 2D image features. Experimental results show the superiority of the proposed Concerto when applying to downstream tasks of semantic and instance segmentation under different tuning settings including linear probing, decoder probing, and full finetuning.", "questions": "-- How is the proposed method compared with other methods that also focus on spatial understanding with joint 2D-3D data like [1, 2, 3, 4]? The comparisons are important to understand which is a better solution for joint 2D-3D spatial understanding.\n\n-- Is the pretrained 2D model also tunable during training? If not, I think it cannot be called \"synergy emerged from joint self-supervised learning\". If so, will the 2D model gets improved from jointly training with the 3D model?\n\n-- Does there exist difference from the self-distillation stage presented in the paper and the self-distillation operation in the Sonata paper? If not, I think this stage cannot be an essential component of the proposed method, because the proposed method is built upon Sonata, while the self-distillation stage is just a part of the original Sonata paper.\n\n-- For the Concerto variant specifically for video, is the only difference of this model and the original model being the training data is constructed from video using VGGT instead of point cloud? If so, why this is treated as a separate variant, instead of adding the video data to the training dataset to enrich the training set, and get a more powerful model with the more diverse training set?\n\n[1] Fan et al. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024.\n\n[2] Yue et al. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. ECCV 2024.\n\n[3] Zhu et al. PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. arXiv:2310.08586.\n\n[4] Wang et al. Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation. ECCV 2024.", "limitations": "Yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 2, "originality": 2, "strengths_and_weaknesses": "**Strengths**:\n\n++ The idea of using Joint Embedding Predictive Architecture (JEPA) to predict 3D features from pretrained 2D encoders is neat and effective solution to me. \n\n++ The experimental settings are comprehensive, including different tuning strategies of linear probing, decoder probing, and full finetuning, demonstrating the broad use case of the proposed Concerto under different circumstances and training budgets. Also, the model almost always outperforms the baseline Sonata, showing superior performance on downstream tasks. \n\n++ The ablation study on multiple aspects including criteria type and weight, the ratio of point clouds and images, *etc.* of the implementation can be useful for guiding towards the optimal configuration of the model. \n\n**Weaknesses:**\n\n-- The paper lacks the comparisons with other methods that also focus on spatial understanding with joint 2D-3D data, essentially the methods listed in the third paragraph of the related work section. More specifically, Large Spatial Model [1], FiT3D [2], PonderV2 [3], GGSD [4] are some open-source options, and also representing the different categories listed in the related work section. If the original works are on 2D segmentation, we can lift the 2D features to 3D using camera poses for comparison in 3D segmentation. This comparison is very important because they are actually the existing works for spatial understanding with joint 2D-3D data, while the current compared methods like MSC, Sonata are all pure 3D self-supervised methods. It will help demonstrate how the proposed solution for joint 2D-3D learning performs compared to alternative solutions in other works like lifting projections, differentiable rendering, direct distillation, *etc.*\n\n-- The paper claims to have synergy between the 2D and 3D models. However, from my understanding of the method, the 2D model is fixed, so the model is essentially getting knowledge from pretrained 2D model to predict a better 3D model, but there is no benefit the other way around. Therefore, I think the synergy from the joint learning paradigm is overclaiming.\n\n-- The 3D intra-model self-distillation part follows the identical approach of previous work Sonata. Therefore, I think it cannot be regarded as part of the proposed method. However, the paper gives a feeling that the proposed method has two steps \"(a) intra-modal self-distillation + (b) 2D-3D joint embedding prediction\". In fact, only (b) is the technique proposed in the paper. \n\n-- It sounds a bit far-fetched to me to connect the proposed method with the human concept learning in cognition. Basically the paper is proposing a paradigm for 2D-3D joint learning, which is just very loosely related to the multi-sensory learning concept of humans illustrated in the paper. It is definitely good to have some bio-inspired motivations or designs in the method, but I think the concept is very loosely related to the proposed method. Otherwise, any method that introduce additional modality in its framework can be related to human multi-sensory learning.\n\n-- In the supplementary the paper mentioned the term \"DINOv2.5\". However, I think this is not a standard term. I could only refer to the Sonata paper where they refer to [5] as DINOv2.5. I think a more standard term for this work is called DINOv2-Reg. Also, the submitted manuscript does not have reference about [5], which makes readers feel confusing.\n\n[1] Fan et al. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024.\n\n[2] Yue et al. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. ECCV 2024.\n\n[3] Zhu et al. PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. arXiv:2310.08586.\n\n[4] Wang et al. Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation. ECCV 2024.\n\n[5] Darcet et al. Vision Transformers Need Registers. ICLR 2024.", "was_revised": true, "final_justification": "My major concern on the performance compared with other spatial understanding methods is well resolved in the rebuttal. The remaining issues are about the contribution and motivation. For paper contribution, since the authors say that the \"intra-modal self-distillation\" part is not a newly proposed component in this paper (but it is included in the method, and make readers feel that it is a new technique), the only technical contribution of this paper is the \"2D-3D joint embedding prediction\", which makes the paper contribution a bit limited. Also, the motivation from human perception for 2D-3D joint modeling is too far-fetched and very loosely related. But anyway, the simplicity and effectiveness of the proposed solution make the strengths of this paper outweigh the weaknesses. Therefore, I would like to raise my score to borderline accept.", "rebuttal": "Thanks for your suggestions. We are sincerely grateful for your efforts and feedback. We hope the concerns are well addressed.\n\n### **W1: Comparisons with Other Methods on Spatial Understanding with Joint 2D-3D Data**\n\nThanks for your suggestion. In response, we’ve included comparisons between Concerto and the methods you mentioned on spatial representation learning:\n\n| Methods             | ScanNet Val | ScanNet200 Val | ScanNet++ Val | S3DIS Area5 |\n| :------------------ | :---------: | :------------: | :-----------: | :---------: |\n| Large Spatial Model |    73.6     |      28.4      |     42.0      |    70.0     |\n| Ponderv2            |    77.0     |      32.3      |     46.0      |    73.2     |\n| FiT3D               |    44.8     |      24.1      |     32.8      |      -      |\n| Concerto            |  **80.7**   |    **39.2**    |   **50.7**    |  **77.4**   |\n\nLarge Spatial Model [1], PonderV2 [2], and FiT3D [3] represent different categories of joint 2D-3D learning, such as lifting projections, differentiable rendering, and direct distillation. For GGSD [4], since it focuses on open-vocabulary tasks using additional text and superpoint information, which is outside the scope of our current work, it is not included in this comparison. We will continue to extend our work to other downstream tasks in the future.\n\n### **W2: Synergy from the 2D and 3D**\n\nThank you for your feedback on our use of the term \"synergy\". To address your concern, we provide results on ScanNet where DINOv2 is fine-tuned with Concerto, showing an improvement in 2D tasks compared to the original DINOv2. This demonstrates that fine-tuning the 2D model alongside the 3D model enhances its performance on 2D tasks.\n\n| 2D | mIoU | mAcc | allAcc |\n| --- | :---: | :---: | :---: |\n| DINOv2 | 63.1 | 75.5 | 82.4 |\n| DINOv2(Concerto) | **64.5** | **77.4** | **83.2** |\n\n\n**Here, when we refer to \"synergy,\" we mean \"1 + 1 > 2\"**: with joint learning, our method allows for stronger and more reliable representations than what can be achieved with single modalities or the naive concatenation of 2D and 3D features. This comparison can be seen in ***Table 1*** of our paper.\n\nWe acknowledge that our initial expression may have caused some misunderstanding about the nature of our method. We will revise the relevant section in the final version to make this clearer and will include the fine-tuning results for DINOv2 in the Appendix.\n\n### **W3: 3D Intra-Model Self-Distillation as Preliminary**\n\nThanks for your suggestion. The Intra-Modal Self-Distillation part is not a new contribution of this work, but rather a preliminary step. We introduced it here for clarity and to help readers better understand the foundational aspects of our method. A preliminary introduction is necessary for those who are not familiar with previous works. The other Reviewer, yoVm, also thinks this preliminary should be explained in more detail. In the final version, we will explicitly clarify that the intra-modal self-distillation is a preparatory step and not the primary contribution of our work. Our main contribution lies in the Cross-Modal Joint Embedding Prediction and its integration with self-distillation.\n\n### **W4: Human Concept Learning in Cognition**\n\nThank you for your advice. We will revise the explanation of human concept learning to improve clarity and the overall flow of the manuscript.\n\nWe introduce the concept of human cognition to help readers better understand the awareness in self-supervised learning with the other modality. Humans are exposed to multi-sensory information from birth, and once they learn a concept, such as an \"apple,\" they can recall its taste when they see the fruit. In real-world applications, however, multi-sensory information is not often available to a system. This raises the question: How can we generate multi-sensory representations from a single modality, much like how humans recall information from different senses in their mind?\n\nOur goal is to present a model that can produce representations derived from both 2D and 3D features with only point cloud input, mimicking the process by which humans recall information from other sensory modalities. This is different from the case where one model receives multi-modal input and outputs corresponding responses.\n\n### **W5: Terms for DINOv2-Reg**\n\nThank you for pointing this out. We will update the term DINOv2-Reg [5] in the final version to ensure consistency and clarity, and include the reference.\n\n### **Q1-3:** \n\nThank you for your questions. For detailed responses, please refer to the corresponding sections addressed in the previous  W1-3.\n\n### **Q4: Concerto Variant Specifically for Videos**\n\nWe appreciate your suggestions. To address the data limitations hindering progress in 3D self-supervised learning, we explore the use of abundant online video data. By employing feed-forward methods like VGGT, we can enlarge our point cloud datasets. To demonstrate the potential of this approach, we introduce a separate variant of Concerto that incorporates this additional video data.\n\nFor quantitative results of VGGT-lifted video data, we provide the results using the base model on ScanNet and ScanNet++:\n\n| Data Scale | ScanNet Val | ScanNet++ Val |\n| --- | :---: | :---: |\n| Concerto(lin.) | 77.3 | 45.7 |\n| +video training data(lin.) | **77.8** | **46.3** |\n\nBelow, we provide mIoU evaluation results on ScanNet Val for different data scales using linear and decoder probing, showing scaling performance on video data. Here, 87k refers to 40k original point clouds and 47k VGGT-lifted point clouds.\n\n| Data Scale | 13k | 23k | 35k | 40k | 87k |\n| --- | --- | --- | --- | --- | --- |\n| lin. | 76.5 | 76.6 | 77.4 | 77.3 | 77.8 |\n| dec. | 78.5 | 78.9 | 79.3 | 79.5 | 79.8 |\n\nThe table above shows that, with additional video data, Concerto can further improve the performance. We believe that with the progress of feed-forward methods, the point cloud self-supervised learning has the potential to scale up to a larger level and finally result in a more reliable and robust spatial representation model. We will append these results to the Appendix in the final version.\n\n[1] Fan et al. Large Spatial Model: End-to-end Unposed Images to Semantic 3D. NeurIPS 2024.\n\n[2] Zhu et al. PonderV2: Pave the Way for 3D Foundation Model with A Universal Pre-training Paradigm. T-PAMI 2025\n\n[3] Yue et al. Improving 2D Feature Representations by 3D-Aware Fine-Tuning. ECCV 2024.\n\n[4] Wang et al. Open Vocabulary 3D Scene Understanding via Geometry Guided Self-Distillation. ECCV 2024.\n\n[5] Darcet et al. Vision Transformers Need Registers. ICLR 2024."}, {"review_id": "Hr80dW2QgN", "reviewer": "Reviewer_yoVm", "summary": "This paper presents Concerto, a minimalist yet powerful framework for learning spatial representations by combining 3D intra-modal self-distillation with 2D-3D cross-modal joint embedding. I found the central idea of connecting 2D and 3D feature spaces particularly novel and compelling. The intuition that humans can abstract and recall spatial concepts across modalities is well-motivated and convincingly translated into the model design.\n\n* The empirical results are somewhat strong: Concerto demonstrates fine improvements over both 2D and 3D SOTA baselines in fine-tuned settings\n---> while the zero shot evaluation is missing and lacking. \n\nOverall, the method is conceptually sound and has fine practical relevance.\n\nThat said, there are several comments that should be addressed prior to publication.", "questions": "Very important questons related also to the Weaknesses:\n\n\n  1. Line 92–94 (Sentence Starting with \"Ideally\"): This sentence is confusing and seems to contradict the surrounding paragraph. Please clarify its intent and explain how it aligns with the rest of the section.\n\n   2. Lines 112–113 – Clarification Needed: It took time to uderstand what is meant by \"linear probing\" in this context. you are projecting point cloud features into the CLIP feature space, given that CLIP already aligns images and text during its pretraining -- the goal is to align the 3d representation with text. The question is \n * why CLIP if you dont use the image embeddings? or do you? \n * can you use any other text feature extractor? \n  * How to do align, what is the loss here?\n I think this pargraph need a better writing. \n\n3. DINOV2 as a Target – Clarification: If your model is trained to predict DINOv2 features, does this imply that DINOv2 defines an upper bound for your performance? If so, how is your method able to outperform it? I found this somewhat confusing. \n\n4. Lines 145–146 – Ambiguity: The phrasing “divide a scene into one point cloud” is unclear. Please revise this for clarity. Do you mean that each scene is represented as a single point cloud - then what does divide mean? If not, please specify the intended meaning.", "limitations": "See my Weaknesses 4 5 and 6.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 3, "originality": 4, "strengths_and_weaknesses": "** Strengths **\n1. Novel and Well-Motivated Idea: The paper introduces a creative and compelling approach that bridges 2D and 3D modalities for spatial representation learning. The core intuition is both original and conceptually sound.\n\n2. Good Experimental Results and Ablation: The experimental evaluation is comprehensive, with solid performance across multiple benchmarks. The ablation studies effectively isolate the contributions of different components (I appriciate this part very mach)*\n\n3. Data Efficiency Results – A Strength: The results on data efficiency are excellent and, in my view, one of the paper’s key strengths. I suggest highlighting this more prominently in both the abstract and conclusion.\n\n** Weaknesses **\n\n1. Writing \n  * Redundancy: The manuscript suffers from a noticeable degree of textual redundancy. I recommend a thorough revision to reduce repetition and improve clarity throughout the paper.\n\n  * Lack of Formal Definitions: While the paper is generally well-written and conceptually clear, it would benefit from a more formal and mathematical presentation. Providing explicit formulas at each step of the pipeline would make the method easier to follow and reproduce.\n\n* Some Statements Are Confusing: Certain parts of the paper are unclear or potentially contradictory. See Questions 1 and 2 in my review for specific examples that should be clarified.\n\n2. Figure 3 – Needs Improvement: This figure is central to the paper, but currently it is unclear and potentially misleading. If it is meant to depict the architecture, it should be expanded with clearer annotations, supporting formulas, and step-by-step explanations. Consider reducing textual redundancy elsewhere in the paper to make room for a more informative and detailed illustration.\n\n3. Intra-Modal Self-Distillation Section: This section is underexplained, despite being a key preliminaries of the method. Please provide more details on how intra-modal self-distillation is implemented and how it contributes to the learned representations.\n\n4. Zero-Shot Segmentation – Untapped Potential: Have you considered evaluating your model on zero-shot segmentation tasks, similar to ConceptFusion [1] or FollowAnything [2], by providing an image or user click to specify the object of interest and not only text?\n\n5. Long-Tail Concepts – Missing Evaluation: It would be informative to test the method on long-tailed or rare object classes, as found in datasets like unCOCO. This would demonstrate robustness in more realistic, open-world scenarios.\n\n6. Comparison to ConceptFusion: ConceptFusion is a closely related work that also aligns 2D and 3D representations and achieves strong results on ScanNet and many other dtasets. A direct comparison would strengthen the paper, especially in light of similar goals. For zero short text, click, images as input.\n\n[1] https://concept-fusion.github.io/\n[2] https://arxiv.org/abs/2308.05737", "was_revised": true, "final_justification": "From the beginning, I liked the paper, particularly the main idea, which I find both timely and important. I had some questions and concerns, but the authors addressed them satisfactorily. Therefore, I vote for acceptance.", "rebuttal": "Thanks for your acknowledgement of our work, especially your recognition of our data-efficient and ablation results. And we hope your concerns are well addressed below.\n\n### **W1: Clarity of Writing**\n\n- Redundancy\n\n    We will polish our paper to make it more concise and cohesive. We would also welcome more specific feedback.\n\n- Lack of Formal Definitions\n\n    Yes, some math formulas will help readers understand. Here we provide the mathematical presentation of our method:\n\n    As shown in Fig. 3 of the paper, we obtain point representations $s_x$ and image representations $s_y$ from the point encoder and image encoder respectively, as\n\n    $$s_x = \\Psi_{\\text{pcd}}(x),$$\n    $$s_y = \\Phi_{\\text{img}}(y).$$\n\n    With camera parameters $z$ as conditions, the correspondence between the points and pixels can be captured, which is further explained in Appendix A.2. First, the intrinsic matrix $K$ and extrinsic matrix $[R|t]$, derived from $z$, are used to project 3D point coordinates $p_x$ to the 2D pixel coordinates $p_y$:\n\n    $$p_y = K[R|t] p_x.$$\n\n    Then, for each image patch $j$, we compute the mean of the features of points falling within it using\n\n    $$\\hat{s}\\_{y,j} = \\frac{1}{|\\mathcal{N}\\_j|} \\sum_{i \\in \\mathcal{N}\\_j} s\\_{x,i},$$\n      \n    where $\\mathcal{N}\\_j$ is the set of indices of points whose projected coordinates $p_y$ fall within the $j$-th image patch, and $|\\mathcal{N}\\_j|$ is the number of such points. $s\\_{x,i}$ is the representation of the $i$-th point. Then, we compute the loss $D(s_y, \\hat{s}_y)$ using cosine similarity\n\n    $$D(s_y, \\hat{s}_y) = \\left\\langle \\frac{s_y}{\\||s_y\\||_2}, \\frac{\\hat{s}_y}{\\||\\hat{s}_y\\||_2} \\right\\rangle$$\n\n    between representations of point clouds and image patches. This process introduces 2D self-supervised features into 3D self-supervised learning and stimulates the point intra-modal self-distillation process.\n\n- Confusing Statement\n\n    Please refer to our responses to Q2 and Q3 for clarification.\n\n### **W2: Figure 3 Improvement**\n\nWe will improve Figure 3 to present our method more clearly. As mentioned in W1, we will include a more detailed version of the formulas to complement the figure.\n\n### **W3: Further Explanation for Intra-Modal Self-Distillation Section**\n\nWe agree that introducing the preliminary work is important for readers, especially those unfamiliar with it. Here’s a more detailed explanation:\n\nIn Intra-Modal Self-Distillation, we build upon the principles of multi-view consistency and masked reconstruction, illustrated in Part (a) of Figure 3. The main challenge in 3D point cloud self-supervised learning is the geometry shortcut, which means networks tend to focus on sparse point cloud geometry and neglect the rich semantic information, such as color and texture. To mitigate this, it is important to make the geometry clues harder for the network to capture. This approach is discussed in detail in the Sonata paper.  In addition to applying masking, cropping, and view rolling techniques, we also adopt the encoder-only structure and feature upcasting strategy from Sonata to reduce the over-reliance on local geometry and incorporate multi-scale features. This preliminary step ensures the point encoder refines its representations with richer semantic information.\n\nWe will revise this section, provide a detailed explanation in the Appendix, and refer readers to the Sonata [1] paper for more information.\n\n### **W4: Zero-Shot Segmentation**\n\nThank you for the suggestion to further improve our work. ConceptFusion [2] and FollowAnything [3] are both remarkable contributions to zero-shot semantic segmentation using existing foundation models(e.g., SAM, DINO, and CLIP) and providing various interfaces for users. However, we focus on building such a foundation model in the 3D domain, like DINOv2 in the 2D domain. The absence of such a 3D foundation model blocks the progress in improving downstream tasks. We acknowledge this as a key area for future work and will continue to push in this direction.\n\nAdditionally, we have noticed that evaluation datasets like unCoCo, used in ConceptFusion and FollowAnything, are currently not available. However, we provide a comparison between our method and ConceptFusion on ScanNet, as mentioned in W6.\n\n### **W5: Long-Tail Concepts**\n\nWe agree that long-tail concepts are crucial for evaluating the generalization ability of model. Although the unCoCo dataset is currently unavailable, we believe that the results on ScanNet200 in our paper can offer insights into the model's performance on long-tail concepts. ScanNet200 includes more rare classes than ScanNet, and our model achieves 37.4\\% mIoU on ScanNet200 using linear probing, compared to the previous SOTA of 29.3\\%. This improvement demonstrates Concerto's effectiveness in handling long-tail objects.\n\n### **W6: Comparison to ConceptFusion**\n\nWe followed the evaluation protocol from ConceptFusion, using the same subset and object classes without background concepts. Below is a comparison between Concerto and ConceptFusion on ScanNet Val:\n\n| Methods | mAcc | mIoU |\n| --- | :---: | :---: |\n| ConceptFusion | 0.63 | 0.58 |\n| Concerto | 0.75 | 0.58 |\n\nHowever, the main focus of Concerto is point cloud representation learning. The language probing serves as an auxiliary evaluation method to demonstrate the potential of our representations. This shows that even without explicit text information, the representations learned by our method can inherently capture conceptual features related to language. With simple alignment, these representations can be transformed into text space. We intentionally use shallow language alignment to avoid complex components, such as decoders, dominating the evaluation results.\n\nFor W4-W6, we will include the related results and discussion in our final version.\n\n### **Q1: Lines 92–94 (Sentence Starting with \"Ideally\")**\n\nThis sentence refers back to Figure 2, which illustrates the \"Apple\" concept in cognition. Humans are exposed to multi-modal information from birth, and once they learn the concept of an apple, they can recall its taste when they see a picture of it. We aim to express a similar idea here. Specifically, while the model is trained using multi-modal joint representation learning, it is capable of outputting representations derived from 2D and 3D information, even when the input is a single modality, such as a 3D point cloud, during inference. This means a lot in real-world applications, because multi-modal input is not always available for a system.\n\nWe will rewrite this part in the final version to erase the confusion.\n\n### **Q2: Lines 112–113 – Clarification**\n\nThis paragraph describes our future expectation for a next-level evaluation criterion for self-supervised learning beyond single modality, which means that although the model is trained without texts, it can simultaneously form concepts akin to human language. Just as humans first learn concepts from the world before using language to express them, a strong self-supervised model should be able to form its own conceptual \"language\". This property can be validated during evaluation on text alignment, where the model’s representations can be aligned with the human semantic space through a simple transformation, such as a single linear layer. As mentioned in the W6, aligning representations with text is not the main contribution of this work but rather an auxiliary evaluation method to validate the quality of Concerto's representations, complementing the semantic segmentation tasks.\n\nThen, come to the question. We here choose the CLIP text space to validate our representations. We align our point cloud representations with the image embeddings from LSeg, using the image as a bridge to connect our 3D point features with text embeddings. If any other text feature extractor is aligned with image pixels, Concerto’s representations can be transformed into that text space as well. The loss function used for alignment is cosine similarity. We freeze the pretrained backbone, add a linear layer on top, and align the output with the image embeddings from the LSeg image encoder.\n\n### **Q3: DINOV2 as a Target – Clarification**\n\nOur model does not aim to simply copy 2D features from DINOv2, which ignores 3D information. Instead, we emphasize the importance of interaction between modalities. This leads to entirely new representations derived from different domains. The core of our method is the stimulation from the 2D modality to enhance intra-modal self-distillation in the 3D domain. As shown in ***Table 1*** of the paper, our method outperforms each single-modality approach as well as their naive concatenation. This demonstrates that, rather than merely combining information from two modalities, Concerto captures richer representations through the interaction between 2D and 3D information.\n\n### **Q4: Lines 145–146 – Ambiguity**\n\nApologies for the ambiguity. What we meant to convey is that a scene with multiple images (>>4) is divided into several data pieces, where each piece consists of one point cloud and 4 images. We will revise this section in the final version to ensure better clarity.\n\n[1] Wu, et al. Sonata: Self-Supervised Learning of Reliable Point Representations. CVPR 2025.\n\n[2] Jatavallabhula, et al. ConceptFusion: Open-set multimodal 3d mapping. RSS 2023\n\n[3] Maalouf, et al. Follow Anything: Open-set detection, tracking, and following in real-time RA-L 2024"}, {"review_id": "9irWU0BaRE", "reviewer": "Reviewer_yNfS", "summary": "This paper proposes the Concerto framework, which fuses image and point cloud modalities through 2D-3D joint self-supervised learning, and combines intra-modal self-distillation (3D point cloud feature refinement) with cross-modal joint embedding prediction (image-point cloud feature alignment) to achieve more consistent spatial representation learning. Experiments show that Concerto significantly outperforms single-modal methods on benchmarks such as ScanNet, with linear detection accuracy improved by 4.8%-14.2%, and achieves SOTA performance of 80.7% mIoU under full fine-tuning. In addition, its generalization and open-world perception potential are verified through video point cloud expansion and CLIP language space mapping.", "questions": "1. **Cross-modal generalization**: In the LiDAR-vision heterogeneous sensor fusion scenario, is an additional calibration step required?\n2. **Long-term tracking performance**: How to solve the drift problem of the video extension version in long sequences?\n3. **Lightweight potential**: Can a parameter compression version (such as distillation or quantization) be launched for mobile devices?", "limitations": "1. **Scene constraints**: Relying on high-quality image-point cloud pairing data, not robust enough for textureless or low-light scenes.\n2. **Semantic gap**: Language mapping only stays at linear projection, and does not model the complex semantic association between spatial concepts and natural language.\n3. **Ecological dependence**: Highly dependent on 2D models such as DINOv2. If other backbones (such as Swin Transformer or SAM) are used, will the performance be stable?", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "**Advantages**:\n1. Multimodal collaborative innovation: simulate human multi-sensory learning mechanism, break through the limitations of single-modal representation through cross-modal alignment (image-point cloud) and intra-modal distillation (point cloud feature optimization), and fill the gap between 2D and 3D features.\n2. Comprehensive performance leadership: In tasks such as semantic segmentation and instance segmentation, Concerto significantly surpasses single-modal models such as DINOv2 and Sonata, and even outperforms feature splicing baselines, proving the effectiveness of multimodal collaboration.\n3. Strong technical universality: supports pure point cloud input, video sequence expansion and language space mapping, adapts to multiple scenarios such as autonomous driving and robots, and has industrial landing potential.\n\n**Disadvantages**:\n1. Modality dependency limitations: cross-modal alignment depends on camera parameters, and performance may degrade in scenarios without calibration or multi-view missing, and the extreme case of pure point cloud without image input has not been verified.\n2. High computational cost: 40k point clouds + 300k images are required for pre-training, and the video memory usage and training time are significantly higher than those of single-modal models. The ablation experiment did not quantify the difference in computational overhead.\n3. Shallow language alignment: CLIP space is only mapped by linear projection, and the zero-shot segmentation accuracy (44.56% mIoU) is far behind the supervised results, and deep semantic fusion needs to be strengthened.", "was_revised": true, "final_justification": "The authors' responses have addressed most of my concerns. Taking into account the overall evaluation of the other reviewers, I choose to maintain my rating as Borderline Accept (4).", "rebuttal": "Thank you for your kind acknowledgment of our work, especially for recognizing our model’s technical universality. We hope the responses provided address your concerns effectively.\n\n### **W1: Modality Dependency Limitations**\n\nIt is crucial to consider the potential lack of data from certain modalities in multi-modal joint learning. To address scenarios where images or calibration data may be missing, we ablate the image usage ratio of the whole datasets(ranging from 0\\% to 100\\%) in the pretraining stage in ***Table 5(b)*** of the paper. The results demonstrate Concerto's ability to adapt to point cloud data even in the absence of image input. Additionally, we include VGGT-lifted point clouds into our datasets, which makes it possible to directly use images to reconstruct without camera calibration. We provide more quantitative results of this variant:\n\n| Data Scale | ScanNet Val | ScanNet++ Val |\n| --- | :---: | :---: |\n| Concerto(lin.) | 77.3 | 45.7 |\n| +video training data(lin.) | **77.8** | **46.3** |\n\nWe will include the results for this variant in the Appendix of the final version.\n\nMoreover, Concerto is robust to data with some degree of imperfection, such as noisy camera parameters, which is an advantage of our method. We begin with point fusion and then learning, which means we perform intra-modal self-distillation and cross-modal joint-embedding prediction at the abstract level, between fused points and image patches, instead of points and pixels. This approach ensures that slight misalignments between points and pixels do not cause significant differences in the loss calculations. Just as humans can understand noisy point clouds, Concerto shows a similar capability.\n\n### **W2: High Computational Cost**\n\nSimilar to DINOv2, Concerto is a pretrained self-supervised learning model that alleviates the heavy burden of downstream task learning. High computational cost at pretraining stage leads to strong and generalizable representations, resulting in data efficiency in downstream tasks. For example, ***Table 2(b)*** shows the strong linear probing results with less than 0.2M training parameters, ***Table 2(c)*** highlights outstanding performance in data-scarce situations, and ***Table 9 of the Appendix*** presents efficient LoRA-based adaptation.\n\n### **W3: Shallow Language Alignment**\n\nThank you for pointing that out. This language alignment is a future expectation for next-level evaluation criterion for self-supervised learning beyond single modality, which means although the model is trained without texts, we hope it can form concepts akin to human language. \n\nSince it is an evaluation protocol, we intentionally use a shallow language alignment with a linear layer to prevent other components, such as complex decoders, from dominating the final results. The detailed explanation behind this is discussed in the ***Pilot Study*** section of the paper.\n\n### **Q1: Cross-Modal Generalization**\n\nCompared with the RGBD camera commonly used in indoor scenes, LiDAR + camera needs standard spatial-temporal synchronization, including same-time triggering (temporal) and intrinsic and extrinsic calibration(spatial). Our methods need no additional calibration.\n\n### **Q2: Long-Term Tracking Performance**\n\nIn our work, we use VGGT to lift videos from the RE10K dataset. The videos in RE10K are divided into short sequences, and VGGT effectively handles these sequences for point cloud generation. Although sometimes the feed-forward reconstruction produces failure cases, Concerto shows robustness when training on these noisy data. We also look forward to further improvements in feed-forward reconstruction methods and will release our code for generating this data.\n\n### **Q3: Lightweight Potential**\n\nThank you for your suggestion! We recognize the demand for lightweight models suitable for mobile devices. Currently, we have models of varying scales, and we plan to release these versions in the future. Below are the quantitative results for these models using linear probing:\n\n| Model Size | ScanNet Val | ScanNet200 Val | ScanNet++ Val | S3DIS Area5 |\n| --- | :---: | :---: | :---: | :---: |\n| 16M(T) | 67.7 | 25.0 | 33.7 | 65.2 |\n| 39M(S) | 76.6 | 34.4 | 43.1 | 71.3 |\n| 108M(B) | 77.3 | 37.4 | 45.7 | 73.5 |\n| 207M(L) | 77.5 | 38.6 | 46.3 | 73.7 |\n\n### **L1 and L2:**\n\nPlease refer to our response to W1 and W3 for detailed clarification.\n\n### **L3: Ecological Dependence**\n\nWe conducted experiments on different 2D encoders, like SigLIP2 [1] and RADIOv2.5 [2], as ***Table 8*** in the Appendix of our paper. The results show that the DINOv2 variant has the strongest performance, while the other encoders show comparable numerical performance.\n\nThe choice of DINOv2 as the 2D encoder in our pipeline is based on its strong performance as a self-supervised model. We observed that different modal representations from self-supervised learning benefit the 3D intra-modal self-distillation significantly. This is analogous to the human cognition process: the learning for one single modality is like an intra-modal self-supervised learning process, while humans learn concepts through multi-modal inputs that interact in the system and result in superior recognition of the world. By integrating self-supervised features from two different modalities in Concerto, it emerges more reliable representations than single modal representations and their naive concatenation.\n\n[1] Tschannen, et al. SigLIP 2: Multilingual vision-language encoders with improved semantic understanding, localization, and dense features. arXiv:2502.14786 (2025).\n\n[2] Heinrich, et al. RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models. CVPR 2025"}]

Author rebuttal:

Meta-review: This paper presents a joint 3D self-supervised learning approach combined with 2D joint embedding predictive learning, demonstrating consistent improvements over alternative approaches across multiple tasks. The reviewers unanimously agreed on the value of the proposed method and the rigour of the presented evidence. The extensive comparisons with other models provide strong support for the added value of Concerto.

However, several presentation issues should be addressed in the camera-ready version. I agree with reviewer NTix that the connection with human perception is tenuous and unclear, and I recommend substantially downplaying or removing these claims in the revised paper. Additionally, as correctly noted by several reviewers, the text would benefit from revision to eliminate repetitive and redundant sections that detract from the clarity of the paper.

Despite these presentation concerns, the technical contributions are solid and the experimental validation is thorough. Given the positive evaluations from all reviewers and with the expectation that the camera-ready paper will address the identified presentation issues, I recommend acceptance.
Decision: Accept (poster)