OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images
Authors: Ye Mao
Junpeng Jing
Krystian Mikolajczyk
Venue: NeurIPS 2024 poster (2024)
Keywords: Open-World
3D Representation Learning
3D Shape Understanding
Primary area: machine_vision
TL;DR: OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images

Abstract: Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point clouds with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this alignment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets highlights the need for effective strategies to transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge in pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on a limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0\% on ModelNet40 and 16.4\% on OmniObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-art models.

OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images
Ye Mao
Junpeng Jing" Krystian Mikolajezyk
Imperial College London
mae.m j.jing23, k. mmikoolajccyyk//tmerralll .c.uk
Abstract
Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point clouds with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this align- ment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets high- lights the need for effective strategies transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0% on ModelNer40 and 16.4% on Om- niObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-an models.
Introduction
3D understanding, including tasks like 3D object classification [1, 2]. 3D scene segmentation
[3], and 3D reconstruction [4, 5], is becoming increasingly pivotal in real-world applications like augmented/virtue reality [6, 7], autonomous vehicles [8]. Traditional 3D models [1, 2, 9. 10, 11]
are closed-world, recognizing only pre-defined categories and struggling with 'unseen' ones. Recent studies aim to leverage Vision-Language Models (e-g., CLIP [12]) develop open-world that generalize beyond 'seen' 3D data, enabling zero-shot handling of various 3D tasks.
models
Existing open-world 3D learning methods can be classified into depth-based and point-based ap- proaches. Depth-based methods [13, 14, 15] convert point clouds into multi-view depth maps and use the pre-trained CLIP image encoder for 3D representations. A significant challenge the domain gap since CLIP is pre-trained on RGB images, not depth maps. To mitigate this gap, methods like [15] introduce an additional depth encoder to align depth features with the image and text features from the pre-trained CLIP encoders, as shown in Fig. 1(a). The images used for feature alignment are produced by rendering synthetic CAD models from various camera viewpoints. Point-based methods [16, 17, 18, 19, 20] directly extract 3D representations from point clouds, thereby bypassing the need for depth map projection. However, they also require an extra point encoder for feature alignment to address format disparities between images and point clouds, as shown in Fig. 1(b). Thus, employing
"Correspondig Author
38th Conference on Neural Information Processing Systems (NcurIPS 2024).

---

(a) Depth-based Method
(b) Point-based Method
(c) OpenDlign (ours)
Align
Align
Align
Ooo
OoO
Text Encoder
Image Encoder
Depth Encoder
Text Encoddr
Image Encoder
Point v Encoder
Text Encoder
Image
Encoder
depth map small airplane."
point cloud of a small aleplane.
dopel smull airlane."
Text
Rendered Image (Training unly)
Depth Map
Text
Rendered Image (Training only)
Point Cloud
Text
Depth-Aligned Image (Training only)
Depth
Map
CAD-rendered
images
Depth-aligned images
Figure 1: Top: Comparison of OpenDlign with traditional open-world 3D learning models. Depth- based (a) and point-based (b) methods employ additional depth or point encoders for pre-training align with CAD-rendered images. Conversely, OpenDlign (c) fine-tunes only the image encoder, aligning with vividly colored and textured depth-aligned images for enhanced 3D representation. Both rendered and depth-aligne images are utilized solely during training. Bottom: Visual comparison and corresponding depth aapth-lligned images in OpenDlign.
between multi-view
an extra 3D encoder for multimodal feature alignment of 3D data, CAD-rendered images, and text modalities is a standard practice in modern open-world 3D learning methods.
Despite widespread usage, we argue that CAD-rendered images fall short of providing the visual diversity and realism necessary for robust multimodal alignment. This limitation arises because CAD models in current open-source datasets [21, 22] often feature simplistic or entirely absent textures. These models also struggle to realistically simulate environmental effects like lighting, shadows, and reflections. Moreover, most CAD models maintain visual coherence across viewpoints, leading to overly consistent textures and colors from all angles. To achieve generalizable 3D representations, each image view for alignment is expected to display significant visual variations (See Fig. 1).
Additionally, pretraining an extra 3D encoder for aligmment may not fully leverage the rich knowledge in CLIP for 3D understanding due to the significant volume discrepancy between 2D and 3D pre- training datasets. Mainstream 3D datasets like ShapeNet [21] and Objaverse [23] contain fewer than I million synthetic 3D objects, significantly less than the vast image datasets such as DFNSB [24] and LAION-SB [25], which include around billion images used to train cutting-edge CLIP models. While direct fine-tuning of CLIP's encoders facilitates more straightforward knowledge transfer, it restricts inputs to depth maps. Yet, developing 3D representations from depth maps is currently less effective than from point clouds for two primary reasons: (1) The current widely used CLIP text prompt templates are tailored for matching with RGB images, not depth maps, and (2) the lack of robust projection method for creating dense depth maps with smooth contours from point clouds.
In this paper, we present OpenDlign, the first 3D open-world framework that develops 3D rep- resentation by aligning with multi-view diffusion model-generatee images, termed depth-aligned images. These images benefit from the stochastic nature of the diffusion model, offering greater texture diversity compared to CAD renderings while maintaining the original 3D data's 's geometric and semantic integrity (See Fig. 1). Remarkably, OpenDlign demonstrates competitive open-world capability by fine-tuning only 6 million parameters of the CLIP image encoder on ShapeNet [21], unleashing CLIP's vast potential in 3D learning (See Fig. 1(c)). The success of this fine-tuning stems from refining the depth map projection pipeline, developing depth-specific text prompts, and introducing a logit aggregation strategy to merge multi-view prediction results. Experimental results reveal that OpenDlign significantly outperforms previous state-of- the----tttttttt (SOTA) models on variety

---

of 3D tasks, including zero-shot/few-shee classification, object detection, and cross- moosl retrieval. In zero-shot classification, OpenDlign achieves accuracy improvements 8.0% on ModelNet40 and 16.4% on OmniObject3DD, the largest real-world 3D shape dataset. Additionally, depth-aligned images markedly enhance the performance of SOTA models, consistently improving results across diverse benchmarks and highlighting their transformative impact on open-world 3D learning pipelines.
The main contributions of this paper are outlined follows:
We introduce depth-aligned images as robust alternative to CAD-rendered images for open-world 3D learning. These images, generated from point cloud-projected depth maps using diffusion model, offer rich and realistic texture diversity across viewpoints.
We propose multimodal alignment framework that robustly aligns depth maps, depth- aligned images, and text through streamlined fine-tuning of the CLIP image encoder.
We develop a contour-aware projection pipeline to produce dense and contour-preserving multi-view depth maps from point clouds.
We present depth-specific text prompts and logit aggregation strategy to boost OpenDlign's zero-shot capabilities and mitigate catastrophic forgetting during alignment fine-tuning.
2 Related Work
2.1 Open-World 3D Representation Learning
Vision-Language Models (VLMs) such CLIP [12] have revolutionized 2D representation learning in open-world settings through contrastive lcarning with large-scale image-text pairs [26, 27, 28, 29]. Building this, recent studies have adapted CLIP for 3D representation learning, achieving impressive performance in diverse 3D zero-shot tasks [18, 19].
PointCLIP [14], as pioneering study, utilizes the CLIP image encoder for extracting 3D repre- sentations from depth maps of point clouds, achieving zero-shot recognition by aligning with text embeddings of semantic categories. To address CLIP's s training bias towards RGB images, Zhu et al. [13] introduced GPT-generated 3D-specific prompts and denser depth map projection, while CLIP2Point [15] pre-trains depth encoder for closer alignment with CLIP's encoders. These meth- ods derive representations from depth maps with noisy contours, causing a loss of key shape features needed for precise recognition. Moreover, their reliance either natural image text prompts or depth-specific prompts generated by GPT-3 [30] for certain categories highlights a lack of versatility in handling diverse 3D contexts. Alternative methods [17, 18, 19, 20] avoid depth map projection by directly aligning point clouds, images, and text using specialized 3D encoders. By scaling up the dataset and encoder sizes, these methods show promise in diverse 3D tasks. However, these methods are limited by their reliance on CAD-rendered images, which have limited texture diversity across views, leading to less generalizable representations. Additionally, the smaller volume of 3D datasets compared to CLIP's training data hinders effective knowledge transfer to point cloud encoders.
In this paper, we substitute rendered images with depth-aligned images generated from diffusion model to enhance texture diversity. We also fine-tune the CLIP image encoder for 3D representation learning instead of training new 3D encoder from scratch, reducing the reliance on large 3D datasets.
2.2 Continual Learning in CLIP Fine-Tuning
Continual Learning (CI.) in CLIP aims to mitigate catastrophic forgetting [31], ensuring retention of zero-shot capabilities across varied data distributions while fine-tuning to new tasks. CL methods fall into three categories: adaptive-plastictty methods [32, 33], replay methods [34, 35], and architecture- based methods [36, 37]. -lastive-plasticity methods limit the plasticity of the essential model parameters for past tasks during fine-tuning. For instance, the IMM-Mean [38] method achieves CL by simply averaging parameters of pre-trained and fine-tuned models for inference, although its efficacy might be limited for complex tasks [39]. Replay methods leverage stored exemplars to enable CLIP to recall previously learned knowledge, while they encounter scalability challenges. Without relying on exemplars, architecture-basee architecture to accommodate new information without losing existing knowledge [39]. In this study, align the depth map with the RGB image by freezing the pre-trained CLIP encoder weights and
CL. methods dynamically adjust the model's
3

---

incorporating a trainable transformer- based branch for encoding depth maps, adhering to architecture- based principles. Inspired by IMM-Mean [38]. we use pre-trained and ane-tuned model weights to compute classification logits from multi-view depth maps for inference.
Methodology
The OpenDlign framework, depicted in Fig. 2, starts with contour-aware projection method that transforms point clouds into multi-view depth maps with preserved contours. These maps guide diffusion model produce depth-aligned images with varied colors and textures, maintaining consistent geometry with the depth maps. OpenDlign then aligns features from depth maps and depth- aligned images by fine-tuning transformer block connected to the pre-trained image encoder. The goal is to align feature embeddings from depth maps and corresponding depth papph-d images using contrastive learning. At inference, 3D representations are composed of embeddings from multi-view depth maps, derived using both fine-tuned and pre-trained encoder states. These embeddings are matched with the depth maps, to generate multi-view logits. These logits are then aggregated to facilitate label prediction a zero-shot setting. In few-shot scenarios, these embeddings can be further refined with logistic regressor. Detailed training and model implementation are provided in Appendix A.2.
text embeddings, which capture the semantic and visual properties of
3.1 Contour-Aware Depth Map Projection
The contour-aware projection method transforms the input point cloud into multi-view depth maps with clear contours. Inspired by the pipeline in [13], this method involves four main steps: Quantize, Densify, Smooth, and Squeeze.
In the Quantize step, for the 3th view of point cloud P.. the 3D coordinates (z, (z, P. normalized to [0, 1] and mapped onto discrete grid G E xxxW where H and correspond to the dimensions required by the CLIP image encoder, and B is pre-defined depth dimension. Next, the Densify step enhances G by updating each voxel to the maximum value within its 7 x 7 x 7 neighborhood, yielding denser map G'. Subsequently, the Smooth step applies bilateral filtering to each voxel 2; in G", adjusting its intensity I, to I, using:
1.
W. 05ES
G., By 1) (14, Lu, 1) W.
1
(I)
where Wi = Ev,es Go, (llg (14, 1o, is the normalization factor that ensures voxel weights sum to 1.0. The Gaussian functions Go, and Go2 adjust the influence of each neighboring voxel bj within the 5 x 5 5 kernel from set S around v; based spatial and intensity differences, enhancing contour sharpness and reducing jagged edges in G'. Finally, the Squeeze step applies the minimal pooling on the depth channel of the smoothed G', then triples the output to mimic RGB intensity, producing the final depth map D E HX
3.2 Depth-Aligned Image Generation
total of 524,700 depth-aligned images are generated from ShapeNet [21], a leading public 3D CAD dataset containing approximately 52,470 models, each annotated with semantic metadata. To generate these images, point cloud of 10,000 points sampled from each CAD model, aligning with prior experimental protocols [18, 17J. For each point cloud, 10 views of depth maps are projected using the proposed contour-aware projection method. Subsequently, the ControlNet vl.1 [40] depth model produces depth-aligned images for each contour-awand depth map view, using the CAD model's metadata as text imput and the inverse of the depth map D. as image generation control. This approach ensures that the generated images remain consistent with the depth maps both geometrically and semantically, while also adding texture diversity across different views. ControlNet utilizes 1 instead of D for controlling image outputs because it is predominantly pre-trained on depth images where brighter regions indicate closer proximity. Please refer to Appendix A.2 for the positive and negative prompts used in ControlNet to achieve high-fidelity depth-aligned image generation.
1

---

(a) 3D Representation Learning via Generated Depth-Aligned Images
Contour-Aware Depth Map Projection
Transfommet
Block
Input Point Cloud
12
Transformer Block
Transformer Block
Transformee Block
hg
hg
Multi- View
Depth Maps
hf
-8
Image Generative Model
escontto
Multi-View Depth-Aligned Images
CLIP Image Encoder
30
16
(b) Zero-Shot 3D Classification
(c) Few-Shot 3D Classification
airplane
View
cup
"A restraccd image. "A deptk (clurs) mondet"
Pestninod Text Ennoder
sofa
Rre-mnined Image Encoder
bike
car
Depth Specific Text
F
F
cup
Logistc
Prosrainal
Regrowe
car
Lncoder
View 1
8y2 <
Fise-tmel
Imege Frooder
table
Fize-tred
View +1
View N
"A depth map of (cor) 30 model."
View
Figure 2: Overview of OpenDlign. In (a), OpenDlign converts point clouds into multi-view depth maps using contour-ave annourr-aware projection, which then helps generate depth-aligned RGB images with diverse textures, geometrically and semantically aligned with the maps. A transformer block, residually connected to the CLIP image encoder, is fine-tuned to align depth maps with depth- aligned images for robust 3D representation. For zero-shot classification (b), OpenDlign aggregates multi-view logits from both pre-trained and fine-tuned encoders for label prediction. For few-shot classification (c), it employs logistic regressor trained on multi-view features from the encoders.
3.3 Multimodal Representation Alignment
OpenDlign aligns depth maps and depth- images by fine-tuning transformer block residually linked to the last block of the CLIP image encoder, using contrastive learning. With CLIP pre-trained to align images and text, OpenDlign implicitly aligns depth maps with the shared image-text space.
Multimodal Feature Extraction. Given a 3D point cloud input, let D = ( represent the set of its N projected depth map views, and R = {R, AN the corresponding set of depth-aligned images. Each image R; is encoded through layers of a pre-trained CLIP image encoder, (Tr(-))fm obtain feature representations 1. = T. (R.). Each depth map D, processed up to layer Tr.-1, yielding preliminary features T (D;)). Subsequently, these depth features are passed through the frozen layer Tr. and its trainable counterpart T7, where only the attention layers for spatial interaction in T7 trainable, as inspired by [41]. This process produces the feature for the ith depth map view
5

---

-T The final feature vectors for multi-view depth maps D and depth-aligned images R are hD N Σm ||TP|| and h N Σ> || respectively.
Loss Functions. The aligmment of h and h is achieved by minimizing composite loss function, comprising the contrastive loss Ccom and the feature distance loss Ldists defined as:
exp (bPh5/T)
exp (h/h,/ /////T
Linall
Σ (i.j)
1 log Σ exp (bPhg/T) 2
1 log
Σ (15)
1h
h2ll2 (2)
2
Σx exp (b/h///))
Cas
In each training batch, (h,: h)) is a positive pair, while (h,, h and (h hnf) are negative pairs where k + i, T is a learnable temperature parameter. The contrastive loss enables learning robust representations by maximizing similarity in positive pairs and minimizing it in negative pairs [42, 43).
3.4 3D Zero-Shot Transfer
The alignment between depth maps and depth-aligned images facilitates 3D zero-shot classification by aggregating multi- view classification logits. Each logit represents the similarity between single-view depth features and text features of category candidates, as illustrated in Fig. 2(b).
Pepth-SSeciffic Text Generation. Depth-specific prompt templates are developed by augmenting a base set of 80 text prompts, initially designed for ImageNet classification", with depth-related keywords such as "depth map", "raytraced image", and "silhouette of [CLASS]". These keywords direct OpenDlign's 's attention to geometric details and contours rather than color or texture. The CLIP Interrogator [44] is a prompt engineering tool that combines CLIP and BLIP [45] to select optimal text prompts for specific images. To identify these keywords, this tool identifies the top 10 prompts that match depth maps from the ShapeNet dataset [21], chosen targeted keywords. For zero-shot inference, we employ our depth-specific templates to generate 80 text descriptions for each label I. encoded by texture encoder F()., normalized, and then merged into a unified text feature Fi via average pooling, calculated as 3 Σ [t.(())]l.
These descriptions (t, ),50
Multi- View Logits Aggregation, To calculate classification logits, we first gather visual features from mue depth maps (V.) aiming align with depth-specific text features of M candidate labels F } JM The feature extraction utilizes dual-encoder strategy: the first half of the views
N/2 [V:), utilize a pre-trained CLIP image encoder, while the second half of the views [Vi) employs fine-tuned encoder. The strategy ensures that OpenDlign maintains its capability to recognize previously identifiable depth maps after learning multimodal alignment via fine-tuning. As shown in Fig. 2(b), the logit for single depth map view is the product of V; and F, with the overall classification logit being the sum of logits across all views, calculated as Σ V.FT.
4 Experiments
4.1
Zero-Shot 3D Classification
We first evaluated OpenDlign under the zero-shot shape classification task four benchmark datasets: ModelNet40 [46], ScanObjectNN [47], OmniObject3D [48], and Objaverse-LVIS [23]. ModelNet40 offers synthetic 3D CAD models in 40 categories. ScanObjectNN provides real-scanned objects in 15 categories from OBJ_ONLY version. OmniObject3D, the largest, includes 5,91 real- scanned objects in 216 categories, well-suited for fine-grained, real-world classification evaluation. Objaverse-LViS contains 1.156 categories for evaluating long-tail classification. Point cloud sizes 10,000 points for NedelNet40 and Objaverse-LVIS. 2,048 for ScanObjectNN, and 4,096 for OmniObject3D. OpenDlign was compared against existing methods, including three depth-based methods: PointCLIP [14], PointCLIP V2 [13]. and CLIP2Point [15], and three point-based methods: ULIP [17]. OpenShape [18], and TAMM [20]. To investigate if depth-aligned images consistently enhance the representational abilities of other 3D open-wort methods, we retrained all OpenShape and TAMM variants using their original CAD-rendered images and some depth-aligned images. Note
Text Prompts for ImageNet: ImageNet Prompt Engineering.

---

Table 1: Zero-shot classification results on ModelNet40 [46], ScanObjectNN [47] and OmniObjeet3D [48]. The best-pet pess-performing results are presented in bold, while the second- best results are underlined. Our models are highlighted in blue
Training
3D Open. World
CLIP
ModelNNer40 [46]
ScanObjeciNN [47]
OmniObjeet3D [48]
Source
Methods
Variant
Tup1
Top3
TopS
Top1
Top3
Top5
Top1
Top3
Tops
2D inferences
PoinCCLIP [14]
ResNet-50
19.3
28.6
34,8
10.5
20.8
30.6
0.3
1.0
1.8
No Training
PointCLIP (2[13]
VIT-B-16
63.6
77.9
85.0
42.2
63.3
74.5
3.9
9.6
14.4
CLIP2Point [15]
VIT-B-32
49.5
71.3
81.2
25.5
44.6
59.4
1.4
3.7
7.1
PoinnniiTB [17]
SLJP [50]
60.4
79.0
84.4
51.5
71.1
80.2
8.4
15.2
19.7
OpenShape SpurseConv [18]
CennShape-SuuseeCom
ViT.bigG-14
52.7
72.7
13.7
24.2
DpeaStaay-PPoitBBRRT [18]
VII-bigG-14
70.3
86.9
91.3
51.3
69.4
71..
13.0
23.3
29.4
TAMM--SprssCon [20]
ViI-bigG-14
74.6
88.2
94.0
57.9
75.3
83.1
TAMM MaMM-PiinBRRTTT [20]
ViT-bigG-14
73.1
88.5
91.9
54.8
74.5
83.3
14.9
26.2
33.4
ShapeNet
OpenShape SpurseConv (+dlign)
WTT
74.9
89.5
94.1
56.3
75.2
85.4
15.0
26.1
32.8
OpenShape- OpenShape-Ponnttzi PointBERT (+dlign)
VIT-bigG--1
73.7
87.1
91.3
52.7
72.4
82.6
13.4
23.7
29.9
TAMM PointERRT (+dlign)
VTT bigG--
73.7
89.1
92.2
57.3
73.6
82.3
15.8
27.4
33.0
OpenDlign-B33 B332
WT--B-32
68.4
86.4
92.6
46.7
72.0
83.0
17.3
29.2
36.3
OpenDiign-B16
VIT-B-16
74.2
90.5
95.4
49.3
74.0
84.4
23.2
37.5
44.3
OpenDiign-LL
VIT-L-14
77.8
93.1
96.4
52.1
74.6
$2.8
27.5
41.3
47.8
OpenDiign
VIT-H-14
82.6
96.2
98.4
59.5
76.8
83.7
31.3
46.7
53.2
OpenShape-SpureeCCon SparseConv [18]
big-
97.8
56.7
78.9
33.7
49.3
OpenShape CaunShape-Piiitlee PoinBERT [18]
big-
84.4
96.5
98.0
52.2
79.7
88.7
34,0
49.7
57.9
ToiMM-PoinBERRT [20]
ViI-bigG-14
55.7
37.1
53.5
Ensemble
Spar--SpareCony [20]
ViT-bigG-14
85.4
96.4
98.1
58.5
81.3
89.5
OpenShape Saaepe-pppeeeCnnee (+dlign)
85.0
96.1
97.9
56.2
78.5
87.8
34.1
50.5
58.5
OpenShupe- PointBERT (+ilign)
WiT-bigG-14
85.4
96.6
98.2
51.1
77.4
88.2 2
35.6
50.4
57.9
TAMM-PoinBBERT (+dlign)
VII-bigG-14
86.2
96.6
97.5
60.5
82.5
90.4
37.5
54,9
62.1
that the depth- depth-aligned images for all model variants were exclusively generated from ShapeNet [21], while the pretraining CAD-rendered images could come from ShapeNet or a large-scale ensemble dataset [18] that includes Objaverse [23], ShapeNet [21], 3D-Future [49], and ABO [22]. Furthermore, we evaluated OpenDlign's scalability by training it with various CLIP variants.
Table 6 shows OpenDlign substantially outperforms existing methods trained on ShapeNet across all benchmarks, exceeding the previous best, TAAMM-SparseCo by margins of 8.0% on ModelNet40,, 1.6% on ScanObjectNN, and 16.4% OmniObject3D in Topl accuracy. Appendix A.5 Table 7 demonstrates OpenDlign' excellence in handling long-tail categories by outperforming previous methods by 21.3% the Objaverse-LVIS dataset. OpenDlign also outperforms the top depth- based method, PointCLIP V2, by 19% on ModelNer40 and 27.4% on OmniObject3D. Notably, OpenDlign outshines all methods pre-trained on the ensemble dataset in the ScanObject3D benchmark. Additionally, its performance increases linearly with the complexity of CLIP variants, outperforming most baselines on ModelNet40 and OmniObject3D, even using the lighter V1T-B-16 CLIP model. Remarkably, using depth-aligned images (+dlign) consistently boosts the performance of OpenShape and TAMM variants pre-trained the ShapeNet dataset across all benchmarks. Despite depth- aligned images being available only for ShapeNet, which comprises just 10% of the ensemble dataset, and
(+dlign) shows 4.8% increase in Topl accuracy on the ScanObjectNN dataset, (+dlign) records 1.6% increase on the OmniObject3D. Note that for parts of the ensemble dataset without depth-aligned images, we used CAD-rendered images instead.
Table 2: Few-shot classification results on ModelNet40 [46], ScanObjectNN [47] and OmniObject3D [48]. Our results are averaged over 10 random seeds.
ModeINevtO [16]
ScanObjecINN [47)
OmniObjeet3D [45]
Model
Shot Shox Shot Shoo 16 Sbot
Shot 2 2-Shor 4 Shot 8 Shott 16 Shot
I Sbot 2 Shot
Shot S S-Sho 16 Shot
PoiecBERTRTT [17]
55.1
OpenShope- Poin [18]
57.5
76.5
80.4
82.1
47.9
55.6
62.7
67.0
72.0
34.5
41.9
45.6
OpenStape- SparseCion [18]
62.8
72.0
78.9
82.9
85.7
47.5
56.3
64.5
68.2
14.0
36.0
37.0
41.5
44.7
48.6
BA [20]
62.4
71.3
31.7
83.8
85.9
48.2
57.1
63.6
72.1
76.5
38.9
41.6
46.1
S0.1
54.2
OpenDlign (ours)
65.6
739
82.9
85.
87.6
48.9
58.5
67.9
74.
79.0
42.1
46.9
55.1
61.9
65.8
4.2 Few-Shot 3D Classification
We then assessed OpenDlign's few-shot classification capability by training logistic regressor with linear probing on features from N-shot, 10-view depth maps. Similar to the zero-shot scenario, extracted multi-view features using both fine-tuned and pre-trained OpenDlign encoders. At
7

---

Table 3: Zero-shot 3D object detection results on ScanNet V2 [51].
Method
Mean
Cahinet
Bed
Chair
Sofa
Table
Door
Window
Counter
Desk
Sink
Bathtub
PointCLIP [1-]
6.00
3.99
4.82
45.16
4.82
7.36
4.62
2.19
1.02
4.00
13,40
6.46
AP25
PoánCTIPP [13]
18.97
23.78
13.22
17.42
21.43
14.54
OpenDlign (ours)
50.72
38.91
67.27
86.33
72.01
58.72
44.58
32.07
50.49
62.04
51.98
64.29
PointCLIP [14]
4.76
1.67
4.35
39.53
3.65
5.97
2.61
0.52
0.42
2445
5:27
1,31
AP50
PoinCCLIP LP [13]
11.53
11.35
10.14
OpenDlign (ours)
37.97
17.04
66.68
73.92
54.96
50,03
24.73
12.84
20.44
41.64
34.17
64,29
inference, the regressor aggregates logits from 10 views to predict the final label. We compared OpenDlign with ULIP [17], OpenShape [18], and TAMM [20]. which extract features for training regressor from their point encoders pre-trained on ShapeNet. Table shows OpenDlign outperforms all baselines across varied few-shot scenarios with to 16 training samples per class. OpenDlign significantly outperforms the leading baseline on the OmniOhject3D dataset, exceeding it by 8.8% and 11.8% in the 4-shot and 8-shot classification, respectively. See Appendix A.5 for more results.
4.3 Zero-Shot 3D Object Detection
OpenDlign's capability in zero-shot 3D object detection was evaluated on the ScanNet V2 dataset [51], which contains richly annotated 3D indoor scenes in 18 object categories. Following the PointCLIP V2 methodology [13], we started with the pre-trained 3DETR-m [11] model identify 3D regions of interest, delineate 3D bounding boxes, and extract points within each box. Finally, we applied OpenDlign to these points to generate our predictions. Table illustrates OpenDlign's zero-shot detection prowess using mean Average Precision (mAP) at IoU thresholds of 0.25 and 0.5, achieving scores of 50.72% and 37.97% respectively. It significantly outperforms PointCLIP V2 by more than 31.75% and 26.44%. Remarkably, OpenDlign can detect the 'Sofa' shape with an AP of 54.96% whereas PointCLIP and V2 score below 10%, demonstrating OpenDlign's 's superior ability to extract robust 3D representations from sparse and noisy point clouds in real-world indoor scenes.
4.4 Cross-Modal Retrieval
3D shapes were retrieved by computing the cosine similarity between the embeddings of query and those generated by OpenDlign, followed by a k-nearest neighbors (kNN) analysis to find the most similar shapes. Fig. showcases OpenDlign's 's ability to match 3D shapes to image and text queries. Column (a) shows its precision in distinguishing sub-categories like grand versus upright pianos from image queries. Column (b) demonstrates successful shape retrieval using distinctive text deseriptions like "Batmobile armored". Notably, averaging image and text query embeddings allows OpenDlign to find shapes that combine elements of both queries. For instance, merging a running horse image with the text "man" retrieves both centaur and running man, as shown in Fig. 3(c). Similarly, combining a house image with "tree" retrieves a treehouse. See Appendix A.6 for more results.
(a) Image Query
(b) Text Query
(c) Image Text Queries
decker hos."
Jondan."
"Spaceship"
'Man'
Tree
Teleyhone box."
"Byuncbue
Figure 3: 3D shape retrieval results. (a) Two most similar shapes for each image query. (b) Most similar shapes for each text query. (c) Two most similar shapes for combined image and text queries.
8

---

Table 4: Ablation study for OpenDlign on ModelNet40 [46] and ScanObjectNN [47]. Accuracy improvements over the baseline (first-row) are highlighted in green.
Contour- Aware Multimodal
Depth-Specifns
Logits
ModalNerr0 [46]
ScanObjeatNN [47]
Projection
Alignment
Texts
Aggregation
Top1
Top3
Top5
Top!
Top3
Top5
K
59.7
79.6
86.3
42.8
66.7
78.4
68.8 (+9.1) 85.8 (+6.2) 91.6
44.6 (+1.8)
68.3 (+1.6)
78,9 (+0.5)
79.2 (+19.5) 94.4 (+14.8) (+11.3)
56.9 (+14.1)
75.5 (+8.8)
83.8 (+5.4)
V
75.9 (+16.2) 91.0 (+11.4) 95.4 (+9.1)
49.3 (46.5)
69.8 (+3.1)
79.2 (+0.8)
80.2 (+20.5) 95. (+15.7) 97.7 (+11.4
58.1 (+15.3)
(+8.5)
84.2 (+5.8)
/
X
81.0 (+21.3) 95.2 (+15.6) (+11.3))))
56.8 (+14.0)
74.6 (+7.9)
81.6 (+3.2)
/
V
82.6 (+22.9) 96.2 (+16.6)
59.5 (+12.))
76.8 (+10.))
83.7 (+5.3)
4.5 Ablation Study
Ablation studies were conducted on zero-shot classification benchmarks to assess the contribution of each component in OpenDlign. Consistently, all OpenDlign variants used in these studies employed OpenCLIP-ViTI ViT-H-14 as their backbone. ShapeNet was the default training dataset for all models.
Contour- Aware Projection. Replacing PointCLIP V2's projection pipeline [13] with our contour- aware version enables pre-trained CLIP to achieve 68.8% zero-shot accuracy on ModelNet40, outperforming several baselines that require extra training (See Table 4). This indicates that CLIP can understand RGB images and depth maps when shape features are highlighted.
Effect of Alignment with Depth-Alignee Images. Table shows that aligning depth maps with depth-aligned images (i.e., depth-dlign) significantly boosts performance, improving Top accuracy by around 10% on ScanObjectNN, with without depth-specific prompts. This indicates that depth-d alignment effectively transfers CLIP's s rich knowledge to interpret depth maps.
Further analysis compared depth-dlige -lignn alignment against three alternatives: depth- depth-rendCAD (aligning depth maps with CAD-rendered RGB images), dlign-text depth (aligning with text before depth-dlign alignment), and depth-text & dlign (simultaneous alignment of depth maps with text and depth-aligned images). Table shows depth-dliig outperforming depth-rendCAD by 6.8% on the ScanObjectNN dataset, confirming concerns that alignment with rendered images may lead to overfitting on specific 3D shapes. Moreover, dlign-text & depth performs worst, suggesting that pre-aligning depth-aligned images with text compromises CLIP's ability to generate robust image representations, thus affecting subsequent depth-dlign alignment efficacy. The superior performance of depth-dlign on ModelNet40 and OmniObject3D compared to depth-text & dlign shows that aligning depth maps with depth-aligned images indirectly aligns with text, making additional text alignment unnecessary and potentially limiting OpenDlign's generalization.
images
Depth-Speciff Texts. Table shows that depth-specific prompts enhance OpenDlign's 's performance, regardless of using multimodal alignment or logit aggregation. This indicates that some recognition inaccuracies arise from processing input data as typical RGB images instead of depth maps.
Logits Aggregation. Results in Table 4 show that multi-view logit aggregation improves zero- shot classification on all datasets by combining logits from pre-trained and fine-tuned encoders. This approach effectively mitigates the catastrophic forgetting problem in OpenDlign's multimodal alignment, enabling recognize 3D objects identifiable by both pre-trained CLIP and OpenDlign.
Table 5: Ablation study on various align- ment strategies. Aligning with text modal-
100
2000 (50)
ity was achieved by fine-tuning the image encoder.
TAMR1-POHIBERT Top-1
25
RodAi
Alignment
MiNeH0 [6]
ScanNN [17]
Ommi3D [18]
TAMMU--Poletiinn Top-1 Acc.
Strategy
Topl
Top5
Top1
Top5
Top1
Top5
Aepth-ennCAAI
78.8
96.8
52.7
82.5
29.4
51.8
Vioepoints
Viewpolts
dlign-text & depth
78.6
96.4
79.6
29.1
51.6
depth-dlign (ours)
depth-tcxt dliga
82.6
79.4
98.4
98.0
60.7
59.5
86.0
83.7
31.3
29.5
53.2
52.7
Figure 4: Efffect of the number of views on OpenDlign's zero-shot performance.

---

Varying Number of Views. OpenDlign, like other depth-based methods, requires extracting multiple embeddings from multi-view depth maps for zero-shot inference. Fig. 7 shows that OpenDlign's zero-shot accuracy on ModelNet40 and OmniOhject3D improves with more depth map views. Notably, OpenDlign achieves top performance, comparable to TAMM-PointBERI, with just two views, balancing latency and effective zero-shot classification.
5 Conclusion
In this study, we introduce OpenDlign, an open-world framework that learns robust 3D representations from multi-view depth maps by efficiently fine-tuning with depth-aligned images, which are more visually diverse than CAD-rendered images. The effectiveness of OpenDlign is validated on various images consistently enhance
3D zero-shot and few-shot tasks. We also show that depth-aligne the performance of existing 3D open-world methods. Future work will explore the application of depth-alignet images in designing open- open-world world models for 3D scenes (See Appendix A.
Limitations. Due to limited computational resources, we cannot generate depth-aligned images from the largest 31D dataset [23], containing around million 3D objects. Retraining 3D open-world models with billions of parameters using these images is also too expensive. Moreover, a data filtering strategy needed remove low-quality depth-aligned images (See details in Appendix A.6).
Acknowledgement
We extend our gratitude Dylan Auty, Maojun Zhang, Ranran Huang, Jiangnan Ye, Ziyang Chen, and Chengzu Li for their valuable discussions and insightful feedback on the early drafts of this work. This research was supported by the Imperial College President's 's PhD Scholarships 2023.
References
[1] C. Qi, H. Su, K. Mo, and L. J. Guibas, "Pointnet: Deep learning point sets for 3d classification and segmentation." in Proceedings of the IEEÉ conference on computer vision and pattern recognition, pp. 652 652-660, 2017.
[2] C. R. Qi, Yi, H. Su, and L. J. Guibas, "Pointnet++: Deep hierarchical feature learning on point sets in metric space," Advances in neural information processing systems, vol. 30, 2017.
[3] L. Tchapmi, C. Choy, I. Armeni, J. Gwak, and S. Savarese, "Segcloud: Semantic segmentation of 3d point clouds," in 2017 international conference on 3D vision (3DV), pp. 537-547, IEEE, 2017.
[4] B. Mildenhall, Srinivasan, M. Tancik, J. Barron, R. Ramamoorthi, and R. Ng, "Nerf: Representing scenes neural radiance fields for view synthesis," Communications of the ACM, vol. 65, no. 1, pp. 99-106, 2021.
[5] J. Jing, Y. Mao, and K. Mikolajczyk, "Match-stereo-videos:: Bidirectional alignment for consis- tent dynamic stereo matching," arXiv preprint arXiv:2403.1075: 2024.
[6] I. Armeni, O. Sener, A. R. Zamir, H. Jiang, Brilakis, M. Fischer, and S. Savarese, "3d semantic parsing of large-scale indoor spaces," in Proceedings of the IEEE conference computer vision and pattern recognition, pp. 1534-1543, 2016.
[7] T. Vu, K. Kim, T. M. Luu, T. Nguyen, and C. D. Yoo, "Softgroup for 3d instance segmentation on point clouds," " in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2708 -008-777,211 2022.
[8] Y. Zeng. Y. Hu, S. Liu, J. Ye, Y. Han, X. Li, and N. Sun, "Rt3d: Real-time 3-d vehicle detection in lidar point cloud for autonomous driving." IEEE Roboties and Automation Letters, vol. 3, 4, pp. 3434-333433 3440, 2018.
[9] H. Zhao, L.. Jiang, Jía, P. Torr, and V. Koltun, "Point transformer." in Proceedings of the IEEE/CVF international conference computer vision, pp. (6259-16268) 2021.
10

---

[10] B. Li, T. Zhang, and T. Xia, "Vehicle detection from 3d lidar using fully convolutional network." arXiv preprint arXiv: 07910, 2016.
[11] I. Misra, R. Girdhar, and A. Joulin, "An end-to-end transformer model for 3d object detection," " in Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 2906-2917, 2021.
[12] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al., "Learning transferable visual models from natural language supervi- sion," " in International conference on machine learning, pp. 8748-8763, PMIR, 2021.
[13] X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao, "Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning." 2023 IEEE/CVF International Conference Computer Vision (ICCV), pp. 2639-2653 2022.
[14] R. Zhang. Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. J. Qiao, P. Gao, and H. Li, "Pointclip: Point cloud understanding by clip." 2022 IEEE/CVE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 8542-8555 2021.
[15] T. Huang, B. Dong, Y. Yang, X. Huang, R. W. H. Lau, W. Ouyang, and W. Zuo, "Clip2point: Transfer clip to point cloud classification with image-depth pre-training," 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 22100-22110, 2022.
[16] D. Hegde, J. Valanarasu, and V. M. Patel, "Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition," 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW), pp. 2020 2020-2030, 2023.
[17] L. Xue, M. Gao, C. Xing, R. Martín-Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, "Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding," in Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition, pp. 1179-1189, 2023.
[18] M. Liu, R. Shi, K. Kuang, Y. Zhu, X. Li, S. Han, H. Cai, F. M. Porikli, and H. Su, "Openshape: Scaling up shape representation towards open-world understanding." ArXiv, vol. abs/2305. 2023.
[19] Zhou, J. Wang, B. Ma, Y.-S. Liu, T. Huang, and X. Wang. "Uni3d: Exploring unified 3d representation at scale," ArXiv, vol. abs/23 10.06773, 2023.
[20] Z. Zhang, S. Cao, and Y.-X. Wang, "Tamm: Triadapter multi-modee learning for 3d shape understanding," arXiv preprint 240.:::2002.84900 2024.
[21] A. X. Chang, T. A. Funkhouser, L. J. Guibas, Hanrahan, Q.-X. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, "Shapenet: An information-rich 3d model repository," ArXiv, vol. abs/ab////0// 512.03012, 2015.
[22] J. Collins, S. Goel, K. Deng, A. Luthra, L. Xu, E. Gundogdu, X. Zhang, T. F. Y. Vicente, Dideriksen, H. Arora, et al., "Abo: Dataset and benchmarks for real-world 3d object un- derstanding." in Proceedings of the IEEE/CVF conference computer vision and pullern recognition, pp. 21 21126-21 136, 2022.
[23] M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi, "Objaverse: A universe of annotated 3d objects," 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 13142-13153, 2022.
[24] A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V. Shankar, "Data filtering networks." ArXiv, vol. Abs/2309.1742. 2023.
[25] C. Schuhmann, R. Beaumont, Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, el al., "Laion-5b: An open large-scale dataset for training next generation image-text models," Advances Neural Information Processing Systems, 2022.
vol. 35, pp.
11

---

[26] X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, "Open-vocabulary object detection via vision and language knowledge distillation," " in International Conference on Learning Representations. 2021.
[27] X. Zhou, R. Girdhar, A. Joulin, P. Krahenbuhl, and I. Misra, "Detecting twenty-thousand classes using image-level supervision." ArXiv, vol. abs/2201 .02605, 2022.
[28] X. Dong, Y. Zheng, J. Bao. T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen, Wen, and N. Yu, "Maskclip: Masked self-distillation advances contrastive language-image pretraining," 2023 IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pp. 10995-11005, 2022.
[29] F. Liang, B. Wu, X. Dai, K. Li, Y. Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, "Open- vocabulary semantic segmentation with mask adapted clip." in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 7061-7070, 2023.
[30] T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, "Language models are few-shot learners," " ArXiv, vol. abs/2005.14165, 2020.
[31] M. McCloskey and N. J. Cohen, "Catastrophic interference in connectionist networks: The sequential learning problem," Psychology of Learning and Motivation, vol. 24, pp. 109-165, 1989.
[32] J. Serrà, D. Surís, M. Miron, and A. Karatzoglou, "Overcoming catastrophic forgetting with hard attention to the task," in International Conference on Machine Learning. 2018.
[33] J. Schwarz, W. M. Czarnecki, J. Luketina, A. and R. Hadsell, "Progress & compress: A scalable framework for continual learning," " ArXiv, vol. abs/ 005/1805.000 803706370, 2018.
Y. W. Teh, R. Pascanu,
[34] D. Isele and A. Cosgun, "Selective experience replay for lifelong learning," ArXiv, vol. abs//802. 10269, 2018.
[35] S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, "icarl: Incremental classifier and representation learning," 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 5533-55535 5542, 2016.
[36] S. Yan, J. Xie, and X. He, "Der: Dynamically expandable representation for class incremental learning," " 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 301300012222 2021.
[37] Yoon, E. Yang, J. Lee, and S. J. Hwang, "Lifelong learning with dynamically expandable networks," ArXiv, vol. abs/1708.01544 2017.
[38] S.-W. Lee, J. Kim, Jun, J.-W. Ha, and B.-T. Zhang, "Overcoming catastrophic forgetting by incremental moment matching," ArXiv, vol. abs/ 703.08475, 2017.
[39] Y. Ding, L. Liu, Tian, J. Yang, and H. Ding, "Don't 't stop learning: Towards continual learning for the clip model," ArXiv, vol. abs/2207.09245 2022.
[40] L. Zhang. A. Rao, and M. Agrawala, "Adding conditional control to text-to-image diffusion models." " 2023 IEEE/CVF International Conference on Computer Vision (ICCV), pp. 3813-3824, 2023.
[41] S. Cho, H. Shin, S. Hong. S. An, S. Lee, A. Arnab, P. H. Seo, and S. Kim, "Cat-seg: Cost aggregation for opca-vocyy semantic segmentation," arXiv preprint 1rXiv:2303.1179) 2023.
[42] Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi, "A theoretical analysis of contrastive unsupervised representation learning." arXiv preprint arXi::1902.09229 2019.
12

---

[43] W. Huang, M. Yi, X. Zhao, and Z. Jiang. "Towards the generalization of contrastive self- supervised learning"" arXiv preprint arXiv:2111.00743, 2021. 2021
[44] pharmapsychotic. "Clip interrogator." https://githhnn htttuub..oo//paaaaaae htt.p//b.cooo/paaaaaapppccttiiic
clip- interrogator, 2022.
[45] Li, D. Li, C. Xiong, and S. Hoi, "Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation," in International conference on machine learning, pp. 12888-12900, PMLR, 2022.
[46] Z. Wu, S. Song, Khosla, Yu, L. Zhang. X. Tang, and J. Xiao, "3d shapenets: A deep representation for volumetric shapes." 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 1912-1920, 2014.
[47] M. A. Uy Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung, "Revisiting point cloud classification: new benchmark dataset and classification model on real-world data," in Proceedings of the IEEE/CVF international conference on computer vision, pp. 2019.
[48] Wu, Zhang. X. Fu, Y. Wang. J. Ren, L. Pan, W. Wu, L. Yang. J. Wang, C. Qian, D. Lin, and Z. Liu, "Omniobject3d: Large-vocan ange-vocabulary 3d object dataset for realistic perception, reconstrue- tion and generation," " 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pp. 803- 803-814,77 2023.
[49] H. Fu, R. Jia, L. Gao, M. Gong, B. Zhao, S. Maybank, and D. Tao, "3d-future: 3d furniture shape with texture," International Journal of Computer Vision, vol. 129, pp. 3313-3337, 2021.
[50] N. Mu, A. Kirillov, D. Wagner, and S. Xie, "Slip: Self-supervision meets language-image pre-training," in European Conference on Computer Vision, pp. 529-544, Springer, 2022.
[51] A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, "Scannet: Richly-annotater 3d reconstructions of indoor scenes," 2017 IEEE Conference on Computer Vision and Puttern Recognition (CVPR), pp. 2432-2443, 2017.
[52] Y. Lu, C. Xu, X. Wei, X. Xie, M. Tomizuka, K. Keutzer, and S. Zhang, "Open-vocabulary point- cloud object detection without 3d annotation," in Proceedings of the IEEE/CVF conference computer vision and pattern recognition, pp. 1190-1199 199, 2023.
[53] Y. Cao, Z. Yihan, H. Xu, and D. Xu, "Coda: Collaborative novel box discovery and cross-modal alignment for open- apen-vocabulary 3d object detection," Advances Neural Information Processing Systems, vol. 36, 2024.
A Appendix supplemental material
A.1 Broader Impact
The broader impact of OpenDlign, a novel 3D open-world framework, includes both potential benefits and negative impacts associated with its deployment and use. Some considerations are unique to OpenDlign due to its requirement to generate multi-view depth-aligned images, while others are similar those of existing 3D open-world methods like OpenShape [18], and TAMM [20].
Positive Impacts. While OpenDlign's primary focus is on 3D shape understanding. it shows significant potential for 3D scene understanding. Existing 3D open- open-world methods are limited by relying on CAD-rendered images for alignment, as CAD models are typically available only for individual objects, not entire scenes. In contrast, OpenDlign generates depth-aligned images using depth maps instead of CAD models. These depth maps can be produced from point clouds obtained either by sampling from CAD models using real- sca-nee point clouds collected via LiDAR. Using real-scanned point clouds allows OpenDlign to have richly textured scene images for multimodal alignment, facilitating the learning of robust 3D scene representations.
Negative Impacts. Biases present in CLIP can be transferred to OpenDlign, potentially resulting in biased outcomes unfair 3D representations of diverse content. Additionally, the generated
13

---

images may exhibit biased and stereotypical traits due to inherent biases in the training data of ControlNet. OpenDlign requires generating a large number of multi-view depth-aligned images for multimodal alignment using a diffusion model, raising concerns about energy consumption. Furthermore, if a larger CLIIP backbone is released in the future, the increasing fine-tuning costs of OpenDlign could exacerbate these energy concerns.
In summary, OpenDlign shares common concerns with existing 3D open-world frameworks. However, its positive impacts are unique and hold significant potential to transform the design of future 3D open-world understanding pipelines.
A.2 Implementation Details
Training Details. OpenDlign was implemented in PyTorch, utilizing the image-text encoders from OpenCLIP-ViT-H-- pre-trained on the DFNSB dataset [24]. OpenDlign uses contour-awar projections to transform point clouds into depth maps with dimensions of 224 x 224 x 3, creating 10 views along the x-axis, ranging from 30 to 330 degrees at intervals of 30 degrees between each pair of views. The multimodal alignment was achieved by fine-tuning 10 epochs on an A100-80 GB GPU, employing the AdamW optimizer and the OneCycle scheduler with peak learning rate of 3 x 10-4 and a batch size of 128. Since we precache the text and image CLIP embeddings of all shapes, training is significantly accelerated, achieving convergence in about 10 hours.
Depth-Aligned Image Generation Details. We employed the ControlNet v1.1 model [40], pre- trained on depth maps, with settings of 20 DDIM steps, 9.0 guidance scale, and a 1.0 control strength for generating depth-aligned images are detailed in Table 6. We discovered that ControlNet [40] is more sensitive to jagged edges in the conditional depth map than Vision- Laniun-Language Models. Our contour-aware projection reduces these jagged edges using bilateral filtering in each depth channel. However, the edges may reappear during the channel squeezing step. To combat this, we applied a 7 x kernel median filter to each projected depth map to further diminish the edges. The output images, initially sized at 256 x 256 x 3, were then downsampled to match the depth maps' dimensions. The entire generation process for ShapeNet [21] spanned 16 days on 8 RTX 6000 GPUs.
aapphedligned images. The prompts for generating high-quality, noise-free
Table 6: Main, positive, and negative prompts guide textually describes the semantic information of the 3D data.
image generation. Metadata
Main Prompt
"A realistic (metadata)."
Positive
"best quality, extremely realistic, very professional, extremely detailed, sharp edge, normal, complete."
Prompts
Negative
"'oo-resolution, very blurry, unrealistic, worst quality, deep depth of field, large depth of field, distorted, cropped, unusual, warped, incomplete."
Prompts
A.3 Discussion
How to ensure multi-view geometry consistency in depth-aligned images? There is no geometric inconsistency because depth-aligned images are generated view by view, each paired with respec- tive depth map. ControlNet's [40] conditional control ensures the generated images maintain the same shape and pose as the input depth map, while text prompt preserves object identity. Since these depth maps originate from the same 3D point cloud, the resulting images remain geometrically consistent across views. Fig. 5 visually demonstrates the generated images are gcometry-ccosistent
Why does increasing texture diversity in multi-view images positively impact open-world 3D learning? First, more effective knowledge transfer is achieved by leveraging inconsistent textures in multi-view generated images, allowing depth maps to align with diverse sets of RGB images, which enhances the transfer of rich 2D knowledge embedded in CLIP to 3D representation learning. Second, generating coometry-consiseee representation learning by focusing on invariant features like object shape, size, and contour, which remain robust to texture variations. Third, similar to previous methods [15, 17, 18), OpenDlign pairs a single-view depth map with a single-view RGB image for contrastive learning, treating each view independently and eliminating the need for texture consistency across views. Lastly, texture features
but
multi-view images benefits 3D
14

---

Figure 5: Examples of multi-view depth maps and their corresponding depth-aligned images.
are generally unreliable in both 3D and 2D data, as shown by the common use of color jittering for data augmentation in 2D contrastive learning. Increasing texture diversity is standard approach to improve the robustness of contrastive learning models.
A.4
New Asset
Our training and evaluation code, along with the depth-specific text prompts, are included in the supplementary material. The generated depth-aligned images and the processed benchmark dataset will be made publicly available provided reviewers upon request for this submission.
A.5 Additional Quantitative Analysis
More Results for Zero-Shot Classification. Table 7 presents the zero-shot classification results of OpenDlign on the long-tail Objaverse-LVIS [23] dataset. Remarkably, even the lightest OpenDlign variant (OpenDlign-B32) outperforms the previous state-of-the-art method by 8.3% in Topl accuracy.
Table 7: Zero-shot classification results Objaverse-LVIS [23]. The best-performing results presented bold, while the second-best results are underlined. Our models are highlighted in blue.
Training
Open- World
CLIP
Objaverse IVVI [23]
Source
Methods
Variant
Top1
Top3
Top5
21) inferences
PoinICLIP [14]
ResNet-50
1.9
4.1
5.8
No Training
PointCLIP [13]
VIT-B-16
4.7
9.5
12.9
CLIP2Point [15]
VIT-B-32
2.7
5.8
7.9
ULIP-PoinBERRT [17]
SLIP [50]
6.2
13.6
17.9
OpenShape-SpurseCae [18]
ViT-bigG-14
11.6
21.8
27.1
OpenShape-PintBERTT [18]
VIT-bigG-14
10.8
20.2
25.0
ShapeNet
TAMM-SpurseCom [20]
ViT-bigG-14
13.6
24.2
29.3
[20]
ViT-higG--14
13.7
24.2
29.2
OpenDlign--332
ViT-B-32
22.0
36.7
43.2
OpenDlign-Bii
VIT-B-16
25.7
42.2
48.8
OpenDlign--.
VFT-L-14
32.6
57.3
OpenDlign
VIT-H-14
35.0
53.1
60.0
More Results for Few-Shot Classification. Table 8 presents the few-shot classification results of OpenDlign the long-tail Objaverse-LVIS [23] dataset. OpenDlign consistently outperforms all baselines from 1-shot to 16-shot scenarios. Surprisingly, although OpenShape demonstrates higher zero-shot classification performance than ULIP, their few-shot results are poorer, indicating potential overfitting during multimodal alignment.
More Results for Zero-Shot 3D Object Detection. We followed the setting in [52] to compare our method with two 3D open-wort world approaches, focusing on object detection tasks OV-3DET [52] and CoDA [53] on the Scannet [51] dataset, shown in Table 9. The results demonstrate that OpenDlign performs comparably the not specifically designed for open-wort oprn-world 3D detection.
CoDA method, which promising given that OpenDlign
15

---

Table 8: Few-shot classification results on Objaverse-LVIS [23].
Objaverse-LVIS [23]
Model
1-Sbot 2-Shot 4-Shot 8-Shot 16-Shot
PLIP-PointBERT [17]
13.3
16.3
20.3
25.3
32.3
OpenShape-Pointee [18]
12.2
DpenShape-SpareeCone [18]
13.4
16.9
19.7
22.9
28.9
TAMM-PoinnBERT [20]
13.5
18.0
22.4
27.3
OpenDlign (ours)
21.4
28.4
34,6
40.0
46.8
Table 9: Comparison of 3D open-vocabulary object detection methods in the same setting as OV-
3DET on ScanNet. 'Mean' represents the average precision across all 20 categories.
Methods
Mcan
toilet
bed
chair
sofa
dresser
table
cabinet
bookshelf
pillow
sink
OV-3DT
42.26
31.50
8.21
14.17
2.98
CoDA
19.32
68.09
44.04
28.72
44.57
3.41
20.23
5.32
0.03
27.95
45,26
OpenDlign (Ours)
41.95
34.13
32.25
12.49
18.12
3.89
25.41
34.53
Methods
hathtub
reffigerator
desk
nightstand
door
curtain
box
lamp
OV-3DE
56.28
10.99
19.27
0.77
0.31
9.59
10.53
3.78
2.11
2.71
CoDA
6.55
12.42
15.15
0.68
7.95
OpenDlign (Ours)
58.20
8.74
17.85
5.49
1.53
11,46
6.82
5.03
1.54
3.17
Ablation Study on Fine-Tuning Strategies. Table 10 demonstrates that fine- theettunig only attention layers of transformer block is adequate for OpenDlign to learn effective 3D representation. More- over, placing the trainable block for fine-tuning an independent branch (residual tuning) crucial, as it preserves the strong capability of pre-trained CLIP in extracting robust image representations from cepth-aligned images.
Table 10: Ablation study on various fine-tuning strategies. 'Residual Tuning' decides whether to tune the transformer blocks connected residually the CLIP encoder. "Trainable Copy" initializes the block randomly or with CLIP parameters. 'Block Count' is the number of trainable blocks.
Residual
Trainable
Block
Lan
MadelNet40 [46]
ScanObjectNN [47]
OmniObject3D [48]
Tuning
Copy
Count
Topl
Top3
Top5
Topl Top3
Top5
Tupl
Top3
TopS
56.8
71.7
77.0
40.0
59.8
72.2
24.4
37.1
43.2
81.8
96.0
98.0
76.4
84.4
45.6
51.8
82.2
95.9
98.4
76.6
83.4
31.0
52.9
82.3
96.5
98.4
58.7
76.6
83.9
30.8
46.2
52.8
82.2
96.3
98.4
58.0
76.3
84.0
30.6
45.7
52.1
82.6
96.2
98.4
59.5
76.8
83.7
31.3
46.7
53,2
80
ScanObjeccNN Accuracy (%)
8b
75
70
85
80
85
TAMM--PintBRRT RTTEEERTTT Tops1 Acc.
45
10
Number Viewpoints
Figure Effect of view count on OpenDlign's 's zero-shot performance on ScanObjectNN [47]
More Results for Varying Number of Views. Fig. 6 demonstrates that OpenDlign's zero-shot
classification performance on the ScanObjectNN dataset generally improves as the number of views increases. Notably, OpenDlign surpasses the leading baseline model, PAMM--PoiBBERT [20], with approximately two views of depth maps. Table presents detailed results of OpenDlign's performance on 16-shot classification with varying view counts across three benchmark datasets. These results further confirm that OpenDlign's 's few-shot performance, like its zero-shot performance, consistently improves with increasing number of views.
16

---

Table 11: Effect of the number of viewpoints for 16-shot classification.
Number of Viewpoints
10
ModelNer40 [46]
74.3
78.3
84.6
86.1
87.4
87.6
SeanOhjectNN [47]
46.0
50.5
60.6
68.8
75.6
79.0
OmniObject3D [48]
40.8
45.7
55.1
59.8
62.5
65.8
A.6 Additional Qualitative Analysis
Effect of Contour-Aware Projection. Fig. 7 visually compares depth maps generated using PointCLIP V2's projection [13] technique and our contour-aware projection method. In the PointCLIP V2 projection, the projected maps very blurry, losing some contour and shape details, evidenced by the bed frame gaps being inaccurately filled. Conversely, our contour-aware projection preserves more of the original objects' contours and structures, accurately showing the bed frame gaps.
PoinICLIP V2
Projection
Contour-Awarr Projection
Figure 7: Comparison between depth maps from PointCLIP V2 [13] and contour-aware projection.
More Visualizations of Cross-Modas Retrieval. More examples of 3D shape retrieval that showcase the cross-modal retrieval capabilities of OpenDlign are illustrated in Fig. 8. The results demonstrate that the 3D shapes retrieved are in semantic alignment with the image and text queries.
(a) Image Query
(b) Text Query
Dinoaar:
"oaseen Cuola"
Figure 8: Additional 3D shape retrieval results. (a) Most similar 3D shapes for each image query. (b) Two most similar 3D shapes for each text query.
Generoted Depth-Aligned
Imaggs
Metadata Text
"Moniter
"Gultr"
"Tenk"
"Washing Machine
"Tuble"
Figure 9; Examples of low-quality generated depth-aligned images.
Examples of Low-Qualite Depth-Aligned Images. Fig. 9 showcases bad examples of depth-aligned images. For instance, the diffusion model may interpret 'monitor' as a person observing something, or it might create unrealistic images such as washing machine made of stone. Additionally, it might depict a tent floating in the sky, which deviates from real-world expectations. These examples underscore the need to eliminate inferior depth-aligned images for more generalized alignment.
17

---

NeurIPS Paper Checklist
1. Claims
Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Question:
Answer: [Yes]
Justification: We clearly highlight our main contributions and the scope this paper in both the abstract and the concluding paragraphs of the introduction. Notably, outline the paper's contributions using bullet-point list for clarity.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results be expected to generalize other settings.
It fine include aspirational goals as motivation long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: have included dedicated "Limitations" section in our paper. For detailed information the limitations of our work, please refer to Section
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on a few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory Assumptions and Proofs
Question: For each theoretical result, does the paper provide the full of assumptions and complete (and correct) proof?
18

---

Answer:
[Yes]
Justification:
See the methodology section
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proof's provided in appendix supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental Result Reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?
Question:
Answer: [Yes]
Justification: All essential components of the OpenDlign architecture are described in the Methodology section 3. The experimental setup for downstream tasks, including zero-shot classification, object detection, and 3D shape retrieval, is detailed in the Experiment section 4. Additionally, Appendix A.2 provides the implementation, training, and image generation details, ensuring the reproducibility of our results.
Guidelines:
The answer NA means that the paper does not include experiments. If the paper includes experiments, No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurlPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) If the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) the contribution is a new model (e.g., a large language model), then there should either be way access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (c.g., to registered users), but it should be possible for other researchers to have some path to reproducing verifying the results.
19

---

5. Open access to data and code
Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
The supplementary material includes the complete code for image generation, model training, and evaluation. All evaluation datasets and depth-aligned image datasets open-sourced.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nin) cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (c.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed run to reproduce the results. See the NeurIPS code and data submission guidelines (https: httpp//pllgggppp/////eeeeeeeeeel //nips "nips.ccppuii wtcppblcc//aaoccccccccccccccecct guwdee/dde//ee /ww.ddeee/oeebmmieeec commissionnl ssisonolicyyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental Setting/Details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary understand the results?
Answer: [Yes]
Justification:
The testing details for various downstream tasks, including zero-shot classifi- cation, few-shot classification, object detection, and cross mossl retrieval, illustrated in Experiment section The paper specifies all the training details, including hyperparameter selection, training resources, and optimizer, in Appendix A.2.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment Statistical Significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: Error bars are not reported due to the computational expense. However, for the few-shot classification analysis, results averaged over 10 random seeds.
Guidelines:
The answer NA means that the paper does not include experiments.
20

---

The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall
with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g.. Normally distributed errors).
should be clear whether the error bar is the standard deviation the standard error of the mean.
It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments Compute Resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: Appendix A.2, we specified that OpenDlign model training was conducted using single A 1100-80 GB GPU, which took 10 hours. Depth-aligned image generation was performed using RTX 6000 GPUs around 16 days.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary failed experiments that didn't It make it into the paper).
9. Code Of Ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https:/ /ttp:////////e/./eeeeee/ee cc wtcp/publc//eee wttph///tthhiiiieeeeeeeeiiee line?
Answer: [Yes]
Justification: The research adheres to the NeurIPS Code of Ethics, ensuring data integrity, transparency, and reproducibility. Both the manuscript and supplementary material follow the code of ethics and remain anonymous until camera-ready.
Guidelines:
The answer NA means that the authors have reviewed the NeurIPS Code of Ethics.
the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g.. if there special consid- eration due to laws or regulations in their jurisdiction).
10. Broader Impacts
Question: Does the paper discuss both potential positive societal impacts and negative socictal impacts of the work performed?
21

---

Answer: [Yes]
Justification:
The broader impacts, both positive and negative, are discussed in Appendix
A.1.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No. they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative socictal impacts include potential malicious unintended uses (c.g.. disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement the quality of diffusion models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (c.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how system learns from feedback over time, improving the efficiency and accessibility ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
The paper does not explicitly pose any risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: All assets used in this paper, including code, pretraining and evaluation datasets, and pre-trained models, properly credited to their original creators owners.
Guidelines:
The answer NA means that the paper does not existing assets.
22

---

The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset is used and, if possible, include URL.
The name of the license ge.g.. CC-BY 4.0) should be included for each asset. For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
assets are released, the license, copyright information, and terms use in the package should be provided. For popular datasets, paperswi thcode. com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
For existing datasets that are re-pan re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New Assets
Question: Are assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [Yes]
Justification: The code for this paper is well-structured with clear usage guidance. All assets have been anonymized. The dataset and checkpoint models will be released upon the paper's acceptance.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/cooe/mooel as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset is used.
At submission time, remember anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
Crowdsourcing and Research with Human Subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
The paper does not involve crowdsourcing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code of Ethies, workers involved data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
23

---

Justification:
The paper does not involve crowdsoureing nor research with human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsoureing nor research with
human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such the institution conducting the review.
24


Reviews:
[{"review_id": "l0ONXWUIdy", "reviewer": "Reviewer_tXWH", "summary": "The paper introduces OpenDlign, a novel approach to 3D open-world learning. Traditional 3D learning models struggle with unseen categories and typically rely on CAD-rendered images, which lack texture diversity and realism, leading to suboptimal alignment and performance. OpenDlign leverages depth-aligned images generated from a diffusion model to enhance texture variation and realism. It seems OpenDlign achieves superior zero-shot and few-shot performance across diverse 3D tasks.", "questions": "Overall the reviewer is concerned about the point 1 and 2 mentioned in the Weaknesses. The reviewer may adjust the score based on the authors' response.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 2, "contribution": 3, "strengths": "1. Overall the core idea of this work makes sense, i.e., instead of using CAD rendering, it uses an off-the-shelf diffusion model to convert 3D assets to 2D images, This increases the diversity of training and should be able to improve the generalisation to unseen data. \n\n2. The performance of the proposed method seems strong.", "weaknesses": "1. Although the method shows a strong performance improvement when using ShapeNet for training, such a gain is not clear when using the \"Ensemble\" dataset for training. For example, in Table 1, TAMM-PointBERT (+dlign) is only higher than TAMM-PointBERT by 1.2% on ModelNet40 Top1, and even worse than on ModelNet40 Top5. Similarly,  when tested on the ScanObjectNN, OpenShape-SparseConv (+dlign) is worse than OpenShape-SparseConv over all metrics. Does it mean the proposed method is not important/beneficial when the training data is large and diverse enough?\n\n2. Probably I missed something, but it seems the authors did not mention the name of the diffusion model they used in the paper. As shown in Figure 2, the \"Multi-View Depth-Aligned Images\" are also fed into CLIP Image Encoder. However, how to ensure multi-view consistency is still an open problem for diffusion models. Usually, such generated multi-view images will have obvious inconsistency. How did the authors address this problem? This might be a contributing factor to the performance drop mentioned in point 1, where methods show decreased performance when using OpenDlign.\n\n3. Section 3.2 indicates that the proposed framework generates a single set of multi-view images for a given 3D asset. It seems logical to generate multiple sets of multi-view images for a 3D asset to enhance training diversity and reduce dependency on image appearance. Is there a specific rationale behind limiting the generation to a single set of multi-view images?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We greatly appreciate the reviewer's insightful feedback and valuable recommendations.\n\n**Weakness 1**: Unclear performance gain on ensemble dataset.\n\nAnswer: We understand the concern regarding the scalability of our methods on large datasets. Here are some explanations and additional experimental results to address this issue:\n\n1. **Low Proportion of Depth-Aligned Images:**  The performance on the ensemble dataset is not significant because only 6\\% of the images are depth-aligned (from ShapeNet), while the rest are CAD-rendered. It is impressive that even using a small portion of depth-aligned images improves the existing SOTA model's performance in most settings.\n\n2. **Small Benchmark Datasets:** Training on the ensemble dataset typically results in larger performance gains on larger datasets like OmniObject3D compared to smaller ones like ScanObjectNN and ModelNet40. This is because the ensemble dataset offers a broader range of categories, which is more advantageous for larger datasets with more labels, as shown in Table 1 L.223.  Specifically, ModelNet40, ScanObjectNN, and OmniObject3D have **40**, **15**, and **216** categories respectively.\n\n3. **Domain Shift in ScanObjectNN:** The performance drop on ScanObjectNN is due to its domain shift with other 3D datasets. ScanObjectNN, being the only real-world point-cloud dataset in our experiments, features more **sparse** and **noisy** point clouds compared to those sampled from 3D meshes (e.g., ModelNet40 and OmniObject3D). This visual difference is demonstrated in Figure 1 of the rebuttal PDF. This domain shift causes previous methods to struggle with improving performance on ScanObjectNN as well. For example, OpenShape-PointBERT increases Top-1 accuracy by 21\\% on OmniObject3D but only by 0.9\\% on ScanObjectNN, as shown in Table 1.\n\n4. **Saturation of ModelNet40 Performance:** ModelNet40 is a relatively easy benchmark, and existing methods have already achieved high performance on this dataset. The 1.2\\% improvement from TAMM-PointBERT to TAMM-PointBERT (+dlign) on ModelNet40 Top-1 accuracy is already more significant than the 0.6\\% improvement of TAMM-PointBERT over OpenShape-PointBERT on the same metric.\n\nTo further validate the benefits of depth-aligned images on a larger dataset, we conducted an additional ablation study. We increased the proportion of depth-aligned images in the training dataset from 6\\% to 10\\%, resulting in a total of 90K multi-view depth-aligned images for training TAMM-PointBERT and OpenShape-SparseConv. The results in the table below can be compared to Table 1 (L.223), where all scores improve, demonstrating that these methods achieve positive gains on all metrics when using more depth-aligned images for training.\n\n| 3D Open-World Model | ModelNet40 Top-1 | ModelNet40 Top-3 | ModelNet40 Top-5 | ScanObjectNN Top-1 | ScanObjectNN Top-3 | ScanObjectNN Top-5 | OmniObject3D Top-1 | OmniObject3D Top-3 | OmniObject3D Top-5 |\n|----------------------|------------------|------------------|------------------|---------------------|---------------------|---------------------|---------------------|---------------------|---------------------|\n| OpenShape-SparseConv (+dlign) | 85.8 (+2.4)     | 96.9 (+1.3)      | 98.4 (+0.6)      | 57.5 (+0.8)         | 79.0 (+0.1)         | 90.0 (+1.4)        | 35.0 (+1.3)         | 52.8 (+3.5)         | 59.6 (+2.2)         |\n| TAMM-PointBERT (+dlign)       | 86.7 (+1.7)     | 97.1 (+0.5)      | 98.5 (+0.4)      | 62.8 (+7.1)         | 83.1 (+2.4)         | 91.7 (+2.8)        | 38.8 (+1.7)         | 56.2 (+2.7)         | 62.8 (+1.0)         |\n\nHowever, we acknowledge that generating depth-aligned images for the entire ensemble dataset is time-consuming and computationally expensive, which will be our focus in future work.\n\n**Weakness 2**: What is the name of the diffusion model? How do you ensure multi-view consistency?\n\nAnswer: The diffusion model used in our work is the ControlNet v1.1 [1] depth model (L.151), which allows us to use depth information to control the pose and shape of generated images.\n\nThere is no multiview inconsistency in terms of geometry because we generate depth-aligned images view by view, each with its respective depth map. ControlNet's conditional control helps maintain the same shape and pose as the input depth map, and a text prompt ensures the generated images have consistent object identity. Since these depth maps come from the same 3D point cloud, they result in consistent depth-aligned images across different views. Figure 2 in the rebuttal PDF provides visual examples to further demonstrate this consistency. Multi-view consistency remains an open problem for diffusion models when generating multi-view images from a single-view input [2,3], rather than from multi-view inputs.\n\nWhile the generated depth-aligned images are geometry-consistent, they are not texture-consistent across views. Each view may have different textures, which we believe positively impacts generalizable representation learning, as discussed in the main paper.\n\n**Weakness 3**: Why not generate multiple sets of multi-view images for a 3D asset to enhance training diversity?\n\nAnswer: Due to computational constraints, we only generated one image per depth map (view) for each 3D asset in our experiments. We agree that generating multiple sets of multi-view images using different random seeds for a 3D asset can further enhance training diversity. This will be considered in our future work.\n\n\n\n[1] ControlNet 1.1: https://github.com/lllyasviel/ControlNet-v1-1-nightly\n\n[2] Shi, Ruoxi, et al. \"Zero123++: a single image to consistent multi-view diffusion base model.\" \n\n[3] Liu, Yuan, et al. \"Syncdreamer: Generating multiview-consistent images from a single-view image.\"", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "biAHNk1tPu", "reviewer": "Reviewer_ky3c", "summary": "The paper introduces **OpenDlign**, a novel framework for open-world 3D representation learning by leveraging depth-aligned images generated from a diffusion model. OpenDlign aims to enhance the realism and texture diversity in the 3D learning process, overcoming the limitations of CAD-rendered images. The method involves fine-tuning the CLIP image encoder with depth-aligned images, achieving superior performance in zero-shot and few-shot classification tasks. Experimental results demonstrate significant improvements over existing state-of-the-art models on various benchmarks like ModelNet40, ScanObjectNN, and OmniObject3D.", "questions": "- More applications: could the authors provide application study in other 3D OV Understand tasks like open-vocabulary 3D object detection methods[1, 2]  \n     &nbsp; \n- Can the authors provide more details on the computational cost and feasibility of generating depth-aligned images for larger datasets?  \n     &nbsp; \n- How do the authors plan to address the potential biases in the depth-aligned images generated by the diffusion model?  \n     &nbsp; \n- Are there any plans to extend the approach to other types of 3D datasets beyond ShapeNet?  \n     &nbsp; \n- Could the authors elaborate on the scalability of the method when using larger CLIP models or other vision-language models?\n\n[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3.  \n[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023", "limitations": "The authors have provided discussions about the limitations.", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- **Innovative Approach**: The introduction of depth-aligned images generated from a diffusion model is a novel and effective way to enhance texture diversity and realism in 3D learning.  \n   &nbsp; \n- **Significant Performance Gains**: The experimental results show substantial improvements in zero-shot and few-shot classification tasks, highlighting the effectiveness of the proposed method.  \n   &nbsp; \n- **Detailed Methodology**: The paper provides a comprehensive description of the methodology, including the contour-aware projection method and the multimodal alignment framework.  \n   &nbsp; \n- **Broader Impact Considerations**: The authors discuss both positive and negative societal impacts, demonstrating awareness of the potential implications of their work.  \n   &nbsp; \n- **Reproducibility**: The paper includes sufficient details on the experimental setup and training process, enhancing the reproducibility of the results.", "weaknesses": "- **Limited Dataset for Depth-Aligned Images**: The generation of depth-aligned images is limited to the ShapeNet dataset, which might restrict the generalizability of the results to other datasets.  \n     &nbsp; \n- **Computational Resources**: The approach requires significant computational resources for generating depth-aligned images and fine-tuning the model, which might not be easily accessible to all researchers.  \n     &nbsp; \n- **Potential Biases**: The paper acknowledges biases in the CLIP and ControlNet models but does not provide detailed strategies for mitigating these biases in the generated depth-aligned images.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's insightful feedback. We’re glad to hear that you found our work innovative, effective and reproducible. \n\n**W1: Limited Dataset for Depth-Aligned Images**\n\nAnswer: Due to limited resources, our generated dataset is currently limited to ShapeNet, as discussed in the limitations section. However, post-submission, we generated more images from the ensemble dataset to enhance our model's generalization capabilities. The table for Reviewer ky3c shows that existing SOTA methods' performance keeps improving on all benchmark datasets as the dataset grows from **52K** to **90K** depth-aligned images.\n\n**W2 & Q2: Computational Resources and Feasibility for Generating Depth-Aligned Images on Large Datasets.**\n\nAnswer: We understand the reviewer's concern about the computational constraints. Below we detail the computational resources necessary for generating depth-aligned images, discuss the feasibility of scaling our method to larger datasets, and confirm data accessibility:\n\n1. **Memory Efficiency**: Multi-view depth-aligned images can be generated on any GPU with 24GB of memory using the ControlNet model, ensuring memory-efficiency.\n\n2. **Time Efficiency**: Initially, generating images took two weeks using eight Quadro RTX 6000 GPUs (released in 2018). With newer RTX 4090 GPUs, which are 2.78 times faster, this process can now be completed in less than a week. The RTX 4090 is a consumer GPU, making it more accessible to researchers.\n\n| GPU Model       | Memory | Time/10-view Image |\n|-----------------|--------|--------------------|\n| Quadro RTX 6000 | 24 GB  | 39s                |\n| RTX 4090        | 24 GB  | 14s                |\n\n3. **Comparison with Rendering**: Previous methods that generate CAD-rendered images using rendering tools (e.g., Blender) also demand considerable computational power and time. Therefore, both CAD-rendered images and depth-aligned image generation face similar computational constraints.\n\n4. **Data Accessibility**: We will open-source the existing depth-aligned image dataset for ShapeNet, allowing direct access for the research community to avoid redundant generation.\n\n5. **Faster ControlNet:** Recent work, like DMD2 [1], could serve as a faster alternative to ControlNet. DMD2 achieves high-quality conditional image generation more efficiently with much fewer inference steps compared to traditional methods like ControlNet. Preliminary results indicate that DMD2's four-step inference process for depth-conditioned image generation is 2.59 times faster than ControlNet, as shown below:\n\n| Model      | Time/10-view Image |\n|------------|--------------------|\n| ControlNet | 14s                |\n| DMD2       | 5.4s               |\n\n**W3 & Q3: Potential Biases in ControlNet.**\n\nAnswer: Firstly, we require further experiments to quantify the effect of bias on the 3D representation robustness. One potential solution to mitigate bias is using a VLM (e.g., GPT-4V) or an image captioning model to evaluate generated image attributes like color, texture, and style. If the generated images have too many consistent attributes (like the same color), the diffusion model can be prompted to regenerate the images using different seeds. We will explore this direction in future work.\n\n**Q1: Application Study on Open-Vocabulary 3D object detection.**\n\nAnswer: In section 4.3, we have demonstrated that OpenDlign outperforms PointCLIP and PointCLIP v2 by a large margin on the open-vocabulary 3D object detection task. Additionally, we followed the setting in [1] to compare our method with OV-3DET and CoDA on scannet dataset. Specifically, we use pseudo-box pre-trained 3DETR to locate objects and OpenDlign (ViT-H-14) to classify objects.\n\nThe results below show that OpenDlign performs comparably to the SOTA CoDA method. This is already promising because our method is not specifically designed for OV 3D detection. For detailed results across all categories, please see Table 1 in the rebuttal PDF.\n\n| Methods   | mAP   | Chair | Sofa  | Dresser | Bathtub | Desk  |\n|-----------|-------|-------|-------|---------|---------|-------|\n| OV-3DET   | 18.02 | 27.06 | 31.50 | 8.21    | 56.28   | 19.27 |\n| CoDA      | 19.32 | 28.72 | 44.57 | 3.41    | 50.51   | 12.42 |\n| OpenDlign | 19.27 | 34.13 | 32.25 | 12.49   | 58.20   | 17.85 |\n\n**Q4: Plans for Extending to Other 3D Datasets?**\n\nAnswer: Yes, we plan to first apply our method to the entire ensemble dataset for a more robust 3D shape understanding. Our next goal is to adapt our approach to 3D scene understanding, inspired by studies like 3D-Vista [2], which have successfully used CLIP for learning 3D scene representations by aligning point clouds, images, and text. We envision our method serving as a strong baseline for this task.\n\n**Q5: Scalability of OpenDlign on Larger CLIPs.**\n\nAnswer: Table 1 in the main paper shows the zero-shot performance of OpenDlign with four CLIP backbones (B-32, B-16, L-14, H14). The short answer is better CLIP in 2D understanding leads to better OpenDlign in 3D understanding. We will elaborate on this relationship in the revised paper. The table below shows the model size and pre-training datasets, and 2D zero-shot classification averaged over 38 datasets of different CLIP backbones (https://github.com/mlfoundations/open_clip/blob/main/docs/openclip_results.csv). ViT-H-14 is one of the most powerful CLIP models so far. We can compare this result with the results in Table 1 to validate OpenDlign's scalability.\n\n| Model      | pretrained | params (M) | Avg acc (\\%)|\n|------------|------------|------------| --------|\n| ViT-B-32   | datacomp_xl | 151.28 | 57.95 | \n| ViT-B-16   | datacomp_xl | 149.62 | 61.47 |\n| ViT-L-14   | datacomp_xl | 427.62 | 66.27 |\n| ViT-H-14-quickgelu   | dfn5b      | 986.11 | 69.61 |\n\n\n[1] Open-vocabulary point-cloud object detection without 3d annotation. (CVPR 2023)\n\n[2] 3d-vista: Pre-trained transformer for 3d vision and text alignment. (ICCV 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6vltZ15Weq", "reviewer": "Reviewer_XK9f", "summary": "This paper proposes OpenDlign, a framework for depth-based 3D understanding by aligning depth-image features through training on generated depth-aligned images, which overcomes the limitations of training on CAD-rendered images. Using point cloud data from ShapeNet, a contour-aware projection method is introduced to produce dense depth maps from the point clouds. To prepare the training data, an off-the-shelf depth-controlled diffusion model is employed to generate the corresponding RGB images from the projected depth maps. Additionally, techniques such as depth-specific text prompts and logit aggregation are adopted to enhance performance. Experiments demonstrate state-of-the-art results in zero/few-shot 3D classification, zero-shot 3D object detection, and cross-modal retrieval tasks.", "questions": "Please refer to the questions in the weaknesses section.", "limitations": "Limitations have been discussed. I don't see any potential negative societal impact in this work.", "rating": 5, "confidence": 3, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "- The main idea of using generated depth-image pairs rather than CAD renderings to align a depth encoder with the image encoder is a reasonable design choice, enhancing generalization and robustness.\n- The experiments are very comprehensive. The proposed method appears to achieve state-of-the-art results across all the evaluated tasks, which is quite impressive. Ablation studies are also very thorough, revealing the contribution of each design choice. However, since I am not an expert in these tasks, I cannot fully confirm the true status of this model's performance compared to other state-of-the-art methods.\n- The paper is generally well-written and easy to follow.", "weaknesses": "- The motivations for the major design choices in this work are questionable. They appear to be driven purely by the evaluation benchmark (on where all methods use point cloud as the input, and some methods convert point cloud into depth later), rather than focusing on training a good encoder specifically for 3D understanding from depth maps. If this is the case, the title could better reflect this by referring to point cloud-image alignment rather than depth-image alignment. Specifically:\n  - Why not use mesh to directly render depth maps? If the starting point is depth-image alignment, there is no need to convert point clouds into depth maps, which is only meaningful if the evaluation benchmarks are based on point cloud inputs. The availability of real depth maps renders one of the major contributions, the contour-aware projection method, completely irrelevant to the topic of this work.\n  - Why not use real images and predict the depth maps using existing depth estimators, such as DepthAnything? This approach should significantly reduce the computation overhead of dataset creation and enable training on even more diverse data. Again, it seems that the current design choices are purely driven by the point cloud-based evaluation benchmark.\n- In L152, the definition of inverse depth is incorrect. Inverse depth, also known as disparity, is 1/D. Depth ControlNet should take the normalized inverse depth (normalized into [0, 1]) as the input, rather than 1 - D.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We appreciate the reviewer's constructive questions and feedback.\n\n**Q1: The motivation of the study is not focused on training a good encoder specifically for 3D understanding from depth maps.**\n\nAnswer:  Our primary motivation is to learn robust point cloud representations for open-vocabulary problems, similar to the goal of PointCLIP [1], as highlighted in its title, **\"Point Cloud Understanding by CLIP\"**. We train a depth encoder mainly to indirectly enhance representation learning from point clouds. While the reviewer's suggestions are beneficial for building a strong depth encoder specifically for depth map understanding, their benefits may be limited when the focus is on point cloud understanding.\n\nAdditionally, we want to clarify any confusion caused by our paper title. By \"depth-aligned images,\" we refer to images generated using depth map information, not the problem of \"depth-image alignment.\" Our approach aims to align point cloud projected depth maps and images, thereby implicitly enhancing **point cloud-image alignment** robustness. Recent works like ULIP and OpenShape use the term \"3D understanding\" broadly without explicitly mentioning \"point cloud,\" but their core objective, like ours, is to learn robust point cloud representations. \n\n**Q2: Why not use mesh to directly render depth maps instead of projecting depth maps from point clouds?**\n\nAnswer: Generating depth maps from point clouds is preferred over using 3D meshes for several reasons:\n\n1. **Real-Time Applicability:** The contour-aware projection method in OpenDlign generates depth maps directly from point clouds, making it more efficient than 3D rendering from meshes. Creating rendered depth maps involves a computationally expensive multi-step process: first, reconstructing the surface from the point cloud to create a 3D mesh, and then rendering the 3D mesh to obtain depth maps. This process introduces noise and makes rendered depth maps impractical for many real-time applications [1].\n\n2. **Consistency with Related Works:** The projection of multi-view depth maps from point clouds is a well-established approach. Studies like PointCLIP, PointCLIP v2, and CLIP2Point also focus on proposing better point cloud-to-depth map projection methods. The motivation of these studies, similar to OpenDlign, is to enhance computational efficiency towards real-time point cloud understanding.\n\n3. **Compatibility with Mainstream Methods:** Mainstream 3D understanding methods, such as PointNet and Point Transformer, operate directly on point clouds. OpenDlign's use of point clouds ensures compatibility and efficient processing with these methods.\n\nIn summary, OpenDlign focuses on point cloud representation learning, not depth map representation learning. If our goal were 3D understanding from depth maps, we could use various methods to obtain them. However, since our focus is on point cloud-based 3D understanding, we constrain our depth maps to be projections from point clouds. We hope these explanations clarify our motivation and approach.\n\n**Q3: Why not use real images and predict the depth maps using existing depth estimators, such as DepthAnything?**\n\nAnswer: Thank you for your insightful question. Using real images and pseudo-depth pairs can indeed increase data diversity. However, a key challenge is the domain discrepancy between depth maps predicted by estimators like DepthAnything and those projected from point clouds. This discrepancy arises because DepthAnything's depth maps are influenced by depth estimation errors, while point cloud-projected maps have noise from depth densification. Training with synthetic mono-depth from DepthAnything might limit the model's ability to generalize to point-cloud projected depth maps due to this domain shift. \n\nTo address this, a potential approach is to pre-train the model with a large dataset of image-pseudo depth pairs and then fine-tune it on projected depth maps. Using multiple depth estimators, such as DepthAnything, MiDaS, and ZoeDepth, to generate pseudo depth can further enhance data diversity and improve robustness. It sounds like a promising direction for our future work.\n\n**Q4: The definition of inverse depth is incorrect.**\n\nAnswer: Thank you for pointing that out. We indeed use normalized inverse depth as conditional control for image generation, as evidenced by the correctly generated images shown in Figure 2 in the rebuttal PDF. We will correct this in the revised manuscript. \n\n[1] PointCLIP: Point Cloud Understanding by CLIP (CVPR 2022)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "6HbSkpCXL4", "reviewer": "Reviewer_kzvy", "summary": "This paper focuses on 3D open-world learning on classification. To address the limitation caused by existing CAD-rendered images in open-world 3D learning, it proposes to generate depth-aligned images from point cloud projected depth maps and diffusion models.", "questions": "Please refer to the weakness part.", "limitations": "No.", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "1. Generating depth-aligned images to assist open-world 3d learning is interesting.\n2.  The method is evaluated on multiple 3D classification tasks.", "weaknesses": "1. It seems that the generated textures among multiple views for a single object are not consistent. Would this largely influence the performance?\n2. Generating images with diffusion models for image/3D understanding is a prevailing area [A]. The author should discuss existing works in the related works. Also, It turns out the method is not novel for me. \n3. 3D understanding uses scene-level points in most downstream applications, please try on 3D scene-level datasets. \n\n[A] Is synthetic data from generative models ready for image recognition?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for the interesting questions.\n\n**Q1: Would inconsistent textures in multi-view images of a single object impact performance?**\n\nAnswer: We believe that inconsistent textures in multi-view images improve model performance in 3D representation learning, rather than hinder it. Here are our supporting arguments and experiment evidence: \n\n**Supporting Arguments:**\n\n1. **Effective Knowledge Transferring**: Inconsistent textures between multi-view generated images allow depth maps to align with diverse sets of RGB images. This diversity enhances the transfer of rich 2D knowledge embedded in CLIP to 3D representation learning.\n\n2. **Invariant Feature Learning**: Generating geometry-consistent but texture-inconsistent multi-view images can be beneficial for learning 3D representations that emphasize object shape, size, and contour, which remain robust to texture variations. This approach makes sense because 3D data primarily provide geometric features rather than color or texture.\n\n3. **Independent View Pairing**: Similar to methods like CLIP2Point, ULIP, and OpenShape, our OpenDlign pairs a single-view depth map with a single-view RGB image for contrastive learning, instead of using all multi-view images of a single object. This means each view is treated independently, eliminating the need to maintain texture consistency across views.\n\n4. **Conventional Augmentation in Contrastive Learning**: Texture features are not considered robust not only in 3D data but also in 2D images, as evidenced by the widely used color jittering for data augmentation in 2D contrastive learning. Improving texture diversity is a conventional strategy to enhance the robustness of contrastive learning models.\n\n**Experiment Evidence:**\n\nTable 1 demonstrates that using generated images with inconsistent textures in (+dlign) settings systematically outperforms texture-consistent CAD-rendered images used by other methods without (+dlign). Furthermore, the ablation study in Table 5 shows that OpenDlign achieves a **3.8\\%** and **6.8\\%** accuracy improvement on the ModelNet and ScanObjectNN datasets, respectively, when using texture-inconsistent images compared to depth-rendCAD with consistent texture.\n\n**Q2: Using diffusion-generated images for 3D understanding is not novel.**\n\nAnswer: We acknowledge the growing interest in using diffusion-generated images for 2D understanding. However, after carefully considering the feedback and reviewing the referenced work, we respectfully disagree with the assertion that our paper's novelty is similar to the referenced work. Below, we outline our reasons:\n\n1. CLIP is designed for 2D representation learning, not 3D understanding. The referenced work [1] also focuses on enhancing CLIP's 2D image understanding using diffusion-generated images. In contrast, our approach transfers CLIP's 2D knowledge into 3D understanding, expanding its application to a new task. Currently, the use of diffusion-generated images in 3D understanding is largely unexplored, as shown by the papers citing [1].\n\n2. The data scarcity challenge in 3D vision is much more severe than in 2D vision. State-of-the-art CLIP models for 2D understanding use billions of real-world image-text pairs, highlighting the substantial volume of pretraining data. The authors [1] recognize the need to explore if diffusion-generated images benefit larger CLIP models pre-trained on extensive datasets. However, the 3D datasets available for training 3D open-world models (i.e., '3D CLIP') are synthetic and contain less than a million samples. Our work mitigates this critical gap, emphasizing its broader impact and significance.\n\n3. The contributions of this study are multifold, including contour-aware depth-map projection, a multimodal alignment framework, depth-aware prompt design, and a continual learning method to mitigate catastrophic forgetting. These innovations consistently result in significant performance improvements, as demonstrated in Table 4. The use of depth-aligned images is one of the key contributions, potentially benefiting all existing 3D open-world methods. \n\n4. In [1], the generated images need only to be consistent with the text prompt. Our work, however, requires images to be consistent with both the object's pose and the text prompt, necessitating a more sophisticated generation process. \n\nFurthermore, we reviewed recent work utilizing diffusion models for various 3D understanding tasks. These studies [2,3] primarily use diffusion features to replace CLIP features for specific downstream applications, such as semantic segmentation [2,3], with very few explicitly generating images. In contrast, our work, similar to PointCLIP and OpenShape, focuses on learning robust 3D representations. These representations are versatile and can enhance various downstream tasks, including point cloud classification and 3D object detection, without being limited to specific applications. 3D classification is just one of the downstream tasks used to evaluate representation robustness.\n\n**Q3: 3D understanding uses scene-level points in most downstream applications, please try on 3D scene-level datasets.**\n\nAnswer: Open-world 3D representation learning is a new field, mainly focusing on object-level understanding due to limited 3D scene data. Our setup fully follows existing methods, and we included additional experiments showing OpenDlign's performance on open-vocabulary 3D detection using the **scene-level ScanNet dataset**, as shown in Table 1 in the rebuttal pdf. As noted in Appendix A.1, we plan to extend our work to scene-level representation learning because CAD-rendered images for multimodal alignment are not always available for 3D scenes.\n\n\n[1] Is synthetic data from generative models ready for image recognition? (ICLR 2023)\n\n[2] Open-Vocabulary 3D Semantic Segmentation with Text-to-Image Diffusion Models. (Arxiv)\n\n[3] Open-vocabulary panoptic segmentation with text-to-image diffusion models.(CVPR 2023)", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:
We thank the reviewers for their detailed and thoughtful feedback on our submission. We are grateful that most reviewers appreciated the soundness of our model design and acknowledged its strong performance on various downstream 3D understanding tasks.

For each reviewer, we addressed their questions point by point in the threads below, but we want to highlight three main points that have caused confusion among reviewers:

1. The depth-aligned images in our approach are consistent with multi-view geometry but not with texture. This design choice aims to more effectively transfer rich 2D semantic knowledge from CLIP for learning robust 3D representations that are invariant to texture or color variations, focusing more on geometric features like object shape and contour.

2. Using depth-aligned images enhances model generalization, regardless of the size of the CLIP backbones or the 3D dataset. This is supported by additional experimental results in the rebuttal PDF and the results in Table 1 of the main paper.

3. The motivation of OpenDlign, similar to the pioneering study PointCLIP, focuses on learning robust 3D representations for point cloud understanding.

If any questions are not answered clearly, we are more than happy to provide further clarification in the discussion session.

Meta-review: The paper leverages an off-the-shelf diffusion model to convert 3D assets into 2D images, increasing the diversity of training data and improving generalization to unseen data. Strong performance was demonstrated on the ModelNet40 and OmniObject3D datasets. Initially, the paper received one reject, two borderline acceptances, and one weak acceptance. The primary concerns raised were the scalability of the methods on large datasets, inconsistent textures in multi-view scenarios, and the novelty of using diffusion-generated images for 3D understanding. However, the authors addressed these concerns effectively in their rebuttal, providing additional experimental results and clarifying the paper's novelty. As a result, the final ratings were three borderline acceptances and one acceptance. Given the consistent shift towards acceptance, the Area Chair recommended the paper for acceptance.
Decision: Accept (poster)