Title: OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images
Venue: NeurIPS 2024
Accepted: yes
Keywords: Open-World

3D Representation Learning

3D Shape Understanding
Abstract: Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point clouds with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this alignment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets highlights the need for effective strategies to transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge in pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on a limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0\% on ModelNet40 and 16.4\% on OmniObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-art models.

OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images

Abstract

Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point clouds with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this alignment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets highlights the need for effective strategies to transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge in pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on a limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0% on ModelNet40 and 16.4% on OmniObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-art models.

1 Introduction

3D understanding, including tasks like 3D object classification [1, 2], 3D scene segmentation [3], and 3D reconstruction [4, 5], is becoming increasingly pivotal in real-world applications like augmented/virtual reality [6, 7], autonomous vehicles [8]. Traditional 3D models [1, 2, 9, 10, 11] are closed-world, recognizing only pre-defined categories and struggling with 'unseen' ones. Recent studies aim to leverage Vision-Language Models (e.g., CLIP [12]) to develop open-world 3D models that generalize beyond 'seen' 3D data, enabling zero-shot handling of various 3D tasks.

Existing open-world 3D learning methods can be classified into depth-based and point-based approaches. Depth-based methods [13, 14, 15] convert point clouds into multi-view depth maps and use the pre-trained CLIP image encoder for 3D representations. A significant challenge is the domain gap since CLIP is pre-trained on RGB images, not depth maps. To mitigate this gap, methods like [15] introduce an additional depth encoder to align depth features with the image and text features from the pre-trained CLIP encoders, as shown in Fig. 1(a). The images used for feature alignment are produced by rendering synthetic CAD models from various camera viewpoints. Point-based methods [16, 17, 18, 19, 20] directly extract 3D representations from point clouds, thereby bypassing the need for depth map projection. However, they also require an extra point encoder for feature alignment to address format disparities between images and point clouds, as shown in Fig. 1(b). Thus, employing

'

A point cloud of

a small airplane.

'

Image

Encoder

Text

Encoder

Point

Encoder

Align

(b) Point-based Method

(a) Depth-based Method

(c) OpenDlign (ours)

'

A depth map of

a small airplane.

'

Image

Encoder

Text

Encoder

Depth

Encoder

Align

Depth

Rendered Image

(Training only)

Text

Point

Cloud

Rendered Image

(Training only)

Text

'

A depth map of

a small airplane.

'

Image

Encoder

Text

Encoder

Align

Depth

Depth-Aligned Image

(Training only)

Text

CAD-rendered

images

Depth-aligned

images

vs.

QO

QO

↑

↑

↑

an extra 3D encoder for multimodal feature alignment of 3D data, CAD-rendered images, and text modalities is a standard practice in modern open-world 3D learning methods.

Despite widespread usage, we argue that CAD-rendered images fall short of providing the visual diversity and realism necessary for robust multimodal alignment. This limitation arises because CAD models in current open-source datasets [21, 22] often feature simplistic or entirely absent textures. These models also struggle to realistically simulate environmental effects like lighting, shadows, and reflections. Moreover, most CAD models maintain visual coherence across viewpoints, leading to overly consistent textures and colors from all angles. To achieve generalizable 3D representations, each image view for alignment is expected to display significant visual variations (See Fig. 1).

Additionally, pretraining an extra 3D encoder for alignment may not fully leverage the rich knowledge in CLIP for 3D understanding due to the significant volume discrepancy between 2D and 3D pretraining datasets. Mainstream 3D datasets like ShapeNet [21] and Objaverse [23] contain fewer than 1 million synthetic 3D objects, significantly less than the vast image datasets such as DFN5B [24] and LAION-5B [25], which include around 5 billion images used to train cutting-edge CLIP models. While direct fine-tuning of CLIP's encoders facilitates more straightforward knowledge transfer, it restricts inputs to depth maps. Yet, developing 3D representations from depth maps is currently less effective than from point clouds for two primary reasons: (1) The current widely used CLIP text prompt templates are tailored for matching with RGB images, not depth maps, and (2) the lack of a robust projection method for creating dense depth maps with smooth contours from point clouds.

In this paper, we present OpenDlign , the first 3D open-world framework that develops 3D representation by aligning with multi-view diffusion model-generated images, termed depth-aligned images . These images benefit from the stochastic nature of the diffusion model, offering greater texture diversity compared to CAD renderings while maintaining the original 3D data's geometric and semantic integrity (See Fig. 1). Remarkably, OpenDlign demonstrates competitive open-world capability by fine-tuning only 6 million parameters of the CLIP image encoder on ShapeNet [21], unleashing CLIP's vast potential in 3D learning (See Fig. 1(c)). The success of this fine-tuning stems from refining the depth map projection pipeline, developing depth-specific text prompts, and introducing a logit aggregation strategy to merge multi-view prediction results. Experimental results reveal that OpenDlign significantly outperforms previous state-of-the-art (SOTA) models on a variety

of 3D tasks, including zero-shot/few-shot classification, object detection, and cross-modal retrieval. In zero-shot classification, OpenDlign achieves accuracy improvements of 8.0% on ModelNet40 and 16.4% on OmniObject3D, the largest real-world 3D shape dataset. Additionally, depth-aligned images markedly enhance the performance of SOTA models, consistently improving results across diverse benchmarks and highlighting their transformative impact on open-world 3D learning pipelines.

The main contributions of this paper are outlined as follows:

We introduce depth-aligned images as a robust alternative to CAD-rendered images for open-world 3D learning. These images, generated from point cloud-projected depth maps using a diffusion model, offer rich and realistic texture diversity across viewpoints.

We propose a multimodal alignment framework that robustly aligns depth maps, depthaligned images, and text through streamlined fine-tuning of the CLIP image encoder.

We develop a contour-aware projection pipeline to produce dense and contour-preserving multi-view depth maps from point clouds.

We present depth-specific text prompts and a logit aggregation strategy to boost OpenDlign's zero-shot capabilities and mitigate catastrophic forgetting during alignment fine-tuning.

2 Related Work

2.1 Open-World 3D Representation Learning

Vision-Language Models (VLMs) such as CLIP [12] have revolutionized 2D representation learning in open-world settings through contrastive learning with large-scale image-text pairs [26, 27, 28, 29]. Building on this, recent studies have adapted CLIP for 3D representation learning, achieving impressive performance in diverse 3D zero-shot tasks [18, 19].

PointCLIP [14], as a pioneering study, utilizes the CLIP image encoder for extracting 3D representations from depth maps of point clouds, achieving zero-shot recognition by aligning with text embeddings of semantic categories. To address CLIP's training bias towards RGB images, Zhu et al. [13] introduced GPT-generated 3D-specific prompts and a denser depth map projection, while CLIP2Point [15] pre-trains a depth encoder for closer alignment with CLIP's encoders. These methods derive representations from depth maps with noisy contours, causing a loss of key shape features needed for precise recognition. Moreover, their reliance on either natural image text prompts or depth-specific prompts generated by GPT-3 [30] for certain categories highlights a lack of versatility in handling diverse 3D contexts. Alternative methods [17, 18, 19, 20] avoid depth map projection by directly aligning point clouds, images, and text using specialized 3D encoders. By scaling up the dataset and encoder sizes, these methods show promise in diverse 3D tasks. However, these methods are limited by their reliance on CAD-rendered images, which have limited texture diversity across views, leading to less generalizable representations. Additionally, the smaller volume of 3D datasets compared to CLIP's training data hinders effective knowledge transfer to point cloud encoders.

In this paper, we substitute rendered images with depth-aligned images generated from a diffusion model to enhance texture diversity. We also fine-tune the CLIP image encoder for 3D representation learning instead of training a new 3D encoder from scratch, reducing the reliance on large 3D datasets.

2.2 Continual Learning in CLIP Fine-Tuning

Continual Learning (CL) in CLIP aims to mitigate catastrophic forgetting [31], ensuring retention of zero-shot capabilities across varied data distributions while fine-tuning to new tasks. CL methods fall into three categories: adaptive-plasticity methods [32, 33], replay methods [34, 35], and architecturebased methods [36, 37]. Adaptive-plasticity methods limit the plasticity of the essential model parameters for past tasks during fine-tuning. For instance, the IMM-Mean [38] method achieves CL by simply averaging parameters of pre-trained and fine-tuned models for inference, although its efficacy might be limited for complex tasks [39]. Replay methods leverage stored exemplars to enable CLIP to recall previously learned knowledge, while they encounter scalability challenges. Without relying on exemplars, architecture-based CL methods dynamically adjust the model's architecture to accommodate new information without losing existing knowledge [39]. In this study, we align the depth map with the RGB image by freezing the pre-trained CLIP encoder weights and

incorporating a trainable transformer-based branch for encoding depth maps, adhering to architecturebased principles. Inspired by IMM-Mean [38], we use pre-trained and fine-tuned model weights to compute classification logits from multi-view depth maps for inference.

3 Methodology

The OpenDlign framework, depicted in Fig. 2, starts with a contour-aware projection method that transforms point clouds into multi-view depth maps with preserved contours. These maps guide a diffusion model to produce depth-aligned images with varied colors and textures, maintaining consistent geometry with the depth maps. OpenDlign then aligns features from depth maps and depthaligned images by fine-tuning a transformer block connected to the pre-trained image encoder. The goal is to align feature embeddings from depth maps and corresponding depth-aligned images using contrastive learning. At inference, 3D representations are composed of embeddings from multi-view depth maps, derived using both fine-tuned and pre-trained encoder states. These embeddings are matched with depth-specific text embeddings, which capture the semantic and visual properties of the depth maps, to generate multi-view logits. These logits are then aggregated to facilitate label prediction in a zero-shot setting. In few-shot scenarios, these embeddings can be further refined with a logistic regressor. Detailed training and model implementation are provided in Appendix A.2.

3.1 Contour-Aware Depth Map Projection

The contour-aware projection method transforms the input point cloud into multi-view depth maps with clear contours. Inspired by the pipeline in [13], this method involves four main steps: Quantize, Densify, Smooth, and Squeeze.

In the Quantize step, for the i th view of point cloud P i , the 3D coordinates ( x, y, z ) ∈ P i are normalized to [0 , 1] and mapped onto a discrete grid G ∈ R H × W × B , where H and W correspond to the dimensions required by the CLIP image encoder, and B is a pre-defined depth dimension. Next, the Densify step enhances G by updating each voxel to the maximum value within its 7 × 7 × 7 neighborhood, yielding a denser map G ′ . Subsequently, the Smooth step applies bilateral filtering to each voxel v i in G ′ , adjusting its intensity I v i to I ′ v i using:

where W v = ∑ v j ∈ S G σ 1 ( ∥ v i -v j ∥ ) G σ 2 ( | I v i -I v j | ) is the normalization factor that ensures voxel weights sum to 1.0. The Gaussian functions G σ 1 and G σ 2 adjust the influence of each neighboring voxel v j within the 5 × 5 × 5 kernel from set S around v i , based on spatial and intensity differences, enhancing contour sharpness and reducing jagged edges in G ′ . Finally, the Squeeze step applies the minimal pooling on the depth channel of the smoothed G ′ , then triples the output to mimic RGB intensity, producing the final depth map D ∈ R H × W × 3 .

3.2 Depth-Aligned Image Generation

A total of 524,700 depth-aligned images are generated from ShapeNet [21], a leading public 3D CAD dataset containing approximately 52,470 models, each annotated with semantic metadata. To generate these images, a point cloud of 10,000 points is sampled from each CAD model, aligning with prior experimental protocols [18, 17]. For each point cloud, 10 views of depth maps are projected using the proposed contour-aware projection method. Subsequently, the ControlNet v1.1 [40] depth model produces depth-aligned images for each contour-aware depth map view, using the CAD model's metadata as text input and the inverse of the depth map 1 D as image generation control. This approach ensures that the generated images remain consistent with the depth maps both geometrically and semantically, while also adding texture diversity across different views. ControlNet utilizes 1 D instead of D for controlling image outputs because it is predominantly pre-trained on depth images where brighter regions indicate closer proximity. Please refer to Appendix A.2 for the positive and negative prompts used in ControlNet to achieve high-fidelity depth-aligned image generation.

(a) 3D Representation Learning via Generated Depth-Aligned Images

Contour-Aware Depth Map Projection

…

Multi-View Depth Maps

CLIP Image Encoder

Transformer

Block

…

…

𝐡

!

1

𝐡

2

3

𝐡

"

3

𝐡

4

3

…

𝐡

!

3

…

…

…

…

…

Transformer Block

…

Transformer Block

Transformer Block

…

𝐡

2

1

…

𝐡

𝟏

𝑹

⋅ 𝐡

𝟏

𝑫

h

'

,

⋅ h

&

-

h

.

,

⋅ h

&

-

h

/

,

⋅ h

&

-

h

&

,

⋅ h

/

-

h

.

,

⋅ h

/

-

h

'

,

⋅ h

/

-

𝐡

"

1

…

h

&

,

⋅ h

'

-

𝐡

𝟐

𝑹

⋅ 𝐡

𝟐

𝑫

h

.

,

⋅ h

'

-

h

/

,

⋅ h

'

-

𝐡

4

1

…

h

&

,

⋅ h

.

-

h

'

,

⋅ h

.

-

h

/

,

⋅ h

.

-

𝐡

𝟑

𝑹

⋅ 𝐡

𝟑

𝑫

𝐡

𝑵

𝑹

⋅ 𝐡

𝑵

𝑫

𝐓

𝟏

𝐓

𝐓

𝑳

𝐓

𝑳

𝒕

Image

…

Multi-View Depth-Aligned Images

唱

叫。

Generative

+

(b) Zero-Shot 3D Classification

(c) Few-Shot 3D Classification

'A depth map

of a {car} 3D model.'

View

1

…

View

!

"

+1

…

airplane

cup

sofa

car

…

Pretrained

Text Encoder

…

'A silhouette of

a {class}.

'A raytraced image,

Kinetic pointillism

'A depth map

of a {class} 3D model'.

Depth-Specific Text

V

!

⋅ F

"

V

#

⋅ F

"

…

𝐅

!

𝐅

𝟑

𝐅

𝟏

𝐅

𝟐

…

𝐕

𝐢

⋅ 𝐅

𝟑

V

!

⋅ F

&

V

!

⋅ F

'

…

𝐕

𝐣

⋅ 𝐅

𝟑

V

#

⋅ F

'

𝐕

𝐢

𝐕

𝐣

V

#

⋅ F

&

1 ≤ 𝑖 ≤ 𝑁/2

Pre-trained

Image Encoder

Fine-tuned

Image Encoder

View

!

"

View

𝑁

…

…

Logistic

Regressor

bike

cup

…

table

car

View

7

"

+1…𝑁

…

View

1…

!

"

…

Pre-trained

Image Encoder

T

!

"

Fine-tuned

Image Encoder

T

!

"

3.3 Multimodal Representation Alignment

OpenDlign aligns depth maps and depth-aligned images by fine-tuning a transformer block residually linked to the last block of the CLIP image encoder, using contrastive learning. With CLIP pre-trained to align images and text, OpenDlign implicitly aligns depth maps with the shared image-text space.

Multimodal Feature Extraction. Given a 3D point cloud input, let D = { D i } N i =1 represent the set of its N projected depth map views, and R = { R i } N i =1 the corresponding set of depth-aligned images. Each image R i is encoded through L layers of a pre-trained CLIP image encoder, { T l ( · ) } L l =1 , to obtain feature representations I R i = T 1 ...L ( R i ) . Each depth map D i is processed up to layer T L -1 , yielding preliminary features T 1 ...L -1 ( D i ) . Subsequently, these depth features are passed through the frozen layer T L and its trainable counterpart T t L , where only the attention layers for spatial interaction in T t L are trainable, as inspired by [41]. This process produces the feature for the i th depth map view

I D i = T 1 ...L ( D i ) + T t L ( T 1 ...L -1 ( D i )) . The final feature vectors for multi-view depth maps D and depth-aligned images R are h D = 1 N ∑ N i =1 ∥ I D i ∥ and h R = 1 N ∑ N i =1 ∥ I R i ∥ , respectively.

Loss Functions. The alignment of h D and h R is achieved by minimizing a composite loss function, comprising the contrastive loss L cont and the feature distance loss L dist, defined as:

̸

In each training batch, ( h D i , h R j ) is a positive pair, while ( h D i , h R k ) and ( h D k , h R j ) are negative pairs where k = i, j . τ is a learnable temperature parameter. The contrastive loss enables learning robust representations by maximizing similarity in positive pairs and minimizing it in negative pairs [42, 43].

3.4 3D Zero-Shot Transfer

The alignment between depth maps and depth-aligned images facilitates 3D zero-shot classification by aggregating multi-view classification logits. Each logit represents the similarity between single-view depth features and text features of category candidates, as illustrated in Fig. 2(b).

Depth-Specific Text Generation. Depth-specific prompt templates are developed by augmenting a base set of 80 text prompts, initially designed for ImageNet classification 2 , with depth-related keywords such as "depth map", "raytraced image", and "silhouette of [CLASS]". These keywords direct OpenDlign's attention to geometric details and contours rather than color or texture. The CLIP Interrogator [44] is a prompt engineering tool that combines CLIP and BLIP [45] to select optimal text prompts for specific images. To identify these keywords, this tool identifies the top 10 prompts that match depth maps from the ShapeNet dataset [21], chosen as targeted keywords. For zero-shot inference, we employ our depth-specific templates to generate 80 text descriptions for each label l . These descriptions { t i } 80 i =1 are encoded by a texture encoder F ( · ) , normalized, and then merged into a unified text feature F l via average pooling, calculated as 1 80 ∑ 80 i =1 ∥ F ( t i ) ∥ .

Multi-View Logits Aggregation. To calculate classification logits, we first gather visual features from multi-view depth maps { V i } N i =1 , aiming to align with depth-specific text features of M candidate labels F = { F i } i M =1 . The feature extraction utilizes a dual-encoder strategy: the first half of the views { V i } N/ 2 i =1 utilize a pre-trained CLIP image encoder, while the second half of the views { V i } N i = N/ 2+1 employs a fine-tuned encoder. The strategy ensures that OpenDlign maintains its capability to recognize previously identifiable depth maps after learning multimodal alignment via fine-tuning. As shown in Fig. 2(b), the logit for a single depth map view is the product of V i and F , with the overall classification logit being the sum of logits across all views, calculated as ∑ N i =1 V i F T .

4 Experiments

4.1 Zero-Shot 3D Classification

We first evaluated OpenDlign under the zero-shot shape classification task on four benchmark datasets: ModelNet40 [46], ScanObjectNN [47], OmniObject3D [48], and Objaverse-LVIS [23]. ModelNet40 offers synthetic 3D CAD models in 40 categories. ScanObjectNN provides real-scanned objects in 15 categories from OBJ_ONLY version. OmniObject3D, the largest, includes 5,911 realscanned objects in 216 categories, well-suited for fine-grained, real-world classification evaluation. Objaverse-LVIS contains 1,156 categories for evaluating long-tail classification. Point cloud sizes are 10,000 points for ModelNet40 and Objaverse-LVIS, 2,048 for ScanObjectNN, and 4,096 for OmniObject3D. OpenDlign was compared against existing methods, including three depth-based methods: PointCLIP [14], PointCLIP V2 [13], and CLIP2Point [15], and three point-based methods: ULIP [17], OpenShape [18], and TAMM [20]. To investigate if depth-aligned images consistently enhance the representational abilities of other 3D open-world methods, we retrained all OpenShape and TAMM variants using their original CAD-rendered images and some depth-aligned images. Note

that the depth-aligned images for all model variants were exclusively generated from ShapeNet [21], while the pretraining CAD-rendered images could come from ShapeNet or a large-scale ensemble dataset [18] that includes Objaverse [23], ShapeNet [21], 3D-Future [49], and ABO [22]. Furthermore, we evaluated OpenDlign's scalability by training it with various CLIP variants.

Table 6 shows OpenDlign substantially outperforms existing methods trained on ShapeNet across all benchmarks, exceeding the previous best, TAMM-SparseConv, by margins of 8.0% on ModelNet40, 1.6% on ScanObjectNN, and 16.4% on OmniObject3D in Top1 accuracy. Appendix A.5 Table 7 demonstrates OpenDlign's excellence in handling long-tail categories by outperforming previous methods by 21.3% on the Objaverse-LVIS dataset. OpenDlign also outperforms the top depthbased method, PointCLIP V2, by 19% on ModelNet40 and 27.4% on OmniObject3D. Notably, OpenDlign outshines all methods pre-trained on the ensemble dataset in the ScanObject3D benchmark. Additionally, its performance increases linearly with the complexity of CLIP variants, outperforming most baselines on ModelNet40 and OmniObject3D, even using the lighter ViT-B-16 CLIP model. Remarkably, using depth-aligned images (+dlign) consistently boosts the performance of OpenShape and TAMM variants pre-trained on the ShapeNet dataset across all benchmarks. Despite depthaligned images being available only for ShapeNet, which comprises just 10% of the ensemble dataset, TAMM-PointBERT (+dlign) shows a 4.8% increase in Top1 accuracy on the ScanObjectNN dataset, and OpenShape-PointBERT (+dlign) records a 1.6% increase on the OmniObject3D. Note that for parts of the ensemble dataset without depth-aligned images, we used CAD-rendered images instead.

4.2 Few-Shot 3D Classification

We then assessed OpenDlign's few-shot classification capability by training a logistic regressor with linear probing on features from N -shot, 10-view depth maps. Similar to the zero-shot scenario, we extracted multi-view features using both fine-tuned and pre-trained OpenDlign encoders. At

inference, the regressor aggregates logits from 10 views to predict the final label. We compared OpenDlign with ULIP [17], OpenShape [18], and TAMM [20], which extract features for training regressor from their point encoders pre-trained on ShapeNet. Table 2 shows OpenDlign outperforms all baselines across varied few-shot scenarios with 1 to 16 training samples per class. OpenDlign significantly outperforms the leading baseline on the OmniObject3D dataset, exceeding it by 8.8% and 11.8% in the 4-shot and 8-shot classification, respectively. See Appendix A.5 for more results.

4.3 Zero-Shot 3D Object Detection

OpenDlign's capability in zero-shot 3D object detection was evaluated on the ScanNet V2 dataset [51], which contains richly annotated 3D indoor scenes in 18 object categories. Following the PointCLIP V2 methodology [13], we started with the pre-trained 3DETR-m [11] model to identify 3D regions of interest, delineate 3D bounding boxes, and extract points within each box. Finally, we applied OpenDlign to these points to generate our predictions. Table 3 illustrates OpenDlign's zero-shot detection prowess using mean Average Precision (mAP) at IoU thresholds of 0.25 and 0.5, achieving scores of 50.72% and 37.97%, respectively. It significantly outperforms PointCLIP V2 by more than 31.75% and 26.44%. Remarkably, OpenDlign can detect the 'Sofa' shape with an AP 50 of 54 . 96% , whereas PointCLIP and V2 score below 10% , demonstrating OpenDlign's superior ability to extract robust 3D representations from sparse and noisy point clouds in real-world indoor scenes.

4.4 Cross-Modal Retrieval

3D shapes were retrieved by computing the cosine similarity between the embeddings of a query and those generated by OpenDlign, followed by a k-nearest neighbors (kNN) analysis to find the most similar shapes. Fig. 3 showcases OpenDlign's ability to match 3D shapes to image and text queries. Column (a) shows its precision in distinguishing sub-categories like grand versus upright pianos from image queries. Column (b) demonstrates successful shape retrieval using distinctive text descriptions like "Batmobile armored". Notably, averaging image and text query embeddings allows OpenDlign to find shapes that combine elements of both queries. For instance, merging a running horse image with the text "man" retrieves both a centaur and a running man, as shown in Fig. 3(c). Similarly, combining a house image with "tree" retrieves a treehouse. See Appendix A.6 for more results.

(b) Text Query

'Batmobile armored.'

'Telephone box.'

'Double-decker bus.'

'Spaceship.'

(a) Image Query

(c) Image + Text Queries

'Man.'

+

'Tree.'

+

4.5 Ablation Study

Ablation studies were conducted on zero-shot classification benchmarks to assess the contribution of each component in OpenDlign. Consistently, all OpenDlign variants used in these studies employed OpenCLIP-ViT-H-14 as their backbone. ShapeNet was the default training dataset for all models.

Contour-Aware Projection. Replacing PointCLIP V2's projection pipeline [13] with our contouraware version enables a pre-trained CLIP to achieve 68.8% zero-shot accuracy on ModelNet40, outperforming several baselines that require extra training (See Table 4). This indicates that CLIP can understand RGB images and depth maps when shape features are highlighted.

Effect of Alignment with Depth-Aligned Images. Table 4 shows that aligning depth maps with depth-aligned images (i.e., depth-dlign) significantly boosts performance, improving Top1 accuracy by around 10% on ScanObjectNN, with or without depth-specific prompts. This indicates that depth-d alignment effectively transfers CLIP's rich knowledge to interpret depth maps.

Further analysis compared depth-dlign alignment against three alternatives: depth-rendCAD (aligning depth maps with CAD-rendered RGB images), dlign-text & depth (aligning depth-aligned images with text before depth-dlign alignment), and depth-text & dlign (simultaneous alignment of depth maps with text and depth-aligned images). Table 5 shows depth-dlign outperforming depth-rendCAD by 6.8% on the ScanObjectNN dataset, confirming concerns that alignment with rendered images may lead to overfitting on specific 3D shapes. Moreover, dlign-text & depth performs worst, suggesting that pre-aligning depth-aligned images with text compromises CLIP's ability to generate robust image representations, thus affecting subsequent depth-dlign alignment efficacy. The superior performance of depth-dlign on ModelNet40 and OmniObject3D compared to depth-text & dlign shows that aligning depth maps with depth-aligned images indirectly aligns with text, making additional text alignment unnecessary and potentially limiting OpenDlign's generalization.

Depth-Specific Texts. Table 4 shows that depth-specific prompts enhance OpenDlign's performance, regardless of using multimodal alignment or logit aggregation. This indicates that some recognition inaccuracies arise from processing input data as typical RGB images instead of depth maps.

Logits Aggregation. Results in Table 4 show that multi-view logit aggregation improves zeroshot classification on all datasets by combining logits from pre-trained and fine-tuned encoders. This approach effectively mitigates the catastrophic forgetting problem in OpenDlign's multimodal alignment, enabling it to recognize 3D objects identifiable by both pre-trained CLIP and OpenDlign.

100

50

OminObject3D Acc.(%)

(%)

06

ModelNet40 Acc.

40

80

TAMM-PointBERT Top-1Acc.

30

Top-1 Acc.

70

Top-3 Acc.

20

Top-5 Acc.

60

Top-5 Acc.

Top-1 Acc.

Top-3 Acc.

TAMM-PointBERT Top-1Acc.

2

4

6

8

10

10

2

4

6

8

10

Number of Viewpoints

Number of Viewpoints

Varying Number of Views. OpenDlign, like other depth-based methods, requires extracting multiple embeddings from multi-view depth maps for zero-shot inference. Fig. 7 shows that OpenDlign's zero-shot accuracy on ModelNet40 and OmniObject3D improves with more depth map views. Notably, OpenDlign achieves top performance, comparable to TAMM-PointBERT, with just two views, balancing latency and effective zero-shot classification.

5 Conclusion

In this study, we introduce OpenDlign, an open-world framework that learns robust 3D representations from multi-view depth maps by efficiently fine-tuning with depth-aligned images, which are more visually diverse than CAD-rendered images. The effectiveness of OpenDlign is validated on various 3D zero-shot and few-shot tasks. We also show that depth-aligned images consistently enhance the performance of existing 3D open-world methods. Future work will explore the application of depth-aligned images in designing open-world models for 3D scenes (See Appendix A.1).

Limitations. Due to limited computational resources, we cannot generate depth-aligned images from the largest 3D dataset [23], containing around 1 million 3D objects. Retraining 3D open-world models with billions of parameters using these images is also too expensive. Moreover, a data filtering strategy is needed to remove low-quality depth-aligned images (See details in Appendix A.6).

References

C. R. Qi, H. Su, K. Mo, and L. J. Guibas, 'Pointnet: Deep learning on point sets for 3d classification and segmentation,' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 652-660, 2017.

C. R. Qi, L. Yi, H. Su, and L. J. Guibas, 'Pointnet++: Deep hierarchical feature learning on point sets in a metric space,' Advances in neural information processing systems , vol. 30, 2017.

L. Tchapmi, C. Choy, I. Armeni, J. Gwak, and S. Savarese, 'Segcloud: Semantic segmentation of 3d point clouds,' in 2017 international conference on 3D vision (3DV) , pp. 537-547, IEEE, 2017.

B. Mildenhall, P. P. Srinivasan, M. Tancik, J. T. Barron, R. Ramamoorthi, and R. Ng, 'Nerf: Representing scenes as neural radiance fields for view synthesis,' Communications of the ACM , vol. 65, no. 1, pp. 99-106, 2021.

J. Jing, Y . Mao, and K. Mikolajczyk, 'Match-stereo-videos: Bidirectional alignment for consistent dynamic stereo matching,' arXiv preprint arXiv:2403.10755 , 2024.

I. Armeni, O. Sener, A. R. Zamir, H. Jiang, I. Brilakis, M. Fischer, and S. Savarese, '3d semantic parsing of large-scale indoor spaces,' in Proceedings of the IEEE conference on computer vision and pattern recognition , pp. 1534-1543, 2016.

T. Vu, K. Kim, T. M. Luu, T. Nguyen, and C. D. Yoo, 'Softgroup for 3d instance segmentation on point clouds,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 2708-2717, 2022.

Y. Zeng, Y. Hu, S. Liu, J. Ye, Y . Han, X. Li, and N. Sun, 'Rt3d: Real-time 3-d vehicle detection in lidar point cloud for autonomous driving,' IEEE Robotics and Automation Letters , vol. 3, no. 4, pp. 3434-3440, 2018.

H. Zhao, L. Jiang, J. Jia, P. H. Torr, and V . Koltun, 'Point transformer,' in Proceedings of the IEEE/CVF international conference on computer vision , pp. 16259-16268, 2021.

B. Li, T. Zhang, and T. Xia, 'Vehicle detection from 3d lidar using fully convolutional network,' arXiv preprint arXiv:1608.07916 , 2016.

I. Misra, R. Girdhar, and A. Joulin, 'An end-to-end transformer model for 3d object detection,' in Proceedings of the IEEE/CVF International Conference on Computer Vision , pp. 2906-2917, 2021.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. , 'Learning transferable visual models from natural language supervision,' in International conference on machine learning , pp. 8748-8763, PMLR, 2021.

X. Zhu, R. Zhang, B. He, Z. Guo, Z. Zeng, Z. Qin, S. Zhang, and P. Gao, 'Pointclip v2: Prompting clip and gpt for powerful 3d open-world learning,' 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 2639-2650, 2022.

R. Zhang, Z. Guo, W. Zhang, K. Li, X. Miao, B. Cui, Y. J. Qiao, P. Gao, and H. Li, 'Pointclip: Point cloud understanding by clip,' 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 8542-8552, 2021.

T. Huang, B. Dong, Y. Yang, X. Huang, R. W. H. Lau, W. Ouyang, and W. Zuo, 'Clip2point: Transfer clip to point cloud classification with image-depth pre-training,' 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 22100-22110, 2022.

D. Hegde, J. M. J. Valanarasu, and V. M. Patel, 'Clip goes 3d: Leveraging prompt tuning for language grounded 3d recognition,' 2023 IEEE/CVF International Conference on Computer Vision Workshops (ICCVW) , pp. 2020-2030, 2023.

L. Xue, M. Gao, C. Xing, R. Martín-Martín, J. Wu, C. Xiong, R. Xu, J. C. Niebles, and S. Savarese, 'Ulip: Learning a unified representation of language, images, and point clouds for 3d understanding,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 1179-1189, 2023.

M. Liu, R. Shi, K. Kuang, Y. Zhu, X. Li, S. Han, H. Cai, F. M. Porikli, and H. Su, 'Openshape: Scaling up 3d shape representation towards open-world understanding,' ArXiv , vol. abs/2305.10764, 2023.

J. Zhou, J. Wang, B. Ma, Y.-S. Liu, T. Huang, and X. Wang, 'Uni3d: Exploring unified 3d representation at scale,' ArXiv , vol. abs/2310.06773, 2023.

Z. Zhang, S. Cao, and Y.-X. Wang, 'Tamm: Triadapter multi-modal learning for 3d shape understanding,' arXiv preprint arXiv:2402.18490 , 2024.

A. X. Chang, T. A. Funkhouser, L. J. Guibas, P. Hanrahan, Q.-X. Huang, Z. Li, S. Savarese, M. Savva, S. Song, H. Su, J. Xiao, L. Yi, and F. Yu, 'Shapenet: An information-rich 3d model repository,' ArXiv , vol. abs/1512.03012, 2015.

J. Collins, S. Goel, K. Deng, A. Luthra, L. Xu, E. Gundogdu, X. Zhang, T. F. Y. Vicente, T. Dideriksen, H. Arora, et al. , 'Abo: Dataset and benchmarks for real-world 3d object understanding,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 21126-21136, 2022.

M. Deitke, D. Schwenk, J. Salvador, L. Weihs, O. Michel, E. VanderBilt, L. Schmidt, K. Ehsani, A. Kembhavi, and A. Farhadi, 'Objaverse: A universe of annotated 3d objects,' 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 13142-13153, 2022.

A. Fang, A. M. Jose, A. Jain, L. Schmidt, A. Toshev, and V. Shankar, 'Data filtering networks,' ArXiv , vol. abs/2309.17425, 2023.

C. Schuhmann, R. Beaumont, R. Vencu, C. Gordon, R. Wightman, M. Cherti, T. Coombes, A. Katta, C. Mullis, M. Wortsman, et al. , 'Laion-5b: An open large-scale dataset for training next generation image-text models,' Advances in Neural Information Processing Systems , vol. 35, pp. 25278-25294, 2022.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui, 'Open-vocabulary object detection via vision and language knowledge distillation,' in International Conference on Learning Representations , 2021.

X. Zhou, R. Girdhar, A. Joulin, P. Krahenbuhl, and I. Misra, 'Detecting twenty-thousand classes using image-level supervision,' ArXiv , vol. abs/2201.02605, 2022.

X. Dong, Y. Zheng, J. Bao, T. Zhang, D. Chen, H. Yang, M. Zeng, W. Zhang, L. Yuan, D. Chen, F. Wen, and N. Yu, 'Maskclip: Masked self-distillation advances contrastive language-image pretraining,' 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 10995-11005, 2022.

F. Liang, B. Wu, X. Dai, K. Li, Y . Zhao, H. Zhang, P. Zhang, P. Vajda, and D. Marculescu, 'Openvocabulary semantic segmentation with mask-adapted clip,' in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pp. 7061-7070, 2023.

T. B. Brown, B. Mann, N. Ryder, M. Subbiah, J. Kaplan, P. Dhariwal, A. Neelakantan, P. Shyam, G. Sastry, A. Askell, S. Agarwal, A. Herbert-Voss, G. Krueger, T. J. Henighan, R. Child, A. Ramesh, D. M. Ziegler, J. Wu, C. Winter, C. Hesse, M. Chen, E. Sigler, M. Litwin, S. Gray, B. Chess, J. Clark, C. Berner, S. McCandlish, A. Radford, I. Sutskever, and D. Amodei, 'Language models are few-shot learners,' ArXiv , vol. abs/2005.14165, 2020.

M. McCloskey and N. J. Cohen, 'Catastrophic interference in connectionist networks: The sequential learning problem,' Psychology of Learning and Motivation , vol. 24, pp. 109-165, 1989.

J. Serrà, D. Surís, M. Miron, and A. Karatzoglou, 'Overcoming catastrophic forgetting with hard attention to the task,' in International Conference on Machine Learning , 2018.

J. Schwarz, W. M. Czarnecki, J. Luketina, A. Grabska-Barwinska, Y. W. Teh, R. Pascanu, and R. Hadsell, 'Progress & compress: A scalable framework for continual learning,' ArXiv , vol. abs/1805.06370, 2018.

D. Isele and A. Cosgun, 'Selective experience replay for lifelong learning,' ArXiv , vol. abs/1802.10269, 2018.

S.-A. Rebuffi, A. Kolesnikov, G. Sperl, and C. H. Lampert, 'icarl: Incremental classifier and representation learning,' 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 5533-5542, 2016.

S. Yan, J. Xie, and X. He, 'Der: Dynamically expandable representation for class incremental learning,' 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 3013-3022, 2021.

J. Yoon, E. Yang, J. Lee, and S. J. Hwang, 'Lifelong learning with dynamically expandable networks,' ArXiv , vol. abs/1708.01547, 2017.

S.-W. Lee, J.-H. Kim, J. Jun, J.-W. Ha, and B.-T. Zhang, 'Overcoming catastrophic forgetting by incremental moment matching,' ArXiv , vol. abs/1703.08475, 2017.

Y. Ding, L. Liu, C. Tian, J. Yang, and H. Ding, 'Don't stop learning: Towards continual learning for the clip model,' ArXiv , vol. abs/2207.09248, 2022.

L. Zhang, A. Rao, and M. Agrawala, 'Adding conditional control to text-to-image diffusion models,' 2023 IEEE/CVF International Conference on Computer Vision (ICCV) , pp. 3813-3824, 2023.

S. Cho, H. Shin, S. Hong, S. An, S. Lee, A. Arnab, P. H. Seo, and S. Kim, 'Cat-seg: Cost aggregation for open-vocabulary semantic segmentation,' arXiv preprint arXiv:2303.11797 , 2023.

S. Arora, H. Khandeparkar, M. Khodak, O. Plevrakis, and N. Saunshi, 'A theoretical analysis of contrastive unsupervised representation learning,' arXiv preprint arXiv:1902.09229 , 2019.

W. Huang, M. Yi, X. Zhao, and Z. Jiang, 'Towards the generalization of contrastive selfsupervised learning,' arXiv preprint arXiv:2111.00743 , 2021.

pharmapsychotic, 'Clip interrogator.' https://github.com/pharmapsychotic/ clip-interrogator , 2022.

J. Li, D. Li, C. Xiong, and S. Hoi, 'Blip: Bootstrapping language-image pre-training for unified vision-language understanding and generation,' in International conference on machine learning , pp. 12888-12900, PMLR, 2022.

Z. Wu, S. Song, A. Khosla, F. Yu, L. Zhang, X. Tang, and J. Xiao, '3d shapenets: A deep representation for volumetric shapes,' 2015 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 1912-1920, 2014.

M. A. Uy, Q.-H. Pham, B.-S. Hua, T. Nguyen, and S.-K. Yeung, 'Revisiting point cloud classification: A new benchmark dataset and classification model on real-world data,' in Proceedings of the IEEE/CVF international conference on computer vision , pp. 1588-1597, 2019.

T. Wu, J. Zhang, X. Fu, Y. Wang, J. Ren, L. Pan, W. Wu, L. Yang, J. Wang, C. Qian, D. Lin, and Z. Liu, 'Omniobject3d: Large-vocabulary 3d object dataset for realistic perception, reconstruction and generation,' 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 803-814, 2023.

H. Fu, R. Jia, L. Gao, M. Gong, B. Zhao, S. Maybank, and D. Tao, '3d-future: 3d furniture shape with texture,' International Journal of Computer Vision , vol. 129, pp. 3313-3337, 2021.

N. Mu, A. Kirillov, D. Wagner, and S. Xie, 'Slip: Self-supervision meets language-image pre-training,' in European Conference on Computer Vision , pp. 529-544, Springer, 2022.

A. Dai, A. X. Chang, M. Savva, M. Halber, T. A. Funkhouser, and M. Nießner, 'Scannet: Richly-annotated 3d reconstructions of indoor scenes,' 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR) , pp. 2432-2443, 2017.

Y. Lu, C. Xu, X. Wei, X. Xie, M. Tomizuka, K. Keutzer, and S. Zhang, 'Open-vocabulary pointcloud object detection without 3d annotation,' in Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pp. 1190-1199, 2023.

Y. Cao, Z. Yihan, H. Xu, and D. Xu, 'Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection,' Advances in Neural Information Processing Systems , vol. 36, 2024.

A Appendix / supplemental material

A.1 Broader Impact

The broader impact of OpenDlign, a novel 3D open-world framework, includes both potential benefits and negative impacts associated with its deployment and use. Some considerations are unique to OpenDlign due to its requirement to generate multi-view depth-aligned images, while others are similar to those of existing 3D open-world methods like OpenShape [18], and TAMM [20].

Positive Impacts. While OpenDlign's primary focus is on 3D shape understanding, it shows significant potential for 3D scene understanding. Existing 3D open-world methods are limited by relying on CAD-rendered images for alignment, as CAD models are typically available only for individual objects, not entire scenes. In contrast, OpenDlign generates depth-aligned images using depth maps instead of CAD models. These depth maps can be produced from point clouds obtained either by sampling from CAD models or using real-scanned point clouds collected via LiDAR. Using real-scanned point clouds allows OpenDlign to have richly textured scene images for multimodal alignment, facilitating the learning of robust 3D scene representations.

Negative Impacts. Biases present in CLIP can be transferred to OpenDlign, potentially resulting in biased outcomes or unfair 3D representations of diverse content. Additionally, the generated

depth-aligned images may exhibit biased and stereotypical traits due to inherent biases in the training data of ControlNet. OpenDlign requires generating a large number of multi-view depth-aligned images for multimodal alignment using a diffusion model, raising concerns about energy consumption. Furthermore, if a larger CLIP backbone is released in the future, the increasing fine-tuning costs of OpenDlign could exacerbate these energy concerns.

In summary, OpenDlign shares common concerns with existing 3D open-world frameworks. However, its positive impacts are unique and hold significant potential to transform the design of future 3D open-world understanding pipelines.

A.2 Implementation Details

Training Details. OpenDlign was implemented in PyTorch, utilizing the image-text encoders from OpenCLIP-ViT-H-14, pre-trained on the DFN5B dataset [24]. OpenDlign uses contour-aware projections to transform point clouds into depth maps with dimensions of 224 × 224 × 3 , creating 10 views along the x-axis, ranging from 30 to 330 degrees at intervals of 30 degrees between each pair of views. The multimodal alignment was achieved by fine-tuning 10 epochs on an A100-80 GB GPU, employing the AdamW optimizer and the OneCycle scheduler with a peak learning rate of 3 × 10 -4 and a batch size of 128. Since we precache the text and image CLIP embeddings of all shapes, training is significantly accelerated, achieving convergence in about 10 hours.

Depth-Aligned Image Generation Details. We employed the ControlNet v1.1 model [40], pretrained on depth maps, with settings of 20 DDIM steps, a 9.0 guidance scale, and a 1.0 control strength for generating depth-aligned images. The prompts for generating high-quality, noise-free depth-aligned images are detailed in Table 6. We discovered that ControlNet [40] is more sensitive to jagged edges in the conditional depth map than Vision-Language Models. Our contour-aware projection reduces these jagged edges using bilateral filtering in each depth channel. However, the edges may reappear during the channel squeezing step. To combat this, we applied a 7 × 7 kernel median filter to each projected depth map to further diminish the edges. The output images, initially sized at 256 × 256 × 3 , were then downsampled to match the depth maps' dimensions. The entire generation process for ShapeNet [21] spanned 16 days on 8 RTX 6000 GPUs.

A.3 Discussion

How to ensure multi-view geometry consistency in depth-aligned images? There is no geometric inconsistency because depth-aligned images are generated view by view, each paired with its respective depth map. ControlNet's [40] conditional control ensures the generated images maintain the same shape and pose as the input depth map, while a text prompt preserves object identity. Since these depth maps originate from the same 3D point cloud, the resulting images remain geometrically consistent across views. Fig. 5 visually demonstrates the generated images are geometry-consistent.

Why does increasing texture diversity in multi-view images positively impact open-world 3D learning? First, more effective knowledge transfer is achieved by leveraging inconsistent textures in multi-view generated images, allowing depth maps to align with diverse sets of RGB images, which enhances the transfer of rich 2D knowledge embedded in CLIP to 3D representation learning. Second, generating geometry-consistent but texture-inconsistent multi-view images benefits 3D representation learning by focusing on invariant features like object shape, size, and contour, which remain robust to texture variations. Third, similar to previous methods [15, 17, 18], OpenDlign pairs a single-view depth map with a single-view RGB image for contrastive learning, treating each view independently and eliminating the need for texture consistency across views. Lastly, texture features

are generally unreliable in both 3D and 2D data, as shown by the common use of color jittering for data augmentation in 2D contrastive learning. Increasing texture diversity is a standard approach to improve the robustness of contrastive learning models.

A.4 New Asset

Our training and evaluation code, along with the depth-specific text prompts, are included in the supplementary material. The generated depth-aligned images and the processed benchmark dataset will be made publicly available or provided to reviewers upon request for this submission.

A.5 Additional Quantitative Analysis

More Results for Zero-Shot Classification. Table 7 presents the zero-shot classification results of OpenDlign on the long-tail Objaverse-LVIS [23] dataset. Remarkably, even the lightest OpenDlign variant (OpenDlign-B32) outperforms the previous state-of-the-art method by 8.3% in Top1 accuracy.

More Results for Few-Shot Classification. Table 8 presents the few-shot classification results of OpenDlign on the long-tail Objaverse-LVIS [23] dataset. OpenDlign consistently outperforms all baselines from 1-shot to 16-shot scenarios. Surprisingly, although OpenShape demonstrates higher zero-shot classification performance than ULIP, their few-shot results are poorer, indicating potential overfitting during multimodal alignment.

More Results for Zero-Shot 3D Object Detection. We followed the setting in [52] to compare our method with two 3D open-world approaches, focusing on object detection tasks OV-3DET [52] and CoDA [53] on the Scannet [51] dataset, as shown in Table 9. The results demonstrate that OpenDlign performs comparably to the state-of-the-art CoDA method, which is promising given that OpenDlign is not specifically designed for open-world 3D detection.

Ablation Study on Fine-Tuning Strategies. Table 10 demonstrates that fine-tuning only attention layers of 1 transformer block is adequate for OpenDlign to learn effective 3D representation. Moreover, placing the trainable block for fine-tuning in an independent branch (residual tuning) is crucial, as it preserves the strong capability of pre-trained CLIP in extracting robust image representations from depth-aligned images.

90

ScanObjectNN Accuracy (%)

65 7055 60 5 50

Top-1 Acc.

Top-3 Acc.

Top-5Acc.

TAMM-PointBERTTop-1Acc.

45

2

4

6

8

10

NumberofViewpoints

More Results for Varying Number of Views. Fig. 6 demonstrates that OpenDlign's zero-shot classification performance on the ScanObjectNN dataset generally improves as the number of views increases. Notably, OpenDlign surpasses the leading baseline model, TAMM-PointBERT [20], with approximately two views of depth maps. Table 11 presents detailed results of OpenDlign's performance on 16-shot classification with varying view counts across three benchmark datasets. These results further confirm that OpenDlign's few-shot performance, like its zero-shot performance, consistently improves with an increasing number of views.

A.6 Additional Qualitative Analysis

Effect of Contour-Aware Projection. Fig. 7 visually compares depth maps generated using PointCLIP V2's projection [13] technique and our contour-aware projection method. In the PointCLIP V2 projection, the projected maps are very blurry, losing some contour and shape details, as evidenced by the bed frame gaps being inaccurately filled. Conversely, our contour-aware projection preserves more of the original objects' contours and structures, accurately showing the bed frame gaps.

PointCLIP V2

Projection

Contour-Aware

Projection

More Visualizations of Cross-Modal Retrieval. More examples of 3D shape retrieval that showcase the cross-modal retrieval capabilities of OpenDlign are illustrated in Fig. 8. The results demonstrate that the 3D shapes retrieved are in semantic alignment with the image and text queries.

(a) Image Query

(b) Text Query

'Aircraft.'

'Dinosaur.'

'Ancient Castle.'

'Monitor'

'Tent'

'Washing Machine'

'Table'

Generated

Depth-Aligned

Images

Metadata Text

Examples of Low-Quality Depth-Aligned Images. Fig. 9 showcases bad examples of depth-aligned images. For instance, the diffusion model may interpret 'monitor' as a person observing something, or it might create unrealistic images such as a washing machine made of stone. Additionally, it might depict a tent floating in the sky, which deviates from real-world expectations. These examples underscore the need to eliminate inferior depth-aligned images for more generalized alignment.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: We clearly highlight our main contributions and the scope of this paper in both the abstract and the concluding paragraphs of the introduction. Notably, we outline the paper's contributions using a bullet-point list for clarity.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors? Answer: [Yes]

Justification: We have included a dedicated "Limitations" section in our paper. For detailed information on the limitations of our work, please refer to Section 5.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See the methodology section 3.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: All essential components of the OpenDlign architecture are described in the Methodology section 3. The experimental setup for downstream tasks, including zero-shot classification, object detection, and 3D shape retrieval, is detailed in the Experiment section 4. Additionally, Appendix A.2 provides the implementation, training, and image generation details, ensuring the reproducibility of our results.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: The supplementary material includes the complete code for image generation, model training, and evaluation. All evaluation datasets and depth-aligned image datasets are open-sourced.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The testing details for various downstream tasks, including zero-shot classification, few-shot classification, object detection, and cross-modal retrieval, are illustrated in Experiment section 4. The paper specifies all the training details, including hyperparameter selection, training resources, and optimizer, in Appendix A.2.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [No]

Justification: Error bars are not reported due to the computational expense. However, for the few-shot classification analysis, our results are averaged over 10 random seeds.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: In Appendix A.2, we specified that OpenDlign model training was conducted using a single A100-80 GB GPU, which took 10 hours. Depth-aligned image generation was performed using 8 RTX 6000 GPUs around 16 days.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The research adheres to the NeurIPS Code of Ethics, ensuring data integrity, transparency, and reproducibility. Both the manuscript and supplementary material follow the code of ethics and remain anonymous until camera-ready.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [Yes]

Justification: The broader impacts, both positive and negative, are discussed in Appendix A.1.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of diffusion models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pre-trained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper does not explicitly pose any risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: All assets used in this paper, including code, pretraining and evaluation datasets, and pre-trained models, are properly credited to their original creators or owners.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [Yes]

Justification: The code for this paper is well-structured with clear usage guidance. All assets have been anonymized. The dataset and checkpoint models will be released upon the paper's acceptance.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper focuses on 3D open-world learning on classification. To address the limitation caused by existing CAD-rendered images in open-world 3D learning, it proposes to generate depth-aligned images from point cloud projected depth maps and diffusion models.

STRENGTHS:
1. Generating depth-aligned images to assist open-world 3d learning is interesting.
2.  The method is evaluated on multiple 3D classification tasks.

WEAKNESSES:
1. It seems that the generated textures among multiple views for a single object are not consistent. Would this largely influence the performance?
2. Generating images with diffusion models for image/3D understanding is a prevailing area [A]. The author should discuss existing works in the related works. Also, It turns out the method is not novel for me. 
3. 3D understanding uses scene-level points in most downstream applications, please try on 3D scene-level datasets. 

[A] Is synthetic data from generative models ready for image recognition?

QUESTIONS:
Please refer to the weakness part.

LIMITATIONS:
No.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2024-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes OpenDlign, a framework for depth-based 3D understanding by aligning depth-image features through training on generated depth-aligned images, which overcomes the limitations of training on CAD-rendered images. Using point cloud data from ShapeNet, a contour-aware projection method is introduced to produce dense depth maps from the point clouds. To prepare the training data, an off-the-shelf depth-controlled diffusion model is employed to generate the corresponding RGB images from the projected depth maps. Additionally, techniques such as depth-specific text prompts and logit aggregation are adopted to enhance performance. Experiments demonstrate state-of-the-art results in zero/few-shot 3D classification, zero-shot 3D object detection, and cross-modal retrieval tasks.\n\nSTRENGTHS:\n- The main idea of using generated depth-image pairs rather than CAD renderings to align a depth encoder with the image encoder is a reasonable design choice, enhancing generalization and robustness.\n- The experiments are very comprehensive. The proposed method appears to achieve state-of-the-art results across all the evaluated tasks, which is quite impressive. Ablation studies are also very thorough, revealing the contribution of each design choice. However, since I am not an expert in these tasks, I cannot fully confirm the true status of this model's performance compared to other state-of-the-art methods.\n- The paper is generally well-written and easy to follow.\n\nWEAKNESSES:\n- The motivations for the major design choices in this work are questionable. They appear to be driven purely by the evaluation benchmark (on where all methods use point cloud as the input, and some methods convert point cloud into depth later), rather than focusing on training a good encoder specifically for 3D understanding from depth maps. If this is the case, the title could better reflect this by referring to point cloud-image alignment rather than depth-image alignment. Specifically:\n  - Why not use mesh to directly render depth maps? If the starting point is depth-image alignment, there is no need to convert point clouds into depth maps, which is only meaningful if the evaluation benchmarks are based on point cloud inputs. The availability of real depth maps renders one of the major contributions, the contour-aware projection method, completely irrelevant to the topic of this work.\n  - Why not use real images and predict the depth maps using existing depth estimators, such as DepthAnything? This approach should significantly reduce the computation overhead of dataset creation and enable training on even more diverse data. Again, it seems that the current design choices are purely driven by the point cloud-based evaluation benchmark.\n- In L152, the definition of inverse depth is incorrect. Inverse depth, also known as disparity, is 1/D. Depth ControlNet should take the normalized inverse depth (normalized into [0, 1]) as the input, rather than 1 - D.\n\nQUESTIONS:\nPlease refer to the questions in the weaknesses section.\n\nLIMITATIONS:\nLimitations have been discussed. I don't see any potential negative societal impact in this work.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces **OpenDlign**, a novel framework for open-world 3D representation learning by leveraging depth-aligned images generated from a diffusion model. OpenDlign aims to enhance the realism and texture diversity in the 3D learning process, overcoming the limitations of CAD-rendered images. The method involves fine-tuning the CLIP image encoder with depth-aligned images, achieving superior performance in zero-shot and few-shot classification tasks. Experimental results demonstrate significant improvements over existing state-of-the-art models on various benchmarks like ModelNet40, ScanObjectNN, and OmniObject3D.\n\nSTRENGTHS:\n- **Innovative Approach**: The introduction of depth-aligned images generated from a diffusion model is a novel and effective way to enhance texture diversity and realism in 3D learning.  \n   &nbsp; \n- **Significant Performance Gains**: The experimental results show substantial improvements in zero-shot and few-shot classification tasks, highlighting the effectiveness of the proposed method.  \n   &nbsp; \n- **Detailed Methodology**: The paper provides a comprehensive description of the methodology, including the contour-aware projection method and the multimodal alignment framework.  \n   &nbsp; \n- **Broader Impact Considerations**: The authors discuss both positive and negative societal impacts, demonstrating awareness of the potential implications of their work.  \n   &nbsp; \n- **Reproducibility**: The paper includes sufficient details on the experimental setup and training process, enhancing the reproducibility of the results.\n\nWEAKNESSES:\n- **Limited Dataset for Depth-Aligned Images**: The generation of depth-aligned images is limited to the ShapeNet dataset, which might restrict the generalizability of the results to other datasets.  \n     &nbsp; \n- **Computational Resources**: The approach requires significant computational resources for generating depth-aligned images and fine-tuning the model, which might not be easily accessible to all researchers.  \n     &nbsp; \n- **Potential Biases**: The paper acknowledges biases in the CLIP and ControlNet models but does not provide detailed strategies for mitigating these biases in the generated depth-aligned images.\n\nQUESTIONS:\n- More applications: could the authors provide application study in other 3D OV Understand tasks like open-vocabulary 3D object detection methods[1, 2]  \n     &nbsp; \n- Can the authors provide more details on the computational cost and feasibility of generating depth-aligned images for larger datasets?  \n     &nbsp; \n- How do the authors plan to address the potential biases in the depth-aligned images generated by the diffusion model?  \n     &nbsp; \n- Are there any plans to extend the approach to other types of 3D datasets beyond ShapeNet?  \n     &nbsp; \n- Could the authors elaborate on the scalability of the method when using larger CLIP models or other vision-language models?\n\n[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3.  \n[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023\n\nLIMITATIONS:\nThe authors have provided discussions about the limitations.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-13"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper introduces OpenDlign, a novel approach to 3D open-world learning. Traditional 3D learning models struggle with unseen categories and typically rely on CAD-rendered images, which lack texture diversity and realism, leading to suboptimal alignment and performance. OpenDlign leverages depth-aligned images generated from a diffusion model to enhance texture variation and realism. It seems OpenDlign achieves superior zero-shot and few-shot performance across diverse 3D tasks.\n\nSTRENGTHS:\n1. Overall the core idea of this work makes sense, i.e., instead of using CAD rendering, it uses an off-the-shelf diffusion model to convert 3D assets to 2D images, This increases the diversity of training and should be able to improve the generalisation to unseen data. \n\n2. The performance of the proposed method seems strong.\n\nWEAKNESSES:\n1. Although the method shows a strong performance improvement when using ShapeNet for training, such a gain is not clear when using the \"Ensemble\" dataset for training. For example, in Table 1, TAMM-PointBERT (+dlign) is only higher than TAMM-PointBERT by 1.2% on ModelNet40 Top1, and even worse than on ModelNet40 Top5. Similarly,  when tested on the ScanObjectNN, OpenShape-SparseConv (+dlign) is worse than OpenShape-SparseConv over all metrics. Does it mean the proposed method is not important/beneficial when the training data is large and diverse enough?\n\n2. Probably I missed something, but it seems the authors did not mention the name of the diffusion model they used in the paper. As shown in Figure 2, the \"Multi-View Depth-Aligned Images\" are also fed into CLIP Image Encoder. However, how to ensure multi-view consistency is still an open problem for diffusion models. Usually, such generated multi-view images will have obvious inconsistency. How did the authors address this problem? This might be a contributing factor to the performance drop mentioned in point 1, where methods show decreased performance when using OpenDlign.\n\n3. Section 3.2 indicates that the proposed framework generates a single set of multi-view images for a given 3D asset. It seems logical to generate multiple sets of multi-view images for a 3D asset to enhance training diversity and reduce dependency on image appearance. Is there a specific rationale behind limiting the generation to a single set of multi-view images?\n\nQUESTIONS:\nOverall the reviewer is concerned about the point 1 and 2 mentioned in the Weaknesses. The reviewer may adjust the score based on the authors' response.\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-14"}