OpenDlign: Open-World Point Cloud Understanding with Depth-Aligned Images

Authors: ye  mao
junpeng  jing
krystian  mikolajczyk
Venue: NeurIPS_cc_2024_Conference
Published: 2024-04-23 22:02:48

Abstract: Recent open-world 3D representation learning methods using Vision-Language Models (VLMs) to align 3D point clouds with image-text information have shown superior 3D zero-shot performance. However, CAD-rendered images for this alignment often lack realism and texture variation, compromising alignment robustness. Moreover, the volume discrepancy between 3D and 2D pretraining datasets highlights the need for effective strategies to transfer the representational abilities of VLMs to 3D learning. In this paper, we present OpenDlign, a novel open-world 3D model using depth-aligned images generated from a diffusion model for robust multimodal alignment. These images exhibit greater texture diversity than CAD renderings due to the stochastic nature of the diffusion model. By refining the depth map projection pipeline and designing depth-specific prompts, OpenDlign leverages rich knowledge in pre-trained VLM for 3D representation learning with streamlined fine-tuning. Our experiments show that OpenDlign achieves high zero-shot and few-shot performance on diverse 3D tasks, despite only fine-tuning 6 million parameters on a limited ShapeNet dataset. In zero-shot classification, OpenDlign surpasses previous models by 8.0\% on ModelNet40 and 16.4\% on OmniObject3D. Additionally, using depth-aligned images for multimodal alignment consistently enhances the performance of other state-of-the-art models.

Main paper: 
 introduction: 
 3D understanding, including tasks like 3D object classification [1, 2] , 3D scene segmentation [3] , and 3D reconstruction [4, 5] , is becoming increasingly pivotal in real-world applications like augmented/virtual reality [6, 7] , autonomous vehicles [8] . Traditional 3D models [1, 2, 9, 10, 11] are closed-world, recognizing only pre-defined categories and struggling with 'unseen' ones. Recent studies aim to leverage Vision-Language Models (e.g., CLIP [12] ) to develop open-world 3D models that generalize beyond 'seen' 3D data, enabling zero-shot handling of various 3D tasks.Existing open-world 3D learning methods can be classified into depth-based and point-based approaches. Depth-based methods [13, 14, 15] convert point clouds into multi-view depth maps and use the pre-trained CLIP image encoder for 3D representations. A significant challenge is the domain gap since CLIP is pre-trained on RGB images, not depth maps. To mitigate this gap, methods like [15] introduce an additional depth encoder to align depth features with the image and text features from the pre-trained CLIP encoders, as shown in Fig. 1(a) . The images used for feature alignment are produced by rendering synthetic CAD models from various camera viewpoints. Point-based methods [16, 17, 18, 19, 20] directly extract 3D representations from point clouds, thereby bypassing the need for depth map projection. However, they also require an extra point encoder for feature alignment to address format disparities between images and point clouds, as shown in Fig. 1 an extra 3D encoder for multimodal feature alignment of 3D data, CAD-rendered images, and text modalities is a standard practice in modern open-world 3D learning methods.Despite widespread usage, we argue that CAD-rendered images fall short of providing the visual diversity and realism necessary for robust multimodal alignment. This limitation arises because CAD models in current open-source datasets [21, 22] often feature simplistic or entirely absent textures. These models also struggle to realistically simulate environmental effects like lighting, shadows, and reflections. Moreover, most CAD models maintain visual coherence across viewpoints, leading to overly consistent textures and colors from all angles. To achieve generalizable 3D representations, each image view for alignment is expected to display significant visual variations (See Fig. 1 ).Additionally, pretraining an extra 3D encoder for alignment may not fully leverage the rich knowledge in CLIP for 3D understanding due to the significant volume discrepancy between 2D and 3D pretraining datasets. Mainstream 3D datasets like ShapeNet [21] and Objaverse [23] contain fewer than 1 million synthetic 3D objects, significantly less than the vast image datasets such as DFN5B [24] and LAION-5B [25] , which include around 5 billion images used to train cutting-edge CLIP models. While direct fine-tuning of CLIP's encoders facilitates more straightforward knowledge transfer, it restricts inputs to depth maps. Yet, developing 3D representations from depth maps is currently less effective than from point clouds for two primary reasons: (1) The current widely used CLIP text prompt templates are tailored for matching with RGB images, not depth maps, and (2) the lack of a robust projection method for creating dense depth maps with smooth contours from point clouds.In this paper, we present OpenDlign, the first 3D open-world framework that develops 3D representation by aligning with multi-view diffusion model-generated images, termed depth-aligned images. These images benefit from the stochastic nature of the diffusion model, offering greater texture diversity compared to CAD renderings while maintaining the original 3D data's geometric and semantic integrity (See Fig. 1 ). Remarkably, OpenDlign demonstrates competitive open-world capability by fine-tuning only 6 million parameters of the CLIP image encoder on ShapeNet [21] , unleashing CLIP's vast potential in 3D learning (See Fig. 1 (c)). The success of this fine-tuning stems from refining the depth map projection pipeline, developing depth-specific text prompts, and introducing a logit aggregation strategy to merge multi-view prediction results. Experimental results reveal that OpenDlign significantly outperforms previous state-of-the-art (SOTA) models on a variety of 3D tasks, including zero-shot/few-shot classification, object detection, and cross-modal retrieval.In zero-shot classification, OpenDlign achieves accuracy improvements of 8.0% on ModelNet40 and 16.4% on OmniObject3D, the largest real-world 3D shape dataset. Additionally, depth-aligned images markedly enhance the performance of SOTA models, consistently improving results across diverse benchmarks and highlighting their transformative impact on open-world 3D learning pipelines.The main contributions of this paper are outlined as follows:• We introduce depth-aligned images as a robust alternative to CAD-rendered images for open-world 3D learning. These images, generated from point cloud-projected depth maps using a diffusion model, offer rich and realistic texture diversity across viewpoints.• We propose a multimodal alignment framework that robustly aligns depth maps, depthaligned images, and text through streamlined fine-tuning of the CLIP image encoder.• We develop a contour-aware projection pipeline to produce dense and contour-preserving multi-view depth maps from point clouds.• We present depth-specific text prompts and a logit aggregation strategy to boost OpenDlign's zero-shot capabilities and mitigate catastrophic forgetting during alignment fine-tuning.2 Related Work 
open-world 3d representation learning: 
 Vision-Language Models (VLMs) such as CLIP [12] have revolutionized 2D representation learning in open-world settings through contrastive learning with large-scale image-text pairs [26, 27, 28, 29] . Building on this, recent studies have adapted CLIP for 3D representation learning, achieving impressive performance in diverse 3D zero-shot tasks [18, 19] .PointCLIP [14] , as a pioneering study, utilizes the CLIP image encoder for extracting 3D representations from depth maps of point clouds, achieving zero-shot recognition by aligning with text embeddings of semantic categories. To address CLIP's training bias towards RGB images, Zhu et al. [13] introduced GPT-generated 3D-specific prompts and a denser depth map projection, while CLIP2Point [15] pre-trains a depth encoder for closer alignment with CLIP's encoders. These methods derive representations from depth maps with noisy contours, causing a loss of key shape features needed for precise recognition. Moreover, their reliance on either natural image text prompts or depth-specific prompts generated by GPT-3 [30] for certain categories highlights a lack of versatility in handling diverse 3D contexts. Alternative methods [17, 18, 19, 20] avoid depth map projection by directly aligning point clouds, images, and text using specialized 3D encoders. By scaling up the dataset and encoder sizes, these methods show promise in diverse 3D tasks. However, these methods are limited by their reliance on CAD-rendered images, which have limited texture diversity across views, leading to less generalizable representations. Additionally, the smaller volume of 3D datasets compared to CLIP's training data hinders effective knowledge transfer to point cloud encoders.In this paper, we substitute rendered images with depth-aligned images generated from a diffusion model to enhance texture diversity. We also fine-tune the CLIP image encoder for 3D representation learning instead of training a new 3D encoder from scratch, reducing the reliance on large 3D datasets.Continual Learning (CL) in CLIP aims to mitigate catastrophic forgetting [31] , ensuring retention of zero-shot capabilities across varied data distributions while fine-tuning to new tasks. CL methods fall into three categories: adaptive-plasticity methods [32, 33] , replay methods [34, 35] , and architecturebased methods [36, 37] . Adaptive-plasticity methods limit the plasticity of the essential model parameters for past tasks during fine-tuning. For instance, the IMM-Mean [38] method achieves CL by simply averaging parameters of pre-trained and fine-tuned models for inference, although its efficacy might be limited for complex tasks [39] . Replay methods leverage stored exemplars to enable CLIP to recall previously learned knowledge, while they encounter scalability challenges. Without relying on exemplars, architecture-based CL methods dynamically adjust the model's architecture to accommodate new information without losing existing knowledge [39] . In this study, we align the depth map with the RGB image by freezing the pre-trained CLIP encoder weights and incorporating a trainable transformer-based branch for encoding depth maps, adhering to architecturebased principles. Inspired by IMM-Mean [38] , we use pre-trained and fine-tuned model weights to compute classification logits from multi-view depth maps for inference. 
methodology: 
 The OpenDlign framework, depicted in Fig. 2 , starts with a contour-aware projection method that transforms point clouds into multi-view depth maps with preserved contours. These maps guide a diffusion model to produce depth-aligned images with varied colors and textures, maintaining consistent geometry with the depth maps. OpenDlign then aligns features from depth maps and depthaligned images by fine-tuning a transformer block connected to the pre-trained image encoder. The goal is to align feature embeddings from depth maps and corresponding depth-aligned images using contrastive learning. At inference, 3D representations are composed of embeddings from multi-view depth maps, derived using both fine-tuned and pre-trained encoder states. These embeddings are matched with depth-specific text embeddings, which capture the semantic and visual properties of the depth maps, to generate multi-view logits. These logits are then aggregated to facilitate label prediction in a zero-shot setting. In few-shot scenarios, these embeddings can be further refined with a logistic regressor. Detailed training and model implementation are provided in Appendix A.2.The contour-aware projection method transforms the input point cloud into multi-view depth maps with clear contours. Inspired by the pipeline in [13] , this method involves four main steps: Quantize, Densify, Smooth, and Squeeze.In the Quantize step, for the i th view of point cloud P i , the 3D coordinates (x, y, z) ∈ P i are normalized to [0, 1] and mapped onto a discrete grid G ∈ R H×W ×B , where H and W correspond to the dimensions required by the CLIP image encoder, and B is a pre-defined depth dimension. Next, the Densify step enhances G by updating each voxel to the maximum value within its 7 × 7 × 7 neighborhood, yielding a denser map G ′ . Subsequently, the Smooth step applies bilateral filtering to each voxel v i in G ′ , adjusting its intensity I vi to I ′ vi using:EQUATIONwhereW v = vj ∈S G σ1 (∥v i -v j ∥)G σ2 (|I vi -I vj |)is the normalization factor that ensures voxel weights sum to 1.0. The Gaussian functions G σ1 and G σ2 adjust the influence of each neighboring voxel v j within the 5 × 5 × 5 kernel from set S around v i , based on spatial and intensity differences, enhancing contour sharpness and reducing jagged edges in G ′ . Finally, the Squeeze step applies the minimal pooling on the depth channel of the smoothed G ′ , then triples the output to mimic RGB intensity, producing the final depth map D ∈ R H×W ×3 .A total of 524,700 depth-aligned images are generated from ShapeNet [21] , a leading public 3D CAD dataset containing approximately 52,470 models, each annotated with semantic metadata. To generate these images, a point cloud of 10,000 points is sampled from each CAD model, aligning with prior experimental protocols [18, 17] . For each point cloud, 10 views of depth maps are projected using the proposed contour-aware projection method. Subsequently, the ControlNet v1.1 [40] depth model produces depth-aligned images for each contour-aware depth map view, using the CAD model's metadata as text input and the inverse of the depth map 1 D as image generation control. This approach ensures that the generated images remain consistent with the depth maps both geometrically and semantically, while also adding texture diversity across different views. ControlNet utilizes 1 D instead of D for controlling image outputs because it is predominantly pre-trained on depth images where brighter regions indicate closer proximity. Please refer to Appendix A.2 for the positive and negative prompts used in ControlNet to achieve high-fidelity depth-aligned image generation. V! ⋅ F" V# ⋅ F" … 𝐅 ! 𝐅 𝟑 𝐅 𝟏 𝐅 𝟐 … 𝐕𝐢 ⋅ 𝐅𝟑 V! ⋅ F& V! ⋅ F' … 𝐕𝐣 ⋅ 𝐅𝟑 V# ⋅ F' 𝐕 𝐢 𝐕 𝐣 V# ⋅ F& 1 ≤ 𝑖 ≤ 𝑁/2 𝑁/2 < 𝑗 ≤ 𝑁 Pre-trained Image Encoder Fine-tuned Image Encoder View ! " View 𝑁 … … Contour-Aware Depth Map Projection … Input Point Cloud Multi-View Depth Maps CLIP Image Encoder Transformer Block … … 𝐡 ! 1 𝐡 2 3 𝐡 " 3 𝐡 4 3 … 𝐡 ! 3 … … … … … Transformer Block … Transformer Block Transformer Block … 𝐡 2 1 … 𝐡 𝟏 𝑹 ⋅ 𝐡 𝟏 𝑫 h' , ⋅ h& - h. , ⋅ h& - h / , ⋅ h & - h& , ⋅ h/ - h. , ⋅ h/ - h' , ⋅ h/ - 𝐡 " 1 … h& , ⋅ h' - 𝐡 𝟐 𝑹 ⋅ 𝐡 𝟐 𝑫 h. , ⋅ h' - h / , ⋅ h ' - 𝐡 4 1 … h& , ⋅ h. - h' , ⋅ h. - h / , ⋅ h . - 𝐡 𝟑 𝑹 ⋅ 𝐡 𝟑 𝑫 𝐡 𝑵 𝑹 ⋅ 𝐡 𝑵 𝑫 𝐓 𝟏 𝐓 𝑳#𝟏 𝐓 𝑳 𝐓 𝑳 𝒕 Logistic Regressor bike cup … table car View 7 " + 1 … 𝑁 … View 1 … ! " … Pre-trained Image Encoder Image Generative Model … Multi-View Depth-Aligned Images T ! " Fine-tuned Image Encoder T ! "Figure 2 : Overview of OpenDlign. In (a), OpenDlign converts point clouds into multi-view depth maps using a contour-aware projection, which then helps generate depth-aligned RGB images with diverse textures, geometrically and semantically aligned with the maps. A transformer block, residually connected to the CLIP image encoder, is fine-tuned to align depth maps with depthaligned images for robust 3D representation. For zero-shot classification (b), OpenDlign aggregates multi-view logits from both pre-trained and fine-tuned encoders for label prediction. For few-shot classification (c), it employs a logistic regressor trained on multi-view features from the encoders.OpenDlign aligns depth maps and depth-aligned images by fine-tuning a transformer block residually linked to the last block of the CLIP image encoder, using contrastive learning. With CLIP pre-trained to align images and text, OpenDlign implicitly aligns depth maps with the shared image-text space.Multimodal Feature Extraction. Given a 3D point cloud input, let D = {D i } N i=1 represent the set of its N projected depth map views, andR = {R i } N i=1 the corresponding set of depth-aligned images. Each image R i is encoded through L layers of a pre-trained CLIP image encoder, {T l (•)} L l=1 , to obtain feature representations I R i = T 1...L (R i ).Each depth map D i is processed up to layer T L-1 , yielding preliminary features T 1...L-1 (D i ). Subsequently, these depth features are passed through the frozen layer T L and its trainable counterpart T t L , where only the attention layers for spatial interaction in T t L are trainable, as inspired by [41] . This process produces the feature for the i th depth map viewI D i = T 1...L (D i ) + T t L (T 1...L-1 (D i )).The final feature vectors for multi-view depth maps D and depth-aligned images R areh D = 1 N N i=1 ∥I D i ∥ and h R = 1 N N i=1 ∥I R i ∥,respectively. Loss Functions. The alignment of h D and h R is achieved by minimizing a composite loss function, comprising the contrastive loss L cont and the feature distance loss L dist , defined as:EQUATIONIn each training batch,(h D i , h R j ) is a positive pair, while (h D i , h R k ) and (h D k , h R j )are negative pairs where k ̸ = i, j. τ is a learnable temperature parameter. The contrastive loss enables learning robust representations by maximizing similarity in positive pairs and minimizing it in negative pairs [42, 43] .The alignment between depth maps and depth-aligned images facilitates 3D zero-shot classification by aggregating multi-view classification logits. Each logit represents the similarity between single-view depth features and text features of category candidates, as illustrated in Fig. 2(b) .Depth-Specific Text Generation. Depth-specific prompt templates are developed by augmenting a base set of 80 text prompts, initially designed for ImageNet classification2 , with depth-related keywords such as "depth map", "raytraced image", and "silhouette of [CLASS]". These keywords direct OpenDlign's attention to geometric details and contours rather than color or texture. The CLIP Interrogator [44] is a prompt engineering tool that combines CLIP and BLIP [45] to select optimal text prompts for specific images. To identify these keywords, this tool identifies the top 10 prompts that match depth maps from the ShapeNet dataset [21] , chosen as targeted keywords. For zero-shot inference, we employ our depth-specific templates to generate 80 text descriptions for each label l. These descriptions {t i } 80 i=1 are encoded by a texture encoder F (•), normalized, and then merged into a unified text feature F l via average pooling, calculated as 1 employs a fine-tuned encoder. The strategy ensures that OpenDlign maintains its capability to recognize previously identifiable depth maps after learning multimodal alignment via fine-tuning. As shown in Fig. 2 (b), the logit for a single depth map view is the product of V i and F, with the overall classification logit being the sum of logits across all views, calculated asN i=1 V i F T . 
zero-shot 3d classification: 
 We first evaluated OpenDlign under the zero-shot shape classification task on four benchmark datasets: ModelNet40 [46] , ScanObjectNN [47] , OmniObject3D [48] , and Objaverse-LVIS [23] . ModelNet40 offers synthetic 3D CAD models in 40 categories. ScanObjectNN provides real-scanned objects in 15 categories from OBJ_ONLY version. OmniObject3D, the largest, includes 5,911 realscanned objects in 216 categories, well-suited for fine-grained, real-world classification evaluation. Objaverse-LVIS contains 1,156 categories for evaluating long-tail classification. Point cloud sizes are 10,000 points for ModelNet40 and Objaverse-LVIS, 2,048 for ScanObjectNN, and 4,096 for OmniObject3D. OpenDlign was compared against existing methods, including three depth-based methods: PointCLIP [14] , PointCLIP V2 [13] , and CLIP2Point [15] , and three point-based methods: ULIP [17] , OpenShape [18] , and TAMM [20] . To investigate if depth-aligned images consistently enhance the representational abilities of other 3D open-world methods, we retrained all OpenShape and TAMM variants using their original CAD-rendered images and some depth-aligned images. Note Table 1 : Zero-shot classification results on ModelNet40 [46] , ScanObjectNN [47] and OmniObject3D [48] . The best-performing results are presented in bold, while the second-best results are underlined. Our models are highlighted in blue . that the depth-aligned images for all model variants were exclusively generated from ShapeNet [21] , while the pretraining CAD-rendered images could come from ShapeNet or a large-scale ensemble dataset [18] that includes Objaverse [23] , ShapeNet [21] , 3D-Future [49] , and ABO [22] . Furthermore, we evaluated OpenDlign's scalability by training it with various CLIP variants. We then assessed OpenDlign's few-shot classification capability by training a logistic regressor with linear probing on features from N -shot, 10-view depth maps. Similar to the zero-shot scenario, we extracted multi-view features using both fine-tuned and pre-trained OpenDlign encoders. At inference, the regressor aggregates logits from 10 views to predict the final label. We compared OpenDlign with ULIP [17] , OpenShape [18] , and TAMM [20] , which extract features for training regressor from their point encoders pre-trained on ShapeNet. Table 2 shows OpenDlign outperforms all baselines across varied few-shot scenarios with 1 to 16 training samples per class. OpenDlign significantly outperforms the leading baseline on the OmniObject3D dataset, exceeding it by 8.8% and 11.8% in the 4-shot and 8-shot classification, respectively. See Appendix A.5 for more results.OpenDlign's capability in zero-shot 3D object detection was evaluated on the ScanNet V2 dataset [51] , which contains richly annotated 3D indoor scenes in 18 object categories. Following the PointCLIP V2 methodology [13] , we started with the pre-trained 3DETR-m [11] model to identify 3D regions of interest, delineate 3D bounding boxes, and extract points within each box. Finally, we applied OpenDlign to these points to generate our predictions. 3D shapes were retrieved by computing the cosine similarity between the embeddings of a query and those generated by OpenDlign, followed by a k-nearest neighbors (kNN) analysis to find the most similar shapes. Fig. 3 showcases OpenDlign's ability to match 3D shapes to image and text queries. Column (a) shows its precision in distinguishing sub-categories like grand versus upright pianos from image queries. Column (b) demonstrates successful shape retrieval using distinctive text descriptions like "Batmobile armored". Notably, averaging image and text query embeddings allows OpenDlign to find shapes that combine elements of both queries. For instance, merging a running horse image with the text "man" retrieves both a centaur and a running man, as shown in Fig. 3(c ). Similarly, combining a house image with "tree" retrieves a treehouse. See Appendix A.6 for more results. Ablation studies were conducted on zero-shot classification benchmarks to assess the contribution of each component in OpenDlign. Consistently, all OpenDlign variants used in these studies employed OpenCLIP-ViT-H-14 as their backbone. ShapeNet was the default training dataset for all models.Contour-Aware Projection. Replacing PointCLIP V2's projection pipeline [13] with our contouraware version enables a pre-trained CLIP to achieve 68.8% zero-shot accuracy on ModelNet40, outperforming several baselines that require extra training (See Table 4 ). This indicates that CLIP can understand RGB images and depth maps when shape features are highlighted.Effect of Alignment with Depth-Aligned Images. Table 4 shows that aligning depth maps with depth-aligned images (i.e., depth-dlign) significantly boosts performance, improving Top1 accuracy by around 10% on ScanObjectNN, with or without depth-specific prompts. This indicates that depth-d alignment effectively transfers CLIP's rich knowledge to interpret depth maps. Depth-Specific Texts. Table 4 shows that depth-specific prompts enhance OpenDlign's performance, regardless of using multimodal alignment or logit aggregation. This indicates that some recognition inaccuracies arise from processing input data as typical RGB images instead of depth maps.Logits Aggregation. Results in Table 4 show that multi-view logit aggregation improves zeroshot classification on all datasets by combining logits from pre-trained and fine-tuned encoders. This approach effectively mitigates the catastrophic forgetting problem in OpenDlign's multimodal alignment, enabling it to recognize 3D objects identifiable by both pre-trained CLIP and OpenDlign.Table 5 : Ablation study on various alignment strategies. Aligning with text modality was achieved by fine-tuning the image encoder.Alignment MNet40 [46] ScanNN [47] Omni3D Varying Number of Views. OpenDlign, like other depth-based methods, requires extracting multiple embeddings from multi-view depth maps for zero-shot inference. Fig. 7 shows that OpenDlign's zero-shot accuracy on ModelNet40 and OmniObject3D improves with more depth map views. Notably, OpenDlign achieves top performance, comparable to TAMM-PointBERT, with just two views, balancing latency and effective zero-shot classification. 
conclusion: 
 In this study, we introduce OpenDlign, an open-world framework that learns robust 3D representations from multi-view depth maps by efficiently fine-tuning with depth-aligned images, which are more visually diverse than CAD-rendered images. The effectiveness of OpenDlign is validated on various 3D zero-shot and few-shot tasks. We also show that depth-aligned images consistently enhance the performance of existing 3D open-world methods. Future work will explore the application of depth-aligned images in designing open-world models for 3D scenes (See Appendix A.1).Limitations. Due to limited computational resources, we cannot generate depth-aligned images from the largest 3D dataset [23] , containing around 1 million 3D objects. Retraining 3D open-world models with billions of parameters using these images is also too expensive. Moreover, a data filtering strategy is needed to remove low-quality depth-aligned images (See details in Appendix A.6).depth-aligned images may exhibit biased and stereotypical traits due to inherent biases in the training data of ControlNet. OpenDlign requires generating a large number of multi-view depth-aligned images for multimodal alignment using a diffusion model, raising concerns about energy consumption. Furthermore, if a larger CLIP backbone is released in the future, the increasing fine-tuning costs of OpenDlign could exacerbate these energy concerns.In summary, OpenDlign shares common concerns with existing 3D open-world frameworks. However, its positive impacts are unique and hold significant potential to transform the design of future 3D open-world understanding pipelines. 
code of ethics: 
 Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines? 
broader impacts: 
 Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?• The authors should cite the original paper that produced the code package or dataset.• The authors should state which version of the asset is used and, if possible, include a URL. • The name of the license (e.g., CC-BY 4.0) should be included for each asset.• For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided. • If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset. • For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided. • If this information is not available online, the authors are encouraged to reach out to the asset's creators. 13. New Assets Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets? Answer: [Yes] Justification: The code for this paper is well-structured with clear usage guidance. All assets have been anonymized. The dataset and checkpoint models will be released upon the paper's acceptance.Guidelines:• The answer NA means that the paper does not release new assets.• Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc. • The paper should discuss whether and how consent was obtained from people whose asset is used.  
 
 Appendix: Training Details. OpenDlign was implemented in PyTorch, utilizing the image-text encoders from OpenCLIP-ViT-H-14, pre-trained on the DFN5B dataset [24] . OpenDlign uses contour-aware projections to transform point clouds into depth maps with dimensions of 224 × 224 × 3, creating 10 views along the x-axis, ranging from 30 to 330 degrees at intervals of 30 degrees between each pair of views. The multimodal alignment was achieved by fine-tuning 10 epochs on an A100-80 GB GPU, employing the AdamW optimizer and the OneCycle scheduler with a peak learning rate of 3 × 10 -4 and a batch size of 128. Since we precache the text and image CLIP embeddings of all shapes, training is significantly accelerated, achieving convergence in about 10 hours.Depth-Aligned Image Generation Details. We employed the ControlNet v1.1 model [40] , pretrained on depth maps, with settings of 20 DDIM steps, a 9.0 guidance scale, and a 1.0 control strength for generating depth-aligned images. The prompts for generating high-quality, noise-free depth-aligned images are detailed in Table 6 . We discovered that ControlNet [40] is more sensitive to jagged edges in the conditional depth map than Vision-Language Models. Our contour-aware projection reduces these jagged edges using bilateral filtering in each depth channel. However, the edges may reappear during the channel squeezing step. To combat this, we applied a 7 × 7 kernel median filter to each projected depth map to further diminish the edges. The output images, initially sized at 256 × 256 × 3, were then downsampled to match the depth maps' dimensions. The entire generation process for ShapeNet [21] spanned 16 days on 8 RTX 6000 GPUs. Table 6 : Main, positive, and negative prompts guide depth-aligned image generation. Metadata textually describes the semantic information of the 3D data.Main Prompt "A realistic {metadata}."Positive "best quality, extremely realistic, very professional, Prompts extremely detailed, sharp edge, normal, complete."Negative "low-resolution, very blurry, unrealistic, worst quality, deep depth of field, Prompts large depth of field, distorted, cropped, unusual, warped, incomplete."How to ensure multi-view geometry consistency in depth-aligned images? There is no geometric inconsistency because depth-aligned images are generated view by view, each paired with its respective depth map. ControlNet's [40] conditional control ensures the generated images maintain the same shape and pose as the input depth map, while a text prompt preserves object identity. Since these depth maps originate from the same 3D point cloud, the resulting images remain geometrically consistent across views. Fig. 5 visually demonstrates the generated images are geometry-consistent.Why does increasing texture diversity in multi-view images positively impact open-world 3D learning? First, more effective knowledge transfer is achieved by leveraging inconsistent textures in multi-view generated images, allowing depth maps to align with diverse sets of RGB images, which enhances the transfer of rich 2D knowledge embedded in CLIP to 3D representation learning. Second, generating geometry-consistent but texture-inconsistent multi-view images benefits 3D representation learning by focusing on invariant features like object shape, size, and contour, which remain robust to texture variations. Third, similar to previous methods [15, 17, 18] , OpenDlign pairs a single-view depth map with a single-view RGB image for contrastive learning, treating each view independently and eliminating the need for texture consistency across views. Lastly, texture features with approximately two views of depth maps. Table 11 presents detailed results of OpenDlign's performance on 16-shot classification with varying view counts across three benchmark datasets. These results further confirm that OpenDlign's few-shot performance, like its zero-shot performance, consistently improves with an increasing number of views. A.6 Additional Qualitative Analysis Effect of Contour-Aware Projection. Fig. 7 visually compares depth maps generated using PointCLIP V2's projection [13] technique and our contour-aware projection method. In the PointCLIP V2 projection, the projected maps are very blurry, losing some contour and shape details, as evidenced by the bed frame gaps being inaccurately filled. Conversely, our contour-aware projection preserves more of the original objects' contours and structures, accurately showing the bed frame gaps.Contour-Aware Projection Guidelines:• The answer NA means that the paper does not include experiments.• The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage. • The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute. • The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).Justification: The research adheres to the NeurIPS Code of Ethics, ensuring data integrity, transparency, and reproducibility. Both the manuscript and supplementary material follow the code of ethics and remain anonymous until camera-ready.Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics. • The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).Text Prompts for ImageNet: ImageNet Prompt Engineering. 


Decision: The paper leverages an off-the-shelf diffusion model to convert 3D assets into 2D images, increasing the diversity of training data and improving generalization to unseen data. Strong performance was demonstrated on the ModelNet40 and OmniObject3D datasets. Initially, the paper received one reject, two borderline acceptances, and one weak acceptance. The primary concerns raised were the scalability of the methods on large datasets, inconsistent textures in multi-view scenarios, and the novelty of using diffusion-generated images for 3D understanding. However, the authors addressed these concerns effectively in their rebuttal, providing additional experimental results and clarifying the paper's novelty. As a result, the final ratings were three borderline acceptances and one acceptance. Given the consistent shift towards acceptance, the Area Chair recommended the paper for acceptance.

Reviews: review: paper_summary: paper_summary: The paper introduces OpenDlign, a novel approach to 3D open-world learning. Traditional 3D learning models struggle with unseen categories and typically rely on CAD-rendered images, which lack texture diversity and realism, leading to suboptimal alignment and performance. OpenDlign leverages depth-aligned images generated from a diffusion model to enhance texture variation and realism. It seems OpenDlign achieves superior zero-shot and few-shot performance across diverse 3D tasks.
strength_weakness: strength_weakness: 1. Although the method shows a strong performance improvement when using ShapeNet for training, such a gain is not clear when using the "Ensemble" dataset for training. For example, in Table 1, TAMM-PointBERT (+dlign) is only higher than TAMM-PointBERT by 1.2% on ModelNet40 Top1, and even worse than on ModelNet40 Top5. Similarly,  when tested on the ScanObjectNN, OpenShape-SparseConv (+dlign) is worse than OpenShape-SparseConv over all metrics. Does it mean the proposed method is not important/beneficial when the training data is large and diverse enough?

2. Probably I missed something, but it seems the authors did not mention the name of the diffusion model they used in the paper. As shown in Figure 2, the "Multi-View Depth-Aligned Images" are also fed into CLIP Image Encoder. However, how to ensure multi-view consistency is still an open problem for diffusion models. Usually, such generated multi-view images will have obvious inconsistency. How did the authors address this problem? This might be a contributing factor to the performance drop mentioned in point 1, where methods show decreased performance when using OpenDlign.

3. Section 3.2 indicates that the proposed framework generates a single set of multi-view images for a given 3D asset. It seems logical to generate multiple sets of multi-view images for a 3D asset to enhance training diversity and reduce dependency on image appearance. Is there a specific rationale behind limiting the generation to a single set of multi-view images?
questions: questions: Overall the reviewer is concerned about the point 1 and 2 mentioned in the Weaknesses. The reviewer may adjust the score based on the authors' response.
limitations: limitations: Yes
paper_summary: paper_summary: The paper introduces **OpenDlign**, a novel framework for open-world 3D representation learning by leveraging depth-aligned images generated from a diffusion model. OpenDlign aims to enhance the realism and texture diversity in the 3D learning process, overcoming the limitations of CAD-rendered images. The method involves fine-tuning the CLIP image encoder with depth-aligned images, achieving superior performance in zero-shot and few-shot classification tasks. Experimental results demonstrate significant improvements over existing state-of-the-art models on various benchmarks like ModelNet40, ScanObjectNN, and OmniObject3D.
strength_weakness: strength_weakness: - **Limited Dataset for Depth-Aligned Images**: The generation of depth-aligned images is limited to the ShapeNet dataset, which might restrict the generalizability of the results to other datasets.  
     &nbsp; 
- **Computational Resources**: The approach requires significant computational resources for generating depth-aligned images and fine-tuning the model, which might not be easily accessible to all researchers.  
     &nbsp; 
- **Potential Biases**: The paper acknowledges biases in the CLIP and ControlNet models but does not provide detailed strategies for mitigating these biases in the generated depth-aligned images.
questions: questions: - More applications: could the authors provide application study in other 3D OV Understand tasks like open-vocabulary 3D object detection methods[1, 2]  
     &nbsp; 
- Can the authors provide more details on the computational cost and feasibility of generating depth-aligned images for larger datasets?  
     &nbsp; 
- How do the authors plan to address the potential biases in the depth-aligned images generated by the diffusion model?  
     &nbsp; 
- Are there any plans to extend the approach to other types of 3D datasets beyond ShapeNet?  
     &nbsp; 
- Could the authors elaborate on the scalability of the method when using larger CLIP models or other vision-language models?

[1] Yuheng Lu, Chenfeng Xu, Xiaobao Wei, Xiaodong Xie, Masayoshi Tomizuka, Kurt Keutzer, and Shanghang Zhang. Open-vocabulary point-cloud object detection without 3d annotation. In CVPR, 2023. 1, 3.  
[2] Yang Cao, Zeng Yihan, Hang Xu, and Dan Xu. Coda: Collaborative novel box discovery and cross-modal alignment for open-vocabulary 3d object detection. In NeurIPS, 2023
limitations: limitations: The authors have provided discussions about the limitations.
paper_summary: paper_summary: This paper proposes OpenDlign, a framework for depth-based 3D understanding by aligning depth-image features through training on generated depth-aligned images, which overcomes the limitations of training on CAD-rendered images. Using point cloud data from ShapeNet, a contour-aware projection method is introduced to produce dense depth maps from the point clouds. To prepare the training data, an off-the-shelf depth-controlled diffusion model is employed to generate the corresponding RGB images from the projected depth maps. Additionally, techniques such as depth-specific text prompts and logit aggregation are adopted to enhance performance. Experiments demonstrate state-of-the-art results in zero/few-shot 3D classification, zero-shot 3D object detection, and cross-modal retrieval tasks.
strength_weakness: strength_weakness: - The motivations for the major design choices in this work are questionable. They appear to be driven purely by the evaluation benchmark (on where all methods use point cloud as the input, and some methods convert point cloud into depth later), rather than focusing on training a good encoder specifically for 3D understanding from depth maps. If this is the case, the title could better reflect this by referring to point cloud-image alignment rather than depth-image alignment. Specifically:
  - Why not use mesh to directly render depth maps? If the starting point is depth-image alignment, there is no need to convert point clouds into depth maps, which is only meaningful if the evaluation benchmarks are based on point cloud inputs. The availability of real depth maps renders one of the major contributions, the contour-aware projection method, completely irrelevant to the topic of this work.
  - Why not use real images and predict the depth maps using existing depth estimators, such as DepthAnything? This approach should significantly reduce the computation overhead of dataset creation and enable training on even more diverse data. Again, it seems that the current design choices are purely driven by the point cloud-based evaluation benchmark.
- In L152, the definition of inverse depth is incorrect. Inverse depth, also known as disparity, is 1/D. Depth ControlNet should take the normalized inverse depth (normalized into [0, 1]) as the input, rather than 1 - D.
questions: questions: Please refer to the questions in the weaknesses section.
limitations: limitations: Limitations have been discussed. I don't see any potential negative societal impact in this work.
paper_summary: paper_summary: This paper focuses on 3D open-world learning on classification. To address the limitation caused by existing CAD-rendered images in open-world 3D learning, it proposes to generate depth-aligned images from point cloud projected depth maps and diffusion models.
strength_weakness: strength_weakness: 1. It seems that the generated textures among multiple views for a single object are not consistent. Would this largely influence the performance?
2. Generating images with diffusion models for image/3D understanding is a prevailing area [A]. The author should discuss existing works in the related works. Also, It turns out the method is not novel for me. 
3. 3D understanding uses scene-level points in most downstream applications, please try on 3D scene-level datasets. 

[A] Is synthetic data from generative models ready for image recognition?
questions: questions: Please refer to the weakness part.
limitations: limitations: No.

Comments: title: Further reply
comment: Thank you for your feedback. We are glad to know that our rebuttal addresses your most concerns. We will certainly revise our title, abstract and introduction to better highlight "point cloud understanding" in the next version. We apologize again for any confusion caused by the text that may have suggested our focus was on depth understanding rather than point cloud understanding.
Thank you for the rebuttal. I believe most of my concerns can be resolved if "point cloud understanding" is emphasized in the title and abstract. Otherwise, I am uncomfortable with the current title and abstract, as they are misleading; they clearly suggest that the focus is on depth understanding.

As a side note regarding the "Real-Time Applicability" section: If the focus is on training a depth encoder, ShapeNet already provides ground truth meshes, and depth rendering can be extremely fast through simple rasterization. However, for point cloud understanding, the proposed method is reasonable, provided that "point cloud understanding" is clearly indicated in the title and abstract.
Thank you for your feedback. We're pleased to know that our rebuttal has addressed your most concerns. If this resolves the issues, we would greatly appreciate it if you could reconsider your rating. Otherwise, we are always happy to answer further questions that you may have.
Thanks for the authors' rebuttal. My concerns are mostly solved and hence I tend to accept this work.
Thanks for the author's rebuttal. My concerns have been addressed, and I would like to raise my rating to a borderline accept. Please include all contents in the rebuttal to the next version.
Thank you to the authors for the discussion and additional experiments provided during the rebuttal period; most of my concerns have been addressed,  I hope the authors can update the comparative experiments with OV-3DET and CoDA in the final version, as this knowledge will be beneficial to the related fields of the community. Totally speaking, I am inclined to accept this paper and decide to raise my rating to 7. Gook luck :)