DC3DO: Diffusion Classifier for 3D Objects

## Abstract

Abstract Inspired by Geoffrey Hinton’s emphasis on generative modeling (“To recognize shapes, first learn to generate them”), we explore the use of 3D diffusion models for object classification. Leveraging the density estimates from these models, our approach, “Diffusion Classifier for 3D Objects”, dubbed DC3DO, enables zero-shot classification of 3D shapes without additional training. Our method achieves on average 12.5% improvement compared with its multi-view counterparts, demonstrating superior multimodal reasoning compared to discriminative approaches. DC3DO uses a class-conditional diffusion model trained on ShapeNet. We run inferences on chairs and cars pointclouds. This work underscores the potential of generative models in 3D object classification. Code available https://github.com/SGI-2023/3D-Building-Classification .

## 1 Introduction

Recent advancements in deep generative models have yielded state-of-the-art (SOTA) performance in both classification and out-of-distribution (OOD) classification for images []. Deep generative models are increasingly being utilized for discriminative tasks, demonstrating superior effectiveness across various domains, including images [], text [], and tabular data [, ]. This progression builds upon the foundational work of Hinton [], inspired by Oliver Selfridge’s “Pandemonium” model []. While these early researchers focused on generation within the image domain, one could argue that creation should originate in 3D space before extending to the image or text domains. By first generating 3D objects, we can enhance downstream tasks, not only in object classification but also in image and text classification.

Figure: Figure 1: Dataset classes for classification. We performed 3D Classifications tests on cars, chairs, and airplanes. We used multi-view and point cloud representations only for chairs and cars.
Refer to caption: extracted/5788680/images/NewHero1.png

The classification of 3D shapes is increasingly important in fields such as computer vision, robotics, and virtual reality, hence this research. Since traditional methods often struggle to handle the complexity and variability inherent in 3D data, we adopted a diffusion approach []. Diffusion models [], a recent class of likelihood-based generative models, have shown significant promise in various tasks [, , ] by transforming random noise into coherent data samples through an iterative noising and denoising process.

Furthermore, today’s work in diffusion models [, , ] showed unmatched results not only on generative tasks [] but also in classification tasks [, ]. Diffusion models belong to a class of generative models that model the data distribution of the dataset, similar to VAEs [], GANs [, ], EBMs [], Score-based models [].

Therefore, a question arises, Can we use diffusion models for 3D classification tasks? More critically, given their remarkable ability to generate original objects beyond the initial dataset distribution [, , , , ], how do these models perform for out-of-distribution (OOD) data. While these models have excelled on standard benchmarks, they often struggle with novel OOD data, a limitation attributed to biased training datasets that fail to encompass the full spectrum of real-world possibilities. This has been attributed to the biased training data that does not represent all real-world possibilities []. These deep generative models can synthesize strikingly realistic and diverse images, objects, and text and they have shown better performances in zero-shot [, , ], few-shot classification tasks [].

In this research, we explore the application of Denoising Diffusion Probabilistic Models (DDPMs) [] for classifying 3D shapes. Traditional classification methods often fall short with 3D data, requiring novel approaches. Furthermore, 3D data are represented as point clouds [, , ], voxels [, ], signed distance functions [, , ] and multi-view formats []. Inspired by LION [], this research adopts point cloud and voxel [, ] combined with latent representations [] and diffusion models: DC3DO. DC3DO focuses on leveraging the generative capabilities of diffusion models for zero-shot classification []. We compared it against a direct extenstion of the 2D conterpart performed on images []. In a dynamic data landscape, the ability to classify data into previously unseen categories, such as architectural structures [, , ], is of paramount importance. Diffusion models, with their inherent generative strengths, are particularly well-suited for this challenge. Furthermore, by advancing beyond traditional 2D prior models [, , ] and incorporating the LION model [], renowned for generating high-fidelity 3D shapes, we enhance the effectiveness of the Diffusion Classifier in performing discriminative tasks, particularly in 3D object classification. Therefore, our contributions to this field are three-fold:

- •
Novel Method for 3D shape classification: We introduce DC3DO to classify 3D shapes with a diffusion model.
- •
Comparative analysis: We compare our method against multiview 3D representations using a 2D diffusion classifier []. We adapted MVCNN’s [] with its view pooling method to a more U-Net and diffusion classifier-friendly method for a fair comparison.

In these unsupervised settings, diffusion models [] are trained using the objectives of Variational Inference, specifically focusing on maximizing the evidence lower bound (ELBO) [] of the log-likelihood, as described in []. This involves adding noise $\epsilon$ to a sample, using a neural network to predict the noise, and adopting Mean Squared Error (MSE) or L2 loss to compare this predicted noise against a white gaussian noise [].

Given an input $\mathbf{x}$ and a finite set of classes $\mathbf{c}$ that we want to choose from, we can use the model to compute class-conditional likelihoods $p_{\theta}(\mathbf{x}\mid\mathbf{c_{i}})$, where $i$ indicated the class number. Then, by selecting an appropriate prior distribution $p(\mathbf{c_{i}})$ and applying Bayes’ theorem, we can obtain predicted class probabilities $p(\mathbf{c_{i}}\mid\mathbf{x})$. For conditional diffusion models that use an auxiliary input, like a class index for class-conditioned models or a prompt for text-to-image models, we can achieve this by leveraging the ELBO as an approximate class-conditional log-likelihood $\log p(\mathbf{x}\mid\mathbf{c_{i}})$. We repeatedly add noise and compute an estimate of the expected noise reconstruction losses (also called $\epsilon$-prediction loss) for every class, as described in [], please refer to Figure 2.

## 2 Related Work

Multimodal large language models (LLMs) strengths are leveraged in many current works [, , , ]. LLMs can handle diverse tasks through conversational interaction, specifically in the context of 3D objects. Typically, this is achieved by training a 3D shape encoder and aligning it with other modalities (e.g., text, images, and audio). The entire pipeline is then fine-tuned during an instruction-tuning phase, resulting in a model that is better aligned with user requests for specific 3D tasks. This fine-tuning stage is conducted using synthetic datasets or captioning datasets.
These approaches highlight the vast potential of integrating 3D shapes into foundation models, although they still necessitate the fine-tuning of large models. Other methods, such as 3DAxiesPrompts [], enhance images and prompts with additional artifacts to be able to exploit the 2D vision abilities of LLM for 3D objects.

PEVA-Net [] employs a pre-trained CLIP model in a multiview pipeline to classify 3D objects in zero-shot or few-shot environments. It leverages CLIP’s zero-shot classification abilities for each view of the 3D object, subsequently aggregating these results to make the final prediction. Although this approach effectively exploits the zero-shot capabilities of vision-language models (VLMs), transforming 3D shapes into multiview images is an oversimplification that can lead to suboptimal results.

TAMM [] demonstrates that when aligning 3D object representations with other modalities, the image modality contributes less significantly than the text modality. To address this, their method learns to separate visual features from semantic features within the 3D object representation, enabling a more effective alignment with the other modalities and enhancing performance in downstream tasks. These findings suggest that the alignment between modalities for integrating 3D representations into existing methods can sometimes be inadequate []. Regarding 3D representation learning, Zhang et al. [] takes a different approach and incorporates 2D guidance. Their work, dubbed I2P-MAE [], learns advanced 3D representations, achieving state-of-the-art performance on 3D tasks and significantly lowering the need for large-scale 3D datasets. On the contrary concurrent work, DiffCLIP [] demonstrates that the integration of CLIP and diffusion models for 3D classification facilitates zero-shot classification, achieving state-of-the-art results. This methodology utilizes a pre-training pipeline that incorporates a Point Transformer for few-shot 3D point cloud classification, wherein the CLIP model extracts style-based features of the class, synergistically combined with image features. While DiffCLIP [] used Point Transformer we used LION, a latent point-voxel [, , ] representations that leverages a hierarchical two stages diffusion process with state of the art generative performances. Following the line of latent and implicit representations, Xin et al. [] used a Classifier Score Distillation (CSD) method, which utilizes an implicit classification model for generation.

## 3 Methodology

In this section, we present and compare two distinct approaches for 3D object classification: Multi-View Diffusion Classifier (Section 3.1) and DC3DO (Section 3.2). The first approach, the Multi-View Diffusion Classifier, is designed to harness the power of diffusion models while maintaining the architecture of Diffusion Classifier []. The second approach, DC3DO, integrates the advanced generative capabilities of LION [] with diffusion-based classification, targeting zero-shot classification of complex 3D shapes like cars and chairs.

Our goal is to thoroughly assess the performance of these methods in comparison to traditional and state-of-the-art techniques. The Multi-View Diffusion Classifier (MVDC) offers an alternative to the widely-used MVCNN by employing a majority vote mechanism across multiple 3D views. On the other hand, DC3DO leverages LION’s robust generative framework (Section 3.3).

### 3.1 Multi-View Diffusion Classifier (MVDC)

3D objects can be effectively represented as a series of images, providing a straightforward baseline for extending previous work [] to the 3D domain. By simply aggregating multiple views of the same object, we can adapt existing diffusion-based classification techniques for 3D shapes. For our experiments, we utilized the ShapeNet dataset [], focusing on a specific subset of $200$ models per class. This selection was made due to the computational intensity of performing $1000$ diffusion steps ($t$) per image ($X_{i}$), which is particularly challenging in environments with limited GPU resources (poor-gpus settings). Especially if each object is represented by $36$ views taken from cameras view 10-degree intervals around a circumference encircling the 3D object, adding to the computational complexity.

To classify a single object, our method proposes a majority vote scheme. Let $X=\{X_{1},X_{2},\ldots,X_{n}\}$ represent the set of $n$ views of a 3D shape. Each view $X_{i}$ is processed individually by the diffusion model to produce a corresponding prediction $y_{i}=f\left(X_{i}\right)$, where $f(\cdot)$ denotes the classification function of the diffusion model. Unlike MVCNN [], which aggregates these views into a global representation through view pooling, our approach maintains the predictions $\left\{y_{1},y_{2},\ldots,y_{n}\right\}$ independently.

The final classification decision $y^{*}$ is made by selecting the most common prediction among the individual predictions, formulated as:

$$ $y^{*}=\operatorname{mode}\left(y_{1},y_{2},\ldots,y_{n}\right)$ $$

This majority vote approach retains the architectural integrity of diffusion models while emphasizing simplicity and interpretability. While MVCNN’s view pooling may enhance performance by combining features, our goal is to assess the classification power of diffusion models in their unaltered form. Here the 2D Images are processed and encoded into feature maps $\mathbf{i}\in\mathbb{R}^{H\times W\times C}$, where $H$ is the height, $W$ is the width, and $C$ is the number of channels in the image ($512\times 512\times 3$ as the highest resolution in these experiments).

### 3.2 Diffusion Classifier for 3D Objects (DC3DO)

DC3DO consists on the main contribution of our research. Our model combines LION [] with diffusion classifier [] for zero-shot classification. By utilizing LION’s ability to generate diverse 3D shapes and feeding them into the diffusion classifier, we achieve precise categorization of 3D cars and chairs. This section details DC3DO, emphasizing the advantages of diffusion models, the processing of 3D shapes, and the integration of the LION model to enhance classification accuracy.

### 3.3 Integrating LION with Diffusion Classifiers

LION leverages a hierarchical latent space to effectively capture both global and local features of 3D structures, see Figure 2. This hierarchical approach ensures comprehensive encoding of both macro and micro features of 3D object structures.

Figure: Figure 2: Methods comparison. We extended the Diffusion Classifier [] paper to a multi-view [] settings and we compare with our DC3DO model, based on []
Refer to caption: extracted/5788680/images/Diagram.png

#### 3.3.1 Integration with Hierarchical Latent Space

LION’s hierarchical latent space encodes 3D point clouds $\mathbf{x}\in\mathbb{R}^{3\times N}$, where $\mathbf{x}$ consists of $N$ points (2048) with $xyz$-coordinates in $\mathbb{R}^{3}$, into a dual-layered latent representation. This representation includes:

- •
Global Latent Space: This vector-valued latent space, denoted as $\mathbf{z}_{0}\in\mathbb{R}^{D_{\mathbf{z}}}$, captures the overall structure and large-scale features of the building. It captures the overall spatial structure of the 3D shape, including its large-scale features.
- •
Local Point-Structured Latent Space: This latent space, denoted as $\mathbf{h}_{0}\in\mathbb{R}^{(3+D_{\mathbf{h}})\times N}$, represents a point cloud-structured latent consisting of $N$ points with $xyz$-coordinates in $\mathbb{R}^{3}$ and additional $D_{\mathbf{h}}$ latent features per point. This layer captures detailed and fine-grained features.

#### 3.3.2 Integration with Diffusion Models

The integration process involves several key steps:

##### Encoding

The 3D point cloud data $\mathbf{x}$ is encoded into the global latent space using LION’s PVCNN encoder. We found that the global latents contained enough information about the shape and high level features of the object, for the purpose of classification. Also, it is a much smaller latent space compared to the the local point structured latent space, making it easier to work with - considering the diffusion process requires multiple inference steps per sample.

##### Diffusion Process

After encoding, the data undergoes a first diffusion process (global latent). This involves a fixed forward procedure where Gaussian noise is iteratively added (1000 steps) to the latent representations $\mathbf{z}_{0}$ and $\mathbf{h}_{0}$, resulting in the diffused latents $\mathbf{z}_{t}$ and $\mathbf{h}_{t}$. The forward diffusion process is defined as:

$$ $\displaystyle\mathbf{z}_{t}=\alpha_{t}\mathbf{z}_{0}+\sigma_{t}\mathbf{ \epsilon},\quad\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (1) $$

$$ $\displaystyle\mathbf{h}_{t}=\alpha_{t}\mathbf{h}_{0}+\sigma_{t}\mathbf{ \epsilon},\quad\mathbf{\epsilon}\sim\mathcal{N}(\mathbf{0},\mathbf{I})$ (2) $$

where:

- •
$\mathbf{z}_{0}$ and $\mathbf{h}_{0}$ are the initial latent representations capturing the global and local features of the 3D shape, respectively.
- •
$\mathbf{z}_{t}$ and $\mathbf{h}_{t}$ are the diffused latent representations at timestep $t$.
- •
$\alpha_{t}$ and $\sigma_{t}$ are coefficients that control the amount of the original signal and the noise added at each timestep, respectively.
- •
$\mathbf{\epsilon}$ is the Gaussian noise sampled from a standard normal distribution $\mathcal{N}(\mathbf{0},\mathbf{I})$, which introduces randomness to the latent representations.

##### Denoising and Classification

In the pipeline, a deep neural network, conditioned on class labels $\mathbf{c}$, performs the denoising of the perturbed data. The denoising process involves reversing the forward diffusion to retrieve the latent representations $\hat{\mathbf{z}}_{0}$ and $\hat{\mathbf{h}}_{0}$ that best match the original data distribution $\mathbf{x}$. Specifically, the network learns to approximate the posterior distributions $q_{\phi}(\mathbf{z}_{0}|\mathbf{x},\mathbf{c})$ and $q_{\phi}(\mathbf{h}_{0}|\mathbf{x},\mathbf{z}_{0},\mathbf{c})$ by minimizing the reconstruction error.

The classification is then performed by evaluating the likelihood $p_{\theta}(\mathbf{x}_{0}\mid\mathbf{c})$ of the denoised data $\hat{\mathbf{z}}_{0}$ and $\hat{\mathbf{h}}_{0}$ belonging to specific classes. This likelihood is computed as follows:

$$ $\displaystyle p_{\theta}(\mathbf{x}_{0}\mid\mathbf{c})=\int_{\mathbf{x}_{1:T}} p(\mathbf{x}_{T})\prod_{t=1}^{T}p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t}, \mathbf{c})\ \mathrm{d}\mathbf{x}_{1:T}$ (3) $$

where

- •
$p_{\theta}(\mathbf{x}_{0}\mid\mathbf{c})$ is the class-conditional likelihood of the original data $\mathbf{x}_{0}$ given the class label $\mathbf{c}$.
- •
$\mathbf{x}_{T}$ represents the final diffused state, typically modeled as a standard Gaussian distribution.
- •
$p_{\theta}(\mathbf{x}_{t-1}\mid\mathbf{x}_{t},\mathbf{c})$ denotes the learned reverse process that denoises the data at each timestep $t$, conditioned on the class label $\mathbf{c}$.

The model assesses the denoised data to determine the most likely building category by evaluating which class-specific denoising process best corresponds to the introduced noise, ultimately assigning the data point to the class with the highest likelihood.

#### 3.3.3 Text-Conditioned Diffusion

Our model employs multi-modal [] approach to integrate diverse data modalities, providing a comprehensive approach to 3D building classification. The diffusion process is condition on a text prompt ”[C]” (”car”, ”chair”, and ”airplane” ). We added an additional text prompt to provide a diffusion process to classify the model as ”non-car”, ”non-chair”, and ”non-airplane”.

The integrated modalities include:

- •
3D Point Cloud Data: The primary representation of 3D shapes, capturing spatial distribution and structural details.
- •
Textual Descriptions: Supplementary information describing the architectural features and styles of buildings. These are encoded into vector representations $\mathbf{t}\in\mathbb{R}^{d}$.

By integrating these modalities, our model achieves a more informed representation of the data, which improves classification accuracy and robustness. The integration of LION with diffusion models utilizes the combined strengths of both techniques, allowing for precise and reliable classification of 3D building structures.

## 4 Experimental Results

### 4.1 MVDC - 2D Results

In our baseline evaluation, we utilized a multi-view diffusion classifier on the ShapeNet dataset, focusing on three classes: cars, chairs, and airplanes. This approach uses multiple views of 3D shapes to enhance classification accuracy by taking advantage of the rich spatial information in the dataset. The process involved encoding 3D shapes into latent representations using a pretrained VAE, adding Gaussian noise, and employing a UNet model for denoising and classification. This way, we captured the intricate details of 3D shapes and effectively categorized them by adaptively selecting the most promising samples based on predicted errors, optimizing overall classification performance.

**Table 1: Zero-shot classification accuracy (%) of DC3DO on ShapeNet for cars, airplanes, and chairs. We performed the comparison only on the first $200$ subsamples models, each with 6 views and 200 sampling steps. For the Multi View Diffusion we used only the six frontal views, and image resolutions of $64\times 64$.**
| Method | Accuracy |  |
| --- | --- | --- |
|  | Car | Chair |
| MVDC (100 models) | 65.7% | 32.3% |
| MVDC (200 models) | 64.8% | 31.5% |
| DC3DO-100m (ours) | 100% | 36% |
| DC3DO-200m (ours) | 100% | 49% |

MVCNN [] utilized 36 fixed cameras, with objects placed in a canonical pose. The cameras were positioned at uniform intervals, with a 10-degree rotation between each, defined by their position parameters $(X,Y,Z)$. To manage computational constraints, we downscaled the images to $64\times 64$ and reduced the number of views per object from 36 to 6. The selected views were the frontal ones, corresponding to camera angles of $10^{\circ},20^{\circ},30^{\circ},340^{\circ},350^{\circ}$, and $360^{\circ}$, in line with the ShapeNet view settings.

As explained in [], frontal camera positions generally yield higher accuracy. Therefore, we focused on these 6 specific camera positions for our experiments.

To compute the accuracy of multi-view classification, we employed a majority vote mechanism across the $n$ views of each 3D mesh, where $n=6$. Let $y_{i}\in\{0,1\}$ represent the binary prediction for each view $X_{i}$, where 1 corresponds to the prediction “car” and 0 corresponds to “not car”. The final classification $y^{*}$ for each object is determined as:

$$ $y^{*}=\begin{cases}1&\text{ if }\sum_{i=1}^{n}y_{i}\geq\frac{n}{2}\\ 0&\text{ otherwise. }\end{cases}$ $$

In this setup, if the number of votes for ”car” (i.e., $\sum_{i=1}^{n}y_{i}$ ) is greater than or equal to 3, the object is classified as ”car.” For each class $c$, we performed a binary classification, distinguishing between class $c$ and all other classes. The accuracy $A_{c}$ for each class $c$ is calculated as:

$$ $A_{c}=\frac{\text{ Number of correctly classified objects in class }c}{\text{ Total number of objects in class }c}.$ $$

Finally, the mean per-class accuracy $\bar{A}$ is computed by averaging the binary classification accuracies across all classes:

$$ $\bar{A}=\frac{1}{C}\sum_{c=1}^{C}A_{c},$ $$

where $C$ is the total number of classes.

### 4.2 DC3DO Inference

Since DC3DO is based on LION, we took its model weights publicly available. LION has been trained on specific classes from the ShapeNet dataset, with the model weights for the “chairs” and “cars” categories publicly available to the research community. Consequently, we utilized these pretrained models in our experiments. Due to computational constraints, we set the number of diffusion steps for both the Multi-View Diffusion Classifier and LION to 200 steps.

For our experiments, we employed a batch size of $1$ for the “cars” and “chairs” categories. Currently, it takes approximately 20 seconds to classify each object.

**Table 2: Best 3 chairs per accuracy**
| Model | Render | Accuracy |
| --- | --- | --- |
| Chair 142 |  | 35% |
| Chair 143 |  | 34% |
| Chair 188 |  | 33% |

**Table 3: Worst 3 chairs per accuracy**
| Model | Render | Accuracy |
| --- | --- | --- |
| Chair 116 |  | 20% |
| Chair 70 |  | 20% |
| Chair 87 |  | 20% |

Figure: (a) Car #184 Acc: 30%

### 4.3 Ablation Studies

To gain deeper insights into the contributions of different components in our model, we conducted ablation studies by systematically disabling specific features and quantifying their impact on performance.

For the MVDC model, let $S\times S$ denote the image size and $n$ the number of views. We observed that the diffusion process time $T$ increases non-linearly with both $S$ and $n$. Specifically, $T\propto S^{2}\times n$, indicating that larger image sizes $S$ and an increased number of views $n$ result in significantly slower processing.

We ran inference at various image sizes to study its run time performance and relationship to the model performance. First, we confirmed that the inference time grows exponentially with larger image size, for a $512\times 512$ resolution image and 500 sampling steps, the processing time was approximately 1.5 minutes per image, making it infeasible to evaluate at larger scale. Moreover, when we reduced the image size to $S=64\times 64$ or $S=128\times 128$, the classifier’s performance degraded severely. The model exhibited a tendency to collapse, consistently predicting a single class $c$ regardless of the input views, suggesting that the classifier lost its ability to differentiate between classes under reduced image resolutions.

**Table 4: Ablation studies about image resolutions. Inference time and accuracy analysis of MVDC model on 3 classes from shapenet: Airplane, Car, and Chair. The sample size is fixed at 200 steps.**
| Image | Inference | Accuracy |  |  |
| --- | --- | --- | --- | --- |
| Resolution | Time | Airplane | Car | Chair |
| $64\times 64$ | 1h03m | 66.7% | 64.8% | 31.5% |
| $128\times 128$ | 2h13m | 33.7% | 66.7% | 67.0% |
| $256\times 256$ | 7h05m | 99.3% | 98.7% | 99.% |

## 5 Limitations

One of the primary limitations of our approach is the computational cost. The 3D diffusion process currently requires approximately $20$ minutes per object on a T4 GPU, making it a time-intensive task. Similarly, the multi-view approach, while effective, is also relatively slow due to the independent processing of each view.

Regarding the Multi-View Diffusion Classifier, a significant limitation is that the views are processed individually and then aggregated through a majority vote, rather than being combined into a global latent vector as in the approach used by MVCNN []. This method of independent view processing may not fully capture the holistic structure of 3D shapes, which could be better represented through a more integrated multi-view approach.

Due to time and computational constraints, we limited our experiments to 200 shapes per category. With access to more powerful GPUs and additional resources, future work could extend these experiments to a larger number of objects, potentially providing more comprehensive results.

## 6 Discussion and Future Work

The high classification accuracy on ID data indicates that the model effectively captures the distinguishing features of various 3D objects.

The hierarchical latent space of LION played a crucial role in accurately representing both global and local features of 3D shapes, contributing to the model’s overall performance compared the multi-view (see Section 5 for more details). The diffusion process further enhanced the model’s ability to denoise and classify complex 3D structures, providing a reliable mechanism for zero-shot classification.

These results highlight the potential of integrating generative models like LION with diffusion classifiers for advanced 3D shape analysis and classification tasks, particularly in scenarios involving diverse and unseen data. In fact, in this work, we delved into 3D diffusion models and present our method that enables zero-shot classification of 3D shapes in a robust manner. For future works, we wish to explore 3D diffusion capabilities in state-of-the-art multimodal methods such as ULIP-2 [], integrated with PointBERT [] architectures similar to the concurrent work []. We believe this will enhance the performance of these architectures and make them capable of 3D understanding.

## 7 Conclusion

In this paper, we propose a model that seamlessly integrates LION [] with a diffusion classifier [] to achieve accurate classification of 3D cars and chairs. The model’s success is driven by the hierarchical latent space and diffusion process, which together enable precise representation and classification of complex 3D shapes from the ShapeNet dataset []. Our approach, named DC3DO, demonstrates a 12.5% improvement on average compared to multi-view methods, highlighting the potential of generative models in 3D object classification. This work suggests that future research could adapt generative models to discriminative tasks, potentially leading to enhanced classification and regression performance.

## 8 Acknowledgments

We express our gratitude to Alexander C. Li and Le Xue for their insightful initial discussions. We also extend our thanks to the mentors and volunteers of the Summer Geometric Initiative (SGI) at MIT for their invaluable guidance. A special thank you to Professor Justin Solomon for his coordination and funding support. We are also grateful to Google Cloud for providing credits and sponsorship, and to Paul Guerrero for generously sharing the render file used in the previous SGI.

## Peer Reviews

confidence: 5
contribution: 2
initial_rating: 3
presentation: 3
questions: 1. Why no more categories are validated for a classifier? Please refer to Weaknesses 1.

2. Why no other baseline method is compared? Please refer to Weaknesses 2. Especially for OOD settings, experiment without comparisons has very limited value.
soundness: 2
strengths: The idea of building a 3D diffusion classifier is attractive. The paper compared two methods of combining 3D representation and 2D diffusion classifier. 

The paper is well-written and clear to follow.
summary: This paper proposes a Diffusion Classifier designed for 3D objects (DC3DO). DC3DO combines the LION model with the 2D Diffusion Classifier to build a class-conditioned 3D Diffusion Classifier based on latent encoded from the 3D shape and points of objects. 

The proposed DC3DO is mainly compared with the Multi-View Diffusion Classifier(MVDC), which combines the aggregated multi-view features and the 2D Diffusion Classifier. 

Out-of-distribution performance is evaluated on unseen datasets other than the training dataset.
weaknesses: 1. Limited number of classes are validated: only “chairs” and “cars” are evaluated for classification performance. Even though the current 3D datasets are relatively smaller than 2D datasets, two categories are insufficient for validating a classifier considering the MVCNN (Su et al., 2015) was validated on 40 classes. 

2. Limited comparison baselines: at least MVCCN is closely related to MVDC in this paper and as a frequently mentioned baseline method, can I know why the authors didn't compare the main classification results with MVCNN? And also other standard baseline 3D classification models are expected.

3. Limited novelty: The main contribution of DC3DO is a simple combination of LION and Diffusion Classifier. Considering the combination is the same way as turning a 2D diffusion model into a classifier as Diffusion Classifier (Li et al., 2023), the novelty of this paper is limited.

4. Misuse of the term “zero-shot classification”: zero-shot classification is supposed to generalize a classifier to unseen "classes". In this paper only 3D models of the same class from unseen datasets/sources are validated as OOD data, it should be best described as a domain generalization instead of a zero-shot classification. Can the author provide further clarification on this?

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 3
questions: Please see the weakness session.

I think the paper needs serious revision to include more experimentations and to reduce the inference time overhead.
soundness: 1
strengths: 1. The paper is easy to follow, the presentation of the work is good.
2. Figures are clear and easily understandable, and they help reader conceive the main idea the paper is trying to present.
3. Interesting illustrations of certain categories with high and low prediction accuracies of car and chair
summary: This paper proposes DC3DO, a pipeline that adopts 2D diffusion types of models to zero-shot 3D object classification. DC3DO is extended from previous work LION that adopts diffusion models to 2D classification model by introducing multi-view diffusion components. The model is tested on ShapeNet (only on chairs and cars categories).
weaknesses: 1. The biggest problem of this paper is the lack of enough experimentation. The experiments are only conducted on two selected categories (car and chairs) from one dataset (ShapeNet), which is definitely not enough for a paper at this conference. Also, not enough baselines are considered, and the paper only compares to baseline MVDC, which is not a recent work. The current experiment results cannot support the claim of the paper.

2. The paper lacks technical novelties. MVDC seems only extend LION to multi-view without much structural and strategical changes. It seems to be a naive multi-view extension of LION. 

3. The paper fails to discuss and tackle the potential drawback of its proposed solution. As I can see, the multi-view diffusion process will take a long time on every image, which will be much slower than some other 3D object classification models. The paper should analyze the tradeoff between performance and efficiency compared to more typical 3D object classification models. Does the performance boost worth the increasing inference time? 

4. The claim that this method is robust is also not sufficiently supported by the experiment. The experiment does not compare to any other baseline at all to show the proposed method is more robust.

---

confidence: 4
contribution: 2
initial_rating: 3
presentation: 2
questions: 1. For the baseline what is the renderer setup? What texture/illumination is used when rendering the images?
2. Why do only equations on pages 4 and 8 have equation numbers?
soundness: 1
strengths: 1. The method is straightforward and easy to understand.
2. Exploring the 3D object understanding and classification seems a worth study topic.
summary: This paper proposes a diffusion-based 3D object classification approach. Specifically, they adapt the LION (diffusion-based model for point cloud to mesh generation) into a classification model that defines P(x|c) where x is the given 3D geometry, and c is the object class. They conduct an experiment on a small subset of the ShapeNet dataset and evaluate it on both ShapeNet and ModelNet. The proposed approach seems reasonable, however, the experiment setup is not sound enough to demonstration the advantages of the proposed approach.
weaknesses: 1. The experiment setup, especially for the baseline MVCNN is confusing, and the accuracy for the baseline seems problematic. According to Line 253, the classification is defined as a close-set classification problem that uses the class given the largest P(x|c) as the prediction, however, in line 309, the baseline is evaluated as a binary classification problem (whether belongs to category car or not). Also, for both 3-class classification and binary classification problem the accuracy of Chiar in Table 1 is lower than random guess (33.3% or 50%), which seems the baseline is totally not working. Thus, the experiment results seem not meaningful.
2. The proposed method is incremental. But compared to weakness 1, this is not a significant weakness.

---

confidence: 5
contribution: 2
initial_rating: 3
presentation: 2
questions: What if the multi-view diffusion classifier is fine-tuned on point cloud rendering? Will it significantly change the performance?
soundness: 1
strengths: The writing of the introduction looks fine.
summary: This paper explores the use of 3D diffusion models for zero-shot 3D object classification. This paper applies the diffusion classifier (Li et al., 2023a) on a point cloud diffusion model (Zeng et al., 2022a). The experiments show that the proposed method outperforms the baseline, which applies the diffusion classifier directly on multi-view renderings, and classifies based on voting.
weaknesses: 1. The baseline is not well designed. The original diffusion classifier (Li et al., 2023a) is based on StableDiffusion, which was trained on natural images rather than point cloud rendering. It is expectable that directly applying the diffusion classifier on the point cloud rendering gives bad results.
2. The main experiments are not thorough. The model is only tested on cars and chairs in ShapeNet. Seven years ago, PointNet [A] has already set a good benchmark for point cloud classification.
3. The out-of-distribution experiments are barely reasonable. Both the in-distribution and out-of-distribution data are just chairs, though from different datasets, ShapeNet VS ModelNet.
4. Considering the accuracy of classifying chairs is less than 50%, it is not proper to describe the model as "accurate" in multiple places in the paper, such as line 190, 433, 485, and 501.
5. This paper contains useless paragraphs. For example, line 440-443, it says increasing the image size and number of views can slow down the processing time of the multi-view classifider, which is too obvious.
6. The writing is confusing. In the introduction, it says MVDC is just a baseline and DC3DO is the proposed model. However, the whole ablation study is about how the input image size affects the performance and inference time of MVDC.
7. Classifying each object takes approximately 20 seconds.

[A] Charles R. Qi, Hao Su, Kaichun Mo, Leonidas J. Guibas, "PointNet: Deep Learning on Point Sets for 3D Classification and Segmentation", CVPR 2017.

Meta-review: This paper presented an approach for 3D (generative) diffusion models for 3D classification. While this is an interesting application for this idea for 3D, the reviewers raised several common concerns. Specifically, the method evaluation is only on a few classes, and although this is due to the base model being limited, this does make the approach difficult to advocate for. The reviewers also felt the contribution in this work is limited in context of Diffusion Classifier. Ultimately, the reviewers all recommend rejection, and the AC sees no reason to overturn this consensus.

Decision: Reject