UniL: Point Cloud Novelty Detection through Multimodal Pre-training

Authors: yuhan  wang
mofei  song
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-04 13:22:10

Abstract: 3D novelty detection plays a crucial role in various real-world applications, especially in safety-critical fields such as autonomous driving and intelligent surveillance systems. However, existing 3D novelty detection methods are constrained by the scarcity of 3D data, which may impede the model's ability to learn adequate representations, thereby impacting detection accuracy. To address this challenge, we propose a Unified Learning Framework (UniL) for facilitating novelty detection. During the pretraining phase, UniL assists the point cloud encoder in learning information from other modalities, aligning visual, textual, and 3D features within the same feature space. Additionally, we introduce a novel Multimodal Supervised Contrastive Loss (MSC Loss) to improve the model's ability to cluster samples from the same category in feature space by leveraging label information during pretraining. Furthermore, we propose a straightforward yet powerful scoring method, Depth Map Error (DME), which assesses the discrepancy between projected depth maps before and after point cloud reconstruction during novelty detection. Extensive experiments conducted on 3DOS have demonstrated the effectiveness of our approach, significantly enhancing the performance of the unsupervised VAE method in 3D novelty detection. The code will be made available.

Main paper: 
 introduction: 
 Novelty detection, also known as "novel class detection" [33, 34] , primarily focuses on identifying semantic shifts. It aims to detect any test samples that do not fall into any training category, as the term "novel" generally refers to the unknown, new, and something interesting [63] . Distributional shifts compared to the training data can significantly impact model performance, posing potential threats or risks. For instance, in safety-critical applications such as autonomous driving systems [25, 65] , the model's ability to detect and reject unknown samples becomes crucial, as it must return control to the driver. In the realm of 2D analysis, the field of novelty detection has reached a relatively advanced stage. However, within the 3D domain, despite significant advancements in visual understanding [13, 16, 22, 26, 30, 31, 57] , novelty detection remains nascent and has received little attention from researchers. Antonio et al. [3] present the first benchmark for 3D Open Set learning (3DOS), which includes various tasks of increasing difficulties regarding semantic shifts and encompasses both in-domain and cross-domain scenarios. Building on this foundation, the authors of 3DOS conducted an extensive survey of methods for out-ofdistribution (OOD) detection and Open Set recognition across 2D and 3D domains, evaluating them on the proposed benchmark.However, we observed a significant limitation in those methods: the scarcity of available 3D data. Compared to 2D data, acquiring and annotating 3D data is typically more expensive and time-consuming, resulting in a limited availability of annotated datasets. This limitation can lead to inadequately trained models that may struggle to represent all possible scenarios and variations. Moreover, the increased complexity of feature extraction from 3D data may cause suboptimal representation, thus further impacting the model's performance in novelty detection.To address this challenge, we opted to integrate information from other modalities of point cloud data, transferring knowledge from pre-trained multimodal models. Among the methods evaluated on A black and white photo of the chair. There is a chair in the scene. A low resolution photo of the chair. The visualization of UniL. To address the issue of insufficient data in 3D novelty detection tasks, UniL adopts a pretraining approach to allow the 3D encoder to learn more information. By aligning point cloud, depth map, and text features simultaneously, the 3D encoder can achieve better performance in subsequent novelty detection task.Text Encoder Image Encoder3DOS [3] , only VAE [35] is unsupervised, making it our chosen baseline. Specifically, our approach consists of three phases: pretraining the 3D encoder, training the VAE, and performing novelty detection. These phases are illustrated in Fig. 1 , which outlines the workflow of our approach. Firstly, to tackle the challenge of limited 3D data, we incorporate information from both text and image domains, constructing a Unified Learning Framework that facilitates the 3D encoder in assimilating knowledge from these modalities, called UniL. To further enhance the alignment accuracy in the feature space, we designed the Multimodal Supervised Contrastive Loss (MSC Loss), which fully utilizes the label information of point cloud data. This enables the 3D encoder to focus more on the similarity among samples from the same category, forming tighter clusters in the feature space. Secondly, during the VAE training phase, we employ the pre-trained encoder and utilize its encoded features to fit the mean and variance. After reparameterization sampling, the features are input into the decoder to reconstruct the input point cloud. Finally, to uncover underlying novel patterns, we integrate information from the 2D domain and introduce the Depth Map Error (DME). The computation of DME involves projecting the reconstructed point cloud onto a depth map and calculating the error between it and the original depth map. By comparing the depth maps before and after reconstruction, we can quantify the quality of the point cloud reconstruction and extract potential novel patterns. This approach not only compensates for the shortcomings of limited 3D data but also fully utilizes the correlations between different modalities, thereby improves the robustness and accuracy of novelty detection. After thorough evaluation on the benchmark introduced in 3DOS [3] , the effectiveness of our approach has been demonstrated. Specifically, our method achieved top-1 performance on SN1 (hard), with a notable improvement of 2.0% in AUROC and a reduction of 5.5% in FPR95. On SR1 (easy), we observed a reduction in FPR95 of 0.9%, while on SR1 (hard), we achieved a notable improvement in AUROC of 1.9%.• To address the issue of insufficient 3D data, we constructed a Unified Learning Framework (UniL) to assist the 3D encoder in learning knowledge from other modalities. • We proposed Multimodal Supervised Contrastive Loss (MSC Loss), which utilizes label information to further enhance the alignment accuracy of features across different modalities.• In order to incorporate information from different modalities for novelty detection, we introduced Depth Map Error (DME), a simple yet efficient approach that detects potential novelties by evaluating the projection error of depth maps both before and after point cloud reconstruction. 
related work 2.1 3d point cloud learning: 
 There are mainly two streams for learning from point cloud. One is projecting point clouds into voxels [36, 49] or images [24, 66] and then using 2D/3D convolutions for feature extraction, given the irregular and unordered nature of point cloud structures. The other one is directly processing point cloud data, with PointNet [41] being the first neural network to adopt this method. It can effectively learn and extract features from unordered point sets, which significantly influences point-based 3D networks. DGCNN [59] proposed a dynamic graph structure for performing convolution operations on unordered point sets. PointMLP [32] is a simple feed-forward residual MLP network that hierarchically aggregates the local features extracted by MLPs while eliminating the need for intricate local geometric extractors. Moreover, recent work has adopted self-supervised pretraining methods for 3D understanding. Point-BERT [67] encodes point cloud data into text sequences and applies self-attention mechanism to learn the semantic relationships and representations between points. Point-MAE [39] directly processes the point cloud by masking out 3D patches and then predicting them back using L2 loss. PointGPT [7] applies the principles of GPT [44] to the generation of point cloud data, acquiring robust 3D representations through pretraining on autoregressive generation tasks. While these methods have shown effectiveness, their full potential remains unrealized due to the limited availability of 3D data.Multimodal pretraining aims to learn universal representations by simultaneously processing multiple types of data, such as images, text, speech, etc. This paradigm helps models understand the semantic correlations between different modalities, thereby achieving better performance in various downstream tasks.CLIP [43] trains the model by contrastive learning between images and text, allowing natural language to understand visual concepts. As the first successful multimodal learning model, CLIP holds significant importance in advancing research on multimodal learning and understanding the semantic relationships between images and text. Inspired by CLIP-based adaption methods [12, 28, 68] , PointCLIP V2 [69] and CLIP2Point [18] convert point clouds into 2D forms through projection and rendering, and then apply the powerful generalization capability of CLIP to zero-shot classification. ULIP [61] aims to directly adapt the paradigm of CLIP to learn a unified representation space for point clouds, language, and images. Drawing inspiration from ULIP, we aimed to incorporate the information of point cloud in other modalities, enhancing the representation capabilities of the 3D encoder. Unlike ULIP, which neglects the semantic similarity of point cloud data in the language modality, we introduce the Multimodal Supervised Contrastive Loss (MSC Loss) to refine feature alignment accuracy.F ! # F! $ F ! % F! & F ' " ...Currently, novelty detection methods primarily focus on densitybased, classification-based, distance-based, and reconstruction-based approaches. Among these, density-based methods [1, 9, 23, 40, 48, 70] identify novel samples by measuring the density around the data points, since novel samples typically reside in sparse regions. Classification methods [45, 47, 52, 58] attempt to assign test samples to predefined categories, while samples that cannot be classified can be considered as novel samples. As novelties are assumed to be distant from the training data, distance-based methods [37, 51, 53] assess novelty by computing the distance between a test sample and the training samples. In reconstruction-based methods [4, 8, 21, 38, 54, 62] , the model's ability to reconstruct known patterns is leveraged to detect data points that significantly differ from the training data. However, these methods are primarily designed for traditional 2D data, and therefore perform poorly when transferred to the 3D domain. [35] marks the pioneering attempt to address anomaly detection in 3D point clouds of general objects, introduces an framework built upon a variational autoencoder.Following the work of [35] , [2] proposed a novel unsupervised approach for detecting novelty in 3D point cloud, utilizing a general feature extractor for point clouds and a one-class classifier.Considering that these methods rely solely on 3D data for detection, we propose the Depth Map Error (DME) to assess the novelty of point clouds in the 2D domain. 
method: 
 In this section, we will provide a detailed exposition of our method. Firstly, we pretrain the 3D encoder within the UniL framework, enhancing feature alignment accuracy through MSC Loss. Following this, we load the pretrained 3D encoder and train the VAE to reconstruct input point cloud samples. Lastly, during the novelty detection phase, we introduce additional 2D visual information and employ DME to measure the disparity between the projected depth maps of point clouds before and after reconstruction. The overall framework is illustrated in Fig. 2 .Limited data may impose constraints on the model's ability to learn meaningful and accurate feature representations, thereby impacting its performance and generalization capability. To address this issue, we adopt a multimodal pretraining approach. UniL leverages rich information from data in various modalities and transfers it to 3D encoder by aligning multimodal features within a unified feature space, enhances the encoder's learning capacity and improves feature representation.To obtain a unified semantic space integrating point clouds, vision, and language, we followed ULIP and created a dataset consisting of point clouds, images, and textual descriptions. For each CAD model indexed as 𝑖, we constructed a triplet sample 𝑇 𝑖 consisting of a point cloud 𝑃 𝑖 , a depth map collection {𝐷 𝑖 𝑣 } 𝑁 𝑣=1 , and a text description set 𝑆 𝑖 . After uniformly sampling 𝑁 points from the original point cloud of CAD model, we applied standard point cloud data augmentation techniques, including random point drop, scaling, shifting, and rotation.Next, we choose the Realistic Projection proposed in [69] instead of the rendering to obtain depth maps of point clouds, as depth maps can more accurately reflect the geometric shapes and spatial structures of objects. Specifically, 3D voxel grid 𝐺 ∈ R 𝐻 ×𝑊 ×𝐷 is assigned different depth values to represent the depth information of the point cloud 𝑃 𝑖 = (𝑥, 𝑦, 𝑧) by utilizing:EQUATIONwhere 𝐻 , 𝑊 , 𝐷 denote the spatial resolutions of G and 𝑠 ∈ (0, 1] is a scale factor. Following this, local minimum pooling operations are applied to densify the grid, and the depth values are reassigned to occupy previously sparse voxels. Next, a non-parametric Gaussian kernel is employed for the purposes of smoothing and filtering, compressing the depth dimension to produce the final projected depth map. We project each point cloud from N different viewpoints to generate a collection of depth maps denoted as{𝐷 𝑖 𝑣 } 𝑁 𝑣=1 for point cloud 𝑝 𝑖 .Finally, following ULIP's work, we utilize the category name of each point cloud during pretraining, using 63 prompts such as "an image of [category]" and an additional prompt "a point cloud model of [category]" to accommodate the 3D modality. At the pretraining stage, a set of textual descriptions 𝑆 𝑖 is generated by applying 64 templates to the category name of each point cloud.3.1.2 Feature Extraction. In pretraining, acquiring high-quality representations from other modalities is crucial, as it necessitates aligning the features of the 3D modality with these representations to gain knowledge from other modalities. We choose CLIP [43] as the teacher for our 3D encoder and freeze the parameters of its visual and textual encoders. By leveraging CLIP's reliable representations, we bridge the gap between the limited availability of 3D data and the rich multimodal knowledge encapsulated in CLIP.For the visual modality, we input each depth map of {𝐷 𝑖 𝑣 } into the visual encoder 𝐸 𝑖 (•) of CLIP, and obtain the depth feature collection for sample i as follows:EQUATIONDrawing from the insights of [18] , we furthur employ the proposed Adapter to dynamically fuse the depth feature set {𝐹 𝑖 𝑣 } 𝑁 𝑣=1 across multiple views. Subsequently, the image-domain feature 𝑓 𝐼 𝑖 for sample 𝑖 can be obtained via the following formulation:EQUATIONwhere 𝑓 1 and 𝑓 2 represent two-layer MLP networks, and 𝛼 𝑣 denotes the dynamic fusion coefficients for the 𝑣-th view.For the linguistic modality, we employ CLIP's text encoder 𝐸 𝑖 (•) to process the descriptions 𝑆 𝑖 , generating a set of text representations. Following this, we employ average pooling on the resulting set, yielding the text-domain feature representation 𝑓 𝑇 𝑖 for each sample 𝑖, expressed as:EQUATIONwhere the 𝐴𝑉 𝐺 (•) denotes the average pooling operation. Within the VAE framework, the decoder receives features that have been re-parameterized. Recognizing that aligning features across different training paradigms may lead to unstable training and potential information loss, we choose to directly utilize the global features obtained from the encoder. To convert the encoded 3D feature into a multimodal embedding space, we incorporate a projection network after the 3D encoder 𝐸 𝑝 (•). This allows us to formulate the final 3D features 𝑓 𝑃 𝑖 as follows:EQUATIONwhere 𝑃 𝑖 is the augmented point cloud, and 𝑃𝑟𝑜 𝑗 (•) is a single-layer MLP.The objective of pretraining is training the 3D point cloud encoder 𝐸 𝑝 (•) to align the 3D features of sample 𝑖 with its image and text features. Thus, ULIP investigates the feasibility of transferring 2D contrastive learning to 3D domain by adopting a contrastive loss similar to CLIP to achieve alignment between 3D and image features:EQUATIONwhere 𝑖 and 𝑗 are indices of samples, and 𝜏 is a learnable temperature parameter. Similarly, the alignment between 3D and text features is formulated as: ). (7) Although the contrastive loss used in ULIP is relatively effective in aligning feature representations from different modalities, simply maximizing the logits on the main diagonal of the similarity matrix between 3D and text is not sufficiently accurate. This is because point clouds from the same category share identical prompts, and these prompts, after being encoded by the frozen CLIP, yield identical textual features. Hence, inspired by the supervised contrastive learning [19] , we propose a novel Multimodal Supervised Contrastive Loss (MSC Loss), which modifies the original self-supervised 3D-to-text loss to:EQUATIONEQUATIONwhere 𝑖, 𝑗, 𝑘 are indices of the samples, 𝑦 represents the label information of the samples, and 𝜏 is a learnable temperature parameter. This loss function indicates that 3D features belonging to the same class are brought closer together in the embedding space, while simultaneously pushing apart 3D features from different classes. Therefore, the ultimate training objective is to train the 3D encoder 𝐸 𝑝 (•), minimizing the MSC Loss:EQUATIONThrough Multimodal Supervised Contrastive learning, the 3D encoder can acquire more discriminative feature representations, leading to samples from same category being closer while scattering those of different categories further apart.During the training of the VAE, known categories of point clouds are provided as inputs. By minimizing the disparity between the original data and the reconstructed data, the VAE can discern patterns and structures in normal data. Consequently, when confronted with unseen samples, their reconstruction error tends to be notably higher compared to that of normal samples. EQUATIONEQUATIONEQUATIONwhere 𝑓 1 and 𝑓 2 are single-layer FC layers. Next, we sample from the distribution N (𝜇 𝑖 , 𝜎 𝑖 2 ) to obtain the reparameterized feature 𝑍 𝑖 :EQUATIONwhere 𝜀 is sampled from N (0, 1). Finally, 𝑍 𝑖 is fed into the decoder 𝐷 𝑝 inspired by [64] , to obtain the reconstructed point cloud p𝑖 :EQUATION3.2.2 Training Objective. Existing literature has introduced two permutation-invariant metrics for comparing unordered point sets: Chamfer Distance and Earth Mover's Distance [11, 46] . Following the setup of [35] , we utilize CD to compute the reconstruction error 𝐿 𝑟𝑒𝑐 for point cloud sample 𝑃 𝑖 . This choice is motivated by the faster convergence and lower computational cost of CD compared to EMD. 𝐿 𝑟𝑒𝑐 is calculated as follows:EQUATIONFollowing the traditional VAE [20] approach, the KL divergence is utilized as the fitting loss to minimize the discrepancy between the Gaussian distribution N (0, 1) and N (𝜇, 𝜎 2 ) derived from the original point cloud 𝑃 𝑖 . The KL divergence is defined as:EQUATIONAdditionally, [35] employ a second KL divergence to measure the difference between the Gaussian distribution N (0, 1) and N ( μ, σ2 ) obtained by inputting the reconstructed point cloud P𝑖 into the network:EQUATIONTherefore, the overall training objective is defined as:EQUATION)To assess whether a sample is novelty, [35] adapt the reconstruction error calculated using Chamfer Distance as its anomaly score. However, VAE is a versatile generative model that samples from a latent space to generate data. Since the latent space is continuous, even if the input data is unseen, the model might find similar points in the latent space and decode them to generate reconstructions similar to the input data, resulting in a relatively small reconstruction error.To tackle this challenge, we introduce the Depth Map Error (DME) as a score to incorporate information from other modalities of the point cloud. DME quantifies the disparity in depth maps between the original and reconstructed point clouds. This approach aims to mitigate the aforementioned challenge, providing a more effective measure to capture novelties. Therefore, the score used for novelty detection can be formalized as:EQUATION)where 𝐷𝑀𝐸 (•) represents the function computing the mean squared error between pixels of two depth maps, and 𝑤 is the scoring coefficient.Table 1 : Results on the Synthetic Benchmark track. Each column title indicates the chosen known class set, the other two sets serve as unknown. "DGC" refers to the backbone network DGCNN [59] , while "PN2" denotes the backbone network PointNet++ [42] .  
experiment: 
 To illustrate the advantages of utilizing the pretrained 3D encoder through UniL, we conducted experiments on 3DOS. First, we outline the experimental settings, including datasets and implementation details. Following this, we present the quantitative results of 3D novelty detection on three benchmarks. Finally, we conduct an analysis and demonstration of the proposed components, validating their effectiveness.We use the following dataset employed for building the benchmark proposed in 3DOS [3] : ShapeNetCore [5] contains of 51,127 CAD models from 55 common object categories. In 3DOS, ShapeNetCore v2 is used with the official training (70%), validation (10%), and testing (20%) splits. All point clouds are uniformly sampled from mesh surfaces, normalized to fit within a unit cube centered at the origin and consistently aligned. In the benchmark setting, similar semantic categories, like cellphone and telephone, are merged to obtain 54 classes.ModelNet40 [60] comprises 12,311 CAD models from 40 categories. 3DOS adopts the dataset partitioning of [42] , which consists of 9,843 training samples and 2,468 testing samples. Each point cloud is uniformly sampled from the surfaces of synthetic CAD models and scaled to fit within a unit cube centered at the origin.ScanObjectNN [55] is a dataset of scanned 3D objects from the real world, unlike ShapeNetCore and ModelNet40, consists of 2,902 samples from 15 categories. These samples can be divided into two categories based on the presence of background: OBJ_ONLY and OBJ_BG. In 3DOS, the original OBJ_BG split is considered, wherein 3D scans are impacted by acquisition artifacts. These samples are already in the form of point clouds, with each containing 2048 points, representing both foreground and background objects along with other interacting elements.4.2.1 Pre-training with UniL. We uniformly sample 2048 points from each sample and generate projected depth maps and textual descriptions. CLIP is utilized to acquire multimodal embedding, while both the image and text encoder in our experiment remain frozen, akin to ULIP. During pretraining, only the parameters in the 3D encoder and projection network are trainable. We train Unil for 150 epochs with a batch size of 64 and a learning rate of 1e-4. The optimizer used is AdamW, with a weight decay of 0.1.During VAE training, we used publicly available code provided by [35] , a practice also adopted by 3DOS [3] . Specifically, we randomly sampled 2048 points from each point cloud and applied data augmentation techniques including scaling, translation transformations, and random rotation around the upaxis. The Adam optimizer was employed with a weight decay of 1e-6, and the learning rate was set to 1e-3. Training was conducted with a batch size of 100 for 300 epochs.In the Synthetic Benchmark, due to the relatively abundant data of ShapeNetCore, we fine-tuned the entire 3D encoder. However, in the Synth to Real Benchmark, where the source domain data distribution differs from that of the target domain, updating the encoder was found to lead to catastrophic forgetting. Hence, we opted to freeze the entire encoder. For the Real to Real Benchmark, due to the limited data of ScanObjectNN and to prevent overfitting, we froze the low-level semantic layers of the encoder and only updated the high-level semantic layers. sets: SN1, SN2, and SN3. The ten categories of ModelNet40 corresponding to ScanObjectNN are divided into SR1 and SR2, while the remaining five categories in ScanObjectNN form SR3.. When conducting novelty detection across the three benchmarks, we set the scoring coefficients w to 0.35, 0.3, and 0.15 respectively. Tab. 1 presents the results of the Synthetic benchmark, where our approach outperforms all other methods on SN1(hard), achieving top-1 performance. In detail, our approach demonstrates a 2.0% increase in AUROC and a reduction of 5.5% in FPR95 compared to DGCNN + cosine proto. In SN2(med) and SN3(easy), although not reaching the top-1, our method enables the only unsupervised method VAE to show competitive results.The results of our method on the Synth to Real benchmark are provided in Tab. 2. Due to the domain shift, performance degradation was observed in all methods. However, despite this challenge, we managed to further reduce FPR95 by 0.9% on SR1 (easy). In the Real to Real benchmark, the authors of 3DOS did not evaluate VAE. Hence, we supplemented this result and included our method in Tab. 3 for comprehensive comparison. Notably, our method achieves a 1.9% increase in AUROC on SR1 (hard) compared to the previous state-of-the-art method PointNet++ + cosine proto. These performance improvements are attributed to the rich information brought by pretraining as well as the capability of DME to uncover hidden novel patterns. Regarding other tasks, our method fell short of achieving the SOTA. This is because VAE [35] is the only method trained without the use of labels, unlike other methods evaluated in 3DOS [3] . Despite utilizing label information during pretraining, aligning multimodal features alone did not provide the 3D encoder with discriminative power comparable to methods trained through classification. Although our method did not achieve the state-ofthe-art in every task, it significantly improved the performance of the original VAE across all tasks. In order to investigate the exact contributions of MSC Loss on the pretraining phase and DME on novelty detection, independent ablation studies of the two modules are conducted in our approach.With the aim of evaluating model performance in novelty detection, AUROC and FPR95 metrics are utilized as quantitative evaluation criteria across three benchmarks. The comparison results across the three benchmarks are respectively illustrated in Tab. 4, Tab. 5, and Tab. 6.4.4.1 ULIP Loss vs. MSC Loss. First, we conducted an ablation study to explore the impact of using different loss functions during pretraining. Our findings suggest that performing coarse-grained feature alignment during the pretraining stage can enhance the model's performance in subsequent novelty detection tasks. However, with the incorporation of MSC Loss, the model exhibited further improvements across different tasks on the three benchmarks, achieving average increases of 0.7%, 2.0%, and 2.4% in AUROC, respectively. This is attributed to MSC Loss leveraging label information to more effectively cluster data of the same category in the feature space, thereby enhancing the discriminative capability of the 3D encoder. As illustrated in Fig. 3 , this approach enables data from the same category to acquire more similar features, thus facilitating the formation of denser clusters.  
conclusion: 
 In this paper, we introduce UniL, a multimodal pretraining framework tailored for 3D novelty detection. During pretraining, we propose the MSC Loss to better assist the 3D encoder in feature representation by leveraging label information. In the novelty detection phase, we utilize DME to measure the disparity between depth maps projected before and after reconstruction of point clouds, thus further enhancing the performance of unsupervised VAE. Through extensive experiments on the benchmark proposed in 3DOS [3] , we validate the efficacy of our approach in 3D novelty detection tasks and provide valuable insights for future research. 
 
 Appendix:  


Decision: The paper presents a well-organized and clearly written study on novelty detection using CLIP. The motivation is reasonable, focusing on learning and aligning multimodal information, particularly 3D information. The proposed method is novel, relatively complete, and appears effective in representing 3D objects with depth information. The method is evaluated on multiple benchmarks, with good performance in experimental data. The paper also proposes using image and text modalities to enhance the encoding of point clouds for scarce data, addressing key issues in 3D visual understanding.  All reviewers support accepting this paper, recognizing its novelty and clear writing. The authors address some concerns raised by reviewers during the rebuttal. The ACs agree with the reviewers and thus suggest accepting this paper.

Reviews: review: paper_summary: paper_summary: Compared to 2D data, the scarcity of available 3D data leads to the suboptimal representation of trained models. To address this challenge, authors propose a Unified Learning Framework (UniL) which aligns visual, textual, and 3D features within the same feature space for novelty detection.
strength_weakness: strength_weakness: The paper is well-organized and easy to follow.  
The experiments are extensive.  
The idea that proposes CLIP in novelty detection is interesting.
limitations: limitations: The motivation for detailed implementations, e.g., MSC Loss, and DME, is unclear.  
The novelty is limited, as  Realistic Projection and contrastive loss are utilized in existing works. The author should highlight the novelty. 
The performances do not reach SOTA, compared with PN2+Cosine Proto.
paper_summary: paper_summary: This paper explores anomaly detection in 3D objects based on multimodal learning. This paper first propose the UniL framework to extract features from multi-view images, text descriptions, and 3D objects, and then use contrastive learning to align information between different modalities with the proposed Multimodal Supervised Contrastive Loss (MSC Loss) for pre-training a robust 3D visual encoder. Then it trains a Variational Auto-Encoder (VAE) based on the pre-trained 3D visual encoder to reconstruct the point cloud of the 3D object. Furthermore, this paper uses VAE and the proposed Depth Map Error (DME) for anomaly detection. Various experimental results basically prove the effectiveness of the proposed methods.
strength_weakness: strength_weakness: 1. The writing and presentation of this paper is clear, the structure is reasonable, and the figures and tables are clearly presented.

2. The motivation of this paper is reasonable. Multimodal information, especially 3D information, needs to be learned and aligned.

3. The method proposed in this paper is novel and relatively complete, and seems to be able to represent 3D objects with depth information well.

4. The proposed method is evaluated on multiple benchmarks.
limitations: limitations: I still have some concerns about this paper:

1. For the proposed UniL, the paper does not conduct much exploration on the design of text input. Are these natural language-based inputs robust to the entire model? More ablation experiments are better.

2. For training VAE, I think it is very important to get good point cloud synthesis capabilities. For the output of the decoder, can their depth information be synthesised well? You should have some analysis here, which is very important for the proposed Depth Map Error (DME) in the next step.

3. For anomaly detection, we have more common metrics to represent anomaly scores, such as energy score, or knn distance. Will the proposed Depth Map Error (DME) be better than them?

Additionally, for the pre-trained multimodal feature space, although the experimental results are evaluated on multiple benchmarks, I am concerned that the alignment of the three modalities is difficult to generalize sufficiently, and I would like to get some more discussion here.
paper_summary: paper_summary: This paper proposes a new Unified Learning Framework(UniL) for novelty detection. It argues that the scarcity of 3D data may lead to training bias and provides a more efficient 3D point cloud encoder trained by fusing 2D deepth images, texts and 3D point clouds. For novelty detection, built on unsupervised VAE, this paper designs a new MSC Loss to improve the compactness of clustering. A depth map error is additionally introduced to novelty score, reducing the error caused by potential distribution consistency. The effectiveness of the proposed method is verified by experiments, and the effectiveness of MSC Loss and DME is verified by ablation experiments.
strength_weakness: strength_weakness: - This article is very well written and clearly illustrated. 
- For scarce point cloud data, this article proposes to use image and text modalities to enhance the encoding of point clouds.
limitations: limitations: - Although the multimodal encoder is able to encode 3D point clouds better, it would be useful to provide  the experimental validation of the comparison between proposed encoder and other encoders.
- The method proposed in this article can far outperform the original unsupervised VAE method, but still falls short of other supervised methods.
paper_summary: paper_summary: This paper proposed a Unified Learning Framework for facilitating novelty detection to address the impact of 3D data scarcity on detection accuracy. In addition, authors introduced a new Multimodal Supervised Contrastive Loss (MSC Loss) and Depth Map Error(DME).
strength_weakness: strength_weakness: Being able to find key issues in 3D visual understanding and give solutions. The performance in the experimental data seems to be good.
limitations: limitations: 1.The methodology of this paper is not innovative enough and the network is highly approximate to ULIP.
2.In pre-training, due to the limited amount of data, catastrophic forgetting can occur if the image and text encoders of CLIP are updated. When applying ULIP to downstream tasks, this leads to a significant performance degradation, and the authors' choice to freeze the weights of CLIP's visual and text encoders seems to solve the catastrophic forgetting. Please ask the authors what approach they utilize to address the problem of error accumulation in continuous learning.
3.The authors state that the MSC Loss enables samples from the same class to form denser clusters in the feature space, but the difference between the ULIP Loss and the MSC Loss is not that great from the visualization in Figure 3, which looks more like homologous scatter aggregation.