DAC: 2D-3D Retrieval with Noisy Labels via Divide-and-Conquer Alignment and Correction

Authors: encoder  encoder
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-19 13:04:02

Abstract: With the recent burst of 2D and 3D data, cross-modal retrieval has attracted increasing attention recently. However, manual labeling by non-experts will inevitably introduce corrupted annotations given ambiguous 2D/3D content, leading to performance degradation. Though previous works have addressed this issue by designing a naive division strategy with hand-crafted thresholds, their performance generally exhibits great sensitivity to the threshold value, implying their poor robustness in real-world scenarios. Besides, they fail to fully utilize the valuable supervisory signals within each divided subset. To tackle this problem, 
we propose a Divide-and-conquer 2D-3D cross-modal Alignment and Correction framework (DAC), which comprises Multimodal Dynamic Division (MDD) and Adaptive Alignment and Correction (AAC). Specifically, the former performs accurate sample division by adaptive credibility modeling for each sample based on the compensation information within multimodal loss distribution. Then in AAC, samples in distinct subsets are exploited with different alignment strategies to fully enhance the semantic compactness and meanwhile alleviate over-fitting to noisy labels, where a self-correction strategy is introduced to improve the quality of representation by mining the valuable supervisory signals from multimodal predictions as well. Moreover. To evaluate the effectiveness in real-world scenarios, we introduce a challenging noisy benchmark, namely Objaverse-N200, which comprises 200k-level samples annotated with 1156 realistic noisy labels. Extensive experiments on both traditional and the newly proposed benchmarks demonstrate the generality and superiority of our DAC, where DAC outperforms state-of-the-art models by a large margin (i.e., with +5.9\% gain on ModelNet40 and +5.8\% on Objaverse-N200).

Main paper: 
 learning with noisy labels.: 
 Existing LNL methods could be roughly classified into robust architecture, robust loss, and sample selection approaches. The robust architecture-based methods focus on modeling the noise transition matrix with a noise adaptation layer [7, 15, 33] . However, these methods can not identify false-labeled examples, by treating all the samples equally. For robust loss-based methods, existing works [31, 37] leverage the self-prediction of the model to effectively refine the noisy labels to exploit the valuable semantics in the noise sample. Furthermore, various methods [39, 44] adopt an extra metacorrector to obtain higher-quality corrected labels. However, they require a small clean dataset for evaluation, which is not realistic in real-world scenarios. Sample selection-based methods focus on selecting clean examples from a noisy dataset. Among them, the small loss trick is a popular selection criterion [6, 17, 30] . Based on the small-loss trick, DivideMix [27] adopts GMM to model the loss distribution, thus better distinguishing clean and noisy samples. However, these sample selection methods could not be directly applied to the field of cross-modal retrieval due to the heterogeneity and geometric gap across 2D and 3D data. Additionally, when confronting complex and high-ratio noise scenarios, the GMM does not work well due to the largely overlapping of loss distribution and the confirmation bias issues [17] . 
methodology 3.1 overview: 
 For convenience, we first clarify the notations within our DAC framework. Specifically, the noisy multimodal dataset is denoted asD = M 𝑗 𝑀 𝑗=1 = X 𝑗 , Y 𝑗 𝑀 𝑗=1, where 𝑀/𝐾 denote the number of modality and classes respectively. M 𝑗 can be represented asM 𝑗 = 𝑥 𝑗 𝑖 , 𝑦 𝑗 𝑖 𝑁 𝑖=1, where 𝑁 is the sample number, 𝑥 𝑗 𝑖 /𝑦 𝑗 𝑖 denote the 𝑖-th sample and noisy label in M 𝑗 . The feature extractor for M 𝑗 is represented as 𝜙 𝑗 , and the feature 𝑧 𝑗 𝑖 = 𝜙 (𝑥 𝑗 𝑖 ). For simplicity, we denote the multimodal data pair for 𝑖-th sample as(𝑥 𝑖 , 𝑦 𝑖 ) = ({𝑥 𝑗 𝑖 } 𝑀 𝑗=1 , {𝑦 𝑗 𝑖 } 𝑀 𝑗=1). Then we give out an overview of DAC framework. As shown in Fig. 3 , DAC consists of two main components: Multimodal Dynamic Division (MDD) and Adaptive Alignment and Correction strategy (AAC). Specifically, the 2D and 3D samples in D are first fed into MDD to perform adaptive credibility modeling for each sample by considering multimodal loss distribution. Then these samples are divided into clean/noisy subsets based on the estimated credibility with adaptive thresholds. Afterward, in AAC, samples in the clean set are directly utilized for cross-modal alignment to fully harvest the discriminative information. For samples in the noisy set, we leverage the inherent supervisory signals in their multimodal predictions to perform label purification, which can mitigate the negative impact of noisy labels. Then the purified labels are used for the same alignment strategy as the clean set to further enhance the representation compactness.To clarify our MDD more clearly, we first simply introduce the classification loss. For noisy sample (𝑥 𝑗 𝑖 , 𝑦 𝑗 𝑖 ), to alleviate the semantic discrepancy across different modalities in label space, a classification loss is utilized to optimize the representation under label noise. Specifically, a shared classifier layer 𝐹 𝑢 is proposed to obtain the classifier prediction 𝐹 𝑢 (𝑧 𝑗 𝑖 ; 𝜃 𝑢 ) of each modality of the sample, and then a conventional cross-entropy (CE) loss is utilized to optimize the network, which is formulated as: (𝑖=1,𝑗=1) of the noisy dataset. However, performing sample division based on the unimodal loss distribution overlooks the complementary information in the multimodal data, which is not sufficient to deal with complex noisy scenarios. In addition, unimodal loss distribution of true-labeled and false-labeled examples largely overlaps in complex noise scenarios. To tackle these issues, we propose a multimodal loss distribution, which could adaptively capture the valuable semantics in the multimodal data and generate a more discriminative distribution for sample division. Specifically, we first fuse the multimodal features {𝑧 𝑗 𝑖 } 𝑀 𝑗=1 for each sample (𝑥 𝑖 , 𝑦 𝑖 ) by a fusion layer 𝜓 , which is formulated as:EQUATIONEQUATIONwhere ẑ𝑖 is the fused feature. Then, a multimodal classifier 𝐹 𝑚 is adopted to calculate the multimodal prediction 𝐹 𝑚 ( ẑ𝑖 ; 𝜃 𝑚 ) for each noisy sample (𝑥 𝑖 , 𝑦 𝑖 ) and the multimodal loss 𝑙 𝑖 is calculated as:EQUATIONTo further avoid confirmation bias issues [17] , we jointly utilize the shared classifier 𝐹 𝑢 and the multimodal classifier 𝐹 𝑚 to calculate the final multimodal loss distribution 𝑙 = {𝑙 𝑖 } 𝑁 𝑖=1 , and the final multimodal loss 𝑙 𝑖 is formulated as:𝑙 𝑖 = 𝐿 (𝐹 𝑚 ( ẑ𝑖 ; 𝜃 𝑚 ) , 𝑦 𝑖 ) + 1 𝑀 𝑀 ∑︁ 𝑗 𝐿 𝐹 𝑢 𝒛 𝑗 𝑖 ; 𝜃 𝑢 , 𝑦 𝑗 𝑖 (4)Both 𝐹 𝑢 and 𝐹 𝑚 are optimized using Eq. 1. Sample credibility. Due to the memorization effect of DNNs [3] , clean samples usually converge at a faster pace than noisy ones, thus the multimodal loss distribution 𝑙 tends to be bimodal during the training stage. Hence, we adopt a two-component Gaussian Mixture Model (GMM) to fit the multimodal loss distribution 𝑙, thereby estimating the credibility of each sample. The GMM is defined as:EQUATIONwhere N (𝑙 | 𝜇 𝑡 , Σ 𝑡 ) is the Gaussian probability density function with mean 𝜇 𝑡 and covariance Σ 𝑡 , and 𝜋 𝑡 is the weight for 𝑡-th Gaussian component and we have 𝑇 𝑡 =1 𝜋 𝑡 = 1, and 𝑇 is the number of components of GMM. In our case, 𝑇 = 2.Then, we use the Expectation Maximization (EM) procedure [10] to fit the two components of GMM. Considering the efficiency, we iterate the EM procedure with 10 iterations to update the parameters(i.e. 𝜋 𝑡 , 𝜇 𝑡 , Σ 𝑡 ) of GMM. Finally, we obtain the credibility 𝛾 𝑖 of each sample through the posterior probability:EQUATIONwhere N (𝑙 𝑖 | 𝜇 1 , Σ 1 ) denotes the Gaussian component with a smaller mean (smaller loss).With the adaptive credibility modeling, we could divide the noise data into clean set 𝑆 𝑐 and noisy set 𝑆 𝑛 :EQUATION)where 𝛼 is a threshold for clean-noise separation.Following the sample division, we employ adaptive alignment techniques to manage the distinct subsets, 𝑆 𝑐 and 𝑆 𝑛 . To address the noisy samples in 𝑆 𝑛 , a self-correction mechanism is introduced, which leverages the model's multimodal predictions to refurbish the corrupted labels, thereby boosting the semantic compactness and discrimination of the learned representations. Adaptive Sample Alignment. We employ different alignment strategies to deal with the samples in different subsets(i.e. 𝑆 𝑐 ,𝑆 𝑛 ).For the samples in 𝑆 𝑐 , due to the high reliability of the labels, we directly use the labels for semantic alignment which could effectively mitigate the semantic gap across multimodal representations. Previous works adopt the center learning [12, 22] to achieve semantic alignment, which compacts the different modalities of samples to corresponding semantic centers in the common feature space. However, they optimize the center learning by a center loss in the form of absolute errors [12, 22] , which is hard to optimize and limits the compactness of representation. To tackle these issues, we propose a contrastive center loss to optimize center learning, which is formulated as:L 𝑠𝑒𝑚 = - 1 𝑀𝑁 ∑︁ 𝑥 𝑖 ∈𝑆 𝑀 ∑︁ 𝑗 log 𝑒 1 𝜏𝑐 𝒄 (𝑘=𝑦 𝑗 𝑖 ) 𝑇 𝒛 𝑗 𝑖 𝐾 𝑛=1 𝑒 1 𝜏𝑐 (𝒄 𝑛 ) 𝑇 𝒛 𝑗 𝑖 , 𝑆 = 𝑆 𝑐 /𝑆 𝑛 (8)where 𝑐 𝑛 is the learnable shared clustering center of 𝑛-th category in the common space, 𝜏 𝑐 is a temperature parameter.For the samples in 𝑆 𝑛 , due to the little discriminative information in their labels, we utilize them for training with instance alignment, which reduces the inherent gap between 2D and 3D data from the instance-based perspective. Specifically, we adopt a Multi-modal Dataset Objects Classes ModelNet10 [40] 4899 10 ModelNet40 [40] 12311 40 ShapeNet [5] 51190 55 ScanObjectNN [35] 2902 15 Objaverse-N200 194800 1156Modal Gap loss(MG) [20] to optimize the cross-modal representation of samples, which is formulated as:L 𝑖𝑛𝑠𝑡 = - 1 𝑀𝑁 ∑︁ 𝑥 𝑖 ∈𝑆 𝑀 ∑︁ 𝑗 log 𝑀 𝑘 𝑒 1 𝜏𝑚 𝒛 𝑘 𝑖 𝑇 𝒛 𝑗 𝑖 𝑁 𝑙 𝑀 𝑚 𝑒 1 𝜏𝑚 𝒛 𝑚 𝑙 𝑇 𝒛 𝑗 𝑖, 𝑆 = 𝑆 𝑐 /𝑆 𝑛 (9) 𝜏 𝑚 is a temperature parameter. Due to the randomness of the category centers 𝒄 𝑛 caused by the random initialization, we also apply the instance alignment based on Eq. 9 for samples in 𝑆 𝑐 to further alleviate the inherent gap across different modalities. Self-Correction(SC). To exploit useful semantic information from the noisy set 𝑆 𝑛 , a self-correction strategy is proposed, which mines the valuable semantics from the classifier's self-prediction. Specifically, we utilize the classifier's prediction to correct the label of the sample in 𝑆 𝑛 . To obtain more reliable corrected labels, we correct the mislabeled samples based on the prediction of the multimodal classifier 𝐹 𝑚 . For each sample (𝑥 𝑖 , 𝑦 𝑖 ) in 𝑆 𝑛 , the corrected label ŷ𝑖 is a soft label and is written as:EQUATIONHowever, due to the unstable training process, it is not reliable to refurbish the corrupted labels with the prediction of a single epoch. Therefore, we use the idea of Exponential Moving Average(EMA) to promote the reliability of our label correction. At epoch e, the moving-average corrected label over multiple training epochs isEQUATIONwhere 𝜇 = 0.9. In addition, for the corrected sample (𝑥 𝑖 , arg max( ŷ𝑒 𝑖 )), we also employ semantic alignment based on Eq. 8 to further enhance the compactness of representation. Overall loss. The complete loss function of our DAC can be formulated as:EQUATION) where 𝜆 1 , 𝜆 2 are balance parameters for three losses. 
objaverse-n200: 
 To evaluate the effectiveness of our model in real-world scenarios, we developed a realistic and noisy benchmark, Objaverse-N200, built upon the recently proposed 3D dataset, Objaverse [9] . Due to the high expense and time-consuming nature of annotation, there is no category annotation for the object in Objaverse. To tackle this issue, we utilize a large pre-trained model Uni3D [46] to assign categories to these objects. Specifically, we first employ Uni3D for zero-shot classification, where unlabeled objects are assigned to their corresponding categories based on the category set derived from Objaverse-lvis. Then we collect the top 200 samples of each category to build the 3D dataset. Finally, we attain a real-world noisy 3D benchmark Objaverse-N200 containing about 200k 3D objects and annotated with 1156 realistic noisy labels, and the noise ratio of Objaverse-N200 is about 50%. The comparison between Objaverse-N200 and traditional 3D datasets is shown in table 1. Objaverse-N200 surpasses prior datasets by over an order of magnitude in size and number of categories. More information about Objaverse-N200 can be referred to our supplementary materials. 
experiments 5.1 experimental setup: 
 Noisy test benchmarks. We conduct extensive experiments on three traditional 3D datasets (3D MNIST [41] , ModelNet10, Model-Net40 [40] ) and our realistic noisy dataset Objaverse-N200. Following [12] , we consider two noisy settings: Symmetric/Asymmetric noise, and use the same evaluation protocol. The noise rates for symmetric/asymmetric noise are [0.2, 0.4, 0.6, 0.8] and [0.1, 0.2, 0.4] respectively. It's important to note that Objaverse-N200 inherently constitutes a realistic noisy benchmark. Due to the space limitation, we only present the results of ModelNet10/40 and Objaverse-N200. More experimental results and details can be referred to our supplementary materials. Implementation details. For the traditional datasets, we adopt the ResNet18 [18] /DGCNN [38] /MeshNet [11] as the backbone network for RGB/point cloud/mesh feature extraction. Then, all the features are projected to 256-D with two fully connected layers. The training epochs are 400 epochs with an initialized learning rate of 0.0001 and the learning rate dropped per 100 epochs with factor 10. For Objaverse-N200, we adopt the pre-trained image encoder and point cloud encoder in Uni3D [46] to extract the image and point cloud feature. Then, the features are projected to 1024-D with two fully connected layers. The training epochs are 40, while the initialized learning rate is 0.0001 and the learning rate is decayed with factor 10 at epochs 20 and 40. For all the datasets, the batch size is 128 and the optimizer is Adam [25] with a momentum of 0.9 for all the noisy benchmarks. 𝜏 𝑐 and 𝜏 𝑚 are set as 0.22 and 1.0 respectively. For the fusion layer 𝜓 , we investigate various fusion structures (Add, Concat, Attention). Due to space limitation, the investigation of 𝜓 can be referred to our supplementary materials and we use the Concat (feature concatenation) as our final 𝜓 .Results on traditional benchmarks. To evaluate our method, we conduct detailed experiments on ModelNet10 and ModelNet40 [40] as shown in Tab. 2 and Tab. 3. As RONO employs a supervised pre-trained point cloud encoder [38] for feature extraction, which may bias the comparison, we also conduct experiments based on unsupervised pre-trained point cloud encoder [1] , denoted by * in the results. And non- * denote using supervised pre-trained point cloud encoder, consistent with RONO's protocol [12] . From these results, we could obtain the following observations: 1) Improvements on mAP. Overall, our model(DAC/DAC * ) achieved superior results compared to both unsupervised (DCCAE, UCCH, etc.) and supervised (MRL/MRL * , RONO/RONO * , etc.) methods across different noisy ratios. Specifically, our model achieves 5.8% ↑ on ModelNet40 under 80% symmetric noise and 3% ↑ on ModelNet40 under 20% asymmetric noise. 2) Robustness. The results reveal that previous [36] 0.703 0.703 0.703 0.703 0.693 0.693 0.693 0.693 0.593 0.593 0.593 0.593 0.572 0.572 0.572 0.572 DGCPN [42] 0.765 0.765 0.765 0.765 0.759 0.759 0.759 0.759 0.705 0.705 0.705 0.705 0.697 0.697 0.697 0.697 UCCH [21] 0.771 0.771 0.771 0.771 0.770 0.770 0.770 0.770 0.755 0.755 0.755 0.755 0.739 0.739 0.739 0.739 AGAH [16] 0.862 0.821 0.805 0.756 0.867 0.827 0.801 0.743 0.807 0.817 0.778 0.778 0.799 0.800 0.779 0.761 DSCMR [43] 0.849 0.851 0.838 0.675 0.842 0.825 0.810 0.661 0.867 0.831 0.811 0.656 0.866 0.819 0.804 0.651 MRL [20] 0.887 0.869 0.867 0.859 0.871 0.865 0.861 0.854 0.848 0.846 0.838 0.811 0.843 0.844 0.838 0.799 CLF [22] 0.884 0.856 0.803 0.741 0.867 0.840 0.798 0.743 0.871 0.855 0.820 0.757 0.878 0.852 0.813 0.758 CLF+MAE [14] Table 5 : Performance on the Objacerse-N200 datasets.Objaverse-N200 Img → Pnt Pnt → Img Openshape [28] 0.274 0.257 MRL [20] 0.260 0.252 CLF [22] 0.238 0.192 CLF [22] +MAE [14] 0.237 0.192 RONO [12] 0.276 0.279 Ours 0.334 0.338 on the realistic noisy benchmark: Objaverse-N200. The comparison results are shown in Table 5 . The results reveal that most prior methods yield lower performance compared to unsupervised approaches, indicating the complexity and challenge posed by realistic noise and these methods struggle to deal with the real-world noise. In contrast, our DAC model outperforms the unsupervised method openshape [28] significantly by 6.0%, which demonstrates the effectiveness of our model against real-world noise. Generality of MDD. To validate the generality of our MDD, we integrate it with various popular traditional methods, including MRL [20] , CLF [22] , and RONO [12] . As shown in Table 4 , MDD consistently brings substantial improvements across various noise ratios for these methods. Notably, MDD boosts CLF's performance by an impressive 49.6% on ModelNet10 under 80% symmetric noise, which demonstrates its superior capacity to handle severe noise. Consequently, these results confirm the generality of our MDD in enhancing existing methodologies.Investigation on Division and Self-Correction. We evaluate the effectiveness of our proposed division strategy, MDD, and selfcorrection strategy in AAC in Fig. 4 . As depicted in 4(a) and (b), our methods consistently exhibit high accuracy in both dividing and correcting noisy samples across distinct datasets with varying noise Investigation on Multimodal loss distribution. We present experimental results to investigate our proposed multimodal loss distribution in Fig. 5 (a) and (b). Our results revealed the following key insights: 1) Mitigating overfitting of unimodal data. By comparing the unimodal and multimodal loss distributions, we noticed that false-labeled samples typically exhibit lower losses in the unimodal case, which can lead to overfitting and confirmation bias. Conversely, in the multimodal loss distribution, the normalized loss of these false-labeled samples is generally above 0.5, indicating a higher level of robustness 2) Enhancing reliability of sample division. The multimodal loss distribution exhibits a smaller overlap between its components compared to the unimodal one, indicating a higher reliability in sample division. This is attributed to the effective fusion of complementary information from different modalities. More experiments of loss distribution can be referred to our supplementary materials. Ablation Study. To assess the effectiveness of MDD and AAC, we conducted detailed ablation studies on ModelNet10 and Mod-elNet40, as presented in Table 6 . Comparing row 1 with rows 6, 7, and 8, adopting MDD to divide the noisy dataset before alignment significantly boosts model performance (9.9% and 10.1% ↑ for Sym-4% and Sym-80%), highlighting the effectiveness and robustness of our MDD. By comparing row 1 with row 2 and row 3, we observe that it is insufficient to divide the noisy dataset based on unimodal data. When adopting the fused multimodal feature with 𝐹 𝑚 , both the performance of ModelNet10 and ModelNet40 [40] can be improved significantly, especially under serve noisy scenes, due to its adaptive capture of semantics. Moreover, combining both 𝐹 𝑢 and 𝐹 𝑚 leads to additional performance gains (6.8% ↑ on ModelNet40 (Sym-80%)) as it addresses the confirmation bias of the single classifier(𝐹 𝑢 /𝐹 𝑚 ). Additionally, our AAC could further improve model performance (3.6% on ModelNet40 with 80% symmetric noise), evidencing the effectiveness of adaptive alignment and the reliability of corrected labels. These results demonstrate the effectiveness of each component. Parameter sensitivity. To evaluate the influence of threshold 𝛼 on RONO [12] and our DAC, we conducted experiments on Model-Net40 [40] with 40% symmetric noise. As shown in Fig. 5 (c) and (d), RONO [12] demonstrates significant sensitivity to 𝛼, resulting in poor robustness and sub-optimal performance. Conversely, our DAC exhibits superior and stable performance across various 𝛼 values, indicating its robustness to the threshold 𝛼. And, we set 𝛼 as 0.5 in our experiments.Further comparison with RONO. To thoroughly evaluate the performance of our proposed DAC, we conducted extensive retrieval and visualization experiments across three modalities (image, point cloud, and mesh) on ModelNet10 and ModelNet40 [40] , comparing it with the state-of-the-art RONO [12] . The results, presented in Tab.7 and Fig. 6 , show that our DAC consistently outperforms RONO across various noisy ratios, demonstrating the robustness and effectiveness of our DAC. 
conclusion: 
 In this paper, we propose a novel robust Divide-and-conquer Alignment and Correction framework (DAC) for noisy 2D-3D crossmodal retrieval. Specifically, our DAC employs a novel Multimodal Dynamic Division strategy to dynamically divide the noisy dataset based on the multimodal loss distribution. With the help of division, we design an Adaptive Alignment and Correction strategy to enhance the semantic compactness of representation while mining the implicit semantics in noisy samples. In addition, we introduce a challenging realistic noisy benchmark: Objaverse-N200. We conduct extensive experiments on both traditional 3D model datasets and Objaverse-N200 to demonstrate the superior robustness and effectiveness of our DAC against synthetic and realistic noise. Our method also can be integrated with existing methods as a plug-andplay strategy to achieve further improvement. 
: 
 data [8, 9] . An essential aspect of 3D data analysis and understanding is retrieving relevant 3D representations from given 2D/3D query input. This technique finds broad applications in virtual reality [13] , autonomous driving [26, 32] , and robotic manipulation [4, 34] . However, due to their complex geometrical structures, high dimensions, and irregular distribution, 3D data typically present more challenges compared to 2D data. Moreover, with the proliferation of large volumes of 3D data, data annotation is becoming increasingly expensive and time-consuming, leading to label noise issues in 2D-3D cross-modal retrieval. Consequently, learning with noisy labels has emerged as a crucial problem to address in 2D-3D cross-modal retrieval.Previous methods for cross-modal retrieval could be categorized into two families: unsupervised-based and supervised-based. The former [12, 20, 21] focuses on aligning the cross-modal features from instance-based views, thus mitigating the inherent heterogeneity gap across different modalities. As for the latter ones [19, 22, 24, 43] , they resort to a shared projection network for semantic gap reduction and adopt a traditional center loss to maximize the crossmodal correlation and minimize the intra-class variation. However, these methods generally suffer from significant performance degradation when applied in noisy scenarios. Thus, to tackle this issue, various methods are proposed to eliminate the negative impact of noise labels by designing Robust Clustering loss and employing sample division [12, 20] .However, the Robust Clustering loss [20] does not consider the sample-wise credibility and treats all the noise samples equally, which may fail in complex noisy scenarios. Inspired by the recent progress of sample division based methods [6, 17, 27, 30] in Learning with Noisy Labels (LNL), we attempt to adopt a similar division scheme to adaptively utilize the inherent information within each sample. However, progress made by division strategies in 2D LNL may fail to be observed in cross-modal tasks due to the task-wise discrepancy. Thus we first investigate the impact of the division strategy on the cross-modal retrieval performance. Specifically, we visualize the loss distribution of training without/with sample division in Fig. 2 (a) and (b) respectively. It can be observed that models trained without sample division tend to over-fit on corrupted labels, exhibiting small losses for all false-labeled samples. Conversely, models trained with sample division showcase a distinct bimodal loss distribution. Such a phenomenon verifies the importance of sample division in noisy cross-modal retrieval.Though the recent RONO [12] already integrates sample division to address the noisy issue, it adopts a simple hand-crafted threshold to distinguish true and false labeled samples as shown in Fig. 1 . We empirically find that such a naive scheme showcases great sensitivity to parameter variations and lacks generalization in complex scenarios. Specifically, as shown in Fig. 2 (c), it can be observed that the division performance of RONO significantly deteriorates with the noise ratio increasing. Moreover, RONO performs sample division independently in each modality, and thus fails to utilize the potential complementary information of two modalities, leading to its inferior performance as shown in Fig. 2 (c ). Additionally, to further verify the importance of the complementary effect, we conduct experiments by adopting the Gaussian Mixture Model (GMM) as the dynamic division strategy. As shown in Fig. 2 (d) , unimodal-based strategies (i.e. Image or Point cloud), generally yield sub-optimal division accuracy due to the insufficient information within each modality.To address the aforementioned problems, we propose a novel Divide-and-conquer 2D-3D cross-modal Alignment and Correction framework (DAC). Specifically, we introduce a Multimodal Dynamic Division strategy(MDD), which adaptively captures the valuable noise-free semantic information in different modalities and constructs a multimodal loss distribution based on the cross-modal features. Subsequently, the credibility of each sample is adaptively modeled based on the multimodal loss distribution. With the credibility of each sample, we dynamically categorize the samples into clean and noisy sets. Then, we adopt a Adaptive Alignment and Correction strategy(AAC) to conquer the samples in different subsets. Specifically, the samples in the clean set and noisy set are utilized for semantic and instance alignment respectively to eliminate the cross-modality gap while alleviating over-fitting to noisy labels. In addition, for the samples in the noisy set, a self-correction strategy is introduced to correct their corrupted labels with the model's multimodal predictions, which could further enhance the discrimination of representation. Moreover, we introduce a challenging realistic noisy benchmark: Objaverse-N200, which comprises 200klevel samples annotated with 1156 realistic noisy labels to evaluate the generalization ability of our model in real-world scenarios. In general, our contributions can be summarized as follows:• We propose a novel and robust framework for 2D-3D crossmodal retrieval with noisy labels, namely DAC, which performs Divide-and-Conquer alignment for different noisy samples based on the dynamic-estimated credibility of each sample and adaptively utilize the supervisory information within different subsets. and relationships within multimodal data. Methods such as Canonical Correlation Analysis (CCA) [2, 21, 36, 42, 45] and Multi-View Learning [23, 29] were widely used to learn common representations across modalities. In addition, [20] adopts Multimodal Contrastive loss (MC) to maximize the mutual information between different modalities. On the other hand, supervised-based approaches leverage labeled data to explicitly learn the semantic correlations between modalities. [43] proposes to learn modality-invariant representations by both label and common representation supervision. Moreover, [22] utilize center loss to joint train the components of the cross-modal framework to find optimal features. However, most of the above works rely on well-labeled data, which cannot directly be transferred to noisy 2D-3D scenarios. 
 
 Appendix:  


Decision: All four reviewers agree that this paper introduces a new benchmark, Objaverse-N200, and clearly articulates its merit and originality. The paper is well-written, and the experiments are relatively sufficient, effectively demonstrating the method's effectiveness. Based on these points, I recommend acceptance.

Reviews: review: paper_summary: paper_summary: This paper addresses the challenges in cross-modal retrieval of 2D and 3D data, specifically focusing on performance degradation due to corrupted annotations from non-expert labeling. Existing methods using naive division strategies and hand-crafted thresholds show poor robustness and fail to fully utilize supervisory signals. The authors propose the Divide-and-Conquer 2D-3D cross-modal Alignment and Correction framework (DAC), which includes Multimodal Dynamic Division (MDD) for accurate sample division and Adaptive Alignment and Correction (AAC) for enhancing semantic compactness and mitigating overfitting. DAC also introduces a self-correction strategy to improve representation quality. To test their approach, the authors introduce a new benchmark, Objaverse-N200. Extensive experiments show that DAC significantly outperforms state-of-the-art models.
strength_weakness: strength_weakness: 1. The proposed MDD and AAC methods make sense and appear to be effective.

2. The experiments are comprehensive, effectively showing the superiority of the proposed method.
limitations: limitations: 1 The discussion on methods related to 2D-3D retrieval is missing in the related work section, which is crucial for the completeness of the paper. If it is mentioned in the section, it should be explicitly highlighted.

2 There are several areas that can be improved. For instance, punctuation is needed after formulas, there is an extra punctuation mark in the title of Section 3.3, the title of Table 4 lacks punctuation, and the notation "𝑀/𝐾" on line 327 looks like division.

3 I am still unclear about the necessity of using GMM. It would be helpful to provide more explanation on this aspect.
paper_summary: paper_summary: Past approaches with fixed thresholds for data division show limited robustness. 
The Divide-and-Conquer 2D-3D Alignment and Correction (DAC) framework introduces Multimodal Dynamic Division (MDD) for adaptive sample division based on loss compensation, and Adaptive Alignment and Correction (AAC) to refine representations via subset-specific strategies and self-correction, combating noisy labels. 
To validate real-world performance, the authors present Objaverse-N200, a benchmark with 200,000 samples and 1156 noisy labels, where DAC demonstrates remarkable superiority, outperforming current models significantly.
strength_weakness: strength_weakness: The author's ideas are simple and easy to understand
The writing is clear.
The experiments are relatively sufficient and prove the effectiveness of the method.
limitations: limitations: There are several issues that need to be addressed below.

1. In line 493, why can the multimodal classifier correct wrong labels? If the multimodal classifier does not classify well enough, does it correct the error? Is the multimodal classifier actually a teacher model?
2. Can the Gaussian component be set to T=3 or T=4?
3. This algorithm requires 10 iterations to update the GMM parameters. Will this cause the training and testing time of DAC to be higher than that of RONO?
4. For pseudo-label generation technology, more related work still needs to be discussed. For example [1][2].

[1] DistillHash: Unsupervised Deep Hashing by Distilling Data Pairs
[2] Dual-Stream Knowledge-Preserving Hashing for Unsupervised Video Retrieval
paper_summary: paper_summary: This paper investigates the problem of 2D-3D retrieval with noisy labels and provide a divide-and-conquer alignment and correction for this problem.
strength_weakness: strength_weakness: 1. The motivation is clear.
2. The methodology is reasonable.
3. The experiments are thorough.
limitations: limitations: The divide-and-conquer strategy resembles some existing works. Please provide a discussion with them.
paper_summary: paper_summary: The paper studies cross-modal retrieval with noisy label. Compared to previous approach, the main contribution is to adopt the posterior probability of GMM to judge the real or fake sample label, which is called as Divide-and-Conquer alignment for different noisy samples.

Extensive experiments on both traditional and the newly proposed benchmarks demonstrate the effectiveness.
strength_weakness: strength_weakness: 1. The paper introduces the posterior probability of GMM to divide the samle into two sets, which is S_c and S_n.
2. Experiments is ok for me, especially on the introduced challenging realistic noisy benchmark: Objaverse-N200.
limitations: limitations: 1. The upper and lower subscripts in the main text of the paper and the upper and lower subscripts in the figure are represented differently, which can easily cause confusion during reading.
2. The S in formula 8 is unclear.
3. The fusion layer is learnable or not? From experiment, it seems only adopting Concat, which is better than attention and so on?
4. The process of using GMM model to determine the authenticity of labels is something like hard assignment, why dont consider about soft assignment?