# Multi-Modal 3D Shape Clustering with Dual Contrastive Learning
## 3 The Proposed Method
### Representation-Level Dual Contrastive Learning  
For each 3D shape, different views within the multi-view modality and the corresponding point cloud share consistent semantics while containing complementary 3D shape information across both views and modalities. Simultaneously performing cross-view contrastive learning within multi-view modality and cross-modal contrastive learning between point cloud and multi-view modalities for consistent representation is conducive to capturing more discriminative 3D shape information from a comprehensive understanding of 3D shapes. To this end, an RDCL module that adopts both cross-view and cross-modal contrastive representation learning was developed to capture discriminative 3D shape features for clustering. Figure 1: The architecture of the proposed DCL-Net for multi-modal 3D shape clustering. First, for the \(i\)-th 3D shape in a given mini-batch \(D=\{I_{i}^{v},P_{i}\}_{i=1...n}^{v=1...V}\) of size \(n\), two views \(\{I_{i}^{v_{1}},I_{i}^{v_{2}}\}\) from different angles in the multiple views and the corresponding point cloud \(P_{i}\) are selected as the inputs for the network. Among them, \(I_{i}^{v_{1}}\) and \(I_{i}^{v_{2}}\) are arbitrarily chosen from the \(V\) views, thereby providing more cross-view and cross-modal combinations for the network learning. After that, to capture latent 3D shape features from the inputs \(\{I_{i}^{v_{1}},I_{i}^{v_{2}},P_{i}\}\), a three-branch multi-modal feature extractor including a shared view encoder \(f_{\theta_{1}}(\cdot)\) and a point cloud encoder \(f_{\theta_{p}}(\cdot)\) is employed. The shared view encoder maps \(I_{i}^{v_{1}}\) and \(I_{i}^{v_{2}}\) into the view features \(F_{i}^{v_{1}}\) and \(F_{i}^{v_{2}}\) respectively, while the point cloud encoder is responsible for mapping \(P_{i}\) into the point cloud feature \(F_{i}^{p}\). The mapping processes are calculated as follows:  
\[F_{i}^{v_{1}}=f_{\theta_{1}}(I_{i}^{v_{1}}), \tag{1}\]  
\[F_{i}^{v_{2}}=f_{\theta_{1}}(I_{i}^{v_{2}}), \tag{2}\]  
\[F_{i}^{p}=f_{\theta_{p}}(P_{i}), \tag{3}\]  
where \(\theta_{l}\) and \(\theta_{P}\) denote the parameters of the view encoder and the point cloud encoder, respectively. Afterward, to effectively ensure the discrimination of the learned 3D shape features, a representation projection head \(g_{\phi}(\cdot)\) is adopted in the RDCL module to further project the learned features into the representation space:  
\[Z_{i}^{v_{1}}=g_{\phi}(F_{i}^{v_{1}}), \tag{4}\]  
\[Z_{i}^{v_{2}}=g_{\phi}(F_{i}^{v_{2}}), \tag{5}\]  
\[Z_{i}^{p}=g_{\phi}(F_{i}^{p}). \tag{6}\]  
Considering the semantic consistency among different views and the corresponding point cloud, multiple view representations and the point cloud representation of the same 3D shape should maintain higher similarities than those of different 3D shapes. Therefore, the view representations and point cloud representation of the same 3D shape need to be taken as positives to be pulled together, while those of different 3D shapes need to be regarded as negatives to be pushed apart. In view of this, a representation-level cross-view contrastive loss \(L_{RCV}\) and a representation-level cross-modal contrastive loss \(L_{RCM}\) are simultaneously applied to the representation space to ensure both the cross-view representation consistency within the multi-view modality and the cross-modal representation consistency between the point cloud and multi-view modalities. Specifically, the representation-level cross-view contrastive loss for the input view \(I_{i}^{v_{1}}\) is calculated as follows:  
\[L_{RCV_{i}}^{v_{1}}=-\log\frac{\exp(s(Z_{i}^{v_{1}},Z_{i}^{v_{2}})/\tau_{R})} {\sum\limits_{j=1}^{n}[\exp(s(Z_{i}^{v_{1}},Z_{j}^{v_{1}})/\tau_{R})+\exp(s(Z_{ i}^{v_{1}},Z_{j}^{v_{2}})/\tau_{R})]}, \tag{7}\]  
where \(\tau_{R}\) is the representation-level temperature parameter and is generally set to 0.5.