# Multi-Modal 3D Shape Clustering with Dual Contrastive Learning
## 3 The Proposed Method
### Assignment-Level Dual Contrastive Learning  
The ADCL module was designed to simultaneously ensure cross-view clustering assignment consistency within the multi-view modality and cross-modal clustering assignment consistency between the point cloud and multi-view modalities, thus further boosting the compactness of the learned 3D shape features for clustering. Specifically, for the extracted view and point cloud features \(F_{i}^{v_{1}}\), \(F_{i}^{v_{2}}\), and \(F_{i}^{P}\) of the \(i\)-th 3D shape, the ADCL module further maps them into soft labels using an assignment projection head \(h_{\psi}(\cdot)\) with the following process:  
\[Y_{i}^{v_{1}}=h_{\psi}(F_{i}^{v_{1}}), \tag{12}\]  
\[Y_{i}^{v_{2}}=h_{\psi}(F_{i}^{v_{2}}), \tag{13}\]  
\[Y_{i}^{p}=h_{\psi}(F_{i}^{p}). \tag{14}\]  
Let \(Y^{u}=[Y_{1}^{u},Y_{2}^{u},\dots,Y_{n}^{u}]\) denote the outputs of the assignment projection head for the \(n\) 3D shapes in the mini-batch and \(G^{u}\) denote the transposed matrix of \(Y^{u}\), where \(u\in\{v_{1},v_{2},p\}\). In \(G^{u}\), the \(i\)-th column vector denotes the soft label of the \(i\)-th 3D shape and the \(k\)-th row vector denotes the clustering assignment distribution of the cluster \(k\). Considering that both different views within the multi-view modality and the corresponding point cloud of the same 3D shape contain consistent semantics, the obtained clustering assignment distributions should be similar within the multi-view modality and between the point cloud and multi-view modalities. Namely, the index of different views and the corresponding point cloud that are assigned to a particular cluster should be consistent. To this end, an assignment-level cross-view contrastive loss \(L_{ACV}\) and an assignment-level cross-modal contrastive loss \(L_{ACM}\) are simultaneously applied to the clustering assignments. The assignment-level cross-view contrastive loss for the \(G_{k}^{v_{1}}\) is calculated as follows:  
\[L_{ACV_{k}}^{v_{1}}=-\log\frac{\exp(s(G_{k}^{v_{1}},G_{k}^{v_{2}})/\tau_{A})}{ \sum\limits_{j=1}^{c}[\exp(s(G_{k}^{v_{1}},G_{k}^{v_{1}})/\tau_{A})+\exp(s(G_{k }^{v_{1}},G_{l}^{v_{2}})/\tau_{A})]}, \tag{15}\]where \(\tau_{A}\) denotes the assignment-level temperature parameter and is generally set to 1.0. \(G_{k}^{p_{1}}\) and \(G_{l}^{p_{1}}\) are the \(k\)-th and \(l\)-th row vectors of \(G^{p_{1}}\), respectively. Then, the \(L_{ACV}\) is calculated by computing the assignment-level cross-view contrastive loss for each cluster:  
\[L_{ACV}=\frac{1}{2c}\sum\limits_{k=1}^{c}(L_{ACV_{k}}^{p_{1}}+L_{ACV_{k}}^{p_{2 }}), \tag{16}\]  
where \(c\) denotes the number of clusters. Similar to the representation-level cross-modal contrastive loss, the \(L_{ACM_{k}}\) is also obtained by computing the \(L_{ACM1_{k}}\) between \(G_{k}^{p}\) and \(G_{k}^{p_{1}}\) and the \(L_{ACM2_{k}}\) between \(G_{k}^{p}\) and \(G_{k}^{p_{2}}\). In this way, the bias of the consistent clustering assignments toward a particular view is effectively removed, thus further boosting the robustness of the clustering assignments. The assignment-level cross-modal contrastive loss for the \(G_{k}^{p_{1}}\) is calculated as follows:  
\[L_{ACM1_{k}}^{p_{1}}=-\log\frac{\exp(s(G_{k}^{p_{1}},G_{k}^{p})/\tau_{A})}{ \sum\limits_{l=1}^{c}\left[\exp(s(G_{k}^{p_{1}},G_{l}^{p_{1}})/\tau_{A})+\exp (s(G_{k}^{p_{1}},G_{l}^{p})/\tau_{A})\right]}. \tag{17}\]  
Then, the \(L_{ACM}\) is naturally expressed as:  
\[L_{ACM}=\frac{1}{2c}\sum\limits_{k=1}^{c}(L_{ACM1_{k}}^{p_{1}}+L_{ACM1_{k}}^{p _{1}}+L_{ACM2_{k}}^{p_{2}}+L_{ACM2_{k}}^{p}). \tag{18}\]  
Afterward, the overall assignment-level dual contrastive loss \(L_{ADCL}\) of the ADCL module is obtained by summing \(L_{ACV}\) and \(L_{ACM}\):  
\[L_{ADCL}=L_{ACV}+L_{ACM}. \tag{19}\]  
As can be seen from Equation (19), the \(L_{ADCL}\) simultaneously maximizes the cross-view and cross-modal assignment consistency of the same cluster, while minimizing that of different clusters.