SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathcal {L} _ {m c} = - \frac {1}{n c} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {c} \left(\mathbf {Y} _ {i, j} \log \left(\mathbf {P} _ {i, j}\right) + \left(1 - \mathbf {Y} _ {i, j}\right) \log \left(1 - \mathbf {P} _ {i, j}\right)\right) \mathbf {G} _ {i, j}, \tag {10}
$$
where  $\mathbf{G}$  is introduced to mask the unknown label in the calculation of  $\mathcal{L}_{mc}$ .
Combined with cross-channel contrastive loss Eq. (1), reconstructed loss Eq. (2), graph embedding loss Eq. (7), and weighted multi-label classification loss Eq. (10), our total loss function can be expressed as:
$$
\mathcal {L} _ {\text {a l l}} = \mathcal {L} _ {\text {m c}} + \alpha \mathcal {L} _ {\text {g c}} + \beta \mathcal {L} _ {\text {c c c}} + \gamma \mathcal {L} _ {\text {r e}}, \tag {11}
$$
where  $\alpha, \beta$ , and  $\gamma$  are corresponding penalty parameters. We show the detailed training process in algorithm 1.
3.1 Datasets and metrics
Consistent with existing iMvWMLC methods [26, 25, 20], we adopt five famous multi-view multi-label datasets² to validate our model, i.e., Corel5k [32], Pascal07 [33], ESPGame [34], IAPRTC12 [35], and MIRFLICKR [36]. Six kinds of features are extracted from these datasets as their six views, namely GIST, HSV, Hue, Sift, RGB, and LAB feature. The number of samples in these datasets ranges from 4999 to 25000, and the number of categories ranges from 20 to 291. Please refer to the appendix for more detailed information about the five datasets.
Following plenty of existing works [26, 25, 23], six metrics are selected to evaluate all comparison methods, i.e., Ranking Loss (RL), Average Precision (AP), Hamming Loss (HL), adapted area under curve (AUC), OneError (OE), and Coverage (Cov). To compare performance more intuitively, we actually report results with respect to 1-HL, 1-RL, 1-OE, and 1-Cov, so for all metrics, higher values indicate better performance.
<sup>2</sup>http://lear.inrialpes.fr/people/guillaumin/data.php
(a) Corel5k
(b) Pascal07
(c) ESPGame
(d) IAPRTC12
Figure 2: Experimental results of nine methods on the five full databases without any missing views or labels. The center of the radar map shows the worst results and the vertexes mean the best results on the six metrics.
(e) MIRFLICKR
3.2 Incomplete multi-view weak multi-label dataset setting
Analogous to previous works [26, 25, 20], we manually construct incomplete multi-view weak multi-label datasets based on above five complete datasets to evaluate the performance of all methods in the case of missing views and missing labels. Concretely,  $50\%$  of instances on each view are randomly selected as unavailable instances, which are replaced by '0' value. And we ensure that there is no invalid sample in the dataset, i.e., each sample holds at least one unmissing view. For the weak labels, we set half of the positive and negative tags for each category to be unknown.  $70\%$  of samples with missing views and missing labels are randomly selected as the training set. The construction of incomplete data is repeated many times to reduce the randomness of experiment.
3.3 Comparison methods
In order to evaluate the advancement of our framework, eight top comparison methods are selected in our experiment, i.e., C2AE [37], GLOCAL [38], CDMM [18], DM2L [39], LVSL [23], NAIM3L [25], iMVWL [26], and DICNet [20], to compare with our MTD on the five incomplete multi-view weak multi-label datasets. To date, few MvMLC methods can handle missing views and weak labels at the same time, thus we introduce some related multi-label classification methods in our experiments. To be more specific, of the nine methods, iMVWL, NAIM3L, DICNet, and our MTD are perfectly applicable to the iMvWMLC task. C2AE, GLOCAL, and DM2L are single-view multi-label classification methods that can deal with incomplete labels, so we conduct these methods on each view and record the best result. CDMM and LVSL are the MvMLC methods unable to handle any data missing, so we simply complete the missing views using the mean of available instances and fill the unknown tags with '0'. The statistical information of competitors is shown in the appendix.
3.4 Experimental results and analysis
To evaluate the performance of our method in scenarios with missing views and labels, we compare it against eight state-of-the-art algorithms on five datasets. The mean and standard deviation (decimal form in the bottom right) of the results are reported in Table 1. In addition to the six performance metrics mentioned above, we also calculate the average ranking of each algorithm based on these
(a) Epoch 0
(c) Epoch 120
(d) Epoch 180
Figure 3: A random sample's channel similarity heat maps across all channels on Corel5k dataset with half of missing views and labels. S_1- S_6 and O_1- O_6 denote shared features and view-proprietary features on six views, respectively. With the increase of training epoch, the similarities of features on shared and proprietary channels show the expected trend, that is, the shared features across views gradually converged, while the similarities of "shared-proprietary" and "proprietary-proprietary" feature pairs gradually decreased.
metrics, denoted as 'Ave.R', for intuitive comparison. Based on the results in Table 1, we make the following observations:
1. Our MTD is compelling, outperforming comparison methods on almost all metrics across all five datasets. In addition to the average ranking of 1.17 on the IAPRTC12 dataset, it
ranks 1st on the other four datasets. Whether compared with traditional methods (such as GLOCAL and NAIM3L) or deep methods (such as C2AE and DICNet), our MTD shows excellent compatibility with missing views and weak labels.
2. We can find that methods based on deep neural networks generally perform better than traditional methods. Specifically, our MTD, DICNet, and CDMM are consistently ranked in the top 3 across all five datasets, which can be attributed to the strong feature extraction capabilities of deep encoders and the flexible multi-view fusion strategy they employ.
3. In the comparison among traditional methods, the method that is specifically designed for double incompleteness shows obvious advantages compared to other methods. This indicates the necessity of considering possible missing views and labels during the model design process.
In addition to experimenting on datasets with  $50\%$  missing views and missing labels, we also perform comparison experiments on the full dataset without any missing data and use radar charts to show the results in Fig. (2). For more intuitive observation, we set the range for each axis to the min-max values of the nine methods on the corresponding metric. It can be seen from Fig. (2) that the performance of our method on the complete datasets is still outstanding, which verifies the good adaptation of it.
To demonstrate that our cross-channel contrastive loss works as expected, we plot a random sample's channel similarity heat maps across all channels in different training epochs on the Corel5k dataset with half of missing views and labels in Fig. (3). Each patch of the heat map shows the cosine similarity of corresponding two instances, i.e.,  $i$ -th sample's heat map  $\mathbf{H}_i = \mathbf{C}\mathbf{C}^T$ , where  $\mathbf{C} = (s_i^{(1)}, s_i^{(2)}, \dots, s_i^{(m)}, o_i^{(1)}, o_i^{(2)}, \dots, o_i^{(m)})^T \in \mathbb{R}^{2m \times d_e}$ .  $s_i^{(1)} \in \mathbb{R}^{d_e}$  and  $o_i^{(1)} \in \mathbb{R}^{d_e}$  are the instances on shared and view-proprietary channels of first view of sample  $i$ , respectively. In Fig. (3), it is evident that in the initial state, the similarities between different channels are relatively small. However, as training progresses, the similarities between instance pairs on the first  $m$  shared channels increase rapidly, while the similarities for the remaining channels are close to zero. This observation indicates that our cross-channel contrastive loss is functioning as expected. Additionally, we notice that there is a shared feature on channel 3 that is not well-aggregated. This may be due to the third view of this sample sharing less information with the other views.
3.5 Ablation study
Table 2: Ablation results of our MTD on Corel5k dataset with  $50\%$  missing views,  $50\%$  missing labels. 's_ch' denotes the conventional single-channel network and 'd_ch' denotes our double-channel framework. 'w/o' means 'without'.
Method	AP	1-HL	1-RL	AUC	1-OE	1-Cov
s_ch baseline	0.396	0.988	0.888	0.891	0.469	0.738
d_ch baseline	0.408	0.988	0.890	0.893	0.482	0.744
d_ch+Lgc	0.409	0.988	0.890	0.893	0.488	0.744
d_ch+Lre+Lgc	0.413	0.988	0.890	0.894	0.487	0.746
MTD	0.415	0.988	0.893	0.896	0.491	0.750
MTD w/o mask	0.397	0.987	0.888	0.891	0.466	0.738
s_ch w/o mask	0.372	0.987	0.881	0.884	0.449	0.721
d_ch w/o mask	0.387	0.987	0.885	0.889	0.452	0.732
Our model contains multiple loss functions and strategies, in order to further study the effectiveness of each component, we perform extensive ablation experiments on the Corel5k dataset with  $50\%$  missing labels and views, and report results in Table 2. Specifically, we take the common single-channel model [20, 40] as a baseline architecture and extend it to a two-channel baseline framework. We iteratively add our individual loss functions  $\mathcal{L}_{gc}$ ,  $\mathcal{L}_{re}$ , and  $\mathcal{L}_{ccc}$  into it and remove the masking operation from different versions. From Table 2, we observe that all designs play a role to varying degrees, especially the fragment masking mechanism's performance increase on the Corel5k dataset is the most noticeable. This confirms that the random masking mechanism to raw data, commonly used in the image and text analysis domain, is also effective in the field of multi-view learning. However, this strategy still needs to be further studied for the underlying mechanism of action.
3.6 Implementation details
Our TMD is implemented by Pytorch and MindSpore frameworks on Ubuntu operating system with a single RTX 3090 GPU and an i7-12900k CPU. The learning rate is set to 0.1 and the Stochastic Gradient Descent (SGD) optimizer is chosen for training model. The batch size and momentum are 128 and 0.9 for all five datasets. Please refer to the appendix for the setting of other hyperparameters. The code can be found at https://justsmart.github.io/.
In this paper, we propose a novel masked two-channel decoupling framework (MTD) for the iMvWMLC task. Our MTD decouples single-channel features commonly used in previous methods into two-channel features for consensus and complementarity learning, and we design a cross-channel contrastive loss for this setting. Additionally, we employ a random fragment masking strategy during the training phase to reduce redundancy of raw data, inspired by image and text masking strategies, which leads to an obvious performance boost. Furthermore, we introduce a label-guided graph constraint approach to ensure that the learned embedding features maintain structure information among samples. Our extensive comparison and ablation experiments demonstrate that MTD outperforms other advanced methods and is compatible with arbitrary incomplete multi-view and weak multi-label data. Looking back at our work, although we propose many strategies and techniques to advance the frontier research progress of iMvWMLC, the complexity of this issue leaves room for further optimization, such as considering multi-label correlation and missing data recovery as directions of future works.
This work is supported in part by the Shenzhen Higher Education Stability Support Program under Grant No. GXWD20220811173317002, in part by the Chinese Association for Artificial Intelligence (CAAI)-Huawei MindSpore Open Fund under Grant No. CAAIXSJLJJ-2022-011C.
[1] Yangbangyan Jiang, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Dm2c: deep mixed-modal clustering. In International Conference on Neural Information Processing Systems, pages 5888-5892, 2019.
[2] Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. Advances in neural information processing systems, 34:2148-2159, 2021.
[3] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 5177-5186, 2019.
[4] Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1134–1143, 2021.
[5] Jie Wen, Zheng Zhang, Lunke Fei, Bob Zhang, Yong Xu, Zhao Zhang, and Jinxing Li. A survey on incomplete multiview clustering. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022.
[6] Chengliang Liu, Zhihao Wu, Jie Wen, Yong Xu, and Chao Huang. Localized sparse incomplete multi-view clustering. IEEE Transactions on Multimedia, 2022.
[7] Xinwang Liu, Miaomiao Li, Chang Tang, Jingyuan Xia, Jian Xiong, Li Liu, Marius Kloft, and En Zhu. Efficient and effective regularized incomplete multi-view clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2634-2646, 2020.
[8] Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. Multi-modal cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia, 2022.
[26] Qiaoyu Tan, Guoxian Yu, Carlotta Domeniconi, Jun Wang, and Zili Zhang. Incomplete multiview weak-label learning. In International Joint Conference on Artificial Intelligence, pages 2703-2709, 2018.
[27] Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dólár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[28] Jie Wen, Zheng Zhang, Zhao Zhang, Zhihao Wu, Lunke Fei, Yong Xu, and Bob Zhang. Dimc-net: Deep incomplete multi-view clustering network. In ACM International Conference on Multimedia, pages 3753-3761, 2020.
[29] Peng Zhou, Liang Du, Yi-Dong Shen, and Xuejun Li. Tri-level robust clustering ensemble with multiple graph learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11125-11133, 2021.
[30] Peng Zhou, Liang Du, Xuejun Li, Yi-Dong Shen, and Yuhua Qian. Unsupervised feature selection with adaptive multiple graph learning. Pattern Recognition, 105:107375, 2020.
[31] Kun Zhan, Feiping Nie, Jing Wang, and Yi Yang. Multiview consensus graph clustering. IEEE Transactions on Image Processing, 28(3):1261-1270, 2018.
[32] Pinar Duygulu, Kobus Barnard, Joao FG de Freitas, and David A Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In European Conference on Computer Vision, pages 97-112, 2002.
[33] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The Pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-308, 2009.
[34] Luis Von Ahn and Laura Dabbish. Labeling images with a computer game. In SIGCHI Conference on Human Factors in Computing Systems, pages 319-326, 2004.
[35] M Henning, D Thomas, et al. The iapr benchmark: A new evaluation resource for visual information systems. In International Conference on Language Resources and Evaluation, pages 1-11, 2006.
[36] Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In ACM International Conference on Multimedia Information Retrieval, pages 39-43, 2008.
[37] Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In AAAI Conference on Artificial Intelligence, volume 31, 2017.
[38] Yue Zhu, James T Kwok, and Zhi-Hua Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering, 30(6):1081-1094, 2017.
[39] Zhongchen Ma and Songcan Chen. Expand globally, shrink locally: Discriminant multi-label learning with missing labels. Pattern Recognition, 111:107675, 2021.
[40] Jie Wen, Chengliang Liu, Shijie Deng, Yicheng Liu, Lunke Fei, Ke Yan, and Yong Xu. Deep double incomplete multi-view multi-label learning with incomplete labels and missing views. IEEE transactions on neural networks and learning systems, PP, 2023.
Table 3: Detailed statistics about five multi-view multi-label databases.
Database	# Sample	# Category	# View	# Category/#Sample
Corel5k	4999	260	6	3.40
Pascal07	9963	20	6	1.47
ESPGame	20770	268	6	4.69
IAPRTC12	19627	291	6	5.72
MIRFLICKR	25000	38	6	4.72
These datasets are uniformly extracted six kinds of feature as six views from five image datasets, i.e., GIST, HSV, Hue, Sift, RGB, and LAB. Table 3 shows the statistical information of the five datasets.
B Comparison Methods
Table 4: Information of comparison multi-label classification methods. 'Multi-view' means the method is designed for multi-view data; 'Missing-view' and 'Missing-label' represent the corresponding method is able to handle incomplete views and weak labels.
Method	Sources	Multi-view	Missing-view	Missing-label
C2AE[37]	AAAI '17	X	X	✓
GLOCAL[38]	TKDE '17	X	X	✓
CDMM[18]	KBS '20	✓	X	X
DM2L[39]	PR '21	X	X	✓
LVSL[23]	TMM '22	✓	X	X
iMVWL[26]	IJCAI '18	✓	✓	✓
NAIM3L[25]	TPAMI '22	✓	✓	✓
DICNet[20]	AAAI '23	✓	✓	✓
C Hyperparameters study
The overall loss of our model mainly contains 3 hyperparameters, i.e.,  $\alpha$ ,  $\beta$ , and  $\gamma$ . In order to study the impact of these three hyperparameters on the performance of the model, we iteratively set different values and report the corresponding AP values in Fig. 4:
In Fig. 4, we show the AP value with respect to different parameter selections on Corel5k and Pascal07 datasets with  $50\%$  missing views and labels. Specifically, from Fig. 4a and 4b, we recommend selecting  $\alpha$  from the ranges of  $[1e - 4,1e - 3]$  and  $[1e - 4,1e0]$  for Corel5k and Pascal07 datasets respectively; selecting  $\beta$  from the ranges of  $[1e - 3,1e - 1]$  and  $[1e - 3,1e - 1]$  for Corel5k and Pascal07 datasets respectively. For convenience, we uniformly set  $\alpha = 0.4$  and  $\beta = 0.4$  in our experiments. For  $\gamma$ , our method is not sensitive to it, so we set  $\gamma = 0.1$  for all five datasets.
From Fig. 4d and Fig. 4e, it can be seen that too large or too small mask rate is not conducive to achieving optimal performance. Based on experiments, we recommend setting a mask rate from 0.2 to 0.4 (0.25 is used for all datasets in our experiments for convenience).
D Time cost
To study the training and test efficiency of our MTD, we report the time cost of the training and test phases of the six supervised methods on the Corel5k dataset with  $70\%$  training samples in Table 5. The convergence setting of the model plays a critical role in determining the training time required. Therefore, we measure the running times of all methods under their default convergence conditions.
(c) Corel5k and Pascal07
(d) Corel5k
(e) Pascal07
Figure 4: AP value v.s. hyperparameters  $\alpha$  and  $\beta$  on the (a) Corel5k and (b) Pascal07 datasets; AP value v.s. hyperparameter  $\gamma$  on the (c) Corel5k and Pascal07 datasets; AP value v.s. hyperparameter  $\sigma$  on the (d) Corel5k and (e) Pascal07 datasets. The two datasets are with  $50\%$  missing-label rate,  $50\%$  missing-view rate, and  $70\%$  training samples.
Table 5: Time efficiency of training and testing phases of the nine methods on the Corel5k dataset with  $70\%$  training samples. (Unit: second)
Phase\Method	C2AE	GLOCAL	CDMM	DM2L	LVSL	iMVWL	NAIML	DICNet	MTD
Training	170.24	154.42	16.02	713.37	63.73	165.82	143.63	313.89	687.52
Test	0.04	0.89	1.73	0.04	0.64	0.02	0.01	0.05	0.15
For methods that process a single view at a time, we report both the total training time for all views and the test time for a single view. For DICNet and MTD, we report the time spent in 100 epochs. In order to ensure consistent hardware conditions, all tests are carried out on the same computer. From the results shown in Table 5, it can be observed that our deep learning-based method requires longer training times, but reaches higher performance than other methods.