Masked Two-channel Decoupling Framework for Incomplete Multi-view Weak Multi-label Learning
Authors: Chengliang Liu
Jie Wen
Yabo Liu
Chao Huang
Zhihao Wu
Xiaoling Luo
Yong Xu
Venue: NeurIPS 2023 poster (2023)
Keywords: Incomplete Multi-view Weak Multi-label Learning
Multi-view learning
Multi-label Classification
TL;DR: A Framework for Incomplete Multi-View Weak Multi-label Classification with Masking Mechanism

Abstract: Multi-view learning has become a popular research topic in recent years, but research on the cross-application of classic multi-label classification and multi-view learning is still in its early stages. In this paper, we focus on the complex yet highly realistic task of incomplete multi-view weak multi-label learning and propose a masked two-channel decoupling framework based on deep neural networks to solve this problem. The core innovation of our method lies in decoupling the single-channel view-level representation, which is common in deep multi-view learning methods, into a shared representation and a view-proprietary representation. We also design a cross-channel contrastive loss to enhance the semantic property of the two channels. Additionally, we exploit supervised information to design a label-guided graph regularization loss, helping the extracted embedding features preserve the geometric structure among samples. Inspired by the success of masking mechanisms in image and text analysis, we develop a random fragment masking strategy for vector features to improve the learning ability of encoders. Finally, it is important to emphasize that our model is fully adaptable to arbitrary view and label absences while also performing well on the ideal full data. We have conducted sufficient and convincing experiments to confirm the effectiveness and advancement of our model.

Masked Two-channel Decoupling Framework for Incomplete Multi-view Weak Multi-label Learning
Chengliang Liu', .ie ; Yabo Liu', Chao Huang", Zhihao Wu', Xiaoling Luo", Yong Xu
School of Computer Science and Technology, Harbin Institute of Technology, Shenzhen 2 School Cyher Science and Technology. Sun Yat-sen University (Shenzhen Campus) Coolleee of Computer Science and Software Engineering. Shenzhen University (1iuc11996, horatio (oratio_ng)@1111 mg)ee1......... com, [jiewen_pr) huangchao_ @126. com, yaboliu. ug@gmail. com, xiaolingluooBouttlokk com, yongxu@ymail com
Abstract
Multi-view learning has become popular research topic in recent years, but research on the cross-application of classic multi-label classification and multi-view learning is still in its early stages. In this paper, we focus on the complex yet highly realistic task of incomplete multi-view weak multi-label learning and propose a masked two-channel decoupling framework based on deep neural networks solve this problem. The core innovation of our method lies in decoupling the single- channel view-level representation, which is common in deep multi-view learning methods, into shared representation and a also design cross-channel contrastive loss enhance the semantic property of the two channels. Additionally, we exploit supervised information to design label- guided graph regularization loss, helping the extracted embedding features preserve the geometric structure among samples. Inspired by the success of masking mechanisms in image and text analysis, we develop a random fragment masking strategy for vector features to improve the learning ability of encoders. Finally, it is important to emphasize that our model is fully adaptable to arbitrary view and label absences while also performing well on the ideal full data. We have conducted sufficient and convincing experiments to confirm the effectiveness and advancement of model.
representation. We
Introduction
With the increasing popularity of multi-view learning, researchers have discovered that heterogeneous multi-source features obtained from multiple sensors or techniques can provide a richer and more diverse description samples [T-4). For example, features can be extracted using ieale-invariant feature transform (SIFT) operator and deep neural networks from different perspectives of an image, or features can be extracted from multimedia data, such as images and texts, express higher-level semantic objects 5 Many multi-view learning methods are based on the assumptions of multi- view semantic consistency and feature complementarity g They aim to obtain unique sample labels from multi-source features to maintain the unity of multiple views at the semantic level and to leverage the unique advantages of each view to compensate for limitations of single-view observation [10-12] For instance, Nie et al. emphasized the importance of learning a cross-view similarity matrix in their work [13]. Li et proposed to learn a common graph in the spectral embedding space for final clustering [14]. Han et al. attempted to compute confidence scores of samples on each view to facilitate classification predictions across multiple views [15]. Lin et
"Corresponding author
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

al. proposed dual contrastive prediction for incomplete multi-view representation learning, which performs contrastive learning at both the instance level and category level to ensure cross-view consistency and recover missing information [66.. However, the instance-level contrastive loss of this method only aggregates cross-view representations in the potential space from the perspective of consistency, ignoring the view-private information.
On the other hand, as a classical classification problem, multi-label classification has occupied an important area of pattern recognition for a long time [7.. However, under the upsurge of multi-view learning, new composite problem, i.e., multi- -iew multi-label classification (MvMLCC, are attracting increasing attentions from researchers [18 19I. As the name suggests, MvMLC can be seen as a supervised subtask within the scope of multi view learning and can also be considered as an extension of traditional augle-view multi-label classification with respect to feature diversity. As a result, existing MvMI.C methods not only take into account the basic characteristics of multi-view learning
but also consider the requirements of downstream multi-label classification 200 18 21. For example, Zhao et al. proposed a framework called CDMM that mines multi-view consistency and diversity information, which uses the -chhmid Independence Criterion [22] to enhance the diversity of each view and obtain consistent classification results through late fusion [8]. Another advanced non-aligned MvMLC model proposed by Zhao et al. is LVSL (learn view-specific labels), which not only considers the local geometric structure in the original feature space but also introduces a
learnable label correlation matrix with low-rank structure [23]
However, these ideal MvMLC methods ignore the possibility of missing multi-view features and labels, making it difficult to achieve good performance on incomplete multi-view weak multi-label data, in which sample's ' partial views and tags are unknown [2 25). In recent years, researchers have proposed some incomplete multi-view weak multi-label classification (iMvWMLC) frameworks to address the adverse effects caused by missing views and labels. For instance, Tan et al. proposed an incomplete multi-view weak label learning model (iMVWL) that decomposes multi-view features into a latent common representation and multiple view-specific basis matrices, and connects the latent representation with labels through mapping matrix (26). Additionally, two prior missing indicator matrices are introduced to skip the missing views and labels. Recently, Li et al. developed a non-aligned iMvWMLC model based on traditional matrix factorization, named NAIM3L, to align multiple views in the label space and impose global high-rank and local low-rank constraints on the problem, Li et al. introduced composite indicator matrix for NAIM3L, containing view and label missing information. Another representative deep iMvWMLC framework is DICNet proposed by Liu et al., which applies the contrastive learning technique to aggregate cross-view instances of the same sample and separate instances belonging to different samples (20) Although DICNet has achieved remarkable improvement, there is non- negligible category collision problem, i.e., instances belonging to similar samples will be forcibly separated due to this contrastive learning mechanism.
predicted multi-label matrix 225 For the double
To tackle these issues, we present the Masked Two-channel Decoupling framework (MTD for short), capable of handling cases where partial views and labels are both missing. The motivation behind this framework is that we both want the underlying representations of the same sample in multiple views to be consistent, while also wanting each view to display its own characteristics, creating difficult balancing act [00]. This is especially challenging when each view of the same sample is represented by only one feature vector. To address this, we explicitly decouple each view's latent feature into two types of features, i.e., shared feature and view-proprietary feature, via two-channel encoders. We also propose novel cross-channel contrastive loss that narrows the distance between positive pairs (shared instances in different views of the same sample) and expands the distance between negative pairs (any cross-view and channel instance pairs of a sample, except positive pairs). Note that our cross-channel contrastive loss does not involve the comparison among samples, so it can well avoid the above class conflict problem and improve the framework's feature decoupling ability. Additionally, we draw inspiration from famous image patch masks 2 and introduce random mask fragments in the input data to guide the encoder's learning of valuable information. Lastly, we design a weak label-gue -geieed graph regularization loss to preserve the geometric structure in the embedding space, accounting for the non-equidistance among samples in label space. In summary, our paper makes three key contributions:
1. We propose novel masked two-channel decoupling framework (MTD) for the WMvWML.C task, which extracts shared and private features for each view separately and enforces this
2

---

contrastive learning
D
ATT
E
predictions
Lactivate
E2
E2
classifier
5
0.2
0.1
0.7
Dm
Fm
Em
shared channel
masked input
proprietary channel
x'
299
random mask
label
feature
feature reconstruction
masking random fragment
label-guide graph regularization
Figure I: Main architecture of our MTD. Masked data (x) Jm is real input of the MTD. And the cross-channel contrastive loss aims to enhance the semantic 'shared-proprietary" channels.
design with cross cross-channel contrastive loss. At the same time, this framework is compatible with arbitrary incomplete multi-view and weak multi-labed data.
2. To our best knowledge, we are the first to apply random fragment mask the field of multi- view learning and achieve significant performance gains, which supports a new multi-view vector data enhancement mechanism for the communication.
3. Different from existing graph regularization based approaches, utilize supervised label information to build a more reliable topological graph, inducing the embedding features extracted by the encoders preserve the geometric structure among samples.
Method
In this section, comprehensively introduce model from the following four aspects, namely, two- channel decoupling framework, weak label-guided graph regularization, random fragment masking mechanism, and multi-label classification. First, for the convenience of description, we briefly introduce the formal problem definition and common notations.
2.1 Problem definition
In this section, we define the iMvWMLC task as follows: Given multi-view multi-label data ( {[X(+) Jv=1 ()()) where (1) is v-th view's feature matrix containing samples and is the number of views. The i-th row of (() Ruxd, denotes the instance of i-th sample e-th view with dimension de Y e {0, 1]nxe is the label matrix with e categories and Y = denotes the i-th sample is tagged j-th class. For missing views, introduce W E {0. 10
the missing-view index matrix, where Wi, = represent j-th view of i-th sample is available, otherwise W., 0. For missing labels, we let G (0, 1) be the missing- label index matrix, whose element G 1 means i-th sample's j-th label is known, otherwise G 0. Finally, we fill the unavailable views and the unknown labels with '0' in the data pre-processing step get final incomplete multi-view feature ( === and weak label matrix Y. Our goal is to train neural network that can perform multi-label classification reasoning on incomplete multi-view data. Subscript access B B and B.j mean the element, row, and column of any matrix B.
2.2 Two-channel decoupling framework
As we know, due to the diversity of multi-view data acquisition methods, each view original data may have different feature dimensions, which is not conducive to the parallel execution of deep networks. this end, we map the heterogeneous raw data into a unified embedding space with dimension de through stacked encoders. Different from conventional deep multi-view networks,

---

employ two groups of multi- perceptrons as two channels, shared channel and view-privan channel, to extract shared information and proppiettyy information from raw data, respectively, i.c. {E X'()) + ( J==1 and {E? x + O J==1. E and Eς denote the shared feature encoder and view-proprietary feature encoder of view u, respectively. X represents the masked input (See Section 2.4 ( RTxd, and O
(v) Rnxd= are the extracted multi-view shared feature matrix and view-proprietan feature matrix of n samples. The architectures of these two channels the same but serve different purposes. Specifically, E tries mine common features across views, which preserve the sample's basic properties held by all views, while E focuses on extracting features unique each view. This design decomposes the discriminative information of each view into two parts, which can satisfy the multi-view consistency and complementarity assumptions at the same time. Of course, it is difficult achieve the above purposes with only two encoding channels due to the lack of guidance, so we propose multi-view cross cross-channel contrastive loss Lece to guide the separation of these two kinds of features as follows:
2
Σ
[m (s. .0 101)²+++ Σ Σ
(1)
1 1m (o, (o,") of
3N
Lece
#= Usu
(I)
Σ 1 |m (8 (0,")) ,((" + 1)/2
where S is the similarity measure function: S(z, y) | and |m is the condition function: 1m = 1 if condition {T: W..WWs, 1} is true, 0 otherwise. N = En. W... Wi denotes the number of valid instance pairs. That is to say we only consider instance pairs that are not missing on both sides. a!") and o/") are features corresponding to i-th sample in the shared feature ((r) and new-proprietary feature O(+), respectively. According to Eq. (L). our cross- cross-channel contrastive loss consists of two parts, namely, the numerator is the mean of negative pairs' similarities and the denominator is that of positive pairs' similarities. To be more specific, we pair instances from 2N channels together, where positive pairs consist of shared features from different views, and the remaining pairs are considered negative. We aim to maximize the similarity between the positive pairs of shared features and minimize the similarity between the negative pairs. This approach fulfills the design objective of the dual-channel model, which is to encourage consistency between shared features from different views while maintaining clear distinction between each view's 's view- nee-prropiettary feature and other shared or view nee-proopy features.
To further enhance the feature extraction capability of the encoders, we introduce stacked decoders to restore the embedding features extracted by the two-channel encoders original feature space, i.e., (D, ((()) E Ruxd. + (()) E Raxd, )m J==I, where D, is the decoder corresponding v-th view. x(") is the reconstructed feature. Finally, measure the reconstruction quality using the weighted mean square error loss:
m
n
L
X (0) 2
W i,.
where W is introduced to mask the unavailable instances. With the (( and (o) on each view, can naturally perform cross-view fusion to obtain the unique shared representation and Ô of all samples 28 20
m
W. L.
S.
5
W 1.
O
(3)
W
v=1
v=1
Successively, we try to combine shared information and proprietary information to learn consistent representations of samples. Here, unlike the commonly used addition or connection operations, we adopt a novel feature interaction approach, i.c., enhancing the shared information via the proprietary information:
Zzj = ((Ō.g) Si,5:
where 0 sigmoid activation function and Z E Rud is the final fusion representation.
2.3 Weak label-guided graph regularization
Many unsupervised multi-view learning methods apply graph regularization technique to maintain the intrinsic structure of data, based on the assumption that two similar samples in the original feature

---

space should also be similar in the latent space, which has shown good performance in unsupervised scenarios (2 BT. When transferring this assumption to multi-label classification tasks with the characteristics of supervised tasks, we can get new assumption: two similar samples in the label space should also be similar in the embedding space. Based on this, we attempt to guide the feature extraction process of the encoders by imposing graph constraint with the supervised information. First, we calculate similarity matrix according to the weak label matrix Y:
Ci. (U:D)) C2. (U.D)) +
T
(5)
where T E [0, _| is the sample similarity graph, describing the similarity between any two samples. Ct, G. is the number of labels available for both samples i and y4 and yj denote the i-th and j-th rows of Y. And v) is a constant, empirically set to 100 for simplicity. Taking into account the fact that Cej is much larger than y.y, most datasets, when the number of categories is large, even if two samples only have one same label, it means that they are much more similar than other sample pairs.
With such a similarity graph, we can constrain the distance between any two samples in the embedding feature space to achieve the purpose of structure preservation by following loss:
1
Lge
Zi:: Zz TT.
In view of Eq. (6). the more similar two samples are, the greater the penalty weight they are imposed. In order improve the computational efficiency of the model in the GPU, we rewrite Eq. ( in the form of matrix product:
1 LZ), Lge n²
(Z),
where Tr(-) is the trace operation. L denotes the Laplacian matrix calculated by L = D - T, where D is a diagonal matrix with diagonal element D,.; Σj= Tuj-
2.4 Masking random fragment of feature
As we all know, the original data contains massive redundant information, which inevitably interferes with the feature extraction process of the model. Inspired by the well-known masked autoencoders (MAE) [27] that randomly mask image patches, we attempt to apply masking mechanism to the input mueti-view vector data. To this end, for any view v, we initialize matrix M(²) E {0, 1 whose elements are all at the same size as the raw data X("), and randomly sample n integers b2.,,,..., bn] from the range of [1, dg - !] as the mask starting point of each instance in view v, where / is the mask length. Then the 6,-th to (b, 1)-th bits of feature M[ dR are set to 0 to get the final mask matrix M()), This matrix stores consecutive mask segments starting at random, enabling local dcactivation of feature vectors. We use m mask matrices to mask the original input data, and get corresponding masked feature matrices as the input of the MTD:
(X 1 (x(") (t) ( pm
Je=1
(8)
where O means the element-wise multiplication.
At first glance, this is similar to MAE masking random image patches to encourage the model to learn useful features. The difference is that our MTD only partially masks the input features but does not do anything to the encoders, i.c., the encoders still works as if processing the unmasked data. In addition, in our model, we uniformly set the mask rate o l/d, 0.25 for all datasets instead of larger masking rate like MAE, because we notice the difference that the information density of vector data is much larger than that of image data. Extensive experiments are performed in Section 3.5 to verify the effectiveness of this masking approach.
2.5 Weighted multi-label classification and overall loss function
With the fused embedding representation Z, a fully connected layer employed as classifier to map Z to the label space to get the prediction result:
P (ZW + +B),
(9)
5

---

Algorithm Training process of MTD
Input: Incomplete multi-view data (X m
v=11 missing-view index matrix W. weak label Y,
missing-label index matrix G.
Output: Prediction results P.
I: Initialize model parameters and set hyperparameters (a, 3, learning rate, and training epochs
e). 2: t=
3: while < do
4:
Construct random feature mask matrices {M
5: Compute masked input data (X'(v) jm by Eq. (8).
6: Extract shared embedding feature {S (v) u=1 and view-private embedding feature {0 (O(r)) (0())
by two-channel encoders {E, mm
( u=1 and {E, fmm11 respectively.
7:
Compute fused shared embedding feature S and fused eew-proprietary feature Ô by Eq.
8:
Compute final fused representation Z by Eq. (4).
9:
Compute similarity graph T by Eq. 5 and corresponding Laplacian matrix L.
10:
Compute cross-channel contrastive loss Leee by Eq. (T) reconstruction loss Lre by Eq.
and graph embedding loss Le by Eq. (
11:
Obtain predictions P by Eq. (9) and compute classification loss L'ne by Eq. Compute total loss Lontt by Eq.
13:
Update network parameters.
=
15: end while
where W E R2d, and B Ra learnable parameters of classifier and P E Rxxe is predicted score matrix. In single-label classification tasks, the cross-entropy loss usually adopted to guide the learning of model, while in multi-label classification tasks, the prediction of each category regarded as an independent binary classification problem. Besides, considering the unknown labels in label matrix,
following multi-label cross-entropy loss main classification loss:
1 ΣE
Lmc
(Y log (Pi.) + log(1 - -Pij))
(10)
TC 1=
where G is introduced to mask the unknown label in the calculation of Lmc-
Combined with cross-channel contrastive loss Eq. ( reconstructed loss Eq. (2), graph embedding loss Eq (7. and weighted multi-label classification loss Eq. (10), our total loss function can be expressed as:
Lall = Le + adge + f.Lec + Yarre
(11)
where B, and y corresponding penalty parameters. We show the detailed training process in algorithm[]
Experiments
3.1 Datasets and metrics
Consistent with existing iMyWMLC methods [26 25 20
adopt five famous multi-view multi-
label datasets to validate our model, i.e., Corel5k [32] Pasca107 [33] ESPGame (34), IAPRTCI2
[35) and MIRFLICKR [36). Six kinds of features are extracted from these datasets as their six views, namely GIST, HSV, Hue, Sift, RGB, and LAB feature. The number of samples in these
datasets ranges from 4999 25000, and the number of categories ranges from 20 291. Please refer to the appendix for more detailed information about the five datasets.
Following plenty of existing works 26 25 23
six metrics selected evaluate all comparison methods, i.c., Ranking Loss (RL), Average Precision (AP), Hamming Loss (HL), adapted area under curve (AUC), OneError (OE), and Coverage (Cov). To compare performance more intuitively, actually report results with respect to I-HLL I-RL, 1-OE, and 1-Cov, so for all metrics, higher values indicate better performance.
http::/leaa: http:///leaa.iiraapps....... httpa//aaa.aaa..o/oooo////aaaooa

---

(087 (327
(04)
(0.20.0331
1671
A-E
(2.07.8
02.A322
0.4...5)
(001
(a) Corel5k
(b) Pascal07
(c) ESPGame
00.4402221
3:ME
0-901
1:M
10:94.0::21
CONN
153
AIC 50
(d) TAPRTC12
MIRFLICKR
Figure 2: Experimental results of nine methods on the five full databases without any missing views or labels. The center of the radar map shows the worst results and the vertexes mean the best results on the six metrics.
3.2 Incomplete multi-view weak multi-label dataset setting
Analogous to previous works 26
20 we manually construct incomplete multi-view weak 25
multi-lahel datasets based above five complete datasets evaluate the performance of all methods in the case of missing views and missing labels. Concretely, 50% of instances on each view are randomly selected unavailable instances, which are replaced by '0" value, And we ensure that there is no invalid sample in the dataset, i.c., each sample holds at least one unmissing view. For the weak labels, we half of the positive and negative tags for each category to unknown. 70% of samples with missing views and missing labels are randomly selected as the training set. The construction of incomplete data is repeated many times reduce the randomness of experiment.
3.3 Comparison methods
In order evaluate the advancement of our framework, eight top comparison methods are selected in our experiment, i.c., C2AE (37), GLOCAL (38] CDMM [18], DM2L [39] LVSL [23], NAIM3L 25). iMVWL 26], and DICNet 20) compare with our MTD on the five incomplete multi-view weak multi-label datasets. To date, few MvMLC methods can handle missing views and weak labels at the same time, thus we introduce some related multi-label classification methods in our experiments. To be more specific, of the nine methods, iMVWL, NAIM3L, DICNet, and our MTD are perfectly applicable to the iMWMMLC task. C2AE, GLOCAL, and DM2L are single-view multi-label classification methods that can deal with incomplete labels, we conduct these methods on each view and record the best result. CDMM and LVSI are the MvMI-C methods unable to handle any data missing, simply complete the missing views using the mean of available instances and fill the unknown tags with '0', The statistical information of competitors is shown the appendix.
Experimental results and analysis
To evaluate performance of our method scenarios with missing views and labels, we compare against eight state-of-the-an algorithms on five datasets. The mean and standard deviation (decimal form in the bottom right) of the results are reported in Table In addition to the six performance metrics mentioned above, we also calculate the average ranking of each algorithm based on these
7

---

Table 1: Experimental results of nine methods on the five databases with 50% missing- view rate, 50% missing-labed rate, and 70% training samples. 'Ave.R' denotes the average ranking of the corresponding method on all six metrics.
Data
Metric Sources
C2AE 37
GLOCAL EEEALLESSTRR DSM T8
DM2L 39
LVSL 21
iMVWL 26
NAIMEL
23
DICNe
X0
MTD
11
TKDE '17
KRS '20
PR '21
TMM '22
UCAT '18
TPAMP22
AAAI "3
AP
0.2276.00
0.285g.com
0.354
0.2622.02
0.342.
0.283p oce
0.309p.oo
0.381e. 204
0.4155.
I-HI.
0.980c.002
0.9676.c0
0.987a
0.9876.cze
0.987o.c 000
0.978p.
0. 0.987
0.988e.
0. 0.988p.ocm
Corelsk
I-RL
0.804g.010
0.84 0.840g.c0s
0.884g
0.843g.000
0.881c.c0 .03
0. 0.865g.ce oz5
0. 0.878p.002
0. 0.882,. 204
0. 3.893
AUC
0.800y.m1
0. 0.843
0.888, oes
0. 0.845p.0s2
0 0.884c.00s .03
0, 0.868p.ecs oco
0, 1.881..os
0. 0.884..om 204
0.896g..om .004
1-OB
0.2466.m:
0.3276.010
0.410s 007
0.3911..00 092955.014
oos
0.311p.om
0.350p
0.468o 207
0.4911.
I-Cov
0.5969.016
0.648o.cov
0.723g 007
0. 0.647s.
0.71 8o.0
0.702p.oe
0. 0.725p.ocs
0.727o.
0. 7.749..cc
Ave.R
8.83
6.33
2.83
6.83
3.83
6.83
4.33
2.17
1,00
0.485%..06 no8
0. 0.4966.cox
0.508g oes
0. 0.4711.cz8
0. 0.504c oos
0.437p.om 0.0
0.488g.ua ocs
0.505p. ox2
0.551. 0044
I-HL.
0.908 00?
0.927p.cm
0.931e om
0. 0.928e.cm
0.930e. coo
0.882p. oce
0.928p.cm o01
0.929p. z01
0.932o. .c11
Pascal07
I-RL
0.745p.009
0.7670.004
0,812, 004
061...s
0.806c.
0.736p. oi5
0. 0.783p.0z4
0.783.. o0s
0.831o.ca oco
AUC
0. 0.765o.om
0. 0.786g.cm
0.838;
0. 0.7799.004
0.832,
0.767p.015
0.811p 001
0.80% 2os
0.85
L.F
0.438o. one
0.443p.om
0.419
0. 0.420n.011
0.419
0.3622.075
0.42
0.4276.
0.459p.oc
I-Cov
0.680p.ou .010
0. 0.700..00
0.759,
0. 16692..oo
0. 0.751c. .000
0.677p.015
0. 0.727p.oce 0.02
0. 0.731e.
0. 0.7840.oca
Ave.R
7.17
5.33
2.83
6.67
3.83
8.83
4.83
4.00
1.00
AP
0,2022
0,221
0.289e
0.2122.000
0.285e.
0.244p
0.246p
0.297o. o00
0. 0,300p ocos
1-HLL
0.971..00 002
0.9622.co on0
0.983o
0.982p.cm
0.983o
0.972
0.963g.cm
0.983e.
0.9833..0
ESPCame
I-RL
0.772 0.7726.006
0.7800.004
0.832,
0.7811.00
0.829, .0:
0. 0.808p..c.
0. 0.818g.0se
0.832,,
0. 0.837
AUC
0.7776.00 5.06
0. 0.784p.co cos
0.836, om1
0.785p.0m
0. 1.833, o02
0. 0.813g. o2e
0, 0,824 ose
0. 0.836, o01
0..8422.. oc2
0.2622. onn
0.317 oos
0.396; oos
0, 0.2944.om oow
0. 0.389 cos
0. 0.343p.m
0.339p.mm
0.439p 007
0.447o
1-Cov
0.4 0.496p.com 0.4970.011
0.574 004
0, 0.488p.02a
0.567c.00s
0. 0.548p
0.571...00
0.593.. o03
0.6022.ou
Ave.R
8.67
7.33
2.33
7.50
3.67
6.17
4.33
1.83
1.00
0.224..00 oor
0.256p.o0 or
0.305, os4
0.234p.o2s
0.304, .04
0.237 ocs
0,261, oon
0,323,. oo1
0.3322. ocs
JAPRIC12
1-IIL
0.965 002
0.960p.co ooe
0.981, 000
0. 0.980p.cr o20
0.981c cop
0. 0.969p.ce oco
0. 0.980p.cm 0c0
0.961. 300
0.981.. oco
1-RL
0.806p 005
0. 0.825o.co 000
0.862: 002
0, 0.8236.05
0.861c 002
0.833p.00
0.848p.0r 001
0.873o. 201
0.875o
AUC
0.807
0.830p.co
0.864.
0.825p.001
0.863.
0.835p
0.850p
0.874o. 200
0.87 6o
1-0B
0.300
0.378p.cm
0.432o
0.340p.os o0e
0.429;
0.352,
0.390p
0. 1.466p.
0.467 4677..0.
I-Cov
0.5236.009
0.534p.00
0.597
0, 0.529b.00m
0.5 0.597g.004
0. 0.564p.0cs
0. 0.5922.004
0.649,.
0. 0.6449..cc
Ave.R
9.00
6.33
2.67
7.50
3.33
6.67
5,00
1.50
1.17
AP
0.505
0.537
0.570s
0.514p.00
0.553
0.490
0.551o oze
0.589 ons
0.607
MIRFLICKR
I-IIL
0.853
0.874p.c0
0.886g
0.878p.001
0.885;
0.839
0.882,
0.888,
0.891..00
1-RL
0.821o.003 003
0.8320.001
0.856,
0.831o.003
0.856c 00:
0.803p.0ce
0.844p.ool
0.863e, 204
0. 0.8750.00 .c2
AUC
0.810b.0x 004
0.828p.om
0.846; om
0. 038
0. 0.844, om
0.7877.01
0,837 om
0. 0.849, 004
0.8622..oz oco
0.505 mzo
0.552 oos
0.631, 054
0.510 aze
0.607, cox
0.511 o22
0.585 oc3
0.637 207
0.655o. oc4
1-Cov
0.590p.00 005
11.055..om
0,640s o0
6.4444.0
0.636e .0:
0.572b.00 013
0,631o.0e
0. 0.652o. 207
0.676p .044
Ave.R
8.17
6.17
3.00
6.83
3.83
8.67
5.00
2.00
1.00
5,655
0.6-111
0,6-0,1
0,60,33
5.3-5,6
0,5-0.
9,1-9,5
0,3-0,6
5,3-5.
0,3-0
5,3-5
0,1-0.6
(a) Epoch 0
(b) Epoch 60
(c) Epoch 120
(d) Epoch 180
Figure A random sample's channel similarity heat maps across all channels on Corel5k dataset with half of missing views and labels. $_1- 1--S_6 and 1_1- O_6 denote shared features and pee-propyyy features on six views, respectively. With the increase of training epoch, the similarities of features on shared and proprietary channels show the expected trend, that is, the shared features across views gradually converged, while the similarities "shared-pree "ronreedrpprrttary""" and cooprictary-proprittte feature pairs gradually decreased.
metrics, denoted as 'Ave.R', for intuitive comparison. Based on the results in Table 1 we make the following observations:
1. Our MTD is compelling, outperforming comparison methods on almost all metries across all five datasets. In addition to the average ranking of 1.17 the IAPRTCI2 dataset, it
8

---

ranks 1st on the other four datasets. Whether compared with traditional methods (such as GLOCAL and NAIM3L) or deep methods (such as C2AE and DICNet), our MTD shows excellent compatibility with missing views and weak labels.
2. We can find that methods based on deep neural networks generally perform better than traditional methods. Specifically, our MTD. DICNet, and CDMM are consistently ranked in the top 3 across all five datasets, which can be attributed to the strong feature extraction capabilities of deep encoders and the flexible multi-view fusion strategy they employ.
3. In the comparison among traditional methods, the method that is specifically designed for double incompleteness shows obvious advantages compared to other methods. This indicates the necessity of considering possible missing views and labels during the model design process.
In addition to experimenting on datasets with 50% missing views and missing labels, we also perform comparison experiments on the full dataset without any missing data and use radar charts to show the results Fig. (2). For more intuitive observation, we set the range for each axis to the min-max values of the nine methods on the corresponding metric. It can be seen from Fig. (2) that the performance of our method on the complete datasets still outstanding, which verifies the good adaptation of
To demonstrate that our cross-channe contrastive loss works expected, we plot random sample's channel similarity heat maps across all channels in different training epochs on the Corel5k dataset with half of missing views and labels in Fig. (3). Each patch of the heat map shows the cosine similarity of corresponding two instances, i.e., i-th sample's ' heat map H; = CCT, where C
shared and vieoppropryetary channels of first view of sample i, respectively. In Fig. (3). is evident (1) (sf"), S ...... si (m)) (1) o { E R2mxx. si E Rd and o; E Rd are the instances on
(2)
that in the initial state, the similarities between different channels are relatively small. However, as training progresses. the similarities between instance pairs on the first m shared channels increase rapidly, while the similarities for the remaining channels are close to zero This observation indicates that our cross-chandn chasnell contrastive loss is functioning as expected. Additionally, we notice that there is a shared feature on channel 3 that is not vegl-aggre This may be due to the third view of this sample sharing less information with the other views.
3.5 Ablation study
Table 2: Ablation results of our MTD on Corel5k dataset with 50% missing views, 50% missing labels. 's_ch' denotes the conventional single-channel network and 'd_ch' denotes our double-channel framework. 'w/o' means 'without'.
Method
AP
1-HI.
1-RI.
AUC
1-OF
1-Cov
s_ch baseline
0.396
0.988
0.888
0.891
0.469
0.738
d_ bascline
0.408
0.988
0.890
0.893
0.482
0.744
d d_ch+tLge
0.409
0.988
0.890
0.893
0.488
0.744
d Lh+
0,413
0.988
0.890
0.894
0.487
0.746
MTD
0.415
0.988
0.893
0.896
0.491
0.750
MTD w/o mask
0.397
0.987
0.888
0.891
0,466
0.738
s_ch w/o mask
0.372
0.987
0.881
0.884
0.449
0.721
d_ch w/o mask
0.387
0.987
0.885
0.889
0.452
0.732
Our model contains multiple loss functions and strategies, in order further study the effectiveness of each component, we perform extensive ablation experiments on the Corel5k dataset with 50% missing labels and views, and report results in Table 2 Specifically, we take the common single-channel model (20 44) as a baseline architecture and extend it to a two-channel baseline framework. We iteratively add our individual loss functions Lge+ Lres and Leee into it and remove the masking operation from different versions. From Table [ we observe that all designs play a role to varying degrees, especially the fragment masking mechanism's 's performance increase on the Corel5k dataset is the most noticeable. This confirms that the random masking mechanism to raw data, commonly used in the image and text analysis domain, also effective in the field of multi-view learning. However, this strategy still needs to be further studied for the underlying mechanism of action.

---

3.6
Implementation details
Our TMD is implemented by Pytorch and MindSpore frameworks on Ubuntu operating system with single RTX 3090 GPU and an 17- 17-12900k CPU. The learning rate is set to 0.1 and the Stochastic Gradient Descent (SGD) optimizer is chosen for training model. The batch size and momentum are 128 and 0.9 for all five datasets. Please refer to the appendix for the setting of other hyperparameters. The code can be found https:/// /tte://jjsssrrrre wetammattgithiie
Conclusion
In this paper, we propose novel masked two-channel decoupling framework (MTD) for the iMvWMLC task. Our MTD decouples single-channel features commonly used in previous methods into two-channel features for consensus and complementarity learning, and design cross-channel contrastive loss for this setting. Additionally, we employ a random fragment masking strategy during the training phase reduce redundancy of raw data, inspired by image and text masking strategies, which leads to an obvious performance boost. Furthermore, we introduce a label-guided graph con- straint approach to ensure that the learned embedding features maintain structure information among samples. Our extensive comparison and ablation experiments demonstrate that MTD outperforms other advanced methods and is compatible with arbitrary incomplete multi-view and weak multi-label data. Looking back at our work, although we propose many strategies and techniques to advance the frontier research progress of iMvWMLC. the complexity of this issue leaves room for further optimization, such as considering multi-label correlation and missing data recovery as directions of future works.
Acknowledgements
This work is supported in part by the Shenzhen Higher Education Stability Support Program under Grant No. GXWD2022081 117333000073310002 part by the Chinese Association for Artificial Intelligence (CAAI)-Huawei MindSpore Open Fund under Grant No. CAAIXSJLI--002-00IC..
References
[1] Yangbangyan Jiang, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang, Dm2c: deep mixed-modal clustering. In International Conference on Neural Information Processing Systems, pages 5888-58 0888-589,220 2019.
[2] Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. Advances in neural information processing systems, 34:2148-2111 21599 2021.
[3] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pages 5177-5186, 2019.
[4] Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially viewealigned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 1134 143, 2021.
[5] Jie Wen, Zheng Zhang, Lunke Fei, Bob Zhang, Yong Xu, Zhao Zhang, and Jinxing Li. A survey on incomplete multiview clustering. IEEE Transactions on Systems, Man, and Cybernetics: Systems, 2022.
[6] Chengliang Liu, Zhihao Wu, Jie Wen, Yong Xu, and Chao Huang. Localized sparse incomplete multi-view clustering. IEEE Transactions on Multimedia, 2022.
[7] Xinwang Liu, Miaomiao Li, Chang Tang, Jingyuan Xia, Jian Xiong. Li Liu, Marius Kloft, and En Zhu. Efficient and effective regularized incomplete multi-view clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(8):2634 2646, 2020.
[8] Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. Multi-modal cross-domain alignment network for vidco moment retrieval. IEEE Transactions on Multimedia, 2022.
10

---

[9] Peng Hu, Dezhong Peng, Yongsheng Sang, and Yong Xiang, Multi-view linear discriminant analysis network. IEEE Transactions on Image Processing, 28(1 (8(11):5322-5600 2019.
[10] Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. Multi-level feature learning for contrastive multi-view clustering. Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition, pages 16051 2022.
[11] Zongho Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. In International Conference Learning Representations, 2021.
[12] Yang Wang. Survey on deep multi-mode modal data analyties: Collaboration, rivalry, and fusion. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM), 17(1s):1-25, 2021.
[13] Feiping Nie, Guohao Cai, and Xuelong Li. Multi-view clustering and eupissupereieed classifica- tion with adaptive neighbours. Proceedings of the AAAI conference on artificial intelligence, volume 31, 2017.
[14] Zhenglai Li, Chang Tang, Xinwang Liu, Xiao Zheng, Wei Zhang, and En Zhu. Consensus graph learning for multi-view clustering. JEEE Transactions Multimedia, 24:2461-2472 2021.
[15] Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on pattern analysis and machine intelligence, 2022.
[16] Yijic Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence, 45(4):4447 2022.
[17] Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions Knowledge and Data Engineering, (6(8):1819-183)) 2013.
[18] Dawei Zhao, Qingwei Gao, Yixiang Lu, Dong Sun, and Yusheng Cheng. Consistency and diver- sity neural network multi-view multi-label learning. Knowledge-Bacd Based Systems, 218:106841, 2021.
[19] Jun Huang, Guorong Li, Qingming Huang, and Xindong Wu. Learning label specific features for multi-label classification. In IEEE International Conference on Data Mining, pages 181-190, 2015.
[20] Chengliang Liu, Jie Wen, Xiaoling Luo, Chao Huang, Zhihao Wu, and Yong Xu. Dicnet: Deep instance-level contrastive network for double incomplete multi-view multi-label classification. In Proceedings of the AAAI conference on artificial intelligence, 2023.
[21] Gengyu Lyu, Xiang Deng, Yanan Wu, and Songhe Feng. Beyond shared subspace: view- specific fusion for multi-view multi-label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 36, pages 7647-7654, 2022.
[22] Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory, pages 63-77. Springer, 2005.
[23] Dawei Zhao, Qingwei Gao, Yixiang Lu, and Dong Sun. Non-aligned multi-view multi-label classification via learning view-specific labels. IEEE Transactions on Multimedia, 2022.
[24] Chengliang Liu, Jie Wen, Xiaoling Luo, and Yong Xu. Incomplete multi-view multi-label learning via label-guided masked view- and aatee transformers. In AAAI Conference on Artificial Intelligence, 2023.
[25] Xiang Li and Songcan Chen. A concise yet effective model for non-aligned incomplete multi- view and missing multi-label learning. IEEE Transactions on Puttern Analysis and Machine Intelligence, 44 44(10):5918 5918599911-59922 2022.

---

[26] Qiaoyu Tan, Guoxian Yu, Carlotta Domeniconi. Jun Wang, and Zili Zhang, Incomplete multi- view weak-label learning. In International Joint Conference on Artificial Intelligence, pages 2703-2709, 2018.
[27] Kaiming He, Xinlei Chen, Saining Xie. Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 16000-16009, 2022.
[28] Jie Wen, Zheng Zhang, Zhao Zhang, Zhihao Wu, Lunke Fei, Yong Xu, and Bob Zhang. Dimc- net: Deep incomplete multi-view clustering network. In ACM International Conference on Multimedia, pages 3753-3761, 2020.
[29] Peng Zhou, Liang Du, Yi-Dong Shen, and Xuejun Li. Tri-level robust clustering ensemble with multiple graph learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pages 11 11125-1111 115--11323 1113,, 2021.
[30] Peng Zhou, Liang Du, Xuejun Li, Yi-Dong Shen, and Yuhua Qian. Unsupervised feature selection with adaptive multiple graph learning. Pattern Recognition, 105:107375, 2020.
[31] Kun Zhan, Feiping Nie, Jing Wang, and Yi Yang. Multiview consensus graph clustering. IEEE Transactions on Image Processing, 28(3):1261-1270) 2018.
[32] Pinar Duygulu, Kobus Barnard, Joao FG de Freitas, and David A Forsyth. Object recognition as machine translation: Learning lexicon for fixed image vocabulary. In European Conference on Computer Vision, pages 97-112, 2002.
[33] Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision, 88:303-308, 2009.
[34] Luis Von Ahn and Laura Dabbish. Labeling images with a computer game. In SIGCHI Conference on Human Factors in Computing Systems, pages 319-326, 2004.
[35] M Henning, D Thomas, et al. The iapr benchmark: Anew evaluation resource for visual information systems. In International Conference Language Resources and Evaluation, pages 1-11, 2006.
[36] Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. ACM International Conference on Multimedia Information Retrieval, pages 39-43, 2008
[37] Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In AAAI Conference Artificial Intelligence, volume 31, 2017.
[38] Yue Zhu, James T Kwok, and Zhi-Huz Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering, 30(6):1081-1094 2017. 2017.
[39] Zhongehen Ma and Songcan Chen. Expand globally, shrink locally: Discriminant multi-label learning with missing labels. Pattern Recognition, 111:107675, 2021.
[40] Jie Wen, Chengliang Liu, Shijie Deng, Yicheng Liu, Lunke Fei, Ke Yan, and Yong Xu. Deep double incomplete multi-view multi-label learning with incomplete labels and missing views. IEEE transactions on neural networks and learning systems, PP, 2023.
12

---

A
Datasets
Table 3: Detailed statistics about five multi-vier multi-label databases.
Database
# Sample
Category
# View
Category/HSample
Corel5k
4999
260
6
3,40
Pascal07
9963
20
6
1.47
ESPGame
20770
268
6
4.69
IAPRTCI2
19627
291
6
5.72
MIRFLICKR 25000
38
4.72
These datasets are uniformly extracted six kinds of feature as six views from five image datasets, i.e., GIST, HSV, Hue, Sift, RGB, and LAB. Table shows the statistical information of the five datasets.
B Comparison Methods
Table 4: Information of comparison multi-label classification methods. 'Multi-view' means the method is designed for multi-view data; "Missing-view and 'Missing-label" represent the correspond- ing method is able to handle incomplete views and weak labels.
Method
Sources
Multi-view
Missing-view
Missing-label
37
AAAI '17
GLOCAL 38
TKDE '17
X
X
CDMM
KBS '20
XXX
DM2L 39
PR '21
X
LVSL 23
TMM '22
iMVWL
IJCAI '18
NAIM3L 25
TPAMI '22
DICNet
AAAI '23
C Hyperparameters study
The overall loss of our model mainly contains 3 hyperparameters, i.e., a, 8, and 7. order to study the impact of these three hyperparameters the performance of the model, we iteratively set different values and report the corresponding AP values in Fig.
In Fig. 4 we show the AP value with respect to different parameter selections on Corel5k and Pascal07 datasets with 50% missing views and labels. Specifically, from Fig. 4a and 4b we recommend selecting from the ranges of [1e - 4, 1e 3] and [e 4, 1e0) for Corel5k and Pascal07 datasets respectively; selecting from the ranges [ie 3, le - 1] and [le 3,
1] for Corel5k and
Pasca107 datasets respectively. For convenience, we uniformly set a 0.4 and / = 0.4 in our experiments. For 7, our method is not sensitive to it, so we set ς = 0.1 for five datasets.
From Fig. 4d and Fig. 4e it can be seen that too large or too small mask rate is not conducive to achieving optimal performance. Based experiments, we recommend setting mask rate from 0.2 to 0.4 (0.25 is used for all datasets in our experiments for convenience).
D Time cost
To study the training and test efficiency of our MTD, we report the time cost of the training and test phases of the six supervised methods on the Corel5k dataset with 70% training samples in Table The convergence setting of the model plays a critical role determining the training time required. Therefore, measure the running times of all methods under their default convergence conditions.
13

---

0.4
0.4
0.6
0.3
0.4
0.4
AP
0.2
0
0.2
1e.4
1e-4
1e.3
1e-3
0.1
1e-2
le-2
2
le-1
2-91
T-8t
1
le0
1e-4
1e.3
1e0
1e.
8
(a) Corel5k
(b) Pascal07
CorelSk
0.55
0.40
0.35
160
16-1
1e-2
16-3
1e-4
16-5
(c) Corel5k and Pascal07
0.41
0.58
0.40
0.391
0.390
0.394
0.390
0.38
0.38
0.539
0.37
0,350
2 0.54
0.53
0.527
0.35
0.52
0.339
0.34
0.51
0.33
0.50
(d) Corel5k
(e) Pasca107
Figure 4: AP value v.s. hyperparameters and 3 on the (a) Corel5k and (b) Pascal07 datasets; AP value v.s. hyperparameter y on the (c) Corel5k and Pascal07 datasets; AP value v.s. hyperparameter the (d) Corel5k and (e) Pascal07 datasets. The two datasets are with 50% missing-label rate, 50% missing- view rate, and 70% training samples.
Table 5: Time efficiency of training and testing phases of the nine methods the Corel5k dataset with 70% training samples. (Unit: second)
Methed
C2AB
GLOCAL
CDMM
DM2L
LVSL
iMVWL
NAIML
DICNet
MID
Phase
Training
170.24
154.42
16.02
713.37
63.73
165.82
143.63
313.89 0.05
687.52
Test
0.04
0.89
1.73
0.04
0.64
0.02
0.01
0.15
For methods that process a single view at a time, we report both the total training time for views and the test time for single view. For DICNet and MTD, report the time spent in 100 epochs. In order to ensure consistent hardware conditions, all tests are carried out on the same computer. From the results shown in Table 5 it can be observed that deep learning- based method requires longer training times, but reaches higher performance than other methods.
14


Reviews:
[{"review_id": "IxOyZ4mJhq", "reviewer": "Reviewer_NrND", "summary": "This paper studies incomplete multi-view weak multi-label learning problem, which is important. The authors propose a masked two-channel decoupling framework based on deep neural networks. They develop cross-channel contrastive loss, a label- guided graph regularization loss, and random fragment masking strategy. The experiments validate the effectiveness.", "questions": "See weakness part", "limitations": "The author seems to lack a comprehensive discussion on the limitations of the proposed method.", "rating": 4, "confidence": 4, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "- The problem is interesting.\n- The techniques in this work seem correct and reasonable.", "weaknesses": "- This work just combines some existing widely-used techniques. Thus the work is a bit incremental, does not provide new insights to me.\n- As stated in abstract, the core is to decouple view-level representation into  shared representation and a view-proprietary representation. This idea has been widely used in existing works.\n- The authors first fill the missing views with noise, and missing label with 0, and then treat it as normal complete view learning problem. To me, this scheme seems not grasp the essence of multi-view weak multi-label learning problem.\n- The aim and construction of contrastive loss in this work seem a bit confusing to me. It is suggested to explain it in more detail.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your efforts in reviewing the manuscript! We respond to the comments below:\n\nW1. This work just combines some existing widely-used techniques. Thus the work is a bit incremental, does not provide new insights to me.\n\nA1. Indeed, we admit that the techniques or ideas used in our method have similar applications in other fields. However, we emphasize more on the combination of these existing ideas and multi-view multi-label classification tasks, that is, solving the problem “Why should them be used on the MvMLC task, How to use them, and What are the benefits?”. \n\nThe following is the difference and significance of the three core technologies involved in this paper and the existing technologies:\n\n1)\tView-specific representation decoupling framework and contrastive loss. Decoupling the features of each view or modality into a shared feature and a proprietary feature is not new in multi-view or multi-modal learning [1,2], however, the crux of the problem is how to achieve this decoupled state, the technical routes and emphases of existing methods are not the same. For instances, work [1] uses an adversarial loss to make it difficult for the model to discern the view origin of shared features, however, the training cost of adversarial learning is high. Our decoupling framework and corresponding contrastive loss avoid this problem well. Work [2] only focuses on reducing the distance between shared feature pairs and increasing the distance between private feature pairs, ignoring the differences between private features and shared features at both the intra-view and inter-view levels. However, our method considers both: maximizing the distance between shared and private features from individual view-specific feature, and the comparison between shared and private feature sets from different views.\n2)\tLabel-guided graph regularization. The Laplacian graph regularization technique is often used in traditional methods in the field of multi-view learning, and at the core of the technique, the construction of Laplacian graph usually only comes from the original feature space and acts on the latent representation of the sample. In our MvMLC task, we boldly combine it with the characteristics of the multi-label task, that is, construct a label-based sample adjacency matrix to replace the adjacency matrix from the original data. The obvious advantage of this design is that the sample label information has clear semantic information while the original data often contains noise and redundancy.\n3)\tMasking random fragment of feature. As another work we are proud of, masking attempts on random segments is our contribution to the multi-view learning community. As far as we know, masking strategy that is currently popular in fields such as images have not been applied to vector data, especially in the field of multi-view or multi-label. And unlike the well-known MAE technology, we do not need to perform additional non-coding processing on the masked data segments, which can be easily transplanted to any multi-view learning network to a large extent, and it is a kind of Plug and play technology.\n\n[1] Multi-View Multi-Label Learning with View-Specific Information Extraction. IJCAI,2019.\n[2] CDDFuse: Correlation-Driven Dual-Branch Feature Decomposition for Multi-Modality Image Fusion. CVPR,2022.\n\nW2. As stated in abstract, the core is to decouple view-level representation into shared representation and a view-proprietary representation. This idea has been widely used in existing works.\n\nA2. Thank you for the comment. As we mentioned in A1, some methods apply the idea of decoupling in multi-view learning. However, the above purpose cannot be achieved by using only two types of encoders. And the key to this idea is to design a suitable strategy to make the features extracted by the two types of encoders meet our assumption, that is, true shared features and proprietary features. Our experiment in Figure 3 also confirms the effectiveness of our strategy.\n\nW3. The authors first fill the missing views with noise, and missing label with 0, and then treat it as normal complete view learning problem. To me, this scheme seems not grasp the essence of multi-view weak multi-label learning problem.\n\nA3. Indeed, we fill missing instances with noise and fill unknown labels with 0 values, however this is just a means of data preprocessing. And the purpose of that is to ensure that multi-view data is dimensionally aligned, which is very important in batch-based training of deep neural networks. It should be noted that although we fill in the missing information beforehand, we do not treat it as complete multi-view learning in subsequent processing. For example, in Eq. (1), we introduce condition $\\Upsilon$ to avoid filled noise from taking part in the calculation of contrastive loss; In Eq. (2) and (3), we introduce missing index matrix $W$ in the process of multi-view fusion to exclude the illegal features corresponding to missing instances. In Eq. (9), we introduce a missing label indicator matrix $G$ to avoid the negative effects of unknown labels.\n\nOn the other hand, we would like to say that there are many ways to deal with missing views and labels, including missing view recovery, pseudo-label filling, etc., and the \"skip\" strategy adopted by our method based on prior missing information is also used by many related methods [3-5].\n\n[3] A Concise yet Effective Model for Non-Aligned Incomplete Multi-view and Missing Multi-label Learning. TPAMI,2022.\n[4] Dicnet: Deep instance-level contrastive network for double incomplete multi-view multi-label classification. AAAI, 2023.\n[5] Expand globally, shrink locally: Discriminant multi-label learning with missing labels, Pattern Recognition, 2021.\n\nResponses to W4 will be added to the comments section after the rebuttal phase begins due to word limitation."}, {"review_id": "MjPvSzxq8F", "reviewer": "Reviewer_8GYm", "summary": "This paper proposes a new framework for incomplete multi-view weak multi-label classification (iMvWMLC), which is a challenging task that involves missing views and labels in the data. The framework consists of four main components: a two-channel decoupling mechanism (called MTD) that extracts shared and view-specific features from each view, a cross-channel contrastive loss that enhances the semantic separation of the two channels, a random fragment masking strategy that reduces the redundancy of the input features, and a label-guided graph regularization loss that preserves the geometric structure among samples. The paper evaluates the framework on five datasets and compares it with eight state-of-the-art methods, showing that it outperforms them on various metrics and is robust to different levels of incompleteness.", "questions": "(1) It would be interesting to know if the authors have experimented with other masking methods, such as random bit masking of features, and what the results of such experiments were.\n\n(2) Availability of code is important for reproducibility and further research. There is no code in the supplement material. Do the authors have any plans to make the code public?\n\n(3) The paper should provide a more detailed discussion on why not all comparison methods were adapted to this task. \n", "limitations": "For missing multi-view data without a priori missing information, this method seems to be difficult to work.", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "(1) As far as I know, research on multi-view multi-label classification is still in its infancy, and this article focuses on this topic very well. \n\n(2) To a certain extent, the logic and expression of this article are clear.\n\n(3) I am interested in masking operations applied to vectorized feature data, most multimodal methods focus more on input real data such as text or images. I agree that the use of the mask mechanism on feature data is instructive.\n", "weaknesses": "(1) The similarity matrix calculated based on weak tags does not take into account the interference caused by unknown tags. The paper should address the potential interference caused by unknown tags in the calculation of the similarity matrix. \n\n(2) The results of the ablation experiment suggest that the so-called contrast loss does not seem to have improved much, at least on the Corel5k dataset. The paper should provide some explanation.\n\n(3) The paper needs to improve the details, as there is a error in section 3.4 where \"TMD\" is incorrectly referenced. The authors could correct this and prevent similar errors.\n\n(4) The reviewer has noticed that there are some recent works aiming at solving the multi-view classification problem. The paper could do a better job by citing the different relevant works, e.g., [1-3], and clarifying the differences between this work and them.\n\n[1] Trusted Multi-View Classification, ICLR’21\n\n[2] Partially View-aligned Representation Learning with Noise-robust Contrastive Loss, CVPR’21\n\n[3] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'23 \n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Your comments are greatly appreciated and we respond to these concerns as follows:\n\nW1: The similarity matrix calculated based on weak tags does not take into account the interference caused by unknown tags. The paper should address the potential interference caused by unknown tags in the calculation of the similarity matrix.\n\nA1: Thank you for your suggestion! In the process of calculating the similarity matrix, we only consider the sample similarity that can be obtained from the existing label information. However, in the calculation process of Eq. (4), unknown labels have an impact on the normalization scale, so we adjusted the calculation method of the similarity matrix to better handle sample labels with unknown tags. The specific adjustments are as follows:\n\n$T_{i,j} = \\frac{C_{ij}\\cdot(y_{i}y_{j}^T)}{C_{ij}\\cdot(y_{i}y_{j}^T)+\\eta}$, where ${T}\\in [0,1]^{n\\times n}$ is the sample similarity graph, describing the similarity between any two samples. $C_{ij} = G_{i,:}G_{j,:}^T$ is the number of label available for both samples $i$ and $j$. $y_{i}$ and $y_{j}$ denote the $i$-th and $j$-th rows of $Y$. And $\\eta$ is a constant, empirically set to 100 for simplicity. Taking into account the fact that $C_{ij}$ is much larger than $y_{i}y_{j}^T$ on most datasets, when the number of categories is large, even if two samples only have one same label, it means that they are much more similar than other sample pairs.\n\nW2: The results of the ablation experiment suggest that the so-called contrast loss does not seem to have improved much, at least on the Corel5k dataset. The paper should provide some explanation.\n\nA2: In the ablation experiment, the contrastive loss did not show a large performance improvement on the corel5k dataset. We analyze that this may be because the cross-view sharing information of many samples in corel5k dataset is not sufficient, that is, the view-specific information occupies a dominant position, leading to difficulties in the aggregation of shared information. Experiments on other datasets show that the contrastive loss has improved in all metrics (Table 1 and Table 2 in new supplementary materials).\n\nW3: The paper needs to improve the details, as there is an error in section 3.4 where \"TMD\" is incorrectly referenced. The authors could correct this and prevent similar errors.\n\nA3: Thank you for your careful review. We have carefully checked the full text and corrected similar typos, and believe that the quality of the paper has been further improved.\n\nW4: The reviewer has noticed that there are some recent works aiming at solving the multi-view classification problem. The paper could do a better job by citing the different relevant works, e.g., [1-3], and clarifying the differences between this work and them.\n\nA4: Thanks! We have added to the manuscript an introduction to these multi-view classification works and the differences from these works in the paper:\n\nThe trusted multi-view classification proposed by Han et al. focuses on the multi-view single-label classification based on confidence learning, trying to learn the classification uncertainty of each view based on evidence theory [1]. However, Dempster-Shafer evidence theory requires the mutual exclusion of labels, which is difficult to satisfy in multi-label classification. Yang et al. proposed a multi-view contrastive learning method with noise-robust loss to solve the partial view alignment problem, which takes aligned data as positive pairs and unaligned data as negative pairs [2]. Our method focuses on missing views rather than multi-view unalignment issue, and relies on the shared-private feature assumption rather than the prior alignment information for the construction of positive and negative pairs. Lin et al. proposed dual contrastive prediction for incomplete multi-view representation learning, which performs contrastive learning at both the instance level and category level to ensure cross-view consistency and recover missing information [3]. However, the instance-level contrastive loss of this method only aggregates cross-view representations in the potential space from the perspective of consistency, ignoring the view-private information.\n\n\n\nQ1: It would be interesting to know if the authors have experimented with other masking methods, such as random bit masking of features, and what the results of such experiments were.\n\nA1: Thanks for your question! In fact, we also tried to mask random bits of vector features, but it didn't work well, and we think that the continuity of key information in the features was badly broken.\n\nQ2: Availability of code is important for reproducibility and further research. There is no code in the supplement material. Do the authors have any plans to make the code public?\n\nA2: Yes, we will make the code public upon acceptance of the paper to help peers reproduce it.\n\nQ3: The paper should provide a more detailed discussion on why not all comparison methods were adapted to this task.\n\nA3: At present, the research on multi-view multi-label classification is in its infancy, and few methods that can handle both view and label incompleteness. In our comparison experiment, only iMvWLL[4], NAIM3L[5], and DICNet[6] can meet our task setting at the same time. Therefore, following the existing methods [4-6], we introduce other methods that can handle incomplete multi-views or partial multi-labels in the experiment to enrich our experiment, which also can demonstrate the necessity of special design for missing views and unknown labels in the network structure.\n\n[4] Incomplete multi-view weak-label learning, IJCAI\n\n[5] A concise yet effective model for non-aligned incomplete multi-view and missing multi-label learning, TPAMI\n\n[6] DICNet: Deep Instance-Level Contrastive Network for Double Incomplete Multi-View Multi-Label Classification, AAAI\n"}, {"review_id": "0a3hQzAVJk", "reviewer": "Reviewer_txHg", "summary": "This paper proposes a general framework for missing multi-view and missing multi-label classification. The main innovation is to  decouple each view feature into two different  shared and private features. The paper also applies random masks to the raw features to make the network learn from limited information, and shows the benefits of this approach with ablation experiments.", "questions": "- Are these masks fixed during training? How exactly did the authors set up the mask?\n- Why the results of iMVWL and NAIM3L are not consensus with original papers?\n", "limitations": "N/A", "rating": 6, "confidence": 5, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "- This paper is clear in terms of writing and presentation. It is easy for the reader to understand the author's motivation.\n\n- The problem addressed by the authors is complex (doubly incomplete data) and poses a greater challenge than methods based on ideally complete data. However, judging from the results, the author solves this problem well.\n\n- The method of using labels to constrain the feature extraction is novel, and the author's local masking of input features has also brought performance growth. I think there are simple and effective techniques that can be extended to other fields.\n\n- The experimental results show that the MTD has significant advantages over the existing multi-view multi-label classification methods.\n", "weaknesses": "- The one-HL metric does not seem to be of much significance because it does not effectively evaluate the performance of different methods according to the author's experimental results.\n- It is inappropriate to appear both ‘Figure’ and ‘Fig.’ in the text.\n- The author should clearly explain the meaning of Figure 3 in the caption.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition and we respond to your comments below:\n\nW1: The one-HL metric does not seem to be of much significance because it does not effectively evaluate the performance of different methods according to the author's experimental results.\n\nA1: Yes, the metric \"1-HL\" does not show strong discrimination in performance evaluation, however, as an old multi-label classification metric, we still include it in the experimental results and to make a comprehensive comparison with existing works (iMvWLL, NAIM3L, and DICNet).\n\nW2: It is inappropriate to appear both ‘Figure’ and ‘Fig.’ in the text.\n\nA2: Thanks for your suggestion, we have checked the full text to ensure the consistency of similar expressions.\n\nW3: The author should clearly explain the meaning of Figure 3 in the caption.\n\nA3: Thanks for your comments, we have added a detailed explanation to the caption of Figure 3 in the manuscript:\n\nA random sample's channel similarity heat maps across all channels on Corel5k dataset with half of missing views and labels. S_1- S_6 and O_1- O_6 denote shared features and view-proprietary features on six views, respectively. With the increase of training epoch, the similarities of features on shared and proprietary channels show the expected trend, that is, the shared features across views gradually converged, while the similarities of \"shared-proprietary\" and \"proprietary-proprietary\" feature pairs gradually decreased.\n\nResponse to some questions:\n\nQ1: Are these masks fixed during training? How exactly did the authors set up the mask?\n\nA1: Yes, these masks are fixed during training. In our model, we uniformly set the mask rate $\\sigma=l/d_v$ to 0.25 for all datasets instead of a larger masking rate like MAE. The reason is that we notice the difference that the information density of vector data is much larger than that of image data. At the same time, we give the parameter analysis of $\\sigma$ in the supplementary material, and the experimental results show that the model achieves the best performance when $\\sigma$ is set to 0.2-0.4.\n\nQ2: Why the results of iMVWL and NAIM3L are not consensus with original papers?\n\nA2: Since we added new metrics (1-OE and 1-Cov) to our experiment that were not provided in the original papers, we reproduced their methods on our incomplete data and our replicated results exceed those in the original papers on some datasets and metrics.\n"}, {"review_id": "0AI9NBVVY5", "reviewer": "Reviewer_XPPT", "summary": "The core innovation of the method lies in decoupling the single-channel view-level representation, which is common in deep multi-view learning methods, into a shared representation and a view-proprietary representation with a cross-channel contrastive loss. The authors have conducted sufficient experiments to verify the effectiveness of the proposed method.", "questions": "Is it efficient to address the missing problem by introducing additional prior information, that is matrix G and W in this paper?", "limitations": " The authors have discussed the limitations and suggested further directions for improvement in the Conclusion.", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "1. The definition of the problem in this paper is clear. The topic is novel, and as far as I know, multi-view multi-label classification methods for the complex situation of missing labels and data are underexplored.\n\n2. The interpretation of extracting multi-view features from two perspectives is intuitive and fits with the fundamental assumption of multi-view/multi-modal learning problems that there are commonalities as well as differences between modalities.\n\n3. The experimental results are sufficient. The authors not only provide the results in the missing case, but also provide the results of the proposed model on the complete datasets, which shows that the model has good learning ability for multi-view data.", "weaknesses": "1. In Eq.(1), the necessary explanation about the coefficient 2 in the numerator is lacking. The authors should also provide more explanation about the motivation of such design.\n\n2. A clarification on the value of mask rate $\\sigma$ is lacking.\n\n3. The authors propose to build two groups of encoder-decoder for different views, but the computational complexity may increase with the number of views quickly.\n\n4.  There are some unclear expressions, for instance, in Section 2.2, the authors say “Different from conventional deep multi-view networks”. So what are the conventional deep multi-view networks? There is no corresponding reference here.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you very much for your recognition and we respond to your comments below:\n\nW1: In Eq.(1), the necessary explanation about the coefficient 2 in the numerator is lacking. The authors should also provide more explanation about the motivation of such design.\n\nA1: Thanks for your comments, we have added more explanation about coefficient 2 in molecular of Eq. (1):\n\nIn addition, considering the efficiency of matrix calculation, we set the coefficient of the contrastive item of shared features and private features to 2 in the numerator part, so that the entire paired similarities can be obtained by a single matrix multiplication, i.e., $[s_i^{(1)}; s_i^{(2)};…; s_i^{(N)}; o_i^{(1)}; o_i^{(2)};…; o_i^{(N)}]\\times [s_i^{(1)}; s_i^{(2)};…; s_i^{(N)}; o_i^{(1)}; o_i^{(2)};…; o_i^{(N)}]=Q\\in \\mathbb{R}^{2N\\times 2N}$. The matrix $Q$ can be divided into four sub-matrices whose size is $N$ by $N$. The sub-matrix in the upper left corner stores the similarity values of the denominator in Eq. (1), the sub-matrix in the lower right corner stores the values of the second item in the numerator, and the remaining two submatrix, which are transposed to each other, store the similarity values of the first item in the numerator. Therefore, we set the coefficient of the first term in the numerator to 2. At the same time, the diagonal elements of $Q$ are removed, so that the number of elements in the numerator and denominator parts is $\\frac{1}{3N^{2}-N}$ and $\\frac{1}{N^{2}-N}$ respectively.\n\nW2: A clarification on the value of mask rate $\\sigma$ is lacking.\n\nA2: Thanks for your comments, as described in our paper, we empirically set $\\sigma$ to 0.25 instead of 0.75 in MAE, considering that vector-based features do not have high redundancy as in image data. At the same time, we give the parameter analysis of $\\sigma$ in the supplementary material, and the experimental results show that the model achieves the best performance when $\\sigma$ is set to 0.2-0.4.\n\nW3: The authors propose to build two groups of encoder-decoder for different views, but the computational complexity may increase with the number of views quickly.\n\nA3: Thank you for your comments. In fact, we designed only 2 encoders for each view (one decoder per view). Since our codec structure is simple, including only 4-5 linear layers, the increase in model parameters brought about by increasing the number of views is also linear.\n\nW4: There are some unclear expressions, for instance, in Section 2.2, the authors say “Different from conventional deep multi-view networks”. So what are the conventional deep multi-view networks? There is no corresponding reference here. \n\nA4: We are sorry that our expression confused you. We have added relevant references to the manuscript:\n\nDifferent from conventional deep multi-view networks[1-3], we employ two groups of multi-layer…\n\n[1] Deep Double Incomplete Multi-View Multi-Label Learning With Incomplete Labels and Missing Views, TNNLS\n\n[2] Multi-level Feature Learning for Contrastive Multi-view Clustering, CVPR\n\n[3] Deep Embedded Complementary and Interactive Information for Multi-View Classification, AAAI\n\nQ1: Is it efficient to address the missing problem by introducing additional prior information, that is matrix G and W in this paper?\n\nA1: Thanks for your question, the missing view indicator matrix $\\mathbf{W}$ and missing label indicator matrix $\\mathbf{G}$ we introduced in the paper can avoid the negative impact of missing information. And our experiments in the new supplementary material confirm that introducing $\\mathbf{W}$ and $\\mathbf{G}$ can effectively improve the model's ability to handle missing views and labels.\n"}]

Author rebuttal:
Thanks to all reviewers, and this is our new supplementary material.

Meta-review: This paper focuses on the complex yet highly realistic task of incomplete multi-view weak multi-label learning, and proposes a masked two-channel decoupling framework based on deep neural networks. The method decouples the single-channel view-level representation into a shared representation and a view-proprietary representation, and designs a cross-channel contrastive loss to enhance the semantic property of the two channels. After full discussion, the review issues were resolved, and the method was deemed to be innovative, well-experimented, and exactly described. I'm willing to accept manuscript.
Decision: Accept (poster)