Title: Masked Two-channel Decoupling Framework for Incomplete Multi-view Weak Multi-label Learning
Venue: NeurIPS 2023
Accepted: yes
Keywords: Incomplete Multi-view Weak Multi-label Learning

Multi-view learning

Multi-label Classification
Abstract: Multi-view learning has become a popular research topic in recent years, but research on the cross-application of classic multi-label classification and multi-view learning is still in its early stages. In this paper, we focus on the complex yet highly realistic task of incomplete multi-view weak multi-label learning and propose a masked two-channel decoupling framework based on deep neural networks to solve this problem. The core innovation of our method lies in decoupling the single-channel view-level representation, which is common in deep multi-view learning methods, into a shared representation and a view-proprietary representation. We also design a cross-channel contrastive loss to enhance the semantic property of the two channels. Additionally, we exploit supervised information to design a label-guided graph regularization loss, helping the extracted embedding features preserve the geometric structure among samples. Inspired by the success of masking mechanisms in image and text analysis, we develop a random fragment masking strategy for vector features to improve the learning ability of encoders. Finally, it is important to emphasize that our model is fully adaptable to arbitrary view and label absences while also performing well on the ideal full data. We have conducted sufficient and convincing experiments to confirm the effectiveness and advancement of our model.

Masked Two-channel Decoupling Framework for Incomplete Multi-view Weak Multi-label Learning

Abstract

Multi-view learning has become a popular research topic in recent years, but research on the cross-application of classic multi-label classification and multi-view learning is still in its early stages. In this paper, we focus on the complex yet highly realistic task of incomplete multi-view weak multi-label learning and propose a masked two-channel decoupling framework based on deep neural networks to solve this problem. The core innovation of our method lies in decoupling the singlechannel view-level representation, which is common in deep multi-view learning methods, into a shared representation and a view-proprietary representation. We also design a cross-channel contrastive loss to enhance the semantic property of the two channels. Additionally, we exploit supervised information to design a labelguided graph regularization loss, helping the extracted embedding features preserve the geometric structure among samples. Inspired by the success of masking mechanisms in image and text analysis, we develop a random fragment masking strategy for vector features to improve the learning ability of encoders. Finally, it is important to emphasize that our model is fully adaptable to arbitrary view and label absences while also performing well on the ideal full data. We have conducted sufficient and convincing experiments to confirm the effectiveness and advancement of our model.

1 Introduction

With the increasing popularity of multi-view learning, researchers have discovered that heterogeneous multi-source features obtained from multiple sensors or techniques can provide a richer and more diverse description of samples [1-4]. For example, features can be extracted using scale-invariant feature transform (SIFT) operator and deep neural networks from different perspectives of an image, or features can be extracted from multimedia data, such as images and texts, to express higher-level semantic objects [5-7]. Many multi-view learning methods are based on the assumptions of multiview semantic consistency and feature complementarity [8, 9]. They aim to obtain unique sample labels from multi-source features to maintain the unity of multiple views at the semantic level and to leverage the unique advantages of each view to compensate for the limitations of single-view observation [10-12]. For instance, Nie et al. emphasized the importance of learning a cross-view similarity matrix in their work [13]. Li et al. proposed to learn a common graph in the spectral embedding space for final clustering [14]. Han et al. attempted to compute confidence scores of samples on each view to facilitate classification predictions across multiple views [15]. Lin et

al. proposed dual contrastive prediction for incomplete multi-view representation learning, which performs contrastive learning at both the instance level and category level to ensure cross-view consistency and recover missing information [16]. However, the instance-level contrastive loss of this method only aggregates cross-view representations in the potential space from the perspective of consistency, ignoring the view-private information.

On the other hand, as a classical classification problem, multi-label classification has occupied an important area of pattern recognition for a long time [17]. However, under the upsurge of multi-view learning, new composite problem, i.e., multi-view multi-label classification (MvMLC), are attracting increasing attentions from researchers [18, 19]. As the name suggests, MvMLC can be seen as a supervised subtask within the scope of multi-view learning and can also be considered as an extension of traditional single-view multi-label classification with respect to feature diversity. As a result, existing MvMLC methods not only take into account the basic characteristics of multi-view learning but also consider the requirements of downstream multi-label classification [20, 18, 21]. For example, Zhao et al. proposed a framework called CDMM that mines multi-view consistency and diversity information, which uses the Hilbert-Schmidt Independence Criterion [22] to enhance the diversity of each view and obtain consistent classification results through late fusion [18]. Another advanced non-aligned MvMLC model proposed by Zhao et al. is LVSL (learn view-specific labels), which not only considers the local geometric structure in the original feature space but also introduces a learnable label correlation matrix with low-rank structure [23].

However, these ideal MvMLC methods ignore the possibility of missing multi-view features and labels, making it difficult to achieve good performance on incomplete multi-view weak multi-label data, in which sample's partial views and tags are unknown [24, 25]. In recent years, researchers have proposed some incomplete multi-view weak multi-label classification (iMvWMLC) frameworks to address the adverse effects caused by missing views and labels. For instance, Tan et al. proposed an incomplete multi-view weak label learning model (iMVWL) that decomposes multi-view features into a latent common representation and multiple view-specific basis matrices, and connects the latent representation with labels through a mapping matrix [26]. Additionally, two prior missing indicator matrices are introduced to skip the missing views and labels. Recently, Li et al. developed a non-aligned iMvWMLC model based on traditional matrix factorization, named NAIM3L, to align multiple views in the label space and impose global high-rank and local low-rank constraints on the predicted multi-label matrix [25]. For the double-missing problem, Li et al. introduced a composite indicator matrix for NAIM3L, containing view and label missing information. Another representative deep iMvWMLC framework is DICNet proposed by Liu et al., which applies the contrastive learning technique to aggregate cross-view instances of the same sample and separate instances belonging to different samples [20]. Although DICNet has achieved remarkable improvement, there is a nonnegligible category collision problem, i.e., instances belonging to similar samples will be forcibly separated due to this contrastive learning mechanism.

To tackle these issues, we present the M asked T wo-channel D ecoupling framework ( MTD for short), capable of handling cases where partial views and labels are both missing. The motivation behind this framework is that we both want the underlying representations of the same sample in multiple views to be consistent, while also wanting each view to display its own characteristics, creating a difficult balancing act [10]. This is especially challenging when each view of the same sample is represented by only one feature vector. To address this, we explicitly decouple each view's latent feature into two types of features, i.e., shared feature and view-proprietary feature, via two-channel encoders. We also propose a novel cross-channel contrastive loss that narrows the distance between positive pairs (shared instances in different views of the same sample) and expands the distance between negative pairs (any cross-view and channel instance pairs of a sample, except positive pairs). Note that our cross-channel contrastive loss does not involve the comparison among samples, so it can well avoid the above class conflict problem and improve the framework's feature decoupling ability. Additionally, we draw inspiration from famous image patch masks [27] and introduce random mask fragments in the input data to guide the encoder's learning of valuable information. Lastly, we design a weak label-guided graph regularization loss to preserve the geometric structure in the embedding space, accounting for the non-equidistance among samples in label space. In summary, our paper makes three key contributions:

We propose a novel masked two-channel decoupling framework (MTD) for the iMvWMLC task, which extracts shared and private features for each view separately and enforces this

𝑋′

(")

𝑋′

($)

𝑂

(%)

𝑂

(")

𝑂

($)

𝑆

(%)

𝑆

(")

𝑆

($)

𝑆 ̅

𝑍

+

+

classifier

predictions

E

!

"

E

#

"

E

$

"

E

!

%

E

#

%

E

$

%

𝑋 %

(")

𝑋 %

($)

𝑋 %

(%)

𝑋

($)

𝑋

(%)

𝑋

(")

+

random mask

𝑋′

($)

𝑋

($)

label-guide graph regularization

label

feature

masking random fragment

feature reconstruction

𝐷

!

"

𝐷

#

"

𝐷

$

"

masked input

contrastive learning

𝑋′

(%)

shared channel

proprietary channel

𝑂 %

0.1

0.2

0.4

0.7

activate

𝑆 ̅

design with a cross-channel contrastive loss. At the same time, this framework is compatible with arbitrary incomplete multi-view and weak multi-label data.

To our best knowledge, we are the first to apply random fragment mask in the field of multiview learning and achieve significant performance gains, which supports a new multi-view vector data enhancement mechanism for the communication.

Different from existing graph regularization based approaches, we utilize supervised label information to build a more reliable topological graph, inducing the embedding features extracted by the encoders to preserve the geometric structure among samples.

2 Method

In this section, we comprehensively introduce our model from the following four aspects, namely, twochannel decoupling framework, weak label-guided graph regularization, random fragment masking mechanism, and multi-label classification. First, for the convenience of description, we briefly introduce the formal problem definition and common notations.

2.1 Problem definition

In this section, we define the iMvWMLC task as follows: Given multi-view multi-label data {{ X ( v ) } m v =1 , Y } ( m ) v =1 , where X ( v ) is v -th view's feature matrix containing n samples and m is the number of views. The i -th row of X ( v ) ∈ R n × d v denotes the instance of i -th sample on v -th view with dimension d v . Y ∈ { 0 , 1 } n × c is the label matrix with c categories and Y i,j = 1 denotes the i -th sample is tagged as j -th class. For missing views, we introduce W ∈ { 0 , 1 } n × m as the missing-view index matrix, where W i,j = 1 represent j -th view of i -th sample is available, otherwise W i,j = 0 . For missing labels, we let G ∈ { 0 , 1 } n × c be the missing-label index matrix, whose element G i,j = 1 means i -th sample's j -th label is known, otherwise G i,j = 0 . Finally, we fill the unavailable views and the unknown labels with '0' in the data pre-processing step to get final incomplete multi-view feature { X ( v ) } m v =1 and weak label matrix Y . Our goal is to train a neural network that can perform multi-label classification reasoning on incomplete multi-view data. Subscript access B i,j , B i, : , and B : ,j mean the element, row, and column of any matrix B .

2.2 Two-channel decoupling framework

As we know, due to the diversity of multi-view data acquisition methods, each view of the original data may have different feature dimensions, which is not conducive to the parallel execution of deep networks. To this end, we map the heterogeneous raw data into a unified embedding space with dimension d e through stacked encoders. Different from conventional deep multi-view networks, we

employ two groups of multi-layer perceptrons as two channels, shared channel and view-private channel, to extract shared information and view-proprietary information from raw data, respectively, i.e., { E S v : X ′ ( v ) → S ( v ) } m v =1 and { E O v : X ′ ( v ) → O ( v ) } m v =1 . E S v and E O v denote the shared feature encoder and view-proprietary feature encoder of view v , respectively. X ′ ( v ) represents the masked input (See Section 2.4). S ( v ) ∈ R n × d e and O ( v ) ∈ R n × d e are the extracted multi-view shared feature matrix and view-proprietary feature matrix of n samples. The architectures of these two channels are the same but serve different purposes. Specifically, E S v tries to mine common features across views, which preserve the sample's basic properties held by all views, while E O v focuses on extracting features unique to each view. This design decomposes the discriminative information of each view into two parts, which can satisfy the multi-view consistency and complementarity assumptions at the same time. Of course, it is difficult to achieve the above purposes with only two encoding channels due to the lack of guidance, so we propose a multi-view cross-channel contrastive loss L ccc to guide the separation of these two kinds of features as follows:

glyph[negationslash]

where S is the similarity measure function: S ( x, y ) = x T y ‖ x ‖ 2 ·‖ y ‖ 2 and ✶ [Υ] is the condition function: ✶ [Υ] = 1 if condition { Υ : W i,u W i,v = 1 } is true, 0 otherwise. N = ∑ u,v W i,u W i,v denotes the number of valid instance pairs. That is to say we only consider instance pairs that are not missing on both sides. s ( v ) i and o ( v ) i are features corresponding to i -th sample in the shared feature S ( v ) and view-proprietary feature O ( v ) , respectively. According to Eq. (1), our cross-channel contrastive loss consists of two parts, namely, the numerator is the mean of negative pairs' similarities and the denominator is that of positive pairs' similarities. To be more specific, we pair instances from 2 N channels together, where positive pairs consist of shared features from different views, and the remaining pairs are considered negative. We aim to maximize the similarity between the positive pairs of shared features and minimize the similarity between the negative pairs. This approach fulfills the design objective of the dual-channel model, which is to encourage consistency between shared features from different views while maintaining a clear distinction between each view's view-proprietary feature and other shared or view-proprietary features.

To further enhance the feature extraction capability of the encoders, we introduce stacked decoders to restore the embedding features extracted by the two-channel encoders to original feature space, i.e., { D v : S ( v ) ∈ R n × d e → ¯ X ( v ) ∈ R n × d v } m v =1 , where D v is the decoder corresponding to v -th view. ¯ X ( v ) is the reconstructed feature. Finally, we measure the reconstruction quality using the weighted mean square error loss:

where W is introduced to mask the unavailable instances. With the S ( v ) and O ( v ) on each view, we can naturally perform cross-view fusion to obtain the unique shared representation ¯ S and ¯ O of all samples [28, 20]:

Successively, we try to combine shared information and proprietary information to learn consistent representations of samples. Here, unlike the commonly used addition or connection operations, we adopt a novel feature interaction approach, i.e., enhancing the shared information via the proprietary information:

where θ is sigmoid activation function and Z ∈ R n × d e is the final fusion representation.

2.3 Weak label-guided graph regularization

Many unsupervised multi-view learning methods apply graph regularization technique to maintain the intrinsic structure of data, based on the assumption that two similar samples in the original feature

space should also be similar in the latent space, which has shown good performance in unsupervised scenarios [29-31]. When transferring this assumption to multi-label classification tasks with the characteristics of supervised tasks, we can get a new assumption: two similar samples in the label space should also be similar in the embedding space. Based on this, we attempt to guide the feature extraction process of the encoders by imposing a graph constraint with the supervised information. First, we calculate similarity matrix according to the weak label matrix Y :

where T ∈ [0 , 1] n × n is the sample similarity graph, describing the similarity between any two samples. C i,j = G i, : G T j, : is the number of labels available for both samples i and j . y i and y j denote the i -th and j -th rows of Y . And η is a constant, empirically set to 100 for simplicity. Taking into account the fact that C ij is much larger than y i y T j on most datasets, when the number of categories is large, even if two samples only have one same label, it means that they are much more similar than other sample pairs.

With such a similarity graph, we can constrain the distance between any two samples in the embedding feature space to achieve the purpose of structure preservation by following loss:

In view of Eq. (6), the more similar two samples are, the greater the penalty weight they are imposed. In order to improve the computational efficiency of the model in the GPU, we rewrite Eq. (6) in the form of matrix product:

where Tr ( · ) is the trace operation. L denotes the Laplacian matrix calculated by L = D -T , where D is a diagonal matrix with diagonal element D i.i = ∑ n j =1 T i,j .

2.4 Masking random fragment of feature

As we all know, the original data contains massive redundant information, which inevitably interferes with the feature extraction process of the model. Inspired by the well-known masked autoencoders (MAE) [27] that randomly mask image patches, we attempt to apply a masking mechanism to the input multi-view vector data. To this end, for any view v , we initialize a matrix M ( v ) ∈ { 0 , 1 } n × d v whose elements are all 1 at the same size as the raw data X ( v ) , and randomly sample n integers [ b 1 , b 2 , ..., b n ] from the range of [1 , d v -l ] as the mask starting point of each instance in view v , where l is the mask length. Then the b i -th to ( b i + l ) -th bits of feature M ( v ) i, : ∈ R d v are set to 0 to get the final mask matrix M ( v ) . This matrix stores consecutive mask segments starting at random, enabling local deactivation of feature vectors. We use m mask matrices to mask the original input data, and get corresponding masked feature matrices as the input of the MTD:

where glyph[circledot] means the element-wise multiplication.

At first glance, this is similar to MAE masking random image patches to encourage the model to learn useful features. The difference is that our MTD only partially masks the input features but does not do anything to the encoders, i.e., the encoders still works as if processing the unmasked data. In addition, in our model, we uniformly set the mask rate σ = l/d v to 0.25 for all datasets instead of a larger masking rate like MAE, because we notice the difference that the information density of vector data is much larger than that of image data. Extensive experiments are performed in Section 3.5 to verify the effectiveness of this masking approach.

2.5 Weighted multi-label classification and overall loss function

With the fused embedding representation Z , a fully connected layer is employed as a classifier to map Z to the label space to get the prediction result:

Algorithm 1 Training process of MTD

Input: Incomplete multi-view data { X ( v ) } m v =1 , missing-view index matrix W , weak label Y , missing-label index matrix G .

Output: Prediction results P .

1: Initialize model parameters and set hyperparameters ( α , β , γ , learning rate, and training epochs e ).

2: t=0.

3: while t < e do

4: Construct random feature mask matrices { M ( v ) } m v =1 .

5: Compute masked input data { X ′ ( v ) } m v =1 by Eq. (8).

6: Extract shared embedding feature { S ( v ) } m v =1 and view-private embedding feature { O ( v ) } m v =1 by two-channel encoders { E S v } m v =1 and { E O v } m v =1 , respectively.

7: Compute fused shared embedding feature ¯ S and fused view-proprietary feature ¯ O by Eq. (3).

8: Compute final fused representation Z by Eq. (4).

9: Compute similarity graph T by Eq. (5) and corresponding Laplacian matrix L .

10: Compute cross-channel contrastive loss L ccc by Eq. (1), reconstruction loss L re by Eq. (2), and graph embedding loss L gc by Eq. (7).

11: Obtain predictions P by Eq. (9) and compute classification loss L mc by Eq. (10).

12: Compute total loss L all by Eq. (11).

13: Update network parameters.

14: t = t +1 .

15: end while

where W ∈ R 2 d e × c and B ∈ R n × c are learnable parameters of classifier and P ∈ R n × c is our predicted score matrix. In single-label classification tasks, the cross-entropy loss is usually adopted to guide the learning of model, while in multi-label classification tasks, the prediction of each category is regarded as an independent binary classification problem. Besides, considering the unknown labels in label matrix, we use following multi-label cross-entropy loss as our main classification loss:

where G is introduced to mask the unknown label in the calculation of L mc .

Combined with cross-channel contrastive loss Eq. (1), reconstructed loss Eq. (2), graph embedding loss Eq. (7), and weighted multi-label classification loss Eq. (10), our total loss function can be expressed as:

where α , β , and γ are corresponding penalty parameters. We show the detailed training process in algorithm 1.

3 Experiments

3.1 Datasets and metrics

Consistent with existing iMvWMLC methods [26, 25, 20], we adopt five famous multi-view multilabel datasets 2 to validate our model, i.e., Corel5k [32], Pascal07 [33], ESPGame [34], IAPRTC12 [35], and MIRFLICKR [36]. Six kinds of features are extracted from these datasets as their six views, namely GIST, HSV, Hue, Sift, RGB, and LAB feature. The number of samples in these datasets ranges from 4999 to 25000, and the number of categories ranges from 20 to 291. Please refer to the appendix for more detailed information about the five datasets.

Following plenty of existing works [26, 25, 23], six metrics are selected to evaluate all comparison methods, i.e., Ranking Loss ( RL ), Average Precision ( AP ), Hamming Loss ( HL ), adapted area under curve ( AUC ), OneError ( OE ), and Coverage ( Cov ). To compare performance more intuitively, we actually report results with respect to 1-HL , 1-RL , 1-OE , and 1-Cov , so for all metrics, higher values indicate better performance.

AP

[0.31,0.52]

1-HL

[0.97,0.99]

1-RL

[0.87,0.93]

AUC

[0.87,0.93]

1-OE

[0.36,0.63]

1-Cov

[0.73,0.83]

GLOCAL

CDMM

DM2L

LVSL

iMVWL

NAIM3L

DICNet

MTD

AP

[0.47,0.65]

1-HL

[0.89,0.94]

1-RL

[0.76,0.89]

AUC

[0.79,0.90]

1-OE

[0.41,0.56]

1-Cov

[0.70,0.84]

GLOCAL

CDMM

DM2L

LVSL

iMVWL

NAIM3L

DICNet

MTD

(b) Pascal07

AP

[0.25,0.40]

1-HL

[0.96,0.98]

1-RL

[0.80,0.88]

AUC

[0.81,0.88]

1-OE

[0.34,0.55]

1-Cov

[0.54,0.69]

GLOCAL

CDMM

DM2L

LVSL

iMVWL

NAIM3L

DICNet

MTD

(c) ESPGame

AP

[0.25,0.44]

1-HL

[0.95,0.98]

1-RL

[0.84,0.92]

AUC

[0.84,0.92]

1-OE

[0.38,0.60]

1-Cov

[0.58,0.75]

GLOCAL

CDMM

DM2L

LVSL

iMVWL

NAIM3L

DICNet

MTD

(d) IAPRTC12

AP

[0.49,0.68]

1-HL

[0.84,0.91]

1-RL

[0.81,0.91]

AUC

[0.79,0.89]

1-OE

[0.52,0.75]

1-Cov

[0.58,0.72]

GLOCAL

CDMM

DM2L

LVSL

iMVWL

NAIM3L

DICNet

MTD

(e) MIRFLICKR

3.2 Incomplete multi-view weak multi-label dataset setting

Analogous to previous works [26, 25, 20], we manually construct incomplete multi-view weak multi-label datasets based on above five complete datasets to evaluate the performance of all methods in the case of missing views and missing labels. Concretely, 50% of instances on each view are randomly selected as unavailable instances, which are replaced by '0' value. And we ensure that there is no invalid sample in the dataset, i.e., each sample holds at least one unmissing view. For the weak labels, we set half of the positive and negative tags for each category to be unknown. 70% of samples with missing views and missing labels are randomly selected as the training set. The construction of incomplete data is repeated many times to reduce the randomness of experiment.

3.3 Comparison methods

In order to evaluate the advancement of our framework, eight top comparison methods are selected in our experiment, i.e., C2AE [37], GLOCAL [38], CDMM [18], DM2L [39], LVSL [23], NAIM3L [25], iMVWL [26], and DICNet [20], to compare with our MTD on the five incomplete multi-view weak multi-label datasets. To date, few MvMLC methods can handle missing views and weak labels at the same time, thus we introduce some related multi-label classification methods in our experiments. To be more specific, of the nine methods, iMVWL, NAIM3L, DICNet, and our MTD are perfectly applicable to the iMvWMLC task. C2AE, GLOCAL, and DM2L are single-view multi-label classification methods that can deal with incomplete labels, so we conduct these methods on each view and record the best result. CDMM and LVSL are the MvMLC methods unable to handle any data missing, so we simply complete the missing views using the mean of available instances and fill the unknown tags with '0'. The statistical information of competitors is shown in the appendix.

3.4 Experimental results and analysis

To evaluate the performance of our method in scenarios with missing views and labels, we compare it against eight state-of-the-art algorithms on five datasets. The mean and standard deviation (decimal form in the bottom right) of the results are reported in Table 1. In addition to the six performance metrics mentioned above, we also calculate the average ranking of each algorithm based on these

S_1-S_6              O_1-O_6

O_6-O_1              S_6-S_1

0.0

0.2

0.4

0.6

0.8

1.0

(a) Epoch 0

S_1-S_6              O_1-O_6

O_6-O_1              S_6-S_1

0.0

0.2

0.4

0.6

0.8

1.0

(b) Epoch 60

S_1-S_6              O_1-O_6

O_6-O_1              S_6-S_1

0.0

0.2

0.4

0.6

0.8

1.0

(c) Epoch 120

S_1-S_6              O_1-O_6

O_6-O_1              S_6-S_1

0.0

0.2

0.4

0.6

0.8

1.0

(d) Epoch 180

metrics, denoted as 'Ave.R', for intuitive comparison. Based on the results in Table 1, we make the following observations:

Our MTD is compelling, outperforming comparison methods on almost all metrics across all five datasets. In addition to the average ranking of 1.17 on the IAPRTC12 dataset, it

ranks 1st on the other four datasets. Whether compared with traditional methods (such as GLOCAL and NAIM3L) or deep methods (such as C2AE and DICNet), our MTD shows excellent compatibility with missing views and weak labels.

We can find that methods based on deep neural networks generally perform better than traditional methods. Specifically, our MTD, DICNet, and CDMM are consistently ranked in the top 3 across all five datasets, which can be attributed to the strong feature extraction capabilities of deep encoders and the flexible multi-view fusion strategy they employ.

In the comparison among traditional methods, the method that is specifically designed for double incompleteness shows obvious advantages compared to other methods. This indicates the necessity of considering possible missing views and labels during the model design process.

In addition to experimenting on datasets with 50% missing views and missing labels, we also perform comparison experiments on the full dataset without any missing data and use radar charts to show the results in Fig. (2). For more intuitive observation, we set the range for each axis to the min-max values of the nine methods on the corresponding metric. It can be seen from Fig. (2) that the performance of our method on the complete datasets is still outstanding, which verifies the good adaptation of it.

To demonstrate that our cross-channel contrastive loss works as expected, we plot a random sample's channel similarity heat maps across all channels in different training epochs on the Corel5k dataset with half of missing views and labels in Fig. (3). Each patch of the heat map shows the cosine similarity of corresponding two instances, i.e., i -th sample's heat map H i = CC T , where C = ( s (1) i , s (2) i , ..., s ( m ) i , o (1) i , o (2) i , ..., o ( m ) i ) T ∈ R 2 m × d e . s (1) i ∈ R d e and o (1) i ∈ R d e are the instances on shared and view-proprietary channels of first view of sample i , respectively. In Fig. (3), it is evident that in the initial state, the similarities between different channels are relatively small. However, as training progresses, the similarities between instance pairs on the first m shared channels increase rapidly, while the similarities for the remaining channels are close to zero. This observation indicates that our cross-channel contrastive loss is functioning as expected. Additionally, we notice that there is a shared feature on channel 3 that is not well-aggregated. This may be due to the third view of this sample sharing less information with the other views.

3.5 Ablation study

Our model contains multiple loss functions and strategies, in order to further study the effectiveness of each component, we perform extensive ablation experiments on the Corel5k dataset with 50% missing labels and views, and report results in Table 2. Specifically, we take the common single-channel model [20, 40] as a baseline architecture and extend it to a two-channel baseline framework. We iteratively add our individual loss functions L gc , L re , and L ccc into it and remove the masking operation from different versions. From Table 2, we observe that all designs play a role to varying degrees, especially the fragment masking mechanism's performance increase on the Corel5k dataset is the most noticeable. This confirms that the random masking mechanism to raw data, commonly used in the image and text analysis domain, is also effective in the field of multi-view learning. However, this strategy still needs to be further studied for the underlying mechanism of action.

3.6 Implementation details

Our TMD is implemented by Pytorch and MindSpore frameworks on Ubuntu operating system with a single RTX 3090 GPU and an i7-12900k CPU. The learning rate is set to 0 . 1 and the Stochastic Gradient Descent (SGD) optimizer is chosen for training model. The batch size and momentum are 128 and 0.9 for all five datasets. Please refer to the appendix for the setting of other hyperparameters. The code can be found at https://justsmart.github.io/ .

4 Conclusion

In this paper, we propose a novel masked two-channel decoupling framework (MTD) for the iMvWMLC task. Our MTD decouples single-channel features commonly used in previous methods into two-channel features for consensus and complementarity learning, and we design a cross-channel contrastive loss for this setting. Additionally, we employ a random fragment masking strategy during the training phase to reduce redundancy of raw data, inspired by image and text masking strategies, which leads to an obvious performance boost. Furthermore, we introduce a label-guided graph constraint approach to ensure that the learned embedding features maintain structure information among samples. Our extensive comparison and ablation experiments demonstrate that MTD outperforms other advanced methods and is compatible with arbitrary incomplete multi-view and weak multi-label data. Looking back at our work, although we propose many strategies and techniques to advance the frontier research progress of iMvWMLC, the complexity of this issue leaves room for further optimization, such as considering multi-label correlation and missing data recovery as directions of future works.

References

Yangbangyan Jiang, Qianqian Xu, Zhiyong Yang, Xiaochun Cao, and Qingming Huang. Dm2c: deep mixed-modal clustering. In International Conference on Neural Information Processing Systems , pages 5888-5892, 2019.

Erlin Pan and Zhao Kang. Multi-view contrastive graph clustering. Advances in neural information processing systems , 34:2148-2159, 2021.

Zhao-Min Chen, Xiu-Shen Wei, Peng Wang, and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 5177-5186, 2019.

Mouxing Yang, Yunfan Li, Zhenyu Huang, Zitao Liu, Peng Hu, and Xi Peng. Partially view-aligned representation learning with noise-robust contrastive loss. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition , pages 1134-1143, 2021.

Jie Wen, Zheng Zhang, Lunke Fei, Bob Zhang, Yong Xu, Zhao Zhang, and Jinxing Li. A survey on incomplete multiview clustering. IEEE Transactions on Systems, Man, and Cybernetics: Systems , 2022.

Chengliang Liu, Zhihao Wu, Jie Wen, Yong Xu, and Chao Huang. Localized sparse incomplete multi-view clustering. IEEE Transactions on Multimedia , 2022.

Xinwang Liu, Miaomiao Li, Chang Tang, Jingyuan Xia, Jian Xiong, Li Liu, Marius Kloft, and En Zhu. Efficient and effective regularized incomplete multi-view clustering. IEEE Transactions on Pattern Analysis and Machine Intelligence , 43(8):2634-2646, 2020.

Xiang Fang, Daizong Liu, Pan Zhou, and Yuchong Hu. Multi-modal cross-domain alignment network for video moment retrieval. IEEE Transactions on Multimedia , 2022.

Peng Hu, Dezhong Peng, Yongsheng Sang, and Yong Xiang. Multi-view linear discriminant analysis network. IEEE Transactions on Image Processing , 28(11):5352-5365, 2019.

Jie Xu, Huayi Tang, Yazhou Ren, Liang Peng, Xiaofeng Zhu, and Lifang He. Multi-level feature learning for contrastive multi-view clustering. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16051-16060, 2022.

Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification. In International Conference on Learning Representations , 2021.

Yang Wang. Survey on deep multi-modal data analytics: Collaboration, rivalry, and fusion. ACM Transactions on Multimedia Computing, Communications, and Applications (TOMM) , 17(1s):1-25, 2021.

Feiping Nie, Guohao Cai, and Xuelong Li. Multi-view clustering and semi-supervised classification with adaptive neighbours. In Proceedings of the AAAI conference on artificial intelligence , volume 31, 2017.

Zhenglai Li, Chang Tang, Xinwang Liu, Xiao Zheng, Wei Zhang, and En Zhu. Consensus graph learning for multi-view clustering. IEEE Transactions on Multimedia , 24:2461-2472, 2021.

Zongbo Han, Changqing Zhang, Huazhu Fu, and Joey Tianyi Zhou. Trusted multi-view classification with dynamic evidential fusion. IEEE transactions on pattern analysis and machine intelligence , 2022.

Yijie Lin, Yuanbiao Gou, Xiaotian Liu, Jinfeng Bai, Jiancheng Lv, and Xi Peng. Dual contrastive prediction for incomplete multi-view representation learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 45(4):4447-4461, 2022.

Min-Ling Zhang and Zhi-Hua Zhou. A review on multi-label learning algorithms. IEEE Transactions on Knowledge and Data Engineering , 26(8):1819-1837, 2013.

Dawei Zhao, Qingwei Gao, Yixiang Lu, Dong Sun, and Yusheng Cheng. Consistency and diversity neural network multi-view multi-label learning. Knowledge-Based Systems , 218:106841, 2021.

Jun Huang, Guorong Li, Qingming Huang, and Xindong Wu. Learning label specific features for multi-label classification. In IEEE International Conference on Data Mining , pages 181-190, 2015.

Chengliang Liu, Jie Wen, Xiaoling Luo, Chao Huang, Zhihao Wu, and Yong Xu. Dicnet: Deep instance-level contrastive network for double incomplete multi-view multi-label classification. In Proceedings of the AAAI conference on artificial intelligence , 2023.

Gengyu Lyu, Xiang Deng, Yanan Wu, and Songhe Feng. Beyond shared subspace: A viewspecific fusion for multi-view multi-label learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 36, pages 7647-7654, 2022.

Arthur Gretton, Olivier Bousquet, Alex Smola, and Bernhard Schölkopf. Measuring statistical dependence with hilbert-schmidt norms. In International conference on algorithmic learning theory , pages 63-77. Springer, 2005.

Dawei Zhao, Qingwei Gao, Yixiang Lu, and Dong Sun. Non-aligned multi-view multi-label classification via learning view-specific labels. IEEE Transactions on Multimedia , 2022.

Chengliang Liu, Jie Wen, Xiaoling Luo, and Yong Xu. Incomplete multi-view multi-label learning via label-guided masked view- and category-aware transformers. In AAAI Conference on Artificial Intelligence , 2023.

Xiang Li and Songcan Chen. A concise yet effective model for non-aligned incomplete multiview and missing multi-label learning. IEEE Transactions on Pattern Analysis and Machine Intelligence , 44(10):5918-5932, 2022.

Qiaoyu Tan, Guoxian Yu, Carlotta Domeniconi, Jun Wang, and Zili Zhang. Incomplete multiview weak-label learning. In International Joint Conference on Artificial Intelligence , pages 2703-2709, 2018.

Kaiming He, Xinlei Chen, Saining Xie, Yanghao Li, Piotr Dollár, and Ross Girshick. Masked autoencoders are scalable vision learners. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition , pages 16000-16009, 2022.

Jie Wen, Zheng Zhang, Zhao Zhang, Zhihao Wu, Lunke Fei, Yong Xu, and Bob Zhang. Dimcnet: Deep incomplete multi-view clustering network. In ACM International Conference on Multimedia , pages 3753-3761, 2020.

Peng Zhou, Liang Du, Yi-Dong Shen, and Xuejun Li. Tri-level robust clustering ensemble with multiple graph learning. In Proceedings of the AAAI Conference on Artificial Intelligence , volume 35, pages 11125-11133, 2021.

Peng Zhou, Liang Du, Xuejun Li, Yi-Dong Shen, and Yuhua Qian. Unsupervised feature selection with adaptive multiple graph learning. Pattern Recognition , 105:107375, 2020.

Kun Zhan, Feiping Nie, Jing Wang, and Yi Yang. Multiview consensus graph clustering. IEEE Transactions on Image Processing , 28(3):1261-1270, 2018.

Pinar Duygulu, Kobus Barnard, Joao FG de Freitas, and David A Forsyth. Object recognition as machine translation: Learning a lexicon for a fixed image vocabulary. In European Conference on Computer Vision , pages 97-112, 2002.

Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International journal of computer vision , 88:303-308, 2009.

Luis Von Ahn and Laura Dabbish. Labeling images with a computer game. In SIGCHI Conference on Human Factors in Computing Systems , pages 319-326, 2004.

M Henning, D Thomas, et al. The iapr benchmark: Anew evaluation resource for visual information systems. In International Conference on Language Resources and Evaluation , pages 1-11, 2006.

Mark J Huiskes and Michael S Lew. The mir flickr retrieval evaluation. In ACM International Conference on Multimedia Information Retrieval , pages 39-43, 2008.

Chih-Kuan Yeh, Wei-Chieh Wu, Wei-Jen Ko, and Yu-Chiang Frank Wang. Learning deep latent space for multi-label classification. In AAAI Conference on Artificial Intelligence , volume 31, 2017.

Yue Zhu, James T Kwok, and Zhi-Hua Zhou. Multi-label learning with global and local label correlation. IEEE Transactions on Knowledge and Data Engineering , 30(6):1081-1094, 2017.

Zhongchen Ma and Songcan Chen. Expand globally, shrink locally: Discriminant multi-label learning with missing labels. Pattern Recognition , 111:107675, 2021.

Jie Wen, Chengliang Liu, Shijie Deng, Yicheng Liu, Lunke Fei, Ke Yan, and Yong Xu. Deep double incomplete multi-view multi-label learning with incomplete labels and missing views. IEEE transactions on neural networks and learning systems , PP, 2023.

A Datasets

These datasets are uniformly extracted six kinds of feature as six views from five image datasets, i.e., GIST, HSV, Hue, Sift, RGB, and LAB. Table 3 shows the statistical information of the five datasets.

B Comparison Methods

C Hyperparameters study

The overall loss of our model mainly contains 3 hyperparameters, i.e., α , β , and γ . In order to study the impact of these three hyperparameters on the performance of the model, we iteratively set different values and report the corresponding AP values in Fig. 4:

In Fig. 4, we show the AP value with respect to different parameter selections on Corel5k and Pascal07 datasets with 50% missing views and labels. Specifically, from Fig. 4a and 4b, we recommend selecting α from the ranges of [1 e -4 , 1 e -3] and [1 e -4 , 1 e 0] for Corel5k and Pascal07 datasets respectively; selecting β from the ranges of [1 e -3 , 1 e -1] and [1 e -3 , 1 e -1] for Corel5k and Pascal07 datasets respectively. For convenience, we uniformly set α = 0 . 4 and β = 0 . 4 in our experiments. For γ , our method is not sensitive to it, so we set γ = 0 . 1 for all five datasets.

From Fig. 4d and Fig. 4e, it can be seen that too large or too small mask rate is not conducive to achieving optimal performance. Based on experiments, we recommend setting a mask rate from 0.2 to 0.4 (0.25 is used for all datasets in our experiments for convenience).

D Time cost

To study the training and test efficiency of our MTD, we report the time cost of the training and test phases of the six supervised methods on the Corel5k dataset with 70% training samples in Table 5. The convergence setting of the model plays a critical role in determining the training time required. Therefore, we measure the running times of all methods under their default convergence conditions.

0

0.2

1e-4

AP

1e-3

0.4

1e0

-

1e-2

1e-1

,

1e-1

1e-2

1e0

1e-3

1e-4

0

0.1

0.2

0.3

0.4

0

0.2

1e-4

AP

0.4

1e-3

0.6

1e0

-

1e-2

1e-1

,

1e-1

1e-2

1e0

1e-3

1e-4

0

0.1

0.2

0.3

0.4

0.5

(b) Pascal07

0.35

0.40

0.45

0.50

0.55

1e0

1e-1

1e-2

1e-3

1e-4

1e-5

AP

γ

Pascal07

(c) Corel5k and Pascal07

0.391

0.390

0.394

0.399

0.390

0.360

0.339

0.33

0.34

0.35

0.36

0.37

0.38

0.39

0.40

0.41

0.1

0.2

0.3

0.4

0.5

0.6

0.7

AP

𝜎

0.539

0.550

0.547

0.547

0.540

0.527

0.519

0.50

0.51

0.52

0.53

0.54

0.55

0.56

0.57

0.58

0.1

0.2

0.3

0.4

0.5

0.6

0.7

AP

𝜎

(e) Pascal07

For methods that process a single view at a time, we report both the total training time for all views and the test time for a single view. For DICNet and MTD, we report the time spent in 100 epochs. In order to ensure consistent hardware conditions, all tests are carried out on the same computer. From the results shown in Table 5, it can be observed that our deep learning-based method requires longer training times, but reaches higher performance than other methods.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe core innovation of the method lies in decoupling the single-channel view-level representation, which is common in deep multi-view learning methods, into a shared representation and a view-proprietary representation with a cross-channel contrastive loss. The authors have conducted sufficient experiments to verify the effectiveness of the proposed method.\n\nSTRENGTHS:\n1. The definition of the problem in this paper is clear. The topic is novel, and as far as I know, multi-view multi-label classification methods for the complex situation of missing labels and data are underexplored.\n\n2. The interpretation of extracting multi-view features from two perspectives is intuitive and fits with the fundamental assumption of multi-view/multi-modal learning problems that there are commonalities as well as differences between modalities.\n\n3. The experimental results are sufficient. The authors not only provide the results in the missing case, but also provide the results of the proposed model on the complete datasets, which shows that the model has good learning ability for multi-view data.\n\nWEAKNESSES:\n1. In Eq.(1), the necessary explanation about the coefficient 2 in the numerator is lacking. The authors should also provide more explanation about the motivation of such design.\n\n2. A clarification on the value of mask rate $\\sigma$ is lacking.\n\n3. The authors propose to build two groups of encoder-decoder for different views, but the computational complexity may increase with the number of views quickly.\n\n4.  There are some unclear expressions, for instance, in Section 2.2, the authors say “Different from conventional deep multi-view networks”. So what are the conventional deep multi-view networks? There is no corresponding reference here.\n\nQUESTIONS:\nIs it efficient to address the missing problem by introducing additional prior information, that is matrix G and W in this paper?\n\nLIMITATIONS:\nThe authors have discussed the limitations and suggested further directions for improvement in the Conclusion.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-27"}

comments: SUMMARY:
This paper proposes a general framework for missing multi-view and missing multi-label classification. The main innovation is to  decouple each view feature into two different  shared and private features. The paper also applies random masks to the raw features to make the network learn from limited information, and shows the benefits of this approach with ablation experiments.

STRENGTHS:
- This paper is clear in terms of writing and presentation. It is easy for the reader to understand the author's motivation.

- The problem addressed by the authors is complex (doubly incomplete data) and poses a greater challenge than methods based on ideally complete data. However, judging from the results, the author solves this problem well.

- The method of using labels to constrain the feature extraction is novel, and the author's local masking of input features has also brought performance growth. I think there are simple and effective techniques that can be extended to other fields.

- The experimental results show that the MTD has significant advantages over the existing multi-view multi-label classification methods.

WEAKNESSES:
- The one-HL metric does not seem to be of much significance because it does not effectively evaluate the performance of different methods according to the author's experimental results.
- It is inappropriate to appear both ‘Figure’ and ‘Fig.’ in the text.
- The author should clearly explain the meaning of Figure 3 in the caption.

QUESTIONS:
- Are these masks fixed during training? How exactly did the authors set up the mask?
- Why the results of iMVWL and NAIM3L are not consensus with original papers?

LIMITATIONS:
N/A
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 4
DATE: 2023-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a new framework for incomplete multi-view weak multi-label classification (iMvWMLC), which is a challenging task that involves missing views and labels in the data. The framework consists of four main components: a two-channel decoupling mechanism (called MTD) that extracts shared and view-specific features from each view, a cross-channel contrastive loss that enhances the semantic separation of the two channels, a random fragment masking strategy that reduces the redundancy of the input features, and a label-guided graph regularization loss that preserves the geometric structure among samples. The paper evaluates the framework on five datasets and compares it with eight state-of-the-art methods, showing that it outperforms them on various metrics and is robust to different levels of incompleteness.\n\nSTRENGTHS:\n(1) As far as I know, research on multi-view multi-label classification is still in its infancy, and this article focuses on this topic very well. \n\n(2) To a certain extent, the logic and expression of this article are clear.\n\n(3) I am interested in masking operations applied to vectorized feature data, most multimodal methods focus more on input real data such as text or images. I agree that the use of the mask mechanism on feature data is instructive.\n\nWEAKNESSES:\n(1) The similarity matrix calculated based on weak tags does not take into account the interference caused by unknown tags. The paper should address the potential interference caused by unknown tags in the calculation of the similarity matrix. \n\n(2) The results of the ablation experiment suggest that the so-called contrast loss does not seem to have improved much, at least on the Corel5k dataset. The paper should provide some explanation.\n\n(3) The paper needs to improve the details, as there is a error in section 3.4 where \"TMD\" is incorrectly referenced. The authors could correct this and prevent similar errors.\n\n(4) The reviewer has noticed that there are some recent works aiming at solving the multi-view classification problem. The paper could do a better job by citing the different relevant works, e.g., [1-3], and clarifying the differences between this work and them.\n\n[1] Trusted Multi-View Classification, ICLR’21\n\n[2] Partially View-aligned Representation Learning with Noise-robust Contrastive Loss, CVPR’21\n\n[3] Dual Contrastive Prediction for Incomplete Multi-View Representation Learning, TPAMI'23\n\nQUESTIONS:\n(1) It would be interesting to know if the authors have experimented with other masking methods, such as random bit masking of features, and what the results of such experiments were.\n\n(2) Availability of code is important for reproducibility and further research. There is no code in the supplement material. Do the authors have any plans to make the code public?\n\n(3) The paper should provide a more detailed discussion on why not all comparison methods were adapted to this task.\n\nLIMITATIONS:\nFor missing multi-view data without a priori missing information, this method seems to be difficult to work.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-03"}

comments: SUMMARY:
This paper studies incomplete multi-view weak multi-label learning problem, which is important. The authors propose a masked two-channel decoupling framework based on deep neural networks. They develop cross-channel contrastive loss, a label- guided graph regularization loss, and random fragment masking strategy. The experiments validate the effectiveness.

STRENGTHS:
- The problem is interesting.
- The techniques in this work seem correct and reasonable.

WEAKNESSES:
- This work just combines some existing widely-used techniques. Thus the work is a bit incremental, does not provide new insights to me.
- As stated in abstract, the core is to decouple view-level representation into  shared representation and a view-proprietary representation. This idea has been widely used in existing works.
- The authors first fill the missing views with noise, and missing label with 0, and then treat it as normal complete view learning problem. To me, this scheme seems not grasp the essence of multi-view weak multi-label learning problem.
- The aim and construction of contrastive loss in this work seem a bit confusing to me. It is suggested to explain it in more detail.

QUESTIONS:
See weakness part

LIMITATIONS:
The author seems to lack a comprehensive discussion on the limitations of the proposed method.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 2
DATE: 2023-07-12