Deep Incomplete Multi-View Network Semi-Supervised Multi-Label Learning with Unbiased Loss

Authors: quanjiang  li
tingjin  luo
mingdie  jiang
jiahui  liao
zhangqi  jiang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-08 04:46:13

Abstract: Due to the explosive growth in data sources and label categories,
multi-view multi-label learning has garnered widespread attention.
However, multi-view multi-label data often exhibits incomplete
features and few labeled instances alongside a huge number of unlabeled instances, due to the technical limitations of data collection
and high annotation cost of manual labeling in practice. Learning
for such simultaneous missing of view features and labels is crucial
but rarely studied, particularly when the labeled samples with full
observations are limited. In this paper, we tackle this problem by
proposing a novel Deep Incomplete Multi-View Semi-Supervised
Multi-Label Learning method (DIMvSML). Specifically, to improve
high-level representations of missing features, DIMvSML firstly employs deep graph networks to recover the feature information with
structural similarity relations. Meanwhile, we design the structure-specific deep feature extractors to obtain discriminative information
and preserve the cross-view consistency for the recovered data with
instance-level contrastive loss. Furthermore, to eliminate the bias
of the estimate of the risk that the semi-supervised multi-label
methods minimise, we design a safe risk estimate framework with
an unbiased loss and improve its empirical performance by using pseudo-labels of unlabeled data. Besides, we provide both the
theoretical proof of better estimate variance and the intuitive explanation of our debiased framework. Finally, extensive experimental
results on public datasets validate the superiority of DIMvSML
compared with state-of-the-art methods.

Main paper: 
 introduction: 
 Multi-label learning has attracted increasing attention due to its widespread application in areas such as text classification [3, 34] , image annotation [2, 31] , and computer vision tasks [7, 10] . Furthermore, with the exponential increase of data sources and feature extraction methods, it is no longer adequate to describe and analyze instances from a singular perspective [8] . In real-world applications, objects are usually processed in multiple views, like face information captured by diverse sensors and image data stored using both video and audio techniques. Doubtlessly, the utilization of multi-view data enables comprehensive and accurate description of observed instances [9, 45] . Besides, multi-view data offers abundant data presentation modes, which can be combined with multi-label to represent the rich information content and semantic structure of complex data [39, 44] . Therefore, this paper focuses on the multi-view multi-label classification task, namely MVMLC.For MVMLC, many methods have been proposed, such as the manifold regularization MVMLC [25] , potential semantic-aware LSA-MML [41] and label-embedding based method [46] . However, these traditional methods assume that the given data has complete views and labels, which is violated in practice. On the one hand, the heterogeneous data collected from multiple sources may contain missing views due to the quality of storage equipment and the difficulty of storage methods [4] . For instance, in multi-view multimedia annotation tasks, video, audio and subtitle serve as distinct views. It is common to face situations where not all multimedia content encompasses all three views [37] . On the other hand, manual tagging of all labels being both challenging and expensive, label information in real datasets frequently exhibits varying degrees of incompleteness [23] . Clearly, the absence of views and labels detrimentally impacts MvMLC. In recent years, the challenges posed by the two types of missing have received widespread attention and some works which can simultaneously handle both issues have been proposed. iMvWL [33] combined two weighted matrix factorization models into a unified framework to extract the consensus representation and derive a subspace from missing views with multiple labels. NAIML [17] exploited both consensus across multiple views and the global and local structures among multiple labels from rank constraint. By incorporating missing information into the weighted fusion and classification module, the proposed DD-IMvMLC [37] could effectively explore available data and label information to obtain the discriminative feature extractor and classifier. DICNet [21] emphasized the utilization of stacked auto-encoders to exploit the high-level semantic representations of samples. Besides, it introduced an incomplete instance-level contrastive learning to capture consistent representations. Wen et al. [22] proposed LMVCAT, which employed two transformer-style modules for cross-view feature aggregation and multi-label classification and utilized an adaptively weighted view fusion module to obtain view-consistent embedding features.These MVMLC methods under incomplete data only focus on the partial absence of multiple labels and each instance has a subset of labels that can be utilized to infer the missing ones. However, multiview data not only suffers from missing features but also encounters unlabeled instances in reality. For example, the features and labels of tumor patient data are obtained from various examinations and tests [27] . Certain patients may face limitations in undergoing MRI or PET scans, or in providing blood samples for laboratory testing, thereby leading to corresponding views being unavailable. Besides, factors such as research focus, resource constraint, and annotation complexity greatly contribute to patient labels, including tumor types, grading, treatment responses and so on, remaining unannotated. In aggregate, real-world datasets often present simultaneous problems of missing views and numerous unlabeled instances as shown in Fig. 1 . There are few methods available today that can effectively address both issues. We are all aware that in supervised multi-label scenarios, correlations within labels are beneficial for label recovery. However, in semi-supervised problem [24, 32] , where a sample lacks any annotations, solely considering methods for handling missing from label relevance is simplistic. In fact, to address the problem of semi-supervised multi-label under incomplete views, we need to focus on the following three aspects: i) Since the stability of the model will be seriously affected by the limited availability of data, we should consider maximizing the reconstruction of missing information. Rather than establishing losses only on observed data [38, 47] , we should enhance feature semantic from a data recovery perspective and maintain the stability of the subsequent modules [16] . ii) High-level representations should be explored to improve the contribution of features to classification with limited labeled instances. iii) Improperly incorporating unlabeled data to construct an unsupervised loss undermines the unbiased estimation of ideal risk from supervised losses, which makes the algorithm often lack the support of statistical theory [35, 43] . Besides, it introduces bias to the solution of supervised losses optimization, resulting in potential performance degradation of semi-supervised methods. In Fig. 2 , we select ten relatively balanced labels of Yeast to perform the experiment of label distribution simulation and only set 20% of the test data to be labeled. We can observe that when introducing unlabeled data and learning with traditional biased semi-supervised losses, the performance (marked in purple) tends to be worse than using only supervised data (marked in green). Moreover, the harm caused by the bias expands as the number of label categories increases, which needs to be controlled in multi-label learning. Therefore, the third aspect is to improve the framework of loss functions to make semi-supervised model safe and robust.To tackle these problems, we propose a novel deep incomplete multi-view semi-supervised multi-label learning method named DIMvSML. Specifically, for mitigating the negative influence of missing views, DIMvSML employs the Graph Neural Network to recover the missing data by capitalizing on the existing similarity relations. Based on the auto-encoder structures, we design feature extractors and decoder networks to learn high-level semantic and discriminative representations from all views. In addition, to preserve the cross-view consensus, we adopt the instance-level contrastive loss to enhance the mutual information between different views. An unbiased version of loss function is designed to eliminate the risk of latent downgrade due to the introduction of the unlabeled data and we prove theoretically that this framework also has a lower estimate variance. As depicted in Fig. 2 , utilizing our unbiased loss function leads to stable performance compared to the traditional semi-supervised loss when using unlabeled data. During training, the pseudo-labels are assigned to explore the additional supervisory information contained in unlabeled data. The main contributions of our work are summarised as follows:• We propose the DIMvSML to solve this crucial, but rarely studied problem. To our knowledge, this is the first GNNbased multi-view multi-label learning framework capable of handling both incomplete views and few labeled instances. • DIMvSML is a unified framework designed to recover the absent views, preserve the high-level semantic representations with cross-view consensus and provide a safe risk estimation framework simultaneously. • Extensive experimental results present that our DIMvSML outperforms other compared approaches in almost all cases, demonstrating its superiority and effectiveness. 
methodology: 
 Notations and Problem Formulation. Suppose an incomplete multi-view dataset with 𝑁 instances and 𝑉 views, i.e., X = {𝑿 (𝑣) } 𝑉 𝑣=1 , where 𝑿 (𝑣) = {𝒙 (𝑣) 𝑖 } 𝑁 𝑖=1 ∈ R 𝑁 ×𝑑 𝑣 is the 𝑑 𝑣 dimensional feature matrix of the 𝑣-th view. Let 𝒀 ∈ {0, 1} 𝑁 ×𝐶 represent the label matrix and 𝐶 is the number of categories. Besides, 𝒚 𝑖 ∈ {0, 1} 𝐶 is a row vector that denotes the label of the 𝑖-th instance. 𝑴 ∈ R 𝑁 ×𝑉 is an indicator matrix, where 𝑴 𝑖,𝑗 = 1 indicates that the 𝑖-th instance has the feature of the 𝑗-th view, otherwise 𝑴 𝑖,𝑗 = 0 means the feature is missing and set as 'NaN'. For convenience, we define 𝑛 𝑙 and 𝑛 𝑢 as the number of labeled and unlabeled instances, which satisfies 𝑛 𝑙 ≪ 𝑁 and 𝑛 𝑙 + 𝑛 𝑢 = 𝑁 . We also denote L and U as the index spaces for labeled and unlabeled instances, respectively.Obviously, the missing features will affect the learning of complementary and consistency across all views and prevent subsequent modules from using valid information. Furthermore, the scarcity of labeled samples will seriously limit the learning of multi-label semantics, which demands better utilization of unlabeled data to prevent the harm caused by the bias of the semi-supervised loss. To address these challenges, we propose a novel deep learning framework named DIMvSML. The main framework of our DIMvSML is illustrated in Fig. 3 . Specifically, DIMvSML consists of three main modules: (a) GNN-based feature completion module for recovering the incomplete feature information; (b) Multi-view representation learning module for capturing the high-level semantic representations and discriminative information from all views; (c) Safe semisupervised multi-label learning module for providing an unbiased semi-supervised risk estimator with lower variance.Since the absence of features will lead to the poor performance of deep learning [30] , data recovery is required to realize data augmentation. Recently, GNN-based approaches have garnered attention in data recovery owing to their capacity to extract the geometric details embedded within data [19, 36] . Sato [28] further presented theoretical evidence that substantiates the effectiveness of GNNs in recovering hidden features. Therefore, we employ GNNs to recover the missing views by leveraging the similarity relations between the available data.Firstly, we construct the view-specific graph 𝑺 (𝑣) ∈ R 𝑁 ×𝑁 through 𝑘-nearest neighbors (𝑘-NN) algorithm. 𝑺 (𝑣) demonstrates the relations between the corresponding instances of existing data in the 𝑣-th view, where 𝑺(𝑣) 𝑖,𝑗 = 1 means 𝑴 𝑖,𝑣 𝑴 𝑗,𝑣 = 1 and 𝒙 (𝑣) 𝑗 is the neighbor of 𝒙 (𝑣)𝑖 . Considering the consistency across multiple views, similarity relations between instances in existing views are valid for the missing views. Therefore, we transfer the established graph relations to find the available instances that are associated with the missing ones in each view. Then the transferred 𝑘-NN graph can be obtained byEQUATIONwhere operator diag(•) forms a diagonal matrix, and 𝑴 :,𝑣 denotes the 𝑣-th column of the matrix 𝑴. Secondly, we employ 𝑲 (𝑣) as the adjacency matrix and the related existing features as the iuput nodes in each view-specific GNN to recover the missing data. After the propagation of relation information over 𝑲 (𝑣) in the first layer of the GNN, the initially reconstructed data can be obtained byEQUATIONwhere 𝒃 𝑣 and 𝝎 𝑣 denote the bias and transformation matrix of the 𝑣-th view, respectively. In our experiments, we set 𝜎 as the rectified linear unit (ReLU) activation function. Finally, we combine the reconstructed missing data with the existing data to acquire the recovered matrices { 𝑿 (𝑣) } 𝑉 𝑣=1 . Moreover, in order to consolidate the recovery performance, we minimize the rebuilding loss 𝐿 rb only over recovered data as follows:𝐿 rb = 𝑉 ∑︁ 𝑣=1 ∑︁ 𝑲 (𝑣) 𝑖,𝑗 ≥1 𝑿 (𝑣) 𝑖,: -𝒙 (𝑣) 𝑗 2 2(1 -𝑴 𝑖,𝑣 ).(3)Due to the small number of labeled instances in semi-supervised learning (SSL), we need an efficient representation module to simultaneously explore the high-level semantic of features, unique characteristic for each view and substantial connections between features and labels. Therefore, we adopt the deep neural network rather than the shallow linear model for adaptively extracting the advanced representations. Besides, considering that different views have their distinctive characteristics, the feature extraction network 𝐸 (𝑣) (•) and decoder network 𝐷 (𝑣) (•) should be tailored for each view. Following [37, 40] , we adopt the well-known network structure of stacked auto-encoder. These view-specific networks are all composed of multi-layer perceptrons but with different hidden layer dimensions. By constructing such network structure, we can effectively capture the discriminative information inherent in each view. Therefore, we set the structure of both the 𝐸 (𝑣) (•) and 𝐷 (𝑣) (•) as four stacked linear layers with ReLU activation functions i.e., {Linear, ReLU, Linear, ReLU, Linear, ReLU, Linear}. Specifically, for the 𝑑 𝑣 dimensional feature data of the 𝑣-th view, the dimensions of the four linear layers in the encoder network are adaptively set as 0.8𝑑 𝑣 , 0.8𝑑 𝑣 , 1500, and 𝑑, where 𝑑 is the corresponding dimension of the last layer and can be adjusted according to the number of label categories. In reverse, the dimensions of the decoder network 𝐱 𝟏 (v) 𝐱 𝟐 (v) 𝐱 𝟑 (v) 𝐱 𝟒 (v) 𝐱 𝟓 (v) 𝐱 𝟔 (v) 𝐱 𝟏 (V) 𝐱 𝟐 (V) 𝐱 𝟑 (V) 𝐱 𝟒 (V) 𝐱 𝟓 (V) 𝐱 𝟔 (V) 𝐱 𝟏 (1) 𝐱 𝟐 (1) 𝐱 𝟑 (1) 𝐱 𝟒 (1) 𝐱 𝟓 (1) 𝐱 𝟔 (1) 𝐱 𝟏 (v) 𝐱 𝟐 (v) 𝐱 𝟑 (v) 𝐱 𝟒 (v) 𝐱 𝟓 (v) 𝐱 𝟔 (v) 𝐱 𝟏 (V) 𝐱 𝟐 (V) 𝐱 𝟑 (V) 𝐱 𝟒 (V) 𝐱 𝟓 (V) 𝐱 𝟔 (V) 𝐗 1 𝐗 v 𝐗 V Transfer Graph Generation Transfer Graph Generation … … GNN 1 GNN v GNN V (a) GNN-based Feature Completion Module 𝐄 1 𝐄 v 𝐄 V 𝐙 1 𝐙 v 𝐙 V classifier softmax softmax … … 𝐃 1 𝐃 V 𝐋 rc ෩ 𝐗 1 ෩ 𝐗 v ෩ 𝐗 V … … Concatenate 𝐋 rcEQUATIONTo integrate the coded features, we obtain a common representation matrix 𝒁 ∈ R 𝑑𝑉 ×𝑁 by concatenating the view-specific representations, , which maximizes the retention of recovered features while minimizing complexity. For the purpose of effectively exploring the correlations between features and labels, we employ deep network classifier which transforms the feature space into a probability space associated with the labels, i.e., the elements of the output can be regarded as the probability of the instance to the corresponding label. Therefore, we design the classifier as the combination of two stacked linear layers and a Sigmoid activation function, where the dimensions are set as 𝑑/2 and 𝐶. For improving prediction accuracy, we use the cross entropy [5] to guide model training. Suppose 𝑭 ∈ R 𝑁 ×𝐶 is a prediction matrix generated by classifier, we employ the following loss for labeled instances:𝐿 bce = - 1 𝑛 𝑙 𝐶 ∑︁ 𝑖 ∈ L 𝐶 ∑︁ 𝑗=1 𝑌 𝑖 𝑗 log 𝐹 𝑖 𝑗 + 1 -𝑌 𝑖 𝑗 log 1 -𝐹 𝑖 𝑗 . (5)To prevent the introduction of unlabeled data from causing performance degradation, we provide a safe semi-supervised loss function framework in this section. Assume that the shared parameters of the whole network is Θ and the ultimate objective of our training framework is to minimise a ideal risk R over a data distribution 𝑝 (𝑥, 𝑦). Since the distribution 𝑝 (𝑥, 𝑦) is unknowable, we generally minimise a empirical risk R (Θ), which acts as a surrogate for R and is computed on a sample of 𝑁 i.i.d points drawn from 𝑝 (𝑥, 𝑦). Considering supervised losses from each labeled instance, we suppose𝐿 rc = 1 𝑛 𝑙 𝑖 ∈ L 𝐺 (Θ; 𝑥 𝑖 ) and 𝐿 bce = 1 𝑛 𝑙 𝑖 ∈ L 𝑇 (Θ; 𝑥 𝑖 ; 𝑦 𝑖 ).Besides, we let loss 𝐿 (Θ; 𝑥 𝑖 ; 𝑦 𝑖 ) = 𝐺 (Θ; 𝑥 𝑖 ) + 𝜆 1 𝑇 (Θ; 𝑥 𝑖 ; 𝑦 𝑖 ). Then the supervised risk to minimise isEQUATIONThis traditional supervised risk estimate is unbiased and converges wisely to R (Θ). However, a notable limitation of this framework under semi-supervised problem is that a considerable amount of unlabeled data is not utilized. Therefore, we employ the instancelevel contrastive loss [20] on the unlabeled data to maximize the mutual information between the representations of different views.To calculate the mutual information, we use a Softmax activation function 𝜎 𝑠 at the last layer of the encoder and then we obtain that 𝒛(𝑣) 𝑖 = 𝜎 𝑠 (𝒛 (𝑣)𝑖 ), which is treated as a distribution probability vector [14] . In other words, 𝒛 (𝑣) and 𝒛 (𝑣 * ) (1 ≤ 𝑣 < 𝑣 * ≤ 𝑉 ) can be seen as the distribution of two discrete cluster assignment variables over 𝑑 classes. Therefore, we can compute the joint probability distribution as below:EQUATIONThen the mutual information between the 𝑣-th and 𝑣 * -th view can be calculated throughEQUATIONwhere 𝑷 (𝑣) and 𝑷 (𝑣 * ) are the marginal probability distribution of the 𝑣-th and 𝑣 * -th view. In our experiments, we simply fix the balanced parameter 𝛼 to 9. The loss 𝐿 𝑖𝑐𝑙 -under unlabeled data can be obtained by enumerating the mutual information between different views, i.e.,EQUATIONTo facilitate subsequent analysis, we turn 𝐿 icl -into the form 𝐿 icl -= 1 𝑛 𝑢 𝑖 ∈ U 𝐻 (Θ; 𝑥 𝑖 ). The concrete form of the three instance-level losses 𝐺 (Θ; 𝑥 𝑖 ), 𝑇 (Θ; 𝑥 𝑖 ; 𝑦 𝑖 ) and 𝐻 (Θ; 𝑥 𝑖 ) is easily acquired and will be listed in Appendix. After introducing unlabeled data , we aim to minimise the SSL risk :EQUATIONSince unlabeled data lacks the necessary labeled guidance for classification, introducing it often carries the risk of potential performance degradation, especially when the data distribution assumption is not satisfied [18] . Even though the learning methods presented in the Eq. ( 10) can handle some basic SSL problems, the SSL risk estimate is biased or even asymptotic, which not only hinders the use of statistical learning theory, but also damages the actual effect of the model [26] . Moreover, the biased harm becomes prominent when the number of label categories increases. Hence, in multi-label classification, we should compensate for this bias in the loss function. Inspired by Hugo et al. [29] , we obtain the following unbiased version of the SSL estimator:EQUATIONThis framework uses labelled data to annul the bias, which do not rely on data distribution assumption. In addition to being unbiased, this framework also has favorable estimate variance. To measure the variance of the risk estimate in the Eq.( 11), we require information regarding instance tagging. Therefore, we introduce a binary random variable 𝑟 ∼ B (𝜋) that states whether or not a data point is labelled. 𝑟 𝑖 = 1 denotes the 𝑖-th instance is labeled and 𝑟 𝑖 = 0 denotes missing. 𝜋 ∈ (0, 1) is the probability of being labelled. Under the assumption that the missingness of a label is independent of its feature and value, we can obtain the following theorem:Theorem 1. When 𝜆 1 is fixed, the function 𝜆 2 ↦ → V R𝐷𝑒𝑆𝑆𝐿 (Θ) | 𝑟 reaches its minimum for:𝜆 2 * = 𝑛 𝑢 𝑛 Cov(𝐺 (Θ; 𝑥, 𝑦), 𝐻 (Θ; 𝑥)) V(𝐻 (Θ; 𝑥)) + Cov (𝜆 1 𝑇 (Θ; 𝑥, 𝑦), 𝐻 (Θ; 𝑥)) V(𝐻 (Θ; 𝑥)) and at 𝜆 2 * : V R𝐷𝑒𝑆𝑆𝐿 (Θ) | 𝑟 𝜆 2 * = 1 - 𝑛 𝑢 𝑛 𝜌 2 𝐿,𝐻 V R𝐶𝐶 (Θ) | 𝑟 ≤ V R𝐶𝐶 (Θ) | 𝑟 ,where 𝜌 𝐿,𝐻 = Corr(𝐿(Θ; 𝑥, 𝑦), 𝐻 (Θ; 𝑥)).A detailed proof of this theorem is presented in Appendix. From the theorem 1, we can know that the variance of the unbiased estimate R𝐷𝑒𝑆𝑆𝐿 (𝜃 ) is less than that of R𝐶𝐶 (Θ) using only supervised data when 𝜆 1 and 𝜆 2 meet certain conditions. This theorem also guides us to simultaneously adjust 𝜆 1 and 𝜆 2 to achieve a stable risk estimation effect in practice. When our estimate is unbiased and the variance is smaller, we can theoretically ensure that our semi-supervised module is safe when introducing the unlabeled data and no worse than using only supervised data. To validate the correctness of our unbiased framework analysis, we train our DIMvSML on Yeast and split the test dataset into 20% labeled and 80% unlabeled data to calculate the R𝑆𝑆𝐿 (Θ) and R𝐷𝑒𝑆𝑆𝐿 (Θ) risks that we compared to the oracle risk estimate using all the test set. For variance test experiment, we split 50 times the test set to estimate the variance of the risk estimator. Besides, we compute 𝜆 2 * using the entire test set. As shown in Fig. 4 , the result illustrates that R𝐷𝑒𝑆𝑆𝐿 (Θ) is unbiased for any value of 𝜆 2 and its variance can be optimised by adjusting 𝜆 2 when 𝜆 1 is fixed. Besides, it can be seen that V( R𝐷𝑒𝑆𝑆𝐿 (Θ)) is less than V( R𝑆𝑆𝐿 (Θ)) in most cases value of 𝜆 2 * is close to the minimum point calculated from the actual sampling. Due to the capability to supply new labeled data for training, pseudo-label methods have gained significant prominence in deep semi-supervised classification tasks [15] . Therefore, we assign pseudolabels to exploit additional supervisory information and improve model performance. Since the classification confidence exhibited in the early stage of classifier is low, we choose the output of classifier as labels for unlabeled instances at half of the total training epoch. Then all instances are incorporated into training process.The training strategy employed in DIMvSML contains two phases: pre-training and alternative optimization. In the pre-training phase, we only use the rebuilding loss 𝐿 rb to simply train the view-specific GNNs. During the alternate optimization phase, the three proposed modules mutually complement each other, resulting in simultaneous enhancement of classification performance. Denote the loss 𝐿 icl + = 1 𝑛 𝑙 𝑖 ∈ L 𝐻 (Θ; 𝑥 𝑖 ), the overall loss of our DIMvSML in alternative optimization phase be formulated asEQUATIONwhere 𝜆 1 and 𝜆 2 are penalty coefficients. The training process of our DIMvSML is summarized in Appendix. 
experiments 3.1 experimental setup: 
 Datasets. In our experiments, six public multi-view multi-label datasets are selected to validate the proposed method, i.e., Yeast [12] , Corel 5k [6] , VOC 2007 [7] , Esp Game [1] , IAPR TC-12 Comparison Methods. To validate the effectiveness of DIMvSML, we compare it with six state-of-the-art approaches, which can be categorized into two groups: traditional methods and deep methods. Traditional methods include: TM3L [42] , iMvWL [33] , NAIML [17] , while deep methods include DD-IMvMLC [37] , DICNet [21] , LMVCAT [22] . Five of them are introduced in the preliminaries and TM3L is a multi-view multi-label classification method, which can handle partial multi-label data. Noting that except for TM3L, the other five methods can handle both feature and label missing simultaneously. Therefore, the missing views are populated by their average instance calculated from the corresponding available instances of the same view for TM3L in our experiment. For all comparison methods, we will prioritize the parameter settings recommended in the original code implementations or specified in their respective papers. Data Preparation. Each dataset can be divided into training, validation and test sets in the ratio of 7:1:2. To simulate the partial view setting, we randomly remove some views of samples from each set. Concretely, according to the pre-set partial example ratio (PER), PER% instances are randomly selected as incomplete instances, which randomly missing 1 ∼ 𝑉 -1 views (at least one view per instance is available to keep the total number of samples constant). For the SSL situation, according to the pre-set labeled example ratio (LER), we randomly select LER% instances as labeled instances in the training dataset. Implementation Details. The 𝑘-NN graphs are constructed based on the Euclidean distance metric, where the neighbor number 𝑘 is fixed to 10 for all datasets. The Adam optimizer is employed with an initial learning rate of 0.0001 for optimizing the training loss.In addition, Ranking Loss (RL), Accuracy (ACC), Average Precision (AP) and adapted area under curve (AUC) are adopted as four evaluation metrics. All the experimental results are derived from ten independent runs of the methods, and the final average results along with their corresponding standard deviations are presented. Our model is implemented by PyTorch on one NVIDIA GeForce RTX 4090 with GPU of 24GB memory.To comprehensively verify our DIMvSML, we compare it with six competitive methods from two key aspects: i) view missing and ii) label insufficient. For view missing, we fix LER to 20%, while PER is selected in {0%, 10%, 30%, 50%, 70%, 90%}. For label insufficient, we fix PER to 50%, while LER is chosen in {15%, 20%, 25%, 30%, 35%, 40% 45%}. The statistical results are presented in Table 1 , Fig. 5 and Fig. 6 . Tabel 1 displays the four metrics with LER fixed at 20% and PER fixed at 50%, while Fig. 5 and Fig. 6 show the AUC when LER and PER change respectively. The additional results of Rankingloss and AP are shown in Appendix.Regarding the missing view, we can find that 1) When PER= 0%, DIMvSML achieves the best performance on all datasets, which indicates that the proposed method is also stable and effective in the classification tasks under complete views. 2) With PER increasing from 10% to 90%, DIMvSML still outperforms the other six methods. It shows our method adequately addresses the problem of missing views and is beneficial for feature completion. 3) DIMvSML has the capability to address scenarios where the absence of certain views significantly impacts the classification process. In other words, even when all compared methods fail to deliver satisfactory results, our approach continues to exhibit a considerable effect. For example, DIMvSML and the most competitive method achieve AUC of 70.27% and 61.78% when PER=10% on Corel 5k, revealing an increase of nearly 10% percent. As for label insufficient, we have the following observations: 1) Our method achieves better among all compared methods in almost all cases. 2) Our DIMvSML is robust to few labeled instances since it consistently exhibits relatively promising performance with lower LER. For example, DIMvSML and the second-best method LMVCAT achieve AUC of 74.79% and 73.8% when LER=45% on IAPR TC-12. As LER=15%, the performance of DIMvSML is 71.64% and superior to 61.25% of LMVCAT.The ablation experiments on VOC 2007 and IAPR TC-12 are carried out to thoroughly investigate the impact of the three critical modules of DIMvSML. When the GNN-based feature completion module (𝑆 1 ) is disabled, we employ the average strategy to fill the missing data. When the Multi-view representation learning module (𝑆 2 ) is disabled, we simply concatenate each view and remove the loss 𝐿 rc . As for the safe semi-supervised learning module, we compare the classification performance under the R𝐶𝐶 (Θ) and the R𝐷𝑒𝑆𝑆𝐿 (Θ) loss framework since we focus on whether the introduction of unlabeled data would actually improve performance. The ablation results are listed in Table 2 . We can know that: 1) In the first three rows, performance is reduced when 𝑆 1 and 𝑆 2 are removed respectively, which indicates that our method is effective for data recovery and feature extraction. 2) In the last two rows, the performance of our debiased framework R𝐷𝑒𝑆𝑆𝐿 (Θ) is better than R𝐶𝐶 (Θ) using only supervised data. It demonstrates that our method indeed enhances model performance with the introduction of unlabeled data and provide a reliable effect for semi-supervised classification.We conduct experiments on VOC 2007 and IAPR TC-12 to analyze the sensitivity of 𝜆 1 and 𝜆 2 . Two parameters are selected from the range of {0.01, 0.1, 1, 10, 100} and the joint influence are presented in the heatmap as shown in the Fig. 7 . Since the difference between the best performance and the worst is 22.9 on VOC 2007, we can learn that our method is sensitive to both parameters. The result further validates the Theorem 1 and emphasizes the need to simultaneously adjust 𝜆 1 and 𝜆 2 to approach the condition for the Theorem 1, which ensures stable performance. 
conclusion: 
 To tackle the incomplete multi-view semi-supervised multi-label problem, we propose a novel deep learning based method named DIMvSML in this paper. DIMvSML incorporates both the GNNbased feature completion, view-specific representation extraction network and safe semi-supervised multi-label learning module to preserve discriminative feature and enhance the semantic label information. Besides, we design an unbiased loss to alleviate the bias from large amount of unlabeled data and provide theoretical analysis of our safe risk estimator. Therefore, our DIMvSML can eliminate the negative effect of the incomplete data and use unlabeled information safely for efficient classification. Finally, extensive experimental results on six public datasets demonstrate the effectiveness and superiority of DIMvSML. In the future, we will further extend to solve other multi-label problems under incomplete views, such as class-imbalance and noisy labels etc. 
 
 Appendix:  


Decision: This paper studies semi-supervised multi-view multi-label learning and focuses on the challenging cases with missing views and missing labels. In this paper, the authors proposed a novel network, DIMvSML, which addresses the missing views issue by feature recovery and designs a safe risk estimate framework with an unbiased loss to address the incomplete label learning issue. After the rebuttal phase, four reviewers reach a consistent recommendation to accept the paper. The final recommendation for the paper is acceptance.

Reviews: review: paper_summary: paper_summary: This paper introduces the Deep Incomplete Multi-View Semi-Supervised Multi-Label Learning (DIMvSML) approach to solve the challenges posed by incomplete features and limited labels in multi-view multi-label learning. Specifically, DIMvSML leverages graph neural networks to reconstruct missing view data and employs an unbiased loss to optimize risk estimation. Comprehensive experimental results demonstrate the superior performance of this method.
strength_weakness: strength_weakness: 1. Novelty: The authors designed a deep learning model proposed to recover missing views and obtain unbiased risk estimation for multi-label semi-supervised learning. 
2. Validation: It includes theoretical analysis of the model and its parameters. 
3. Experimental: The experiments demonstrate the method's performance from multiple perspectives, including comparative experiments as well as ablation study and convergence.
limitations: limitations: 1. This paper employs a GNN network to initially restore missing features, followed by leveraging feature representation and contrastive learning techniques. Since subsequent modules hinge on complete features, the restoration process must precede all other operations. Therefore, a detailed explanation of the network's end-to-end training process is warranted, encompassing the various training steps, the utilization of distinct loss functions at different stages, and the precise point at which pseudo-labels are integrated.
2. The paper introduces a novel concept of debiasing in the context of multi-label semi-supervised learning. While traditional semi-supervised methods may still yield satisfactory results without explicit debiasing, the author should elaborate on the underlying reasons for this phenomenon. Additionally, further insight into the risks associated with neglecting debiasing could provide a deeper understanding of its importance.
3. When it comes to the optimization of the network model, this paper lacks crucial details. Specifically, it remains unclear whether mini-batch training is adopted during each epoch and, if so, how batches comprising solely labeled or unlabeled samples are handled. These details are crucial for understanding the network's learning process and potential biases.
paper_summary: paper_summary: In this paper, the authors propose a novel solution with the deep incomplete multi-view semi-supervised multi-label learning method. The method employs graph neural networks to recover feature information through structural similarity relations. Additionally, they design structure-specific deep feature extractors and maintain cross-view consistency for the recovered data using instance-level contrastive loss. Furthermore, to address the bias in risk estimation minimized by semi-supervised multi-label methods, they introduce a safe risk estimation framework with an unbiased loss, improving empirical performance with pseudo-labels of unlabeled data.
strength_weakness: strength_weakness: 1. The authors creatively use graph neural networks to address the problem of incomplete multi-view semi-supervised multi-label learning, providing a new perspective on multi-view learning.

2. A security risk estimation method is presented, with rigorous mathematical proofs ensuring the method's stability on unlabeled data.

3. Extensive experiments demonstrate the effectiveness of the method.
limitations: limitations: 1. The paper appears to have some typos, such as in the method names on lines 224 and 787.

2. The structure of the paper is not sufficiently clear. The implementation details and experimental analysis discussed at the end of Sections 2.2 and 2.3, respectively, would be more appropriately placed in the "Experiments" section.

3. The experimental analysis in the paper is not sufficiently in-depth. For example, in Section 3.4, the authors only conclude that the method is sensitive to parameters. I am also curious why in Figure 5, DIMvSML seems unrelated to changes in LER, while other methods generally increase with increasing LER.

4. In the multi-view representation learning module, the authors fuse the representations of different views by concatenation. Could this approach make it difficult to capture the relationships between views? Have other methods been tried, such as addition, setting learnable parameters, or using attention mechanisms?
paper_summary: paper_summary: This paper introduces a novel Deep Incomplete Multi-View Semi-Supervised Multi-Label Learning method (DIMvSML), addressing the problems of incomplete features and missing labels in multi-view multi-label data. DIMvSML employs graph neural networks to recover missing feature information and designs structure-specific deep feature extractors to capture discriminative information while maintaining cross-view consistency. Additionally, an unbiased loss function is introduced to improve the risk estimation of semi-supervised multi-label methods. Extensive experimental results demonstrate the superiority of DIMvSML over state-of-the-art methods.
strength_weakness: strength_weakness: 1. To address the problem, this paper employs graph neural networks to recover views and utilizes a debiased loss function to enhance the performance of the semi-supervised multi-label learning, And the complementary and consistency across different views are considered in this paper.

2. Experimental results under various feature missing ratios and label annotation ratios validate the effectiveness of the proposed method.
limitations: limitations: 1. DIMvSML uses graph neural networks to recover the information of incomplete multi-view features. What are the benefits of using GNNs compared to other feature recovery methods, and what is the principle behind its ability to recover features?

2. Please clarify the meaning of Eq. (8), and explain how the joint distribution between different views and their marginal distributions are computed in contrastive learning.

3. This paper mentions the strategy of adding pseudo-labels to unlabeled samples to explore additional supervisory information. However, it does not verify the correctness of these pseudo-labels. In scenarios with a large number of label categories, pseudo-labels are likely to be less accurate. Does the presence of incorrect pseudo-labels introduce more noise during network training, potentially affecting the model's performance?
paper_summary: paper_summary: This paper proposes a deep learning framework, DIMvSML, for addressing the common issue of incomplete views and labels in multi-view multi-label data applications. By integrating graph neural networks, the model enhances the accuracy of feature recovery. Additionally, through providing unbiased risk estimation, it not only optimizes label prediction accuracy but also improves the model's stability in solving semi-supervised problems. Experimental results demonstrate that DIMvSML outperforms existing state-of-the-art techniques.
strength_weakness: strength_weakness: In this paper, they conduct the analyses on both methodological and theoretical levels. Methodologically, DIMvSML constructs the model based on GNN networks and unbiased risk estimation losses. Theoretically, it demonstrates that the unbiased loss framework adopted exhibits superior statistical properties. The experimental results including comparisons under various experimental settings, ablation studies, and sensitivity analysis of parameters are provided to a comprehensive analysis of the algorithm's performance.
limitations: limitations: 1. As we know, the problem of multi-label and multi-view data with simultaneous missing views and a large number of unlabeled samples addressed in this paper is very challenging. Is there a real and practical scenario corresponding to this problem?

2. In the introduction section, the author conducts a label distribution simulation experiment on the Yeast dataset and demonstrates that the debiased method outperforms the traditional semi-supervised paradigm. However, it is known that label distribution does not equate to the final prediction accuracy. It is possible to achieve good simulation results by predicting all zeros. Can the author further clarify whether this label distribution simulation experiment can convincingly demonstrate the effectiveness of the debiased framework proposed in this paper, and why?

3. This paper proposes the use of mutual information in the contrastive loss to learn consistency across different views. However, the author does not further explain the source of the mutual information calculation or why it is presented in the form described in the article.