Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations

Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-03-24 09:46:13

Abstract: Insufficient labeled training samples pose a critical challenge in multi-label classification, potentially leading to overfitting of the model. This paper delineates a criterion for establishing a common domain among different datasets, whereby datasets sharing analogous object descriptions and label structures are considered part of the same field. Integrating samples from disparate datasets within this shared field for training purposes effectively mitigates overfitting and enhances model accuracy. Motivated by this approach, we introduce a novel method for multi-label classification termed Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations (NOMWM). Our method strategically amalgamates samples from diverse datasets within the shared field to enrich the training dataset. Furthermore, we project samples from various datasets onto a unified subspace to facilitate learning in a consistent latent space. Additionally, we address the challenge of weak labels stemming from incomplete label overlaps across datasets. Leveraging weak-label indicator matrices and label correlation mining techniques, we effectively mitigate the impact of weak labels. Extensive experimentation on multiple benchmark datasets validates the efficacy of our method, demonstrating clear improvements over existing state-of-the-art approaches.

Main paper: 
 introduction: 
 Multi-label learning stands as a prominent area within machine learning and pattern recognition. This paradigm involves representing each sample with a feature vector while allowing for the simultaneous association of multiple category labels. The overarching objective revolves around inducing a function capable of assigning multiple appropriate labels, drawn from a predefined label set, to unseen instances [30, 34] . The introduction of multi-label learning has spurred significant scholarly interest, leading to the development of numerous efficient algorithms.However, a common challenge arises in scenarios where the available number of labeled samples proves insufficient for training an effective multi-label classification model. Addressing this limitation becomes imperative in the quest to enhance model accuracy. In practical scenarios, datasets often describe akin objects with overlapping label sets, thereby forming what we term as a 'same field' within multi-label data. For instance, consider the multi-label image datasets portraying natural scenes depicted in Figure 1 . These datasets frequently feature analogous objects such as 'beach','sky','ocaen' and 'water', and exhibit comparable label sets. Leveraging datasets within the same field for joint training holds the promise of significantly augmenting the sample pool, thereby effectively mitigating the challenge of insufficient training samples-a common precursor to overfitting in multi-label classification models. This strategy effectively embodies a Non-Overlapped Multi-View Weak-Label Learning framework (sample augmentation multi-label framework), offering a viable solution to counteract overfitting resulting from limited training samples.The Non-Overlapped Multi-View Weak-Label Learning framework encounters two primary challenges. Firstly, samples from distinct datasets often exhibit disparate feature types and lack intersection, as depicted in Figure 2 (c). Consequently, conventional multi-view multi-label classification approaches [22] are unsuitable for direct application. Secondly, corresponding label sets across datasets may not align precisely. For instance, in Figure 1 , data set A includes the label 'clouds,' which is absent in data set B. Thus, when jointly training these datasets, 'clouds' becomes a missing label for images in data set B featuring clouds, exacerbating the weak-label problem. Weak-label learning primarily addresses multilabel learning scenarios with partially relevant label sets [24] [13] [4] .To tackle these challenges, researchers typically explore multiview and weak-label learning strategies. However, existing methodologies struggle to effectively address both challenges simultaneously. Notably, two frameworks closely related to our problem have been proposed in prior works (refer to Figure 2 ). The multi-view multi-label framework [22, 35] (depicted in Figure 2 (a)) concurrently handles multi-view and multi-label classification. Building upon this framework, Tan et al. [25] introduced an extension termed incomplete multi-view multi-label classification (illustrated in Figure 2(b) ), focusing on addressing incomplete views and missing labels concurrently [29] [26] [15] [16] . Nevertheless, these frameworks encounter difficulty in addressing the two challenges posed by sample augmentation multi-label learning (illustrated in Figure 2(c) ). A comparison among the three sub-figures (a, b, c) in Figure 2 reveals that our proposed Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations (NOMWM) offers a more general solution compared to the aforementioned frameworks.To this end, we designed our model to specifically address the aforementioned challenges. Firstly, to leverage samples with diverse features comprehensively, we adopt a multi-view learning approach to jointly handle datasets with varying features. By learning a common subspace, we fuse information from different datasets In this paper, we define that data sets belong to the 'same field' if they describe similar objects and have similar label sets. Data set A and data set B in Figure 1 both describe natural scenes, and they have similar label sets. We consider such two data sets belong to the 'same field' in this paper. The samples of these two data sets do not overlap, and their label sets partially overlap but are not completely consistent.... to enhance model training collectively. Secondly, to contend with inconsistent label sets, we frame the issue as a weak-label learning problem. Specifically, we employ a missing label indicator matrix and label correlation mining techniques. The missing label indicator matrix mitigates the impact of missing labels on the model, while label correlation mining enhances prediction accuracy from a label space perspective. Integration of these aspects facilitates effective sample augmentation for training a more robust multi-label classification model. In summary, the main contributions of this paper are delineated as follows:(1) Proposing a novel multi-label learning framework facilitating effective sample augmentation to alleviate overfitting arising from insufficient training samples in multi-label classification. This framework integrates samples from diverse-featured datasets within the 'same field' and addresses weak-label issues stemming from label set mismatches between datasets.(2) Introducing a novel common subspace learning method to derive a unified representation from disparate training datasets. This method leverages multi-view learning to enhance model prediction accuracy.(3) Introducing a weak-label learning module to tackle label set inconsistencies across datasets. This module incorporates a label indicator matrix and label correlation mining techniques to mitigate the impact of missing labels and enhance framework robustness. 
related work: 
 The landscape of existing multi-label learning algorithms [ [30] , [34] ] broadly divides into two categories: problem transformation methods (PTMs) and algorithm adaptation methods (AAMs). Despite their differences, both PTMs and AAMs grapple with a common challenge in multi-label learning tasks-namely, the scarcity of instances featuring identical features and complete label sets for training classification models. Frequently, practitioners can only access diverse datasets describing similar objects with comparable label sets. Consequently, the crucial issue becomes how to amalgamate instances with differing features and incomplete label sets during the learning process. Our work closely relates to two key aspects: weak-label learning and multi-view multi-label learning, which we review as follows.In weak-label learning, a large number of samples have incomplete labels, focusing on multi-label learning scenarios with partially relevant label sets. Noteworthy methodologies include WELL proposed by Sun et al. [24] , which employs a low-rank similarity matrix hypothesis to uncover instance correlations and complete label dissemination. DM2L by Chen et al. [33] leverages the lowrank structure of the label matrix to integrate global and local low-rank label structures alongside discriminative label information. KWLRL by Zhao [32] reconstructs the label semantic space through joint label correlation, leveraging label information consistency and feature-label dependency assumptions, which also extends the linear model to a nonlinear kernel method to handle data separability challenges. Regarding multi-view multi-label learning, LVSL by Zhao et al. [31] tackles non-alignment issues among views and labels through a unified framework, leveraging global and local structural information and view contribution weights. Similarly, NAIM 3 L by Li et al. [11] addresses missing labels, incomplete views, and non-alignment based on global and local label structures, introducing an indicator matrix for handling missing labels and aligning labels in a common space. AIMNet by Liu [14] proposed an attention-driven technique to address the widespread incompleteness issue in multi-view features and labels, which also utilizing statistical weak label correlations and graph attention networks to improve classification accuracy.While the aforementioned methods are closely related to our work, they predominantly address weak-label learning and multiview multi-label learning separately. Directly applying these methods to multi-label learning tasks involving multiple datasets with differing features and incongruent label sets proves challenging. Specifically, we face two primary challenges when addressing sample augmentation in these tasks: (1) effectively leveraging multiple datasets with diverse features lacking sample overlap, and (2) addressing inconsistencies in label sets among datasets involved in sample augmentation. To tackle these challenges concurrently, we propose our method, Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations (NOMWM). Our approach seamlessly integrates datasets with varying features within the 'same field' to learn a multi-label classification model while adeptly handling the issue of incomplete label set matching across multiple datasets. Further details are elaborated in the subsequent section. 
proposed approach: 
 For multi-label classification, let 𝐷 = {(𝑥 𝑖 , 𝑦 𝑖 )} 𝑚 𝑖=1 = {𝑋, 𝑌 } represent a collection of 𝑑-dimensional training instances 𝑋 ∈ R 𝑚×𝑑 paired with their associated labels 𝑌 ∈ {0, 1} 𝑚×𝑛 , where 𝑚 and 𝑛 denote the number of instances and label attributes, respectively. The primary objective of multi-label classification algorithms is to train a predictor 𝑓 : 𝑋 → 𝑌 from 𝐷 during the training stage, enabling the prediction of the label ŷ for a given test instance x.Our proposed Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations (NOMWM) aims to develop a novel multi-label classification model capable of effectively handling datasets with disparate features within the 'same field' to facilitate sample augmentation. Moreover, the model adeptly addresses the weak-label problem, efficiently managing the issue of incomplete label set matching across multiple datasets. The NOMWM model comprises two core components: latent space representation learning (Figure 3 ) and weak-label classification model (Figure 4 ).We leverage the concept of multi-view learning to jointly train multiple datasets featuring distinct features and extract informative patterns from the samples. While traditional multi-view learning primarily focuses on amalgamating information from diverse views within a single dataset, NOMWM extends this concept to integrate different datasets lacking intersection, thus achieving sample augmentation.Within this framework, we denote 𝑋 𝑣 ∈ R 𝑚 𝑣 ×𝑑 𝑣 as the training datasets with the 𝑣th feature vector, 𝑌 𝑣 ∈ {0, 1} 𝑚 𝑣 ×𝑛 𝑣 as the corresponding label sets, 𝑉 𝑣 ∈ R 𝑚 𝑣 ×𝑑 𝑐 and 𝑈 𝑣 ∈ R 𝑑 𝑐 ×𝑑 𝑣 as the latent space representation and transformation matrix, respectively. 𝑊 represents the mapping matrix from the latent space to the label space, 𝑆 𝑣 indicates the similarity between instances in the 𝑣th training dataset, and 𝐿 denotes the similarity between labels. Given the need to amalgamate diverse datasets for learning, we first unify the label sets. Herein, 𝑌 ∈ {0, 1} 𝑚×𝑛 represents the integrated label set. However, due to label set expansion, certain labels present in the original label set 𝑌 𝑣 ∈ {0, 1} 𝑚 𝑣 ×𝑛 𝑣 may become missing labels in the new label set 𝑌 ∈ {0, 1} 𝑚×𝑛 . Thus, we introduce 𝑀 as the indicator matrix to denote missing labels.Then the objective function of our proposed NOMWM can be written as follows.minEQUATIONwhere Φ(𝑋 𝑣 , 𝑉 𝑣 , 𝑈 𝑣 ) and Ψ(𝑉 𝑣 ,𝑊 , 𝑌, 𝑆 𝑣 , 𝐿) denote the losses of latent space representation learning (Figure 3 (a)) and weak-label classification with label correlation (Figure 3 (b)).In the realm of latent space representation learning, our approach entails projecting datasets 𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑠 originating from diverse feature spaces into a unified common subspace denoted as 𝑉 , while preserving the consistency of feature magnitudes across samples within this shared subspace (depicted in Figure 3 (a)).Observing Figure 2 (c), it becomes apparent that integrating insights from multiple disparate datasets is imperative for joint learning. These datasets feature non-overlapping training data and exhibit distinct feature characteristics. Drawing inspiration from multi-view learning, we achieve information amalgamation by learning a common subspace. This involves projecting datasets from various spaces into the shared subspace 𝑉 , thereby streamlining subsequent multi-label classification tasks. To foster greater consistency in the learned latent space representations 𝑉 𝑣 , we impose distribution-based constraints, a concept we delve into further in the subsequent discussion.The feature alignment module stands as a pivotal component of our proposed NOMWM, particularly in scenarios involving datasets with substantial disparities in feature characteristics. Hence, we introduce the feature alignment module of NOMWM, grounded in a distribution-centric perspective. This viewpoint posits that different datasets inherently belong to distinct distributions, necessitating a holistic approach to fusion from a distributional standpoint.In pursuit of aligning multiple datasets from a distributional perspective, we endeavor to minimize the separation between the centers of each dataset in the latent space. To this end, we leverage the center 𝜇 𝑣 of each dataset 𝑉 𝑣 to encapsulate its distribution, imposing constraints to bring these centers into closer proximity. This endeavor enhances the consistency of learned low-dimensional representations, thus facilitating subsequent classification tasks.By leveraging the learning of a common subspace, we harness the diverse features present across samples to their fullest extent. This approach optimally utilizes the informative content of all samples, thereby enhancing the predictive capacity of the model. 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 Y1 L5 L4 L2 L1 L4 L3 L2 L5 L1 L4 L3 L2 L5 L1 L4 L3 L2 L5 Y3 Y2 FY1 FY3 FY2 Y1 Y3 Y2 Y1 Y3 Y2 full-dimension label space L1 L4 L3 L2 L5 L1 L4 L3 L2 L5 L2 L4 L1 L5 L3 L2integrated label space information Figure 4: The specific process of integrating all label sets. We expand these label space matrices 𝑌 𝑣 ∈ 𝑅 𝑚 𝑣 ×𝑛 𝑣 to fulldimension label matrix 𝐹𝑌 𝑣 ∈ 𝑅 𝑚 𝑣 ×𝑛 . Then the integrated label set 𝑌 ∈ 𝑅 𝑚×𝑛 can be obtained by cascading the fulldimension label matrix 𝐹𝑌 𝑣 ∈ 𝑅 𝑚 𝑣 ×𝑛 , and the corresponding label indicator matrix 𝑀 can be obtained.Subsequently, we utilize the latent space representation as input to train the weak-label classification component. In summary, the loss incurred in latent space representation learning can be articulated as follows:Φ(𝑋 𝑣 , 𝑉 𝑣 , 𝑈 𝑣 ) = 𝑠 ∑︁ 𝑣=1 ∥𝑋 𝑣 -𝑉 𝑣 𝑈 𝑣 ∥ 2 𝐹 + 𝛼 𝑠 ∑︁ 𝑣=1 𝑠 ∑︁ 𝑘≠𝑣 ∥𝜇 𝑣 -𝜇 𝑘 ∥ 2 2 (2)where 𝜇 𝑣 is the center of data sets in latent space. 𝑉 𝑣 . 𝑉 𝑣 ∈ 𝑅 𝑚 𝑣 ×𝑑 𝑐 and 𝑈 𝑣 ∈ 𝑅 𝑑 𝑐 ×𝑑 𝑣 are the corresponding latent space representation and transform matrix. Besides, we have the parameter 𝛼 to balance the losses of projection and alignment.To mine the dependencies between features and labels for better common subspace, this study uses the Transformer encoder based on feature embedding and label embedding [10, 28] . In this study, image feature embeddings are represented as 𝑃, each vector 𝑝 𝑖 ∈ 𝑅 𝑑 𝑐 of 𝑃 ∈ 𝑅 𝑑 𝑐 ×𝑠 corresponds to the feature maps from each view of the image. The label embeddings, which are represented as 𝑄, are denoted by 𝑄 = {𝑞 1 , 𝑞 2 , • • • , 𝑞 𝑛 } ∈ 𝑅 𝑑 𝑐 ×𝑛 , with 𝑛 representing the number of label attributes and 𝑑 𝑐 indicating the dimensionality of the word-embedding vector. We consider a set of embeddings 𝐻 = {𝑝 1 , . . . , 𝑝 𝑠 , 𝑞 1 , . . . , 𝑝 𝑛 } as input for the Transformer encoder. With Transformers, the self-attention mechanism learns the importance, or weight, of embedding ℎ 𝑗 ∈ 𝐻 with respect to ℎ 𝑖 ∈ 𝐻 . We input both feature embedding and label embedding information into the Transformer encoder to obtain the new representation.EQUATIONEQUATIONwhere 𝑊 𝑘 is the key weight matrix, 𝑊 𝑞 is the query weight matrix, 𝑊 𝑣 is the value weight matrix, 𝑊 𝑟 and 𝑊 𝑜 are transformation matrices, and 𝑏 1 and 𝑏 2 are bias vectors. We denote the final output of the Transformer encoder as𝐻 ′ = 𝑝 ′ 1 , . . . , 𝑝 ′ 𝑠 , 𝑞 ′ 1 , . . . , 𝑞 ′ 𝑛 .During the model learning process, 𝑉 𝑣 in Eq.( 2) will be obtained after transformer transformation, correspondingly, here the transformer network will be learned based on the loss and gradient passed back from the optimization process.In this section, we delve into our weak-label learning component, devised to address the challenge of weak-label instances stemming from diverse training datasets. We focus on two key aspects: the utilization of an indicator matrix to signify missing labels and the incorporation of label correlation mining inspired by [5] [18] . The central concept behind label correlation mining involves reconstructing the label space through instance and label similarity, thereby enhancing classification performance. Real-world datasets within the 'same field' often exhibit varying label compositions. In other words, the label sets 𝑌 1 , 𝑌 2 , . . . , 𝑌 𝑠 corresponding to these datasets may encompass different labels. Resolving the issue of inconsistent label sets constitutes a pivotal aspect of our multi-label sample augmentation strategy. To this end, we tackle this challenge in two steps. Firstly, we consolidate the label sets 𝑌 𝑣 ∈ {0, 1} 𝑚 𝑣 ×𝑛 𝑣 from different datasets to derive a unified label set 𝑌 ∈ {0, 1} 𝑚×𝑛 . Subsequently, we leverage this integrated label set 𝑌 for weak-label classification.Addressing the integration of all label sets, we adopt a straightforward yet effective approach. We identify all unique labels across the datasets and record their total count as 𝑛. Subsequently, we expand each label space matrix 𝑌 𝑣 ∈ {0, 1} 𝑚 𝑣 ×𝑛 𝑣 to a full-dimensional label matrix 𝐹𝑌 𝑣 ∈ {0, 1} 𝑚 𝑣 ×𝑛 . This involves filling the expanded label positions with 0, as illustrated by label 𝐿 3 in 𝐹𝑌 1 (refer to Figure 4 ). By concatenating these full-dimensional label matrices 𝐹𝑌 𝑣 ∈ {0, 1} 𝑚 𝑣 ×𝑛 , we obtain the integrated label set 𝑌 ∈ {0, 1} 𝑚×𝑛 .However, the integration of full-dimensional label sets 𝐹𝑌 𝑣 ∈ 𝑅 𝑚 𝑣 ×𝑛 may lack precision across all datasets. Despite expanding the label space to encompass 𝑛 labels, not all corresponding datasets have been relabeled, resulting in potential omissions within the integrated label set 𝑌 . For instance, an instance from dataset 𝑋 𝑖 may possess labels absent in its original label set 𝑌 𝑖 but present in another dataset 𝑌 𝑗 . This integration method may inadvertently classify such labels as missing within the integrated set 𝑌 . To rectify this issue, we employ weak-label learning methodologies and label correlation mining in subsequent steps.To address the impact of inaccuracies within the integrated label set 𝑌 , we propose leveraging weak-label learning alongside label correlations mining strategies. This approach encompasses two key facets. Firstly, we employ a missing indicator matrix 𝑀 to flag missing labels, enabling the reduction of their influence on the overall model training error. Secondly, we enhance the precision of weak-label learning by exploring label correlations. Drawing inspiration from [5] , we incorporate a regularization term into our label correlation mining strategy, crafted with a view towards smoothness assumption. This regularization term encapsulates the smoothness of both instance and label similarities.In addition to these strategies, we endeavor to capture the local geometric structures of the feature space by defining an undirected weighted graph for each dataset within 𝑋 1 , 𝑋 2 , . . . , 𝑋 𝑠 . Herein, we denote the weight matrix (i.e., similarity matrix) for 𝑋 𝑣 as 𝑆 𝑣 = (𝑠 𝑖 𝑗 )𝑚𝑣 × 𝑚 𝑣 , where 𝑠 𝑖 𝑗 signifies the similarity measure between samples 𝑥 𝑖 and 𝑥 𝑗 . This similarity measure can be defined as:𝑠 𝑖 𝑗 = 𝑒 - ∥𝑥 𝑖 -𝑥 𝑗 ∥ 2 𝜎 2 (5)where 𝜎 2 is a parameter that can be adjusted.In sum, the loss of weak-label classification with label correlation can be expressed as:Ψ(𝑉 𝑣 ,𝑊 , 𝑌, 𝑆 𝑣 , 𝐿) = 𝛽 ∥(𝑉𝑊 -𝑌 ) • 𝑀 ∥ 2 𝐹 + 𝛾 𝑠 ∑︁ 𝑣=1 ∥𝑉 𝑣 𝑊 -𝑆 𝑣 𝑉 𝑣 𝑊 𝐿∥ 2 𝐹 (6)In the above two subsections, we introduce latent space representation learning and weak-label classification with label correlation respectively. According to the introduction of these two parts, the objective function Eq.( 1) can now be detailed as follows:EQUATION)The proposed NOMWM consists of four different parts. The first part 𝑠 𝑣=1 ∥𝑋 𝑣 -𝑉 𝑣 𝑈 𝑣 ∥ 2 𝐹 is the information fusion for different data sets, and the second part𝑠 𝑣=1 𝑠 𝑘≠𝑣 ∥𝜇 𝑣 -𝜇 𝑘 ∥ 22 is to complete the feature alignment. These two parts correspond to Figure 3(a) . The third part ∥(𝑉𝑊 -𝑌 ) • 𝑀 ∥ 2 𝐹 is to complete the weak label learning based on the label indication matrix 𝑀, and the last part𝑠 𝑣=1 ∥𝑉 𝑣 𝑊 -𝑆 𝑣 𝑉 𝑣 𝑊 𝐿∥ 2𝐹 is the label correlation mining module. These two parts are illustrated in Figure 3(b) . The specific optimization solution for the above model is presented as follows.For the model we built, we use the iterative minimization approach to optimize its loss function. When one of the variables is fixed, the original optimization problem will degenerate into a convex subproblem, which is convenient to solve. We adopt the MANOPT toolbox [1] to implement gradient descent with line search for the update of 𝑊 , 𝑉 𝑣 , 𝑈 𝑣 and 𝐿. The specific steps will be described as follows:When 𝑉 𝑣 , 𝑈 𝑣 and 𝐿 are fixed, the subproblem of Eq.( 7) is convex to 𝑊 , then the gradient corresponding to 𝑊 can be easily obtained and we use gradient descent to update it. Similarly, with𝑊 , 𝑈 𝑣 and 𝐿 are fixed, the subproblem of Eq.( 7) is convex to 𝑉 𝑣 , then the gradient corresponding to 𝑉 𝑣 can be easily obtained and we use gradient descent to update it. With 𝑊 , 𝑉 𝑣 and 𝐿 are fixed, the subproblem of Eq.( 7) is convex to 𝑈 𝑣 , then the gradient corresponding to 𝑈 𝑣 can be easily obtained and we use gradient descent to update it.When 𝑉 𝑣 , 𝑈 𝑣 and 𝑊 are fixed, we have the following equation for 𝐿 by setting the derivative of Eq.( 7) w.r.t 𝐿 to zero,EQUATIONThen the closed-form solution for 𝐿 can be written as follows:EQUATIONwhere 𝑍 = 𝑠 𝑣=1 (𝑆 𝑣 𝑉 𝑣 𝑊 ) ′ (𝑆 𝑣 𝑉 𝑣 𝑊 ) and 𝑍 + indicates the pseudo inverse matrix of 𝑍 . 
experimental result: 
 In this section, we first introduce the data sets we choose and the reasons for choosing them; secondly, we introduce the relevant evaluation metrics; then we introduce the experimental settings. Finally, specific experiments and corresponding analysis are given.To validate the efficacy of our proposed Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations, we have chosen five benchmark datasets for experimentation, aiming to ascertain the effectiveness of our method. Table 1 provides an overview of the statistics pertaining to these datasets, namely Corel5k [6] , IAPRTC12 [7] , ESPGame [8] , Mirflickr [9] , and MS-COCO [12] . For feature extraction in our experiments, we employ various methodologies across these datasets, encompassing Colour [27] , GIST [21] , HOG [3] , SIFT [19] , LBP [20] , and VGG [23] . The former five are traditional features, while the latter constitute deep features, all extensively utilized in the domain of image processing. The combination of these datasets and features serves to comprehensively evaluate the effectiveness of our proposed method across diverse scenarios, detailed further in the Experimental Settings section.It is essential to underscore that our proposed method primarily addresses scenarios characterized by a relative insufficiency of training samples in multi-label classification. NOMWM leverages sample augmentation to tackle the challenge of integrating information from different datasets and handling missing labels in multi-label learning. However, due to the absence of established benchmark datasets directly applicable to validating our method, we resort to integrating and repurposing existing benchmark datasets to emulate varied datasets within the 'same field', as elaborated in the Experimental Settings section. Notably, the datasets utilized in our experiments predominantly revolve around images, owing to the ease of extracting diverse features for conducting simulation experiments. Nonetheless, it's crucial to highlight that the NOMWM method we propose is applicable across various domains beyond images, extending to fields like music, text, and beyond.In the multi-label learning problem, since each sample may have multiple category labels at the same time, the single-label evaluation metrics which are commonly used in traditional supervised learning, such as accuracy, precision, and recall, cannot be directly used for the performance evaluation of the multi-label learning system. Therefore, researchers have successively proposed a series of multi-label evaluation metrics. Here we consider three evaluation metrics, i.e., Macro-F1 and Micro-F1, which are widely used in multi-label learning to evaluate the prediction performance [30] .We propose a simulation experiment based on the division of single dataset to further validate our method. This approach leverages readily available data from four datasets (Corel5k, Mirflickr, IAPRTC12, and ESPGame), already partitioned into training and test sets. For each dataset, we further partition the training set into six sub-datasets and extract six different features: Colour, GIST, HOG, SIFT, LBP, and VGG. These sub-datasets, each possessing distinct features, simulate six independent datasets within the 'same field' for our experiments.Both sets of experiments encompass scenarios with full labels and weak labels. In the former, the focus is on multi-label training datasets with complete label sets, while in the latter, we randomly remove a portion of labels from the label matrix to simulate weak-label learning. Specifically, we randomly nullify 20% of label elements from each sample to generate weak-label datasets.In our experiments, we compare our proposed approach with five state-of-the-art multi-label classification methods: Incomplete Multi-View Weak-Label Learning (iMVWL) [25] , Hybrid Noise-Oriented Multi-label Learning (HNOML) [2] , Low-Rank Multi-View Learning in Matrix Completion for Multi-Label Image Classification (lrMMC) [17] , Expand globally, shrink locally: Discriminant Multi-Label Learning with Missing Labels (DM2L) [33] , and Non-Aligned Incomplete Multi-view and Missing Multi-label Learning (NAIM 3 L) [11] .In the experimental comparisons, iMVWL, lrMMC, and NAIM 3 L focus on multi-view multi-label classification, with adaptations made to accommodate incomplete features by imputing average values. On the other hand, HNOML and DM2L are designed for multi-label classification with single-view data and incomplete label sets, where we incorporate the latent feature 𝑉 learned by our proposed method for experimentation. Parameter selection for these methods involves random holdout of one-fifth of training data for validation. We adopt predefined parameter settings from related works for iMVWL, HNOML, lrMMC, and DM2L, while selecting parameters 𝛼, 𝛽, and 𝛾 from the range 10 -10 , 10 -9 , • • • , 10 10 for our method. All experiments are conducted on a 64-bit Linux workstation with an Intel E5-2650 CPU and 256GB memory.We evaluate NOMWM, NAIM 3 L, iMVWL, HNOML, lrMMC, and DM2L on multi-label data. Tables 2 and 3 present the performance of multi-label classification under scenarios of weak and full labels across four datasets. These tables encompass three evaluation metrics: Macro-F1 and Micro-F1. The second column in each table denotes the features of the test datasets during prediction, reflecting the method's robustness across different feature sets. Our experimental results yield several noteworthy observations:NOMWM consistently outperforms most state-of-the-art methods across the four datasets. For instance, under the full label scenario in Table 2 and 3 , NOMWM achieves improvements of 2.15% (Micro-F1) and 1.55% (Macro-F1) on the IAPRTC12 dataset using the Gist feature. Under weak label conditions in Table 2 and 3 , NOMWM enhances performance by 0.81% (Micro-F1) and 2.18% (Macro-F1) on the ESPGame dataset using the SIFT feature. These results validate the efficacy of our proposed method. Across Micro-F1 and Macro-F1 metrics, NOMWM consistently outperforms existing methods by 1%-3%, showcasing its overall superiority.NOMWM is compared against NAIM 3 L, iMVWL, HNOML, lr-MMC, and DM2L. Among these methods, NAIM 3 L, iMVWL, and lrMMC effectively handle multi-view datasets, while HNOML and DM2L utilize the latent feature 𝑉 derived from our method. Moreover, all four methods adeptly address weak-label scenarios. This comprehensive comparison underscores the superiority of our approach. Our proposed sample augmentation method significantly enhances classification accuracy. Across most cases in the full label scenario, our method consistently outperforms others. By augmenting the model's adaptability to diverse features through data augmentation, NOMWM provides richer information for training the classification model. In weak label scenarios where missing labels disrupt the label set distribution, NOMWM's label indication matrix and label correlation mining module mitigate the impact, ensuring robust classification capabilities.These observations collectively affirm the efficacy and versatility of our proposed NOMWM method in addressing multi-label learning challenges.  
 
 Appendix:  


Decision: This paper proposed a multi-label learning approach to tackle insufficient labeled samples and label inconsistencies across datasets. Features from different datasets are projected into a common space. Label correlation is employed to make multi-label classification predictions.

Pros:
- The paper presents a new approach to weak-label learning, leveraging non-overlapped multi-view data and guided by multiple correlations.

- The proposed method shows promise in improving learning performance with weak labels, supported by comprehensive experimental results.

Cons:

- Original submission only evaluated on image domain. After rebuttal, additional evaluations in text, image and audio modalities are evaluated to demonstrate the effectiveness.

- The introduced method has many hyper-parameters and may require substantial higher complexity which could limit practical applications. In the rebuttal, authors provided complexity analysis and hyper-parameter evaluations to address the concerns.

Given the effective rebuttal and the consensus among reviewers, I recommend acceptance with suggestions for minor revisions to enhance clarity and address practical concerns raised during the review process.

Reviews: review: paper_summary: paper_summary: This work proposes a novel method for learning in non-overlapping mutli-view scenarios with weak labels. The method incorporates two components, (1) a representation learning technique that applies Transformer encoder to fuse information from multiple datasets into a common latent subspace and (2) weak-label learning technique using label-correlation to perform multi-label classification.
strength_weakness: strength_weakness: 1. The objective is clearly explained. 

2. The idea of developing an non-overlapping multi-view weak-label learning method is novel. 

3. The paper writing is easy to follow and read.
limitations: limitations: However, the methodology is not clearly explained, and the experiments should be improved. 

My comments are as follows: 

1.	Regarding the motivation of this work, apart from preventing overfitting, is there any other reason (or necessity) for developing the proposed method? 

2.	It is not clear to me how Transformer encoder is used to optimize the objective in equation (2). Please elaborate. 

3.	In equation (7), the second loss term scales quadratically w.r.t. the number of views. How well do your method scale w.r.t. the number of views, and can this be shown in an experiment?

4.	What is the time complexity of the iterative optimization approach described in Section 3.3?

5.	In section 4, the claim that “NOMWM is applicable across various domains beyond images, extending to fields like music, text, and beyond” should be supported by experiments conducted on other data modalities. 

6.	Can you elaborate on the potential real-world applications of NOMWM?

7.	There are typos in the current manuscript. E.g., “ocaen” in line 71. Please double-check. 

8.	This is a suggestion. The results would be much more convincing if you can visualize the alignment of multiple datasets in the latent space. The current paper also lacks exhibits of qualitative results.
paper_summary: paper_summary: The paper presents a novel framework, Non-Overlapped Multi-View Weak-Label Learning Guided by Multiple Correlations (NOMWM), aimed at addressing the challenges of insufficient labeled samples and label inconsistencies across datasets in multi-label classification tasks. The approach integrates samples from various datasets within a 'same field' and employs techniques like a common subspace learning and weak-label learning to enhance classification accuracy. Extensive experiments demonstrate the superiority of NOMWM over existing methods
strength_weakness: strength_weakness: 1. "Non-Overlapped Multi-View Weak-Label Learning" is a meaningful topic.

2. The authors give a lot of related work in this paper for helping readers understand the background.
limitations: limitations: 1. Some citation error should be corrected, such as "AIMNet by Liu [14]" should be "AIMNet by Liu et al. [14]".

2. The notations and the problem definition are somewhat confusing, so it is suggested that the authors make specific explanations in this paper.

3. The evaluation metric contains only 'F1', as I know, there are many metrics for multi-label classification tasks.

4. Comparison methods are too outdated.

5. The construction of Section 4.2 is not clear.
paper_summary: paper_summary: This paper introduces a new method for multi-label classification, named Non-Overlapped Multi-View Weak-Label Learning (NOMWM), which effectively integrates datasets with diverse features guided by multiple correlations. This method uses sample augmentation within a shared domain to mitigate overfitting due to insufficient training samples, and incorporates a missing label indicator matrix and label correlation mining techniques to address the issue of incomplete labels. Experimental results demonstrate that this method outperforms existing state-of-the-art techniques on multiple benchmark datasets, highlighting its effectiveness and practical utility.
strength_weakness: strength_weakness: 1．Innovativeness: The proposed Non-Overlapped Multi-View Weak-Label Learning (NOMWM) framework is an innovative approach to multi-label classification that opens new avenues for handling multi-view and weak-label issues by integrating features and labels from different datasets.
2．Methodological Strengths: The method enhances sample diversity and richness through the concept of a shared domain, effectively addressing incomplete labels while maintaining feature diversity. This approach significantly improves the model's prediction accuracy and robustness.
limitations: limitations: 1．The article mentions that 𝑉𝑣 can be obtained through information propagation within a transformer, but in the optimization phase, 𝑉𝑣 is updated by fixing other variables. This juxtaposition of arguments can make the article's logic appear somewhat confused and obscure the intended role of the introduced transformer. Furthermore, the integration of deep networks into traditional methods seems to disrupt the model's coherence. I recommend that the author elaborately detail the motivation for introducing the transformer, as well as the specific problems it aims to solve.
2．Although the experiments are comprehensively conducted in the field of image processing, the application and effectiveness of the model in other domains, such as text or audio, have not been explored, leaving the generality of the method to be verified.
3．The paper is relatively weak in theoretical analysis, lacking an in-depth analysis of the model's convergence and generalization performance.
4．The objective function of the model is composed of four parts and involves three hyperparameters. Additionally, there are numerous variables that need to be updated. However, the paper does not analyze the complexity of the algorithm, nor does it compare the computation time with other methods. This raises the question of whether the complexity of the algorithm could become a limitation in practical applications. Furthermore, the paper does not provide a detailed analysis of the specific impact of the three hyperparameters on the performance, despite the relatively large number of hyperparameters involved.
5．The main idea of this paper is to integrate sample sets from different datasets within the same domain to construct a multi-label classifier, aiming to prevent overfitting. However, the issues encountered, such as alignment of different views, handling of missing labels, and characterization of label correlations, are all addressed using previously established loss functions. This gives the impression that the paper largely pieces together ideas from other researchers. Furthermore, based on experimental results, the proposed method does not significantly outperform traditional multi-label multi-view classification methods, with F1 scores on some datasets remaining particularly low.