Multi-Label Learning with Block Diagonal Labels

Authors: leqi  shen
sicheng  zhao
hui  chen
jundong  zhou
pengzhang  liu
yongjun  bao
guiguang  ding
yifeng  zhang
Venue: acmmm_org_ACMMM_2024_Conference
Published: 2024-04-03 12:16:50

Abstract: Collecting large-scale multi-label data with \emph{full labels} is difficult for real-world scenarios. Many existing studies have tried to address the issue of missing labels caused by annotation but ignored the difficulties encountered during the annotation process. We find that the high annotation workload can be attributed to two reasons: (1) Annotators are required to identify labels on widely varying visual concepts. (2) Exhaustively annotating the entire dataset with all the labels becomes notably difficult and time-consuming. In this paper, we propose a new setting, i.e. block diagonal labels, to reduce the workload on both sides. The numerous categories can be divided into different subsets based on semantics and relevance. Each annotator can only focus on its own subset of labels so that only a small set of highly relevant labels are required to be annotated per image. To deal with the issue of such \emph{missing labels}, we introduce a simple yet effective method that does not require any prior knowledge of the dataset. In practice, we propose an Adaptive Pseudo-Labeling method to predict the unknown labels with less noise. Formal analysis is conducted to evaluate the superiority of our setting. Extensive experiments are conducted to verify the effectiveness of our method on multiple widely used benchmarks.

Main paper: 
 introduction: 
 Image classification [21, 32] is a fundamental and important research area in the computer vision community. Unlike single-label classification, images for multi-label classification usually contain complex scenes with several objects [42] [43] [44] . Due to the extensive number of categories, collecting a realistic large-scale dataset with complete labels is notably laborious [10, 20] . Consequently, annotators inadvertently overlook certain objects [37] , resulting in missing labels.Previous works study multi-label classification with missing labels in the settings of partial labels [13] and single positive labels [7] . The simulation of partial labels annotates each image with a fixed number of categories to study unannotated labels. In order to explore the minimal supervision, only one positive label is annotated per image in single positive labels. Both settings are employed to investigate the unannotated labels caused by annotation while neglecting the annotation challenges encountered during the annotation process.As shown in Fig 2 .(a), the general annotation pipeline for multilabel classification is that the annotators first learn the visual concepts and differences of all categories, and then annotate a large number of images with complete labels. The high annotation workload for multi-label classification can be attributed to two reasons. (1) Learning: Annotators are required to identify labels on widely varying visual concepts, which can take a lot of effort. The large . Each row shows the label vector of an image. The annotation matrix consists of all label vectors. For each image, all categories are annotated in (a) full labels, a fixed portion of random categories are annotated in (b) partial labels, one category is annotated with positive label in (c) single positive labels, and a subset of interrelated categories are annotated in (d) our proposed block diagonal labels. Categories can be partitioned into different subsets according to semantics and relevance. The annotators simply focus on their own block, so that the annotators can individually annotate a smaller dataset with a subset of highly relevant labels.size of the data requires a lot of highly skilled annotators. (2) Annotating: As the number of to-be-annotated images and categories grows, exhaustively annotating the entire dataset with all the labels becomes notably difficult and time-consuming.In light of the aforementioned two aspects, we propose a new setting of multi-label classification with block diagonal labels. As depicted in Fig 1 .(d), the block diagonal labels setting exhibits similarity with block diagonal matrices whose diagonal contains blocks of smaller matrices. Each block corresponds to a single coarse label. In both online resources and existing single-label datasets, numerous images are accompanied by coarse labels. Furthermore, well-trained single-label classifiers have the capacity to obtain coarse labels for unannotated images. For these reasons, coarse labels are readily available. Therefore, we propose to annotate the multi-label dataset by utilizing the prior coarse label present in each image. The primary advantage of block diagonal labels is that annotators can purely concentrate on their assigned block, which contains a compact subset of highly relevant labels, significantly smaller than the complete set of labels. As a result, training annotators to learn visual concepts is more efficient, and the number of annotated labels required per image is greatly reduced. Therefore, the proposed block diagonal labels setting can readily obtain a large-scale dataset with a great number of labels.Unknown labels are still the core problem in the block diagonal labels setting. We propose a simple yet effective method to deal with the problem of unknown labels. A naive approach is to use a hard threshold to distinguish true negative labels, but it will inevitably introduce some noise. Thus, we propose an adaptive pseudo-labeling method that dynamically adjusts the threshold to reduce the noise for labeling the unknown labels. Any relationships between categories or prior statistics of data are not involved in our method, which can be regarded as a strong baseline.To evaluate the effectiveness of our proposed setting, we conduct a formal analysis of the annotation workload. In addition, we compare with state-of-the-art methods in our setting. We artificially create the block diagonal labels datasets from VOC2012 [14] , COCO2014 [23] , NUSWIDE [6] . We extend this evaluation on a more extensive scale, utilizing the large-scale dataset OpenImages [22] . The block divisions are modified according to the category hierarchies of the published papers. Our method significantly outperforms the existing baselines and achieves state-of-the-art performance: 81.92% mAP on COCO2014 with 9.31% annotated labels and 63.26% mAP on NUSWIDE with 15.60% annotated labels.The contributions are summarized in three-fold: (1) We argue the difficulty of producing realistic large-scale multi-label datasets from two aspects: learning and annotating, which are neglected in partial labels and single positive labels. (2) We propose a new multi-label classification setting with block diagonal labels where annotators can simply concentrate on their assigned block to reduce the annotation workload and positive labels are guaranteed per image. (3) We propose a strong baseline to adaptively predict unknown labels without any prior of datasets or categories, which achieves state-of-the-art performance. 
related work 2.1 multi-label learning with full labels: 
 There are a lot of remarkable works in the field of multi-label learning. An important direction is to model correlation between labels via graph neural networks [2, 4, 5, 39] or transformer structure [25, 31] . [31] predicts labels through queries and extracts local discriminative features adaptively for different labels. Another key characteristic of this field is the inherent positive-negative imbalance. [29] dynamically focuses on the hard samples and controls the contribution propagated from the positive and negative samples.The setting of [40] significantly differs from ours. In [40] , the candidate set (positive labels) for each image includes both relevant (true positive) and irrelevant (noisy) labels. In contrast, more recent studies have focused on missing labels, where not all positive labels are annotated. Many works [13, 17, 46] focus on the optimization of the loss function. [13] is the first work to train a deep neural network for partial labels, which introduces partial-binary cross entropy. [17] proposes to reject or correct the large loss samples to prevent the model from memorizing the noisy label. [46] proposes Hill to down-weight negatives and self-paced loss correction to correct potential unknown labels. There are some works [1, 7, 16] that estimate unknown labels by the label prior or correlation. Some other works [3, 13, 27, 34] require extra complex architectures.Different from the exiting missing labels setting, the novel block diagonal labels setting is introduced to take the annotators' learning and annotating process into consideration. Instead of modeling label correlation and using the dataset priors or extra architectures, we propose to adaptively predict unknown labels by dynamically adjusting the thresholds in the training process. 
block diagonal labels 3.1 comparison with other settings: 
 In order to reduce the annotation workload, recent works mainly focus on the missing labels setting, which annotates a small subset of positive and negative labels. We demonstrate the difference between the full labels and existing missing labels settings based on the example in Fig 1 . The dataset has 15 images {𝐼 𝑖 } 15 𝑖=1 and 10 categories {𝐶 𝑖 } 10 𝑖=1 . In the full labels setting, all categories are completely annotated for a given image. Positive labels are marked in red and negative labels are marked in blue. Each row shows the label vector for each image. As for the missing labels setting, the unknown labels are annotated in gray. The partial labels and single positive labels settings for missing labels are widely studied by previous works. Fig 1.(b) shows the partial labels setting, which annotates each image with a small number of labels. Related works manually generate the partial labels dataset, where each image is randomly selected with a fixed portion of categories as annotations and the others are assumed to be unknown. Since the number of positive labels per image is small on the existing dataset, partial labels will inevitably produce images without positive labels, which does not match the true missing labels setting. Furthermore, Fig 1 .(c) shows the single positive labels setting, which annotates one positive label per image, and no additional negative or positive labels are provided. This setting has the least supervision on the multi-label task. However, the scarce supervision greatly limits the performance.Our proposed block diagonal labels setting also belongs to missing labels. In each block, the images have the same coarse label as the category subset. In Fig 1 .(d), each image is annotated with only one subset of categories. The labels in the same subset are correlated and the number of labels in each subset is small. This facilitates annotators in easily annotating the specified subset of labels. Accordingly, we assume in the experiment that labels within each subset are fully annotated. When subset 𝑆 𝑖 is annotated by its annotator, the other subsets 𝑆 𝑗 | 𝑗 ≠ 𝑖, 𝑗 = 1, 2, 3 are ignored. As a result, labels of other subsets are considered unknown.In practical applications, an image may contain multiple coarse labels, each of which can be simultaneously annotated by different annotators. Each annotator is responsible for their designated block, enabling the parallel processing of annotations. These findings suggest that our setting is suitable for complex real-world label distributions. However, our study on block diagonal labels investigates a more extreme scenario, where an image is associated with only a single coarse label, as illustrated in Fig 1 .(d).To demonstrate the superiority of our block diagonal labels, we conduct simulation and analysis of different settings, given a dataset of 𝑁 images and 𝑀 categories which are divided into 𝑂 blocks. 𝑁 𝑖 indicates the number of images in block 𝑖 and 𝑀 𝑖 indicates the number of categories in block 𝑖. Each block requires at least one annotator, so we make the assumption that there are a total of 𝑂 annotators.From learning, we assume that the cost of learning category 𝑖 is 𝑝 𝑖 . In the other settings, all annotators are required to learn all the categories, resulting in a cost of 𝑂 × 𝑀 𝑖=1 𝑝 𝑖 . However, in block diagonal labels, each annotator only learns the categories within their assigned block, significantly reducing the cost to 𝑂 𝑗=1 𝑀 𝑗 𝑖=1 𝑝 𝑖 . The comparison is as follows:EQUATIONFrom annotating, we assume that the cost of annotating category 𝑖 is 𝑞 𝑖 . In full labels, the cost is 𝑁 × 𝑀 𝑖=1 𝑞 𝑖 . In block diagonal labels, the cost is 𝑂 𝑘=1 𝑁 𝑘 𝑗=1 𝑀 𝑘 𝑖=1 𝑞 𝑖 . The comparison is as follows:EQUATIONIn the Appendix, we conduct additional experiments for partial labels and block diagonal labels under conditions with a similar number of annotated labels.Exiting partial labels and single positive labels settings only consider missing labels resulting from annotation errors, but ignore the cost during the annotation pipeline. Our proposed setting of block diagonal labels reduces the annotation workload from both learning and annotating. In full labels, the category 𝑐 𝑖 is labelled by positive '1' or negative '-1'. As for missing labels, 𝑐 𝑖 ∈ {1, -1, 0}, where 0 is an "unknown" annotation.In our block diagonal labels setting, all categories are divided into subsets {𝑆 𝑖 |𝑖 = 1, 2, • • • , 𝑘 } based on the semantic and relevance, where 𝑘 is the number of subsets. 𝑠 𝑖 = |𝑆 𝑖 |, 𝑘 𝑖=1 𝑠 𝑖 = 𝑀, where 𝑠 𝑖 is the size of subset 𝑆 𝑖 . The annotators only annotate 𝑆 𝑖 of their own block 𝐵 𝑖 . The final dataset 𝐷 is the union of all block datasets𝐷 𝐵 𝑖 |𝑖 = 1, 2, • • • , 𝑘 .The label vector of an image in block 𝐵 𝑗 is as follows:EQUATIONThe annotation number per image in block 𝐵 𝑗 is 𝑠 𝑗 , where 𝑠 𝑗 ≪ 𝑀. Unknown labels are still the primary concern requiring resolution in block diagonal labels.Given an image 𝐼 with missing labels, its label vector 𝑦 is composed of positive labels 𝑃 𝐼 = {𝑖 |𝑐 𝑖 = 1}, negative labels 𝑁 𝐼 = {𝑖 |𝑐 𝑖 = -1}, and unknown labels 𝑈 𝐼 = {𝑖 |𝑐 𝑖 = 0}, where|𝑃 𝐼 | + |𝑁 𝐼 | + |𝑈 𝐼 | = 𝑀.There are two naive baselines that can be applied in missing labels, which show different strategies for the unknown labels. All Ignore (AI). In this baseline, we simply ignore the unknown labels. In Fig 4 .(a), only positive and negative labels are used as supervisory signals to calculate the loss:EQUATIONwhere the prediction output for category 𝑐 is 𝑝 𝑐 . The ignored labels do not compute gradients to optimize the network and the predictions of those are seen as the ground truth, which misses valuable training signals.All Negative (AN). In a typical multi-label dataset, there are far more negative labels than positive labels. Therefore, we assume all the unknown labels as negative labels in Fig 4 .(b). The loss function is as follows:EQUATIONThere are also some false negatives of the unknown labels, which leads the model to predict the true positive labels incorrectly. Loss Function. The loss function in AI and AN is the Binary Cross Entropy loss, which is defined as:EQUATIONHowever, the severe imbalance between positive labels and negative labels is an important problem for multi-label learning [38] .PASL loss [1] is a modified ASL loss [29] to mitigate the imbalance problem for missing labels:EQUATIONwhere 𝛾 is a focusing parameter, which performs more attention to the hard samples with a larger value. In the following paper, PASL loss is the base loss function whose abbreviation is 𝐿 + (𝑝 𝑐 , 𝛾), 𝐿 -(𝑝 𝑐 , 𝛾) .PASL is based on AN:EQUATIONwhere 𝛾 + < 𝛾 -< 𝛾 𝑢 . Unknown labels are the most frequent and have less confidence than negative labels. 
hard pseudo-labeling (hpl): 
 A naive approach to reduce the false negatives is to use a constant threshold for all categories. If the prediction probability for an unknown label is below the pre-defined threshold, it is likely a true negative. HPL computes a pseudo label for each unknown label using the predictions:EQUATIONwhere 𝑈 𝐼 = 𝑈 𝑁 𝑒𝑔 𝐼 ∪ 𝑈 𝐼𝑔𝑛 𝐼 , and 𝜏 is a pre-defined hard threshold. As illustrated in Fig 4 .(c), the unknown labels are partitioned by a hard threshold. Then, the overall loss function is calculated as follows:EQUATIONwhere 𝑈 𝐼 is replaced by 𝑈 𝑁 𝑒𝑔 𝐼 in Eqn. (8) . Like [1] , 𝑈 𝐼𝑔𝑛 𝐼 is not involved in the final loss, where the positive and negative are really hard to distinguish.[45] indicates that using identical hard thresholds for all categories fails to consider different learning statuses and learning difficulties of different categories. Thresholds should be flexibly updated for each category during the training stage. Unlike [45] for single-label classification, we maintain memory banks of positive predictions to compute the learning status of categories:𝑀𝐵 = {𝑚𝑏 1 , 𝑚𝑏 2 , • • • , 𝑚𝑏 𝑀 },where 𝑚𝑏 𝑖 is the memory bank of category 𝑐 𝑖 . We store the positive predictions of the past 𝑍 mini-batch, where𝑚𝑏 𝑖 = 𝑝 𝑎𝑣𝑔 𝑧 𝑐 𝑖 𝑍 𝑧=1 is updated with first-in-first-out strategy. 𝑝 𝑎𝑣𝑔 𝑧 𝑐 𝑖is the average predictions of positive 𝑐 𝑖 in 𝑧 mini-batch. Note that we only count the predictions of true positive labels 𝑃. Then, the learning status of 𝑐 𝑖 is formulated as follows:EQUATIONwhere 𝛼 𝑐 𝑖 ∈ [0, 1]. Since negative samples are more frequent compared to positive samples, the model tends to predict them with low probabilities. Larger 𝛼 𝑐 𝑖 indicates that the model can better classify the category 𝑐 𝑖 . The flexible threshold is defined as: where 𝜏 𝑡𝑜𝑝 and 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 are the top and bottom boundaries of the threshold, respectively. Small 𝜏 𝑐 𝑖 means 𝑐 𝑖 is hard to learn, indicating that fewer false negatives in unknown labels should be considered in the model optimization. As 𝜏 𝑐 𝑖 grows, more negative samples are encouraged to be learned.EQUATIONInstead of using a fixed value in HPL, dynamic thresholds are used to divide unknown labels:EQUATIONThe final loss of APL is the same as Eqn. (10) . The thresholds are dynamically adjusted at each iteration. As shown in Fig 4 .(d), dynamic thresholds are applied. 
experiments 5.1 experimental settings: 
 Datasets. Several widely used multi-label benchmarks, VOC-2012, COCO2014, NUSWIDE, and OpenImages, are used to analyze the performance of our proposed setting and method. To generate block diagonal labels datasets, we randomly select only one block that has at least a positive label. For fair comparisons, the same random seed is used to create the datasets in all experiments.We show the block divisions on VOC2012, COCO2014, and NUSWIDE in the Appendix. There are 9 blocks, 11 blocks, and 12 blocks for VOC2012, COCO2014, and NUSWIDE, respectively. As for partial labels, we randomly discard a fixed portion of labels per image following [13] . We maintain a similar proportion of known labels in partial labels as in block diagonal labels. The annotation proportions for VOC2012, COCO2014, and NUSWIDE are nearly 11%, 10%, and 16%, respectively. Comparison between block diagonal labels and partial labels is provided in the Appendix. Evaluation Metrics. To fairly compare with other methods, we employ the mean Average Precision (mAP), per-category F1-score (CF1), and overall F1-score (OF1) as evaluation metrics. Implementation Details. We employ the TResNet-Large [30] and ResNet-101 [15] architectures, which are pre-trained on ImageNet [9] dataset, as our backbone networks. Our training configuration modified from [1] . The model is trained for 30 epochs using Adam optimizer [19] with true-weight-decay [26] of 1e-4 and 1-cycle cosine annealing policy [33] . The maximal learning rate is 2e-5 for VOC2012 and 6e-5 for COCO2014 and NUSWIDE, respectively. We scale the learning rate according to the batch size with the formula: lr 𝑠𝑐𝑎𝑙𝑒𝑑 = lr 64 × batchsize. We resize the input images to 448 × 448. For data augmentation, we use the Cutout [11] with a factor of 0.5 and the auto-augment [8] . We use 4 GPUs with batch size 128.For all benchmarks, we use 𝛾 + = 0, 𝛾 -= 4, 𝛾 𝑢 = 6 in Eqn. (10) . We use the HPL in the first epoch as a warmup phase. Hyperparameters in Eqn. (12) are as followed: In VOC2012, 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 = Table 2 : DualCoOp [34] with our proposed APL in block diagonal labels. The results evaluate the effectiveness on powerful vision-languange CLIP [28] . = 0.85. In COCO2014, 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 = 0.25, 𝜏 𝑡𝑜𝑝 = 0.90. In NUSWIDE, 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 = 0.60, 𝜏 𝑡𝑜𝑝 = 0.90. The length of the memory bank 𝑍 is set to 400. Following [1] , we apply the exponential moving average strategy (EMA) with a decay of 0.9997 for higher performance. Unless stated otherwise, the experiment results are without EMA. Compared Methods. For a fair comparison, the training configuration of compared methods is the same as ours. We apply hyperparameter searches to determine the optimal parameters. Only parameters different from the default values of the published paper will be listed.Two naive baselines mentioned in Sec 4.1 of the main paper, all ignore (AI) and all negative (AN), use BCE loss. ASL applies the asymmetric loss for AN, where 𝛾 + = 0, 𝛾 -= 4. MLdecoder uses full-decoding ML-decoder structure as the classification head based on ASL. 𝜏 in SPLC is 0.65. Three versions of LL are conducted and the best performance of them is listed in Table 1 of the main paper. In LL-R, Δ 𝑟𝑒𝑙 is 0.2, 0.3, and 0.1 for VOC2012, COCO2014, and NUSWIDE. In LL-C t , Δ 𝑟𝑒𝑙 is always 0.1. In LL-C p , Δ 𝑟𝑒𝑙 is 0.3, 0.5, and 0.3 for VOC2012, COCO2014, and NUSWIDE. BoostLU is based on LL-C p . PASL applies the partial asymmetric loss (Eqn.8 of the main paper), where 𝛾 + = 0, 𝛾 -= 4, 𝛾 𝑢 = 6. Selective is based on PASL: In VOC2012, 𝜂 = 0.6, 𝐾 = 100. In COCO2014, 𝜂 = 0.6, 𝐾 = 50. In NUSWIDE, 𝜂 = 0.3, 𝐾 = 20. Selective uses AI to estimate the class distribution in the first stage.Table 1 shows the block diagonal labels results on VOC2012, COCO-2014, and NUSWIDE with TResNet-Large and ResNet-101 backbones. We compare our proposed APL with state-of-the-art methods. ASL and MLdecoder-ASL treat unknown labels as negatives in the experiments. In the first stage of Selective, the label prior is estimated. Although Selective with TResNet-Large reaches the best performance on COCO2014, the results in other experiments are unstable. By adopting the TResNet-Large backbone, APL surpasses other methods by 1.09% and 0.82% mAP on VOC2012 and NUSWIDE. APL also achieves comparable performance on COCO2014. APL Table 4 : Comparison with state-of-the-art single positive labels methods. FL denotes full labels, SPL denotes single positive labels, and BDL denotes block diagonal labels. # Positive denotes the number of positively annotated labels and # Annotated denotes the number of all annotated labels. † denotes the exponential moving average strategy. * denotes methods based on vision-language pretrained CLIP. with ResNet-101 outperforms all existing approaches. The experiments with different backbones validate the robustness of our method. Moreover, EMA averages model weights [35] to further improve the performance of APL. Specifically, APL-EMA reaches 81.92% and 63.26% mAP on COCO2014 and NUSWIDE, respectively. The results on OpenImages, which is a large-scale multi-label dataset, are shown in Table 3 . We compare our APL with the two methods that perform well in Table 1 . Our APL outperforms other methods by 1.15% mAP. In addition, our APL with EMA achieves 75.01% mAP.In Table 2 , we conduct further evaluation based on powerful vision-language models. DualCoOp [34] adapts the knowledge acquired in CLIP [28] to multi-label classification with missing labels. Although DualCoOp employs prompts to provide positive and negative contexts, this approach overlooks unknown labels during training which may leads to a sub-optimal issue [1] . Notably, Dual-CoOp enhanced with our APL surpasses the standard DualCoOp by a considerable margin of 6.54% mAP on COCO2014.(1) SPL can be considered an extreme case where each block in BDL contains only one class, converting a multi-label dataset into a single-label dataset. (2) Such extreme setting potentially harm performance in practical applications due to insufficient supervision. Table 4 indicates a notable gap between SPL and FL. However, our APL method in BDL, with 10.71% more positive labels, achieves performance close to FL. (3) We further compare state-of-the-art SPL methods. Despite some methods leveraging vision-language pertained CLIP, our APL method still achieves better results. The Effectiveness of Individual Components. In Table 5 , we validate the effectiveness of each component in block diagonal labels and partial labels. We generate the partial labels datasets with a similar annotation proportion as the block diagonal labels ones, as detailed in the Appendix.Our proposed HPL and APL apply PASL loss to deal with the positive-negative imbalance issue. HPL sets a constant threshold to find true negatives and reduces the number of false negatives to mitigate performance. However, the thresholds of each category should be dynamically adjusted. Our APL flexibly adjusts thresholds for different classes, which improves the performance in all ablation experiments. Hyper-parameters Analysis. We investigate the effectiveness of hyper-parameters. As shown in Fig 6 , the first col shows the impact of 𝜏 𝑡𝑜𝑝 when 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 is fixed and the second col shows the impact of 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 when 𝜏 𝑡𝑜𝑝 is fixed. Despite variations in values across the three datasets, performance remains consistently near the optimal values. In practice, we can initially set 𝜏 𝑡𝑜𝑝 at 0.9 and conduct a grid search for 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 . Once a satisfactory 𝜏 𝑏𝑜𝑡𝑡𝑜𝑚 is obtained, we proceed with a grid search for 𝜏 𝑡𝑜𝑝 . The pseudo labels are marked in blue or red. Blue indicates a correct prediction and red indicates an incorrect prediction. Four cases show that our APL predicts correct labels better than HPL, which demonstrates the superiority of the dynamic threshold. However, in the last case, some errors still occur in the prediction process.The complex scene contains too many small objects, thus confusing the two approaches to predict the pseudo labels. 
conclusion: 
 In this paper, we demonstrated the main challenge of the multi-label dataset construction from two aspects i.e. learning and annotating. We introduced a new setting of multi-label classification with block diagonal labels, which was considered from both sides to reduce the annotation workload. We compared it with the exiting missing labels setting to prove the effectiveness of our setting. Additionally, an adaptive pseudo-labeling method was proposed, which is a simple yet effective method without any label prior or complex architectures. Our APL achieves state-of-the-art performance on different benchmarks. For further studies in our block diagonal labels, we plan to utilize the prior of coarse label present in each image to improve the prediction accuracy of unknown labels.  
 
 Appendix: To evaluate the superiority of block diagonal labels, we compare the experiments of block diagonal labels with those of partial labels. As illustrated in Table 6 , the number of annotated labels is almost the same between block diagonal labels and partial labels. We generate the block diagonal labels datasets by randomly selecting a block of labels. For comparison between these two settings, we generate the partial labels datasets with a similar annotation proportion as the block diagonal labels ones. The positive proportions #𝑃 𝑃𝑜𝑠 of them differ greatly. For example, #𝑃 𝑃𝑜𝑠 of block diagonal labels is greater than that of partial labels by 4.45% on VOC2012. Randomly annotating a fixed percentage of labels in partial labels, which may generate images with all negative labels, will result in a very small number of positives. Fig 8 shows results of the state-of-the-art methods in block diagonal labels and partial labels. The results in our block diagonal labels significantly surpass the results in partial labels on all benchmarks, which proves the superiority of our setting. Especially for SPLC, the performance gaps between these two settings are remarkable. Evident 19.13%, 11.94%, and 8.46% mAP gains are achieved on VOC2012, COCO2014, and NUSWIDE, respectively. The recent approaches treat most unknown labels as negatives. This assumption is mostly correct due to the large number of true negatives. The effect of annotated negative labels is reduced, while the number of positive labels has a significant impact on the final performance. Therefore, the block diagonal labels setting not only reduces the annotation workload from both learning and annotating aspects but also increases the number of positives, leading to a notable enhancement in performance.In addition, on all benchmarks, we find the performance gaps of our APL are relatively small compared to other methods. Our APL outperforms other methods by a large margin in all experiments. Such results prove the effectiveness and robustness of our APL.  


Decision: This paper introduces a method to reduce annotation workload in large-scale multi-label datasets. Concerns were raised regarding experimental fairness and other issues. All reviewers agree on the merits of the work. The ACs support this assessment and recommend accepting the paper. Please incorporate reviewers' suggestions into the camera-ready version to enhance the paper's clarity and include new results from the rebuttal.

Reviews: review: paper_summary: paper_summary: This paper proposes a new setting to reduce the annotation cost.
strength_weakness: strength_weakness: 1.The paper is well-written and easy to follow. 

2.The analysis of the methods section is very good, and the descriptions of various strategies are very detailed.
limitations: limitations: 1.Insufficient research on related work. Based on my knowledge, the setting of biased markers is not limited to the one mentioned in this work. Analyzing the difference between literature [1] and your setting. It is not inevitable that some samples lack positive labels.

2.The discussion and analysis of single positive multi-labels learning are also insufficient, and the references are too outdated, seemingly all before 2022. In fact, many relatively new jobs have already achieved good results [2,3], which is inconsistent with the shortcomings of single positive multiple labeling discussed by the author in Chapter 3.

3.The experimental part is unfair, and the single positive multi labeling method used for comparison is too outdated. The author should compare it with some new methods. In addition to references [2,3], as far as I know, there are still many methods based on vision-language pretained (VLP) models. By utilizing the transfer ability of VLP models, it can effectively compensate for the shortcomings brought by single positive labels. The author's experiments on VLP models are also not comprehensive.

[1] Partial Multi-Label Learning with Meta Disambiguation. KDD 2021.

[2] Revisiting Pseudo-Label for Single-Positive Multi-Label Learning. ICML 2023.

[3] Hierarchical Prompt Learning Using CLIP for Multi-label Classification with Single Positive Labels. MM 2023.
paper_summary: paper_summary: This paper addresses the challenge of annotating large-scale multi-label datasets by proposing a new setting called block diagonal labels. The method aims to reduce the annotation workload by dividing categories into subsets based on semantics and relevance, allowing annotators to focus on specific and highly relevant subsets. The authors also introduce an Adaptive Pseudo-Labeling (APL) method to predict unknown labels via dynamically adjusting thresholds. Extensive experiments on benchmarks like VOC2012, COCO2014, NUSWIDE, and OpenImages demonstrate the effectiveness of the proposed method, achieving significant improvements in mean Average Precision (mAP), per-category F1-score (CF1), and overall F1-score (OF1), with a reduced annotation effort.
strength_weakness: strength_weakness: The introduction of block diagonal labels to reduce annotation workload is an innovative approach that addresses a practical problem in multi-label classification.

The proposed setting significantly reduces the number of annotations required, making it feasible to create large-scale multi-label datasets.

The introduction of APL, which dynamically adjusts label prediction thresholds, enhances model performance, as well as improving the applicability of the block diagonal labels in real-world practice. 

The paper provides a detailed explanation of the methodology, ensuring transparency and reproducibility of the experiments.
limitations: limitations: The effectiveness of the block diagonal labels setting heavily relies on how well the categories are partitioned, which may be challenging in real-world scenarios where label distribution is not as easily controlled or partitioned.

Although the paper shows that the proposed APL works well with the block diagonal labels, it doesn't consider the difference of importance of each class. In practice, certain classes are much more important to be labelled than others, for example, missing labelling a bin might not matter much but missing labelling a pedestrian might cause catastrophic result.
paper_summary: paper_summary: This paper proposed a new multi-label learning method that tackle the high annotation worekload problem with block diagonal labels.
strength_weakness: strength_weakness: - The paper is clearly written and easy to follow.
- The idea is interesting and the solution is technical.
limitations: limitations: - A suggestion. I think that the introduction of both learning and annotating in abstract is not easy to be understood for readers. Maybe, you can say different annotators have different knoelwdge background so that they are good at annotating different types of labels. Or, an annotator to annotate examples with few number of categories would achieves better performance than that with large number of labels.

- The experiments. The number of classes in each datasets is not large. You should conduct experiments on MLL datasets with extreme labels.
paper_summary: paper_summary: The article proposes a novel method called PML-LENFN (Partial Multi-label Learning based on Label Enhancement of Near-Far Neighborhood Information and Nonlinear Guidance) to address the challenges of partial multi-label learning (PML). PML involves accurately predicting multiple labels for instances in multi-label data that contain noise. The proposed method aims to fully utilize the correlation between samples and labels and consider the nonlinear nature of the model.
strength_weakness: strength_weakness: 1. PML-LENFN incorporates label correlation by reconstructing the original binary label information using near and far neighbor information. This enables the model to better handle noise and improve the accuracy of label prediction.

2. By utilizing information from near and far neighbors, PML-LENFN effectively eliminates the influence of noise labels in candidate labels. This enhances the robustness of the model and improves the quality of predictions.

3. PML-LENFN employs nonlinear mapping to capture the nonlinear relationship between features and labels. This is particularly beneficial in real-world scenarios where the underlying model is often curvilinear and cannot be adequately represented by linear models.
limitations: limitations: 1. The font on the picture is so small that it's hard to read.

2. The proposed method involves too many parameters, which seriously affects the practicality of the method.

3. The computational complexity of the proposed method should be given, and the corresponding experiments should be conducted.