Label-Aware Global Consistency for Multi-Label Learning with Single Positive Labels
Authors: Ming-Kun Xie
Jia-Hao Xiao
Sheng-Jun Huang
Venue: NeurIPS 2022 Accept (2022)

Abstract: In single positive multi-label learning (SPML), only one of multiple positive labels is observed for each instance. The previous work trains the model by simply treating unobserved labels as negative ones, and designs the regularization to constrain the number of expected positive labels. However, in many real-world scenarios, the true number of positive labels is unavailable, making such methods less applicable. In this paper, we propose to solve SPML problems by designing a Label-Aware global Consistency (LAC) regularization, which leverages the manifold structure information to enhance the recovery of potential positive labels. On one hand, we first perform pseudo-labeling for each unobserved label based on its prediction probability. The consistency regularization is then imposed on model outputs to balance the fitting of identified labels and exploring of potential positive labels. On the other hand, by enforcing label-wise embeddings to maintain global consistency, LAC loss encourages the model to learn more distinctive representations, which is beneficial for recovering the information of potential positive labels. Experiments on multiple benchmark datasets validate that the proposed method can achieve state-of-the-art performance for solving SPML tasks.

Label-Aware Global Consistency for Multi-Label Learning with Single Positive Labels
Ming-Kun Xie, Jia-Hao Xiao, Sheng-Jun Huang!
College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics MIIT Key Laboratory of Pattern Analysis and Machine Intelligence, Nanjing, 21 106 {mkxie, .j jiahaoxiao, huangs /nuua. edu.
Abstract
In single positive multi-label learning (SPML), only one of multiple positive labels is observed for each instance. The previous work trains the model by simply treating unobserved labels as negative ones, and designs the regularization to constrain the number of expected positive labels. However, in many real-world scenarios, the true number of positive labels is unavailable, making such methods less applicable. this paper, we propose to solve SPML problems by designing a Label-Aware global Consistency (LAC) regularization, which leverages the manifold structure information to enhance the recovery of potential positive labels. On one hand, we first perform pseudo-labeling for each unobserved label based on its prediction probability. The consistency regularization is then imposed on model outputs to balance the fitting of identified labels and exploring of potential positive labels. On the other hand, by enforcing label-wise embeddings to maintain global consistency, LAC loss encourages the model learn more distinctive representations, which is beneficial for recovering the information of potential positive labels. Experiments multiple benchmark datasets validate that the proposed method achieve state-of- of ththhhear art performance for solving SPML tasks.
Introduction
Multi-label learning (MLL) is practical and effective learning framework for tackling objects with complex semantics, where each instance assumed to be associated with multiple class labels. For example, an image may be annotated with labels beach, sea and sky simultaneously. Multi-label learning aims to train classifier that can accurately predict all the relevant labels for unseen instances. has been successfully applied into many real-world applications, such as image annotation (2). scene understanding 22) facial attribute recognition [3
Given the output space is exponentially larger than that of single-label learning, it often requires large number examples with precise annotations to train an effective MLL classifier. However, in many real-world scenarios, it is difficult and costly to collect the precise annotations. To reduce the labeling cost, an alternative solution to ask the annotators to assign the most obvious label to each instance, leading only single positive label available. Such problem has been formulated learning framework called single positive multi-label learning (SPML) [4. In this case, except for the annotated positive label, each of other labels could be positive negative but unknown, thus the learning task becomes much more challenging due to the lack of supervision.
To solve SPML problems, the previous work 4 intuitively designs the "assume negative" (AN) loss, which treats all unobserved labels as negative ones, and trains an MLL classifier with the conventional
Both authors contributed equally this research.
Conrespondence to: Sheng- Jun Huang (huangsj @ nuaa.cdu.cn).
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

Similar
Person
Person
Bike
Horse
Embedding w.rt. observed label
Labeled image
Unlabeled image
Embedding w.tt. unobserved label
SSL case
SPML case
Figure 1: An illustration of the comparison between clustering regularization and our proposed label-aware global consistency regularization.
binary cross entropy (BCE) loss. Unfortunately, AN loss often introduces many false negative labels, leading to a noticeable decrease in performance of the model. To alleviate the negative impact false negative labels, authors further propose a regularization to constrain the number of expected positive labels. However, since the true number of positive labels cannot be accessible many world scenarios, it often requires additional estimation elaborate syper-parameter tuming to determine its true value.
Clustering assumption aims to recover the true labels for unlabeled examples by utilizing the global consistency, i.e., encouraging points on the same structure (typically referred to as a clustering) to be assigned with the same label. The idea clustering assumption has been successfully extended to solve many Unfortunately, applying clustering regularization to solve SPML problems is not as straightforward as SSL case, since it is difficult to measure the underlying similarity between any two examples with multiple labels. For example, in SSL case (see the left side in Figure [ is feasible to distinguish examples from different classes by measuring their similarity based on image-level feature representations. While in multi-label case (see the right side Figure []. it is hard to measure the similarity between any two images, which share the same object (e.g., person), while still have individual objects (e.g., bike and horse).
35 is first proposed to solve emmi-supervised learning (SSL) problems.
wepervesupervised learning problems 23 28
In this paper, we design a Label-Aware global Consistency (LAC) regularization to leverage the manifold structure information for solving SPML problems. On one hand, we first identify the potential positive labels mislabeled by AN loss based on class probabilities predicted by the model. The pseudo-labeling consistency regularization is then used for training model to boost its ability on identifying the potential positive labels. On the other hand, as illustrated in the right side of Figure [] the LAC loss is designed encourage the global consistency among label- wise embeddings, i.e., pulling together intra-class embeddings while pulling apart inter-class embeddings. By leveraging the manifold structure information, the LAC loss enhances the distinctiveness of learned feature representations, which is beneficial for identifying the potential positive labels. Extensive experimen- tal results multiple benchmark datasets demonstrate that our method can achieve state-of -iree----heerrr performance.
2 Related Work
There are a plenty of studies in multi-label learning literature. Existing methods can be roughly classified into three groups. Considering the traditional binary cross entropy (BCE) loss often suffers from the positive- negative imbalance issue, the first kind of methods aims design specific loss functions to alleviate this issue. Among them, asymmetric loss (ASL) [I]] is a representative method that dynamically down-weights and hard-thresholds easy negative examples. The second kind of methods is proposed to capture the label correlations, which are regarded an essential element for improving the practical performance of MI... MI-GGN 2 and variant (30] utilize the graph convolution network (GCN) capture the co-occurrence correlations among labels. Considering an

---

image often contains multiple objects, the last kind of methods is designed to locate areas of interest related to semantic labels by using attention technique.
Traditional multi-label learning assumes that each instance has been annotated with all of its relevant labels. However, in many real-world scenarios, it is costly and difficult precisely assign an image with multiple labels. To solve this issue, the current efforts are mainly dedicated to reducing the labeling cost by training an MLL model with only partial labels. For example,
[ fine-tune the pre-trained model on ImageNet with partial annotations and perform pseudo-labeling to recover the potential positive labels. In [6], authors propose two-stage interactive learning framework that performs similarity learning and CNN training interactively to improve performance of each other. (20) propose an unified framework to exploit instance-level and prototype-level semantic representation for recovering the potential positive labels.
SPML is the extreme case of multi-label learning with partial labels, where only of multiple potential positive labels can be observed. The earliest work intuitively treats all unobserved labels as negative ones, and trains an MLL model by introducing the regularization terms that alleviate the negative impact of false negative labels [4]. The spatial consistency [26] is designed to avoid the bias toward negative prediction by maintaining the consistency of classification maps between the network and exponential moving average (EMA) model. Unlike aforementioned methods, [36] treat unobserved labels as unknown and propose to maximize the entropy of predicted probabilities for unobserved labels. Furthermore, authors combine the asymmetric pseudo laeulo-labeling and self- self-paced strategy obtain more accurate pseudo labels.
Besides SPMI,, a variety of reakky-superviied multi-label learning frameworks have been widely studied, including learning with multi-label noise [4), partial multi-label learning [32 [4], few-shot multi-label learning []] learning with pairwise relevance comparison [33], and semi-supervised multi-label learning (29).
3 The Proposed Method
In the SPML problem, let x E X be a feature vector and ÿ E Y be its corresponding labels, where X' = RR is the feature space and y = {0, 1 is the target space with q possible class labels. Here, d indicates the j-th label is the only observed positive label for instance æ while J, 0 indicates the j-th label cannot be observed. Since only single positive label can be observed, we have Σj= figg =1 for every instance x;, We further denote by y the true label vector for instance x. Let p(y) x)) be the predicted probability distribution over classes and p(y, be the predicted probability of the j-th class for input æ. We use [q] to denote the integer set {1.....9) 9).
Label-wise Embedding Model To obtain the label-wise embedding with respect to every label, existing methods can be roughly divided into two groups, ie., saall--vveagepooliiin (GAP) based methods [2] and attention based methods [8 21 For simplicity, we adopt the attention-baser method to construct the lasel-wise embedding model Assume that the label-wise embedding model is composed of a label-wise embedding decoder f. which can generate a high-dimensional label-wise feature representation (x) with respect to the j-th label of instance and a classification head h, which can output the predicted probability of the j-th class p(y; (x) = hx)) We further denote by g()) non-linear projection head, which transforms a nigh-dimensional embedding fy (I) to low div-dimensional embedding x; = =
In traditional multi-label classification, the most commonly used loss function is binary cross entropy (BCH) loss, which decomposes the original task into multiple binary classification problems. Formally, given a batch of examples ((æ1, y:) y:) the BCE loss can be defined as follows:
6
2
LBCE = -
Vi log(p(,,1x:)) + (1 - -ig) log(1 p(y, (x,))
1=1
However, the conventional BCE loss cannot be directly applied to solve SPML problems, where only a single positive label can be accessible for every instance. Training the neural network with only one positive label often makes model collapse to a trivial solution. To mitigate this issue, an intuitive method is to treat all unobserved labels as negative ones, and then the neural network can be trained with the resulting AN loss 4 given a batch of SPML
The assume negative (AN) loss
3

---

Pseudo-labeling consistency regularization
N(-)
p
Augi
PLLC
Psoudo labels
Embedding encoder fO
Aug,
LAC
8())
Label-wise embeddings
Low Low-dimensional embeddings
A betch of SPML examples
Intre-class connection
Ioter-class
connection
Label-aware global consistency regulartzation
Figure 2: An illustration the proposed learning framework.
examples {(x,, 0:))=r:
CAN b
Vig log(p(y)læ;))
úis) log(1 p(y; (z:)))
Unfortunately, AN loss often introduces a large number of false negative labels, which leads noticeable decrease in generalization performance of the model. To alleviate the negative impact of false negatives, design the following two regularization losses: 1) the pseudo-labeling consistency regularization CPiC to identify the potential positive labels mislabeled by AN loss, 2) the label-aware global consistency loss LLcC to leverage the manifold structure information for learning a more distinctive feature representation. The overall objective function can be defined as:
CAN + APLCCPLC + ALACCLAC
(3)
Figure 2 provides illustration of the proposed learning framework. For batch of SPML ex- amples, we generate the pseudo labels based on their weakly-augmented versions. On one hand, the consistency regularization is conducted between the pseudo labels and model outputs the versions to prevent from over wer-ffttting to false negative labels. On the other hand, the label-awer global consistency is used to learn more distinctive feature representations such that intra-class embeddings are pushed together while inter-class embeddings are pushed away. In the following content, will introduce these two regularization losses in detail.
3.1 Pseudo-Labeling Consistency Regularization
In this section, we propose the pseudo-laheling consistency regularization for recovering the ground- truth labeling information of unobserved labels.
We define the loss LPLC between the pseudo labels and the model predictions. Specifically, for the j-th label of æ;, let p.j = p lugg (æ,)) denote the pseudo labels, where (.g.. is a stochastic weak augmentation. To avoid the trivial solution, let p = p(u; Aug, (x:)) denote the model prediction, where Aug,( (-) is a stochastic strong augmentation. Then, the pseudo-labelin consistency loss can be formulated as:
LPLC
UP 2 810 = 0) log(pi,) + L(P.) < 810ا = 0) log( 1 p(,)
(4)
1=1
where 5 (or 8 1 8) threshold that controls how many positive (or negative) pseudo labels would be used for model training. Here, I(-) is the indicator function that output if the condition

---

satisfies while O, otherwise, and the condition 0ú = 0 is used to ensure the loss CPcC is only imposed on the unobserved labels. Generally, the loss LPcc imposes a consistency regularization on the model such that encourages it to output the same probability with respect to unobserved label for an example even after is strongly augmented. By using a strong augmentation, LPcC would alleviate the over-fittigg issue to false negative labels and improve the ability of identifying potential positive labels.
3.2 Label-Aware Global Consistency Regularization
As discussed in the above section, LPLC can be regarded as consistency regularization on the high-level representation, i.e., the final output of network. With the increase of the depth of the neural network, the impact on network parameters of enforcing the consistency on the high-level representation may be gradually degraded [12, and thus weakens the effectiveness of the proposed loss CPLC- To further alleviate the harmfulness caused by the false negative labels, motivated by the clustering assumption [35], we design label-aware global consistency regularization LLAC to maintain global consistency among label-wise embeddings. The main idea is to pull intra-class label-wise embeddings to be close while pull inter-class label-wise embeddings to be separated. The intuition is that by maintaining the global consistency among label-wise embeddings, LAC enhances the distinctiveness of learned representations, which beneficial for recovering the true labeling information of potential positive labels.
To derive the LLAC loss, for a batch of examples, we first use the label-wise embedding decoder and projection head to produce a set of embeddings = [ S k S B), where B = b is the total number of embeddings for instances with q classes. For notational convenience, for the embedding with respect to j-class of instance 2,, we define the mapping i = ik), = (kk to bridge the embedding index k with its original instance index and class index j.
In order to exploit as more as possible supervised information to learn distinctive representations, we compute the LLAC loss only unobserved labels but also observed ones. To obtain a unified formulation, for the single positive label Dij = 1, we treat its pseudo label as p = although is actually the true label. For any two intra-class label-wise embeddings 2₁ and z by denoting
i = i(k)..j j = (kk and = " = j(k'), we can construct the intra-class connection matrix
of size B x B to connect them:
1 if و and p
2 ö. p. 2
(5)
otherwise
From the equation 55. every embedding a pseudo label larger than the threshold would be treated the anchor embedding and be connected with embeddings from the same class that satisfy the same condition. Since a low prediction probability indicates the corresponding label is likely to be negative, its embedding is invalid and would never be used.
Similarly, we can construct the inter-class connection matrix i of the same size to connect any two inter-class label-wise embeddings:
{1 j and p.j 2 8, p.y. 2
TR
(6)
O otherwise
From the equation (). an anchor embedding is connected with embeddings from other classes with pseudo labels larger than threshold 8.
Next, we can train the label-wise embedding decoder f and the projection head g by enforcing the global consistency among label-wise embeddings. It is expected to pull together an anchor label-wise embedding and its intra-class label-wise embeddings while separate the anchor label-wise embedding from its inter-class label-wise embeddings. To achieve this goal, given the connection matrices I and I, we define label-aware global consistency loss as follows:
B
B
exp(x1 EM/TT)
LLAC = B
TEk log
EB Iss exp(=x 21/T)
k=
It is noteworthy that when l: = k', i.e., an anchor embedding and itself, in order to avoid a trivial solution, we produce two different embeddings 2 and z/6, by imposing two different augmentations to its original training image.
5

---

Memory queue. Inspired of the previous work [14], we introduce a memory queue to store the past K embeddings and their corresponding pseudo labels MQ = { P P E [K]] where p the pseudo label for embedding z It is noteworthy that the memory queue consists of embeddings with respect to both unobserved and observed labels. Then, the connection matrices and T are extended to size of B x (B- + K). This means we would impose the label-aware global consistency regularization between an anchor embedding of the current embedding set and every embedding of the memory queue.
Two-stage training strategy. In our experiments, we find that training model with Eq. in an end- to-end fashion often leads to an unfavorable performance that can be worse than that without LAAC- The main reason is that at the early stage of training, the model suffers from the insufficient training issue, which leads model to produce low-quality label-wise embeddings. Enforcing low-quality label-wise embeddings to maintain global consistency may destroy the feature representation learning and thus significantly degrades model performance. To solve this issue, we propose a two-stage training strategy, which aims to let model learn high-quality label-wise embeddings firstly and then perform LAC regularization. Specifically, at the first stage, we train the model by minimizing AN loss and PLC loss, i.e., LAN + LPLC. until it reaches the convergence. The model is expected to produce sufficiently high-quality label-wise embeddings for the second stage training. At the second stage, we fine-tune the model with Eq consisting of all three losses to learn a more distinctive representation.
4 Experiments
In this section, we first perform experiments to compare our method with state-of-the-an methods; then, conduct ablation studies examine the effectiveness of each component for the proposed method.
4.1 Experimental Settings
Datasets. We perform experiments to evaluate our proposed method on four benchmark datasets: Pascal VOC-2012 (VOC for short) F 10,, MS-COCO-2014 (COCO for short) 17 NUS-WIDE (NUS for short) 5 3 and CUB- CUB-200-200 2011 (CUB for short) 6 27 Table reports the detailed characteristics of four benchmark datasets. Specifically, VOC contains 5,717 training images and 5,823 validation images for 20 classes. We divide the training set into 4,574 training examples and 1,143 validation examples, and then use the original validation set for testing. COCO contains 82,081 I training images and 40,137 validation images for 80 classes. We divide the training set into 65,665 training examples and 16,416 validation examples, and then use the original validation set for testing. NUS is not complete online because of many invalid URLs. We sent a request email to the authors and obtain the complete dataset, which consists of 126,034 training images and 84,226 testing images for 81 classes. Following 44. we merge all images and randomly select 1 150,000 training samples and 60,260 testing examples. Finally, we withhold 30,000 images from training examples for validation. CUB contains 5,994 training images and 5,794 testing images. Following 4 instead of using 200 bird categories as class labels, we assign vector that indicates the presence absence of 312 binary attributes to every image. We divide the original training set into 4,795 training examples and 1,199 validation examples. In order to compare our method with state-of-the-art methods, we use the code shared by [ to generate the training, validation and testing sets. For each dataset, we withhold 20% of the training examples for validation. To construct SPML data, we randomly select one positive label for each training example, while keep the validation and testing sets always fully labeled. Follow 4. We report the mean average precision (mAP) on the testing set by using predictions of the model with the best validation performance.
Comparing methods. Besides the baseline AN loss, we compare the proposed method with the following -f-th-ar- algorithms: WAN A. which introduces weight parameter to down- weight the losses with respect to negative labels; EPR [4), which utilizes an expected positive
ac. wk/pasca//V////
https://cocodatasee org
htm

---

Table 1: The detailed characteristics of benchmark datasets.
Dataset
# +Triining
# Validation
# Testing
# Classes
VOC 4,574
1.143
5,823
20
COCO
65,665
16,416
40,137
80
NUS 120,000
30,000
60,260
81
CUB 4,795
1,199
5,794
312
regularization to mitigate the negative impact of false negative labels; AN-LS, which incorporates the label smoothing technique to AN loss; ROLE [4] which adopts regularized online label estimation
(ROLE) technique to alleviate the negative impact of false negative labels; EN+SCL [26]. which
combines expected negative loss with the spatial consistency loss (SPL); EM+APL 33). which incorporates asymmetric pseudo-labeling (APL) loss into entropy minimization.
Implementation. The label-wise embedding model consists of two components, including back- bone for extracting visual features, and a label-wise embedding decoder for producing label-wise embeddings. Following 44. we use ResNet-50 [15] pretrained on the ImageNet ( the backbone to extract features. Then, the extracted features are fed into the label-wise embedding decoder to produce label-wise embeddings. The label-wise embedding decoder consists of a standard self- attention block and a cross-attention block (25). After obtaining label-wise embeddings, we can feed them into the classification head h consisting of linear layer obtain the final predictions. By feeding label-wise embeddings into the projection head consisting of two linear layers and a ReLU layer, we can obtain the low-dimensional embeddings for performing label-aware global consistency regularization, In particular, the training process consists of two stages, and only the second stage training needs the low-dimensional embeddings produced by the projection head g. Therefore, we freeze the parameters of projection head at the first stage. We resize the resolution of input images to 448 x 448 as in [4) For training images, we use both a weak augmentation (only containing random horizontal flipping) and a strong augmentation (containing Cutout [8] and Rand RandAugment
[19] optimizer with the weight decay of 0.01. The OneCycleL scheduler is used to change the learning rate with the max learning rate of 0.0001. We train the model for 40 epochs with the early stopping. We consider the batch size in the range of {8, 16, 32, 64). At the first stage, set APCL I and determine the threshold 8 from the range of {0.5, 0.6, 0.7, 0.8, 0.9}. At the second stage, there are two extra parameters, including the balancing parameter ALAC the size of memory queue K. We set ALAC 1, and determine K from the range of (512, 1024, 2048, 4096, 8192), respectively. All hyperparameters are determined according to their mAP obtained on the validation set. In Section (4.4, we conduct experiments to analyze the sensitivity for unfixed parameters. Furthermore, we apply exponential moving average (EMA) to model parameters 0 with a decay of 0.9997. We perform all experiments on GelFrce RTX 3090 GPUs The random seed is set to for all experiments
5) For training the model, we use the AdamW
4.2 Comparison Results with the State-of-the-Arts
Table 2alee22eppe reports comparison results between the proposed method and comparing methods in terms of mAP on four benchmark datasets. To make a fair comparison, besides the results reported in their original paper (marked by ResNet-50), we still report the results of comparing methods trained with the label-wise embedding model (LEM) (marked by LEM). It is noteworthy that for EN methods, including LEN+LCL and LEN+ LSCL, in their original paper, only the results on VOC and COCO are reported. Furthermore, we cannot reproduce these two methods based on LEM due to the inaccessibility of source codes. From the table, it can be observed that: 1) AN loss achieves unfavorable performance in almost all cases. This validates that simply using AN cannot effectively solve SPML problems, since it introduces a large number of false negative labels, which are harmful for model training. 2) ROLE method cannot be easily adapted to the label-wise embedding model, since its performance suffers from a significant drop by replacing the original Re with LEM. One possible reason is that we fail to perform "LinearInit." 'hhtt has been used to initialize the weights of the classifier based on given feature representations in its original implementation, since these given feature representations cannot be used by LEM. 3) Our proposed method consistently achieves desirable performance and achieves the best performance on almost all cases except for CUB, where CEM+LAPL outperforms method with minor improvement. In particular, method achieves
7

---

Table 2: Mean average precision (mAP) of each comparing method on four benchmark datasets. The best performance highlighted in bold.
LEM
ResNet-50
Methods
VOC
COCO
NUS
CUB
VOC
COCO
NUS
CUB
CWAN
89.2
73.5
48.5
22.5
86.5
64.8
46.3
20.3
CEPR
88.8
72.7
49.3
23.1
85.5
63.3
46.0
20.0
LAN
87.6
72.3
48.5
18.6
85.1
64.1
42.0
19.1
LAN-LS
88.0
70.9
47.1
16.3
86.5
69.2
50.5
16.6
LROLE 88.1
69.6
44.5
14.2
88.2
69.0
51.0
16.8
LEN LCL
87.6
71.6
LEN + LSCL.
88.0
72.1
LEM 89.2
73.2
48.7
22.2
89.1
70.7
47.2
20.9
LEM + LAPL
89.2
73.1
48.6
23.6
89.2
70.9
47.6
21.8
Ours
89.6
75.6
51.1
23.3
Table 3: The ablation studies on VOC and COCO.
Methods
mAp (%)
Assume negative
PLC
LAC
Two-stage
Memory queue
VOC
COCO
87.6
72.3
89.0
73.7
84.5
65.6
89.2
74.7
89.6
75.6
favorable performance on two large-scale datasets and improve upon the state-of-the-art performance by 2.1% on COCO and 1.8% on NUS in terms of mAP. These results validate that the proposed method can effectively solve SPML problems.
4.3 Ablation Studies
In this section, to further analyze how the proposed method improves performance of SPML, we conduct series of ablation studies on VOC and COCO and report the results on Table We first conduct the experiment to validate the effectiveness for the proposed pseudo-labeling consistency regularization CPLc. The achieved performance is 89.0% VOC and 73.7% COCO by utilizing CPLC that are better than 87.6% and 72.3% achieved by only utilizing AN loss CAN. This discloses that by using Cplc, the model can recover the true positive labels that benefits for model training. To validate the effectiveness of the proposed label-aware global consistency regularization LAC, as discussed in Section 3.22 we train the model with two strategies, including end-to-end training, which directly optimizes the joint loss (3.3) in a pipeline, and two-stage training, which first trains the model with the first two losses until the model converges, and then trains the model with Eq. is can be observed that by adopting the end-to-end training strategy, the performance is degraded from 89.0% 84.5% VOC and 73.7% 65.6% COCO by adding LAC into training while are improved to 89. 2% and 74.7% by utilizing two-stage training. This is because by adopting the end-to-end training strategy, the early stage of model training, is difficult for the model produce high-quality label-wise embeddings for performing LAC regularization. Enforcing low- low-quality label- wise embeddings maintain the global consistency may destroy the feature representation learning and thus lead to noticeable decrease in model performance. Compared to end-to training, the two-stage training strategy primarily learn high-quality label embeddings the first stage, and then utilize the LAC regularization to encourage global consistency among label-wise embeddings, which improves the discrimination ability of model. Finally, we examine the usefulness of memory queue. By using the memory queue, the performance is improved with 0.4% increment from 89 29. on VOC and with 0.9% increment from 74.7%. The main reason that the memory queue provides the model with much more supervision for learning more distinctive representations, which are

---

VOC
COCO
51.2
23.2
CUB
23.0
60.2
75.55
50.3
50.6
0.5
0.6 0.7
0.5
09
05
0.5
0.7
0.5
0.5
0.7
0.5
0.9
0.5
0.6
0.7
0.8
0.5
1hreshold 6
Throsbrld 6
Threshold
Threshold 6
(a) VOC
(b) COCO
(c) NUS
(d) CUB
Figure 3: The performance curve the threshold 5 increases.
$9.50
75.75
15.50
50.75
23 23.00
89
50.50
22.75
59
0.1
0.5
1.0
1.5
2.0
0.1
01.5
1.9
1.5
2.0
0.1
0.5
1.01
1.5
2.0
01
0.5
1
1.5
2.0
Temperature
Tempercture:
Temperture
Tempersture
(a) VOC
(b) COCO
(c) NUS
(d) CUB
Figure 4: The performance curve as the temperature increases.
89.50
VOC
75.6
75.8
51.4
NUS
23.2
51.0
89.25
23.8
75.2
75.0
5058
8162
ills
2006
8192
sD
1034
4096
2045
J0p8
8192
MQ rizo K
MQ sizo
Miq size
MQ size
(a)
VOC
(b) COCO
(c) NUS
(d) CUB
Figure 5: The performance curve as the size of memory queue K increases.
beneficial for the model to identify the potential positive labels. These results convincingly validate that each component of the proposed method makes positive contribution to the final performance.
4.4 Parameter Sensitivity Analyses
In this section, we study the influence of three parameters, L.e., threshold temperature r and the size of memory queue K. Figure 3 Figureganne Figure Figure iillutrree the performance curves of the proposed method as the threshold 8, the temperature т and the size of memory queue change {0.1, 0.5, 1.0, 1.5, 2.0} and (512, 1024, 2048, 4096, 8192]. respectively. As shown the figures, the performance of the proposed method is generally insensitive to these three parameters. Specifically, the largest performance gap with respect to ô, 7 and K are respectively about 0.3% on COCO, 0.2% on NUS and 0.2% on COCO. The results indicate that we can safely set these parameters in a large range in practice.
in the range of (0.5, 0.6, 0.8, 0.9).
4.5
Case Studies
To disclose the mechanism behind the effectiveness of the proposed method for identifying the potential positive labels, Figure 6 visualizes some cases of attention maps on COCO. For every original image in the first (or fourth) column, we illustrate the attention maps of the single observed positive labels and identified positive labels in the next three columns. From the figures, it is interesting to observe that some small objects without annotations can be captured by the proposed method precisely, such as potted plant in the first row, bowl and carrot in the second row as well as clock and tie objects by maintaining the global consistency of embeddings between the object and its intra-class objects. This further indicates LAC can significantly enhance the ability model for identifying the potential positive labels.
last row. These observations disclose the proposed method can capture small

---

Original image
itentton map
Original image
llention map
Observed label
Identiffied labels
Observed label
Idectified labels
vase
potted plant
sink
cat
celi phone
remole
cup
apple
bowl
person
couch
CarTot
person
remote
clock
caku
tiu
knife
Figure 6: Visualization of attention maps on COCO.
5
Conclusion
The paper studies the problem of multi-label learning with single positive labels. In order to solve SPML problems, design the PLC regularization that exploits the model outputs predicted weakly-augmented and strongly-augmented images, and LAC regularization that leverages the manifold structure information recover the labeling information of the potential positive labels. By utilizing these two regularizations, the model is expected to learn a more distinctive representation, which is beneficial for identifying the potential positive labels. Extensive experimental results multiple benchmark datasets validate the proposed method can achieve state-of-the ff-the-an art performance. In the future, plan to improve the performance of SPML by using other structure information. such as label correlations.
Acknowledgments and Disclosure of Funding
This research supported by the National Key R&D Program of China (2000AA00107000), NSFC (62222605, 62076128), and Natural Science Foundation of Jiangsu Province China (BK20211517, BK2022050029).
References
[1] Amit Alfassy, Leonid Karlinsky, Amit Aides, Joseph Shtok, Sivan Harary, Rogerio Feris, Raja Giryes, and Alex M Bronstein. Laso: Label-set operations networks for multi-label few-shot learning. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages
2019.
[2] Zhao-Min Chen, Xiu-Shen Wei, Peng Wang. and Yanwen Guo. Multi-label image recognition with graph convolutional networks. In IEEE Conference on Computer Vision and Pattern Recognition, pages 5177-5186, 2019.
[3] Tat-Seng Chua, Jinhui Tang, Richang Hong, Haojie Li, Zhiping Luo, and Yantao Zheng. NUS- WIDE: real-world web image database from national university of singapore. In Stéphane Marchand-Maillet and Yiannis Kompatsiaris, editors, Proceedings of the 8th ACM International Conference Image and Video Retrieval, 2009.
[4] Elijah Cole, Oisin Mac Aodha, Titouan Lorieul, Pietro Perona, Dan Morris, and Nebojsa Jojic. Multi-label learning from single positive labels. IEEE Conference on Computer Vision and Pattern Recognition, pages 933-942, 2021.
10

---

[5] Ekin Dogus Cubuk, Barret Zoph. Jonathon Shlens, and Quoc Le. Randaugment: Practical automated data augmentation with reduced search space. In Hugo Larochelle, Mare' Aurelio Ranzato, Raia Hadsell, Maria-Florina Balcan, and Hsuan-TTien Lin, editors, Advances in Neural Information Processing Systems 33: Annual Conference on Neural Information Processing Systems, 2020.
[6] Son D. Dao, Ethan Zhao, Dinh Phung, and Jianfei Cai. Multi-label image classification with contrastive learning. CoRR, abs/2107.11626. 2021.
[7] Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. In 2009 IEEE Computer Society Conference Computer Vision 2009.
and Pattern Recognition, pages
[8] Terrance Devries and Graham W. Taylor. Improved regularization of convolutional neural networks with cutout. CoRR, abs/ 04552, 2017.
[9] Thibaut Durand, Nazanin Mehrasa, and Greg Mori. Learning deep convnet for multi-label clas- sification with partial labels. In IEEE Conference on Computer Vision and Pattern Recognition, pages 647-657, 2019.
[10] Mark Everingham, Lue Van Gool, Christopher K. Williams, John M. Winn, and Andrew Zisserman. The pascal visual object classes (VOC) challenge. International Journal of Computer Vision, 88(2):303-333 2010.
[11] Hao Guo, Kang Zheng. Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. In IEEE Conference on Computer Vision and Pattern Recognition, CVPR 2019, pages 729-739.
[12] Hao Guo, Kang Zheng, Xiaochuan Fan, Hongkai Yu, and Song Wang. Visual attention consistency under image transforms for multi-label image classification. In IEEE Conference on Computer Vision and Pattern Recognition, pages 729-739, 2019.
[13] Emily M. Hand, Carlos Domingo Castillo, and Rama Chellappa. Doing the best we can with what we have: Multi-label balancing with selective learning for attribute prediction. In Sheila A. Mcllraith and Kilian Q. Weinberger, editors, Proceedings of the Thirty-Second AAAI Conference on Artificial Intelligence. pages 6878-6885.
[14] Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross B. Girshick. Momentum contrast for unsupervised visual representation learning. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9726-9735, 2020.
[15] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In 2016 IEEE Conference on Computer Vision and Pattern Recognition, pages 770-778, 2016.
[16] Dat Huynh and Ehsan Elhamifar. Interactive multi-label CNN learning with partial labels. In 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9420-9429, 2020.
[17] Tsung-Y Yi Lin, Michael Maire, Serge Belongic, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In European conference on computer vision, pages 740-755, 2014.
[18) Shilong Liu, Lei Zhang, Xiao Yang, Hang Su, and Jun Zhu. Query2label: A simple transformer way multi-label classification. CoRR, abs/2107. 108344 2021.
[19] Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. In 7th International Conference on Learning Representations, 2019.
[20] Tao Pu, Tianshui Chen, Hefeng Wu, and Liang Lin. Semantic-aware representation blending for multi-label image recognition with partial labels. CoRR, abs/2203.022 022122 2022.
[21] Tal Ridnik, Gilad Sharir, Avi Ben-Cohen, Emanuel Ben Baruch, and Asaf Noy. MI-decoder: Scalable and versatile classification head. CoRR, abs/2111.12933. 2021.
11

---

[22] Jing Shao, Kai Kang, Chen Change Loy, and Xiaogang Wang. Deeply learned attributes for crowded scene understanding. In IEEE Conference on Computer Vision and Pattern Recognition, pages 4657-4666, 2015.
[23] Jake Snell, Kevin Swersky, and Richard S. Zemel. Prototypical networks for few-shot learning. In Isabelle Guyon, Ulrkee von Luxburg. Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, pages 4077-4087, 2017.
[24] Feng Sun, Ming-Kun Xie, and Sheng-Jun Huang. A deep model for partial multi-label image classification with curriculum based disambiguation. arXiv preprini 2rXiv:2207.02410, 2022.
[25] Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Isabelle Guyon, Ulrike von Luxburg, Samy Bengio, Hanna M. Wallach, Rob Fergus, S. V. N. Vishwanathan, and Roman Garnett, editors, Advances in Neural Information Processing Systems 30; Annual Conference on Neural Information P Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-66089 6008, 2017.
[26] Thomas Verelst, Paul K. Rubenstein, Marcin Eichner, Tinne Tuytelaars, and Maxim Berman. Spatial consistency loss for training multi-label classifiers from single-label annotations. CoRR, abs/2203.06127, 2022.
[27] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
[28] Haobo Wang, Ruixuan Xiao, Yixuan Li, Lei Feng, Gang Niu, Gang Chen, and Junbo Zhao. Pico: Contrastive label disanbiguation for partial label learning. CoRR, abs/2201 0es//201108984, 2022.
[29] Lichen Wang, Yunyu Liu, Can Qin, Gan Sun, and Yun Fu. Dual relation semi-superrised multi- label learning. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 34, pages 6227-6234, 2020.
[30] Ya Wang, Dongliang He, Fu Li, Xiang Long, Zhichao Zhou, Jinwen Ma, and Shilei Wen. Multi- label classification with label graph superimposing. In The Thirty- Coury-hourth AAAI Conference on Artificial Intelligence, pages 12265-12272, 2020.
[31] Zhouxia Wang, Tianshui Chen, Guanbin Li, Ruijia Xu, and Liang Lin. Multi-label image recognition by recurrently discovering attentional regions. In IEEE International Conference on Computer Vision, pages 464-472, 2017.
[32] Ming-Kun Kun Xie and Sheng-Jun Huang. Partial multi-label learning. In Proceedings the AAAI Conference Artificial Intelligence, volume 32, 2018
the
[33] Ming-Kun Xie and Sheng-Jun Huang. Multi-label learning with pairwise relevance ordering. Advances in Neural Information Processing Systems, 34:23545-23556, 2021.
[34] Ming-Kun Xie and Sheng-Jun Huang. Cemn: A general framework for learning with class- conditional multi-label noise. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2022.
[35] Dengyong Zhou, Olivier Bousquet, Thomas Navin Lal, Jason Weston and Bernhard Schölkopf. Learning with local and global consistency. In Advances in Neural Information Processing Systems 16, pages 321-328. 2003.
[36] Donghao Zhou, Pengfei Chen, Qiong Wang, Guangyong Chen, and Pheng-Ann Heng. Acknowl- edging the unknown for multi-label learning with single positive labels. CoRR, abs/2203.162111 16219, 2022.
12


Reviews:
[{"review_id": "zRUt6SIlKV9", "reviewer": "Reviewer_8Jhn", "summary": "This paper proposes a Label-Aware global Consistency(LAC) regularization to solve the Single Positive Multi-label Learning (SPML) problem, where each instance is only annotated with one of the positive labels related. The proposed Label-Aware global Consistency(LAC) regularization leverages the manifold structure information via encouraging the global consistency among label-wise embedding, i.e., making intra-class embedding closer while keeping away inter-class embedding.  Finally, they conduct extensive experimental results on multiple benchmark datasets, demonstrating that the proposed method can achieve superior performance.", "questions": "Data Report: In Table 1, the mAP of $L_{AN}$ on the dataset COCO is reported as 64.1. But in Table 2 about the ablation study on COCO, the method ‘assuming negative’ is reported as 68.5. From Page 8 Line 267-268, we can know that they are in the same setting using the same method. I wonder why there exists a 4.4% difference between them.\n  \n", "limitations": "None", "rating": 7, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength:\n1. This paper proposes a novel Label-Aware global Consistency(LAC) regularization to solve the Single Positive Multi-label Learning (SPML) problem, where each instance is only annotated with one of the positive labels related. \n2. This paper successfully applies ‘Clustering assumption’ in SPL and achieves good performance.\n3. The visualization of the problem background, the proposed method, and the experiment results are good, which makes the reader easy to understand this work.\n\nWeakness: \n1.  The basis for saying the sentences in Page 5 Line 150-153 needs to be introduced.   \n2. In Page 3 Line 104, it mentions that this paper adopts the attention-based method to construct the label-wise embedding model, but in the following introduction about the proposed method, this point seems to be neglected in the proposed method, which is only related to Section 4.5 Case Study.   \n\n\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "bPE_WBakInA", "reviewer": "Reviewer_Cnop", "summary": "This paper studies a new Multi-Label Learning (MLL) setting, i.e., the Single Positive MLL (SPML), where only one positive label is observed. The authors use AN to learn SPML. Meanwhile, they put forward Pseudo-Labeling Consistency (PLC) regularization and Label-Aware Global Consistency (LAC) regularization to address the false negative problem raising by AN. The experiments show that the proposed method (AN + PLC + LAC) achieves the best performance. ", "questions": "Please refer to my comments above in the Strengths And Weaknesses. ", "limitations": "The authors haven't claimed any limitations. ", "rating": 7, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The main strengths are as follows:\n+ Although AN is not stable for SPML, the authors put forward PLC and LAC to address the false negative problem raising by AN. \n+ The proposed method AN+PLC+LAC is shown to have the best performance compared with exiting SPML methods. The ablation study and case study are clear. \n\nBesides, I also have several questions:\n- PLC, i.e., Eq. (4) has the indicator function, which is not easily to be trained. I expect to see the convergence of the proposed method. \n- The intra-class connection matrix I has a size of $B^2$, which is such an enormous matrix. What exactly is the size of the matrix in your experimental datasets? Is it possible to train with such an enormous matrix?\n- LAC, i.e, Eq. (7) only involves the matrix I without the matrix $\\bar{I}$. Where does $\\bar{I}$ function?", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "Vk3RE16L-hh", "reviewer": "Reviewer_fikh", "summary": "The paper considers the multi-label learning with single positive labels. In such scenario, each image is only associated with a single positive label. To recover the potential positive labels, the paper first pseudo-labels the unobserved labels based on the model predictions. To boost the labeling performance, the method considers the global consistency regularization for label-wise embeddings to learn more distinctive feature representations. The experiments are conducted in multiple benchmark multi-label datasets. Experimental results show the superiority of the proposed method.", "questions": "Questions\n1.\tIn many real-world scenarios, annotators may assign each image with a subset of all ground-truth labels instead of a single positive label. Can the proposed method solve this problem?\n2.\tIn Section 4.4, Figure 3 illustrates the performance curves as the value of threshold increases. It seems that the method only achieves better performance with the increase of threshold on COCO dataset. Authors are suggested to give some detailed discussion about this phenomenon.\n3.\tBesides similarities among embeddings from a class label, there exists similarities among embedding from different labels. Can such similarity information help learn a good classification model? Why?\n", "limitations": "Yes", "rating": 8, "confidence": 4, "soundness": 3, "presentation": 4, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strength\n1.\tThe paper proposes an effective solution to SPML problems. The main contribution of this paper is to utilize the label-aware global consistency to encourage the feature representations to maintain a more compact structure. The proposed method is well motivated and easy to follow.\n2.\tThe experiments are performed on multiple benchmark datasets to compare with state-of-art methods. The proposed method achieves impressive performances and shows significant superiority to the comparing methods. Furthermore, the ablation studies and sensitivity analyses of hyper-parameters are reported. \n\nWeakness\n\n1.\tThe experimental settings are less clear. For example, it is not introduced how to assign a single label for each image. The paper is suggested to provide some details.\n2.\tThe label correlations are not considered in the proposed method. The label correlations have been regarded as a fundamental element for performing multi-label classification. Is it beneficial for improving the performance of SPML?\n3.\tIn experiments, the proposed method shows large superiority on COCO than that on other datasets. Authors are suggested to provide detailed discussions about this observation.\n4.\tThere are some language mistakes. The paper is suggested to be carefully proofread.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ["I don’t know"]}, "extra_text": {}}, {"review_id": "FFUwPVEKZrX", "reviewer": "Reviewer_vE3b", "summary": "This paper deals with the single positive multi-label learning (SPML) task. Different from vanilla multi-label learning, in SPML, only a single (e.g., the most obvious) positive label is annotated. The authors propose a label-aware global consistency loss for the task. With a two-stage optimization, the method achieves good performance on various benchmarks.\n", "questions": "The paper is good in general, and here are some questions on the implementation details.\n1. Does the label embedding in Eq. 7 instance-specific? It is a bit confusing to use $K$ to index them since it is hard to infer where the label embedding comes from.\n2. How about the performance without a pre-trained backbone? Does the performance of the method rely on the pre-trained backbone? The authors may consider showing the performnace without a pre-trained backbone on a certain dataset.\n3. The authors may consider some more criteria to evaluate SPML. For example, to evaluate how many latent labels it could infer from the data.\n4. Will the performance of the model relates to the (maximum) number of positive labels in the dataset. \n\n\n-------------------------post rebuttal----------------------\nI have read the response, and the authors have addressed my concerns. ", "limitations": "The authors have adequately addressed the limitations and potential negative social impact of the work.\n", "rating": 7, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper is easy to read, and the method is simple yet effective. The authors utilizes the manifold instead of the clustering assumptions in SPML, and show the advantage of the asumption based on the empirical experiments. \n\nHere are some suggestions for the paper:\n1. Figure 1 is not clear enough. By comparing the single-label and multi-label cases, the authors may first show the difference between vanilla multi-label learning and SPML. While there are some elements (such as the embeddings and regularizations) in the figure, which make the figure a bit confusing.\n2. The label-wise embedding before Eq. 1 is not clear. The authors use z to denote the label embedding, but z does not exist in Eq. 1. Since the label embedding is one of the most important elements in the global consistency loss, the authors need to make it clear. Does the label embedding useful in vanilla multi-label learning (e.g., in Eq. 1-2 in the paper).\n3. Since the true number of positive labels cannot be accessible in many real-world scenarios, the authors need to discuss how the proposed method can deal with this case before the experiments.\n", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept