Regret Bounds for Multilabel Classification in Sparse Label Regimes
Authors: Robert Istvan Busa-Fekete
Heejin Choi
Krzysztof Dembczynski
Claudio Gentile
Henry William Reeve
Balazs Szorenyi
Venue: NeurIPS 2022 Accept (2022)
Keywords: regret bounds
sparse multilabel classification

Abstract: Multi-label classification (MLC) has wide practical importance, but the theoretical understanding of its statistical properties is still limited. As an attempt to fill this gap, we thoroughly study upper and lower regret bounds for two canonical MLC performance measures, Hamming loss and Precision@$\kappa$. We consider two different statistical and algorithmic settings, a non-parametric setting tackled by plug-in classifiers \`a la $k$-nearest neighbors, and a parametric one tackled by empirical risk minimization operating on surrogate loss functions. For both, we analyze the interplay between a natural MLC variant of the low noise assumption, widely studied in binary classification, and the label sparsity, the latter being a natural property of large-scale MLC problems. We show that those conditions are crucial in improving the bounds, but the way they are tangled is not obvious, and also different across the two settings.

Regret Bounds for Multilabel Classification in Sparse Label Regimes
Róbert Busa-Fekete Google Research busarobi@google. com
Heejin Choi Google heej wow.gnnnneo....oooooooooooooooon com
Krzysztof Dembezyáski
Yahoo Research
Poznan University of Technology
kdembczynski@cs put. paztapoznan.
Claudio Gentile
Henry W. Reeve
Balázs Szörényi
Google Research
University Bristol
Yahoo Research
cgentile@google. com
henry www..eereeerrst...cooo ac.
szoreny:ballazs@gaal..oom
Abstract
Multi-label classification (MLC) has wide practical importance, but the theoretical understanding of its statistical properties is still limited. As attempt to fill this gap, we thoroughly study upper and lower regret bounds for two canonical MLC performance measures, Hamming loss and Precision @ We consider two different statistical and algorithmic settings, a non non-parametric setting tackled by plug-in in classifiers à la k-nearest neighbors, and parametric one tackled by empirical risk minimization operating on surrogate loss functions. For both, we analyze the interplay between natural MILC variant of the low noise assumption, widely studied in binary classification, and the label sparsity, the latter being a natural property of large-scale MLC problems. We show that those conditions crucial in improving the bounds, but the way they are tangled is not obvious, and also different across the two settings.
Introduction
Multi-label classification (MLC) is natural generalization of binary classification and multi-class classification. It can be seen as a composition of several binary problems for the same instances, or as multi-class problem where more than one label can be assigned to given instance. Several applications can be naturally modeled as a multi-label problem, including image and video annotation for multimedia search [13]. tagging text documents for categorization of Wikipedia articles [9], recommendation of bid words for online ads (27). and beyond. MLC is a well-established sub-field of supervised learning, that has generated a tremendous interest, mainly of experimental nature, introducing a lot of interesting and diverse approaches (for example, [1O] 26
17
34
33 19 ) They
have also been described in multiple survey papers across years, e.g., 30 4 The main concerns in MI.C research are label dependencies (marginal and conditional, see the discussion in 12)) label
sparsity 18 34 and minimization of wide spectrum of performance metrics.
Interestingly, the theoretical understanding of MLC learning problems is still limited. This might be due to the daunting presence of multiple and diverse loss functions that can be defined over subsets of labels. Popular loss functions include Hamming loss, Precision@ k, or subset 0/1 loss. Each such MLC loss function has its pros and cons, and there is consensus among researchers what the right performance metric to be used [12,20] as the choice certainly domain specific. The theoretical results obtained far mainly deal with the analysis of Bayes optimal classifiers for
the wide variety ML.C loss functions 12
25
Fisher-like consistency results [TT 16), or
36th Conference on Neural Information Processing Systems (NcurIPS 2022).

---

surrogate regret bounds 22 33 Results concerning generalization bounds have also been published in 6 34 32
We fill the gap in the landscape of theoretical results by providing upper and lower finite-sample regret bounds in MI.C with focus on computational efficiency. We consider two learning setups, non- parametric and parametric one. Within the former, we focus specifically on a k-nearest neighbor (k-NN) classifier, because of its attractive properties in MLC settings. Beyond computational efficiency, a main reason why k-NN is a convenient approach is that it works "out of the box", operating only in the feature space, independent of the number of labels, hence making a very versatile tool for MLC tasks. It has also been studied in MLC, mainly empirically, for example in [35. In the parametric setting, we introduce MLC counterpart to the surrogate loss regret reductions introduced in [3] thereby considering empirical risk minimization solutions applied to convex surrogate loss functions. We restrict our attention to Hamming loss and Precision k. and thoroughly investigate the interplay between two natural assumptions, the low noise condition and the label sparsity, in determining rates of convergence.
Our results build on a number of former contributions. The analysis of k-NN classifiers has been motivated by the results in [29 are related to those adopted for high probability regret bounds in binary classification [5]. To obtain
The proof techniques we rely upon in deriving regret bounds
lower bounds we follow to some extent results of (2). as the proof uses reduction of a sparse MLC problem to binary classification. The most similar results to ours have been published in 8 (7]. In
88. the authors analyze a modified version of Hamming loss which assigns different weights to relevant and irrelevant labels. The paper also assumes sparsity, but that assumption is stronger than ours. The guarantees obtained are in terms of generalization and regret bounds. 7 does not directly use Precision @ k, but the setting analyzed there is similar to the one for Precision @ k. That paper also formulates a margin condition similar to ours. Yet, the regret bounds contained there are not of a high probability type, but only bounds in expectation, similar to 22. Moreover, [7] does not contain (matching) lower bounds. Our results for the parametric case build on [. which contains regret analysis, also under low noise condition, for surrogate losses in binary classification. Our generalization takes additionally multi-label sparsity into account.
Main results. We consider novel label-wise margin (low-noise) conditions that are sufficient to get fast rates. In addition, we assume that at most s out of all m labels can have conditional probability arbitrarily close to 1, whereas for the rest of the labels this probability is upper bounded by a small constant. Such label sparsity assumption is essentially motivated by real-world multi-label problems with large number of potential labels, where only few such labels are assigned to single instances, a paradigmatic example being image or text annotation.
In the non-parametric setting, we consider a MLC variant of k-NN as our method of choice, since this method is rell-understood in the binary case, and can be shown to achieve fast rates for 0/1-loss (which is in fact the binary counterpart to Hamming loss). Under multi-label margin and sparsity assumptions, we present a high probability bound for the Hamming loss regret of k-NN which is of the form (s//m m (log(s/0) (/) /n) (/)////)))/////nn (X(++)))/(2++1) + 6, where s the sparsity parameter, À the smoothness of the regression function with respect to the infinity norm ||-|| oc. is the exponent in the margin condition, and 8 is the desired confidence level. In the special case where no sparsity is assumed we have
m, hence the bound boils down to more "standard" upper bound for the Hamming loss regret. We would like to emphasize that applying a binary regret analysis, such as the one in (5) each of the m labels independently cannot leverage global properties like label sparsity. Thus, in order to take advantage of label sparsity, a more involved regret analysis is needed. Also observe that, by making the margin condition more favorable (that is, by increasing a), the dependence on the sparsity ratio s/m remains unaffected. In case of Precision we do not need label sparsity: a margin condition, that controls the gap between the s-th and the (x + 1)-th largest conditional distribution, is by itself enough to obtain regret bounds that scale gracefully with the number of labels. Under this assumption, we obtain a regret bound for the multi-label k-NN classifier of the form (log(m/8) 8) ()))))(((//)))))))))))))) (x)((()))//(2++ + 5, where 3 is the margin exponent for the s-th and (N + 1)-th largest conditional distribution.
In the parametric setting, we consider regret bounds for function classes with finite VC dimension. Under the margin and sparsity assumptions, we relate the Hamming loss regret Reg,,, of an arbitrary
2

---

MLC classifier to its MLC surrogate loss regret Reg, via bounds of the form
Regen = =0 (s/m)
Reg
where v a parameter related to the curvature of the surrogate loss (e.g., V = for both truncated square and exponential losses). In turn, these bounds can be used in combination with standard uniform convergence results to control Reg,, e.g..
- (d= Loggoog//// (m/0) + +aa
Reg. = O
Vn
where d is the pseudo-dimension of the VC class under consideration, and is its approximation error. Regret bounds for Precision @x are reduced to those for Hamming loss.
As last contribution, derive MLC regret lower bounds for our MLC setups revealing that, at least in the non- con-mammetrii case, our upper bound for Hamming loss is optimal up to a log s factor, and that our regret upper bound for Precision@a: is optimal up to log m factor.
2 Preliminaries and notation
We assume to have feature space X' endowed with metric px, and set of m labels. We denote by y := {0, 1 the set of binary vectors encoding the set of 2 possible label combinations. We shall assume the existence of pair of random variables X X and Y y' with joint distribution P. Let M(X,)) denote the set of all classifiers, which are measurable mappings Φ : X' - y. The goal of the learner is to obtain classifier & E M(X.)) such that ((x) y for typical pairs (x, y) P. The precise sense in which "*(x) xs y is provided by a loss function E y x y > [0, oc), with the interpretation that ((ÿ. y) measures the cost incurred when the learner predicts y while the true (multi-)label is y.
The problem of multi-label classification can be defined as finding (vector-valued) classifier ((x) = (Ø) (a))) that minimizes the expected loss (or risk):
Re( = E [((((X), Y)
the expectation being over X x P. For a given loss function e, the optimal classifier, often called the Bayes (optimal) classifier is defined as o; = argmin JEM(X') Re(Ø). When is clear from the surrounding context, we shall omit the subscript and simply write 6- We can measure the statistical performance of a given classifier á in terms of its regret, henceforth denoted by Reg, (d). quantifying the sub-optimality of d. Specifically, (ø) measures the extent to which the expected loss of o exceeds the expected loss of the Bayes classifier:
Reg : (d) Re(d))
From a purely statistical standpoint, the goal of learning is then finding a classifier & with regret as small as possible, ideally equal to zero.
Computing d" requires prior knowledge of the data distribution P. Yet, P is assumed to be unknown where the pairs
and accessible only through a dataset (or statistical sample) D = - (X', Y') are drawn i.i.d. from P. In what follows, we denote by μ the marginal distribution of P over X, so that (aAA) = P(X E A). Moreover, we define the multi-label regression function 7 X + [0, 11TH by nTx)) (ny (x: je[m) whose j-th component ny (zz) ) is the conditional probability of the j-th label being one: n7 (m) = P(Y, = 1|X æ)).
3 Losses for multi-label classification
We focus on two widely used loss functions for MLC. namely Hamming loss and Precision @ K, which we now define. The Hamming loss L : y x y [0, 1] is defined for pairs y (y))je[m) y and
(ū) (mm) E y by
L (ū.
1 m jE
Σ
T + F,}
3

---

Hence, Lu (Ú. y) simply counts the fraction of mispredicted labels. We observe that for all Φ E M(X )) we have
Reg (6)
Σ
(0)) + (11-0) (zx)) du(x),
(=()))u((())
jE(m]
implying that o, : X + V defined by
(x)= (I(n 2 1/2)))e(mi
is the Bayes optimal classifier for EH. Moreover, it is not hard to see that for all ø E M(X have
Regen (0)
m Σ jE(m]
[(0((z) + (0 (x)))} (2n(2)) dpdx))).
To define the Precision @xc loss, need restrict the predicted vectors lie in V/+ { EV: | = x}. Then we have
la (0.( =
Σ (ET(Ø)
14 1
(2)
where T(y) = tj [m] Ū3 = 11 is the set of labels output by the predictor (corresponding to the labels of value one). Let (f) Rm Yx be a mapping from a vector of real-valued scores to binary vector with x ones. Then, the corresponding risk is:
Ri =f)
Σ ()())du(z).
R. (ET(O)(((w)))
Based on this, one can obtain the Bayes optimal classifier 67 for lüx one that predicts on the given in input the classes with the highest conditional probabilities (x)) as positives, i.e
Dia. (x: == (I{i E T" (m)()))) (e(m]
where 7 18mh + [m]] returns the set of x indices that are largest for a real-valued input vector. The regret of any classifier can be written
Regra (6) = //= Σ
ni(x))
E
n)(x)) du(x).
(n(x)
(o(x))
4 Sparsity and margin condition
We introduce two assumptions which are fairly natural for MLC scenarios. The first one encodes the intuition that in typical MLC settings, when the total number of classes/labels large, only very few labels are active for any given region in the feature space, that is, each item comes with sparse label vector y.
Assumption 4.1 (Sparsity assumption). Define A(s, t) the set of t-approximately s-sparse vectors as follows:
A(s, t) {u + u, Rm ||u||l S ||o/ t)
Take t [0, 1/2) and E N. We shall say that the t-approximate s-sparsity assumption holds if the multi-label regression function ()x)) E A(s, t) for all E X.
always holds with practice we expect to hold with much smaller value of s, possibly coupled with t 0. fact, this assumption generalizes the standard sparsity assumption of "at most relevant labels instance" by adding term for a long tail. Yet, does not introduce a hard constraint, since applies to
Assumption 4.1
and t = 0. but for many distributions encountered in
In the relevant literature on the subject, Precision@*: x often regarded as a utility function defined as Precision@w = 1 law. To be consistent with our discussion, we instead define Precision @x as a loss function.

---

7(x), According to this sparsity assumption, the expected number of labels per instance æ is at most E E y X = I + (m - s) On real world multi-label datasets with large label spaces, the expected number of labels usually less than 100, while the number of possible labels m can be three or four orders of magnitude bigger 21
The next assumption is MLC variant of Tsybakov's margin (or low noise) assumption (e.g., 2 ties up "noise" present on different labels to occur simultaneously on those labels.
Assumption 4.2 (Margin assumption), Let a 2 0 and C. 2 be constants. We shall say that the margin assumption holds with exponent and constant C. if the following holds for all E (0, 1]:
" U {ar E (2)((=) 1| s {)) S Ca.
je(m]
Non-parametric setting
5.1
The k--N regression estimator
Given æ E X, we let {Tn.q
(z))) (T). be enumeration of [ri] such that for each & E [n 1}.
Px x, X x(mal()) px (x, X X(meti(t))
In words, for given x E X, (Tn,( sorts (x1 ..... x in increasing order of px-distance to æ.
The k-nearest neighbor regression estimator fink (a))) where
x + 10, is given by ( ==
(i) Lon
1 Σ Y)nilo)
n ~ (x)
KE[M]
The plug-in multi-label classifier for Hamming loss that based on the k-NN regression function n. is denoted by mn.k and defined as
Pin (x:
(I(1) (x) 1/2)) jE[m]
Similarly, the plug-in in multi-label classifier for Precision @yc loss that is based on the k-NN regression function is denoted by Faa and defined as
n.k Pan (x) = T", (x)))) jE(m)
In this paper, we shall scrutinize the performance these k-NN classifiers in terms of regret under some ussumptions on the classification problem at hand.
Given æ E X and 0. let (x)) denote the open metric ball of radius centered
Assumption 5.1 (Smoothness assumption). Suppose have exponent À (0, 1), constant CA and a semi-norm || on " We shall say that the measure-smoothme assumption holds with parameters X, Cx, if the following holds for all wo, 21 E X:
[ln(=x)) - n1 Ill Cx (Box (xx))
This kind of assumption is typical for k-NN. It says that the regression function 17(x) does not change much in the neighborhood of every input x. This is essentially needed for k-NN convergence. The specific version adopted here is from [5), and was further employed, e.g., in 22) Note that, more generally, a universal rate of convergence cannot be established without assuming regularity properties of the data distribution, such as smoothness via absolute continuity. Results of that kind are beyond the scope of this study. We refer the reader to [14] for details on L, consistency and its connection to the rate of convergence. this paper, we end up using Assumption 5.1 with being the infinity norm, not an arbitrary semi-norm.
5

---

5.2 Regret upper bound for Hamming loss
This multi-label margin assumption is a sufficient condition for fast rate (11/n). Its binary classifica- tion counterpart is well-understood in the binary classification literature.
We have the following high probability upper bound for the regret in terms of Hamming loss for the k--N classifier Pis, which is the main result of this section
Theorem 5.2. Let Assumption 5.7 hold with parameters X. Cx> 0. 1| 11s, Assumption 4.1 hold
with parameters & E N, E [0, 172) and Assumption 4.2 hold with parameters a, C. > 0. Assume
further that k in k-NN satisfies 16(1 - 2 log (og(2m/5) S l I ((1 (BCx) 1/ (n/2) Then,
with probability at least 1 - 8 over D, the regret of the k-NN plug-in in classifier Vin obtained from 3) satisfies
g..
2s C. (A(n, k,
Regiy h
(o. I m
+
where
2k +
log(s/0)
An,,k,,,0) Cx
k
In particular with k = 2 20) (1-)) 22)2)
n(2A/(2211) and n = s (1og((2m/0)
we have
Regen (WE = (
( (log(41/00)
+
A few remarks are in order at this point.
Remark 5.3. (On the assumptions) The regret bound given in Theorem 5.2 requires three assumptions. The smoothness assumption of the regression function (Assumption is natural in the sense that a universal rate of convergence cannot be established without assuming regularity properties of the data distribution (e.g.. [14 15)). Analyzing the impact of different smoothness criteria is beyond the scope of this study. Assumption 41 ensures label sparsity which allows k-NN learning achieve regret of order s/m. Again, this assumption is also well motivated by practice, since most of the MLC task of practical relevance have sparse labels. Finally, Assumption 4.2 ensures fast rates, i.c., faster than 1/ /n. This assumption is commonly used and well-understood in the binary case. Here, we consider a natural extension the multi-label setup which quantifies the probability of observing instances whose conditional probabilities are close to 1/2.
results in with the fast rate upper bound which we would achieve had we applied the regret upper bound for the binary case (e.g.. Theorem 4 in (5) to each conditional Tzr independently, after setting 8/m and taking a union bound. Notice that the binary result can be applied to our setup to each 7) (z)) individually because our smoothness condition is based on I|- so that the smoothness definition for the binary regression function in [5] readily applics to our MLC setting. Yet, we would like to stress that this need not imply that the bound obtained is optimal. Rather, this way of viewing MLC as a set of independent binary problems sheds light on how to construct a set of problem instances on which we can prove lower bounds based the binary result (see Section 5.4 for details). On the other hand, the sparsity condition s < m imposes global constraints on the MLC task that makes the bounds contained in Theorem 5.2 not readily obtainable by simply reducing m independent binary problems.
Remark 5.4. (On MLC bounds based on binary classification) Theorem 5.2 with
a high probability upper bound of the form O (log(m/8)/m)
ACL oo)
2x41
+ 8. This bound coincides
Remark 5.5. (On fast and ultra fast rates) When Xa > 1/2, the exponent of (1/n) log(4s/5) in the bound of Theorem 5.22 becomes larger than 1/2, giving fast rate guarantee. Similarly, when X(a - 1) > 1, the exponent becomes larger than 1, and we obtain ultra fast rates.
Due space limitations, all proof's are in the appendices.

---

5.3 Margin condition for Precision@x
The margin condition for Hamming loss ensures gap of conditional probabilities around 1/2 with a certain probability, which is motivated by the fact that those are the hard-to-classify instances when the target loss the Hamming loss. Starting from this observation, one can come up with margin condition for Precision ax of similar flavor. Let us denote the ith largest margin given by moy (z)). We assume there a gap with some quantifiable probability between the s-th and (N + 1)-th conditional distribution, specified next.
Assumption 5.6 (Margin condition for Precision@a) x). There exist 3 2 0 and Kg 2 such that for all 0 we have
P (X)))(X) - 7)++1) (X) < E) S Kgg
This assumption bears close resemblance to the margin condition introduced in 22 but, clearly enough, in order make the optimal decision for Precision the classifier needs to distinguish between (x)) (z=) and 1) (2x) with a certain quantifiable probability over X. With this assumption ..k hand, one can show the following high probability regret bound for &
Theorem 5.7. Let Assumption 5.1 hold with parameters X, Cx > 0, || ||x00 and Assumption 5.6 hold with parameter > O and Kg. Suppose further that k satisfies 4 (2/8( - k S n/2. Then, with probability at least 8 over D holds that
8)) 25, Region (WEa..) S (g(2A(n, 1+8
where A(n, k) Cx
log log(3m /5) In particular with
h (10og(3m00))//0m)m (/)2m2+))))211) (a1tant)).2///2/2 222/(22+1),
we have
A(L+-) (log(m/0) n
28+1
Regia ta ( E EO
+ 26.
Remark 5.8. Theorem 5.7 does require label sparsity in order to achieve logarithmic dependence on the number of labels m. Label sparsity given in Assumption 4.1 does not help in general in optimizing Precision@as, since when all conditionals the labels with the top-ı marginals, which is needed to make an optimal decision for Precision
smaller than sparsity does not help find
Remark 5.9. The margin condition in Assumption 5.6 is generally inevitable in order to obtain fast rates for Precision@k. To see this, one should delve into the proof of the lower bound in Theorem 3.TT(Appeedii D). where it shown that, for the multi-label problem constructed therein, the margin condition Assumption 4.2 implies the one Assumption 5.6 with a.
5.4 Regret lower bounds
In this section, we present lower bounds for MLC problems with Hamming loss and Precision@k that scale with the label sparsity. In order to do so, introduce two reductions that map binary data to multi-label data and multi-label classifiers to binary classifiers. These reductions are designed so preserve regret (scaled with sparsity for Hamming loss), in that lower bounds for binary classification turn to lower bounds for the multi-label problem with Hamming loss and Precision@
Theorem 5.10. Let us denote by P the class of probability distributions over X x y for which the following assumptions hold: 1) Assumption 5./ holds with some parameters X Cx > 0, Iboo 2) Assumption 4. holds with parameters s E E [0, 1/2); 3) Assumption 4.2 holds with some
parameters az C. 0. Then there exists constant C. > O such that for any 2 1 and any mapping
fn : (X 7)n + MMX,)) it holds that
sup E PEP
RegEn (fw) > Csn
Al 2x+1
where s denotes the sparsity ratio s = 8/m.
Lower bound for Precision @ x can be also obtained as follows.
7

---

Theorem 5.11. Let us denote by P the class of probability distributions over X x y for which the
following assumptions hold: 1) Assumption 5.1 holds with some parameters X, Cx -1 2)
Assumption 5.0holds with some parameters 3 2 0 and K 2 0. Then there exists a constant C > 0 such that for any n 2 1 and any mapping J (X x )) _MXX3) it holds that
sup E PEP
Regtun (fr)
2 On
1
2XFT
Notice that some bound r(6) on the regret that holds with probability (1 - 5). translates into bound r() + on the expected regret in a straightforward manner. Therefore any lower bound on the expected regret also applies when considering high probability bounds. As consequence, the Hamming loss upper bound in Theorem 5.2 is optimal up to a log s/8 factor and the regret bound for Precision@w given in Theorem 5.7 is optimal up to log m/8 factor. It is an interesting open question whether these log factors can be eliminated to get matching regret bound.
Parametric Setting
In this section, we work with multi-label scoring functions in the form of f X + R multi-label scoring function is mapped to multi-label classifier as Φ = (sgn( (fz (<)) (z))) )E] Notice that, for convenience, we now view the multi-label space as {=1] m. instead of {0, 1) Likewise the Hamming loss and Precision @ x will be defined on -= We will work with convex surrogate loss functions with a single argument that applies to a multi-label score function as (f(vi (x)) (Em)) resulting in a vector of losses for œ. The risk of a scoring function f in terms of surrogate loss f then defined as Re(f) 1/m ΣØ= of
and the Bayes f-risk as Ry = inf, Rf) the infimum being taken over all measurable functions. The f-regret of f as usual Reg, (f) = Rf) R₁ In addition, we will assume that the loss functions & are classification calibrated, as recalled the next definition.
Definition 6.1. loss function L: R + R is classification calibrated if for all 1y [0, such that # 1/2 it holds that
inf zER: (2) nf(z)) (1-++)((==)))) > inf m/(=))+ (1- (1=))
ZER
In words, a loss function is dlasiffiationncalibaatee if the sign of the argmin classifier always equals the sign of the Bayes optimal classifier according to true underlying distribution. Examples of classification-calibaaee losses are the exponential loss, the (truncated) squared loss, and the hinge loss.
Our analysis of the regret of Hamming loss and Precision@* x. extends the work of [3] to MLC. In [3, it is shown that for each clasiffication-calibaatee loss, there exists nondecreasing function d [0, 1] + [0, oo) that provides an upper bound on the regret of the 0-1 loss in terms of the regret of the surrogate loss, i.c., (Reego-(())
(f) for all measurable function f X' + R. We extend their 0-1 loss result to Hamming loss and Precision @x under sparsity assumptions. Further results,
including consistency, contained in Appendix E
6.1 Regret upper bound for Hamming loss
We now give regret upper bounds for Hamming loss using classification calibrated losses. These upper bounds scale with the sparsity factor = s/m, and apply to multi-label classifiers whose regret is small enough. Notice that placing small regret constraints on the multi-labe classifier is unavoidable, since a classifier that has very poor performance, c.g. ff (x) = (x) for all j. will always suffer a regret that does not depend on s. Observe that a similar assumption on the performance of the classifiers is also implicitly made in Theorem 5.2 in the form of lower bound
(log / 2)) on the sample size n, which ensures that the plug-in estimator cannot have arbitrarily large regret.
8

---

0.4
0.3
Figure I: Maximum value of in order for Theorem 6.2 to hold. If t is below the curve, an s sparse classifier satisfies the condition in Theorem 6.2 Here, and C. are the constants in Assumption 4.1 Notice that
0.2
0.1
2 1 when mi (x) + for all x and j.
10-3
10
10
Theorem 6.2. Let Assumption 4.1 hold with parameters E N, [0, 1/2) and Assumption 4.2 hold with parameters a and C.. If Regen (0g) (2T -11) then
Reg (61)
a-1
Rege, N (01)
a+
Is
$6
I Rege (f)
(4)
2r
where s and I
Caa c2/(a+11)
(a + 1)/aa/(at1),
To see how this regret bound depends on assume (0) is of the form (0)) = 0", where v > 1.
This implies that v is a strictly convex function. With this in hand, and solving for Regi, (0y), Eq. becomes
Regry (0r) S (I (V- (ite)))) vlat
o4t 2 a-v go Reg
(5)
Thus the Hamming loss regret is upper bounded by power of Reg, multiplied by sparsity factor of the form ga1e. For the sake of illustration, we can instantiate this upper bound for specific losses:
The truncated quadratic loss {(z) = (max (0, =3]) 2 is obtained with
(see The
regret in terms of this surrogate loss boils down to O(s - Reg, (J) 2+ス ). So without margin condition, that is when a 0, this upper bound becomes O Reg, (f)) In contrast to Theorem 5.2] the dependency sparsity ratio here is worse by vo factor. Moreover, this dependency becomes less and less relevant as a increases. Very similar conclusions can drawn for the exponential loss ((z) = e e= since the corresponding c0( = 1 vI 0 (see again 3 can be lower bounded by 8² /2 and upper bounded by 8².
Arbitrary 1: More generally, the exponents of the sparsity and regret factors are related to the curvature of the loss as 0ss ( when no margin condition is imposed. Remark 6.3. In order to apply. Theorem 6.2 needs the regret of o, be below certain threshold, i.c., Rege,, (oy) < s (2 2 This condition can be easily met by requiring Ø to be an sparse classifier, and that parameter in Assumption [4. is not too large. Since it is usually the case that the classifiers employed in MLC are sparse. Theorem 6.2 applies most interesting MLC problems. To see this in detail, let Assumption 4.1 hold, and consider an sparse classifier Φy. The maximum regret w.r.t. ε is clearly at most 2s. Thus, when
(2I 2t)) >
(6)
any sparse classifier will have regret Rege,, (dr) less then the threshold required by Theorem 6.2 Also, notice that (C.) (((+)))()()111))))))))) a)/a a(1+aa) therein depends on Ca and a. Therefore. for an s sparse classifier, (6) is satisfied when is small enough, the maximum value of t depending on a
and Ca. as exemplified in Figure
Remark 6.4. In Appendix H we show that the upper bound of Theorem 6.2 cannot be improved by more than a factor of
Remark 6.5. When working with classes F with finite pseudo-dimension (e.g., (15),, the regret Rege(f) of any f E F can as usual be decomposed into an estimation error Re(f) - Rx( and an approximation error Re( - Ri where f* Re If each component F. of FF has finite pseudo-dimense
d,, and f is the multi-label scoring function that minimizes the
Our notation differs from 3]. Our correspond to their B.

---

multi-label empirical risk minimization w.r.. the surrogate loss function, then the estimation error Re( Rf( can with probability at least 1 - 8 be upper bounded by an expression of the form (imax, d))+ Combined, e.g., with (5) this yields regret upper bounds of the form
[max, d.) + log(m/0) + +R( - R
Regen (0j)
Remark 6.6. One thing to observe is that in the sparse MLC setting what is relevant is not only the dependence on the sample size n, but also the dependence on the sparsity factor s. This is because in such tasks, the total number of classes can even be larger the total number of samples n. Hence, in bounds like 55 or (7), it is the dependence on both s and Rege(]) (and the interplay between the two) that better elucidates the main dependencies.
6.2 Regret upper bound for Precision @K
Recall that Precision @ is ranking metric that ignores the values of the scoring function components at hand, only the ranking of these values is relevant. For instance, unlike risk Re()), shifting the value of f's components by the same amount does not change Reax (Tx As consequence, low Regaax (T*(() does not imply low Reg.(f Thus, to remedy this problem, we re-center the values to make it invariant to shifting values. We denote by f' the centered version of i..., f'(x) = (ff(2) - (1/2) (f1)(z) + f(++1(<))) jE(m] Regea. (T")) For simplicity, we assume that Var X. [ww)(ar) of fi++1(()) to remove ties. (One simple way avoid ties is to add infinitesimal random noise to the values of Π) At this point, we can rely on Theorem 662 obtain an upper bound on Regeax (J)
of the scoring function so that fin] (x) ) > and f(x+1](2))
Notice that Reg (Tx
terms of the surrogate loss regret (f), when combined with the following result.
Theorem 6.7. Let O S 8 S R. If Assumption 4.1 holds with parameters & E N, t E 1/2). then
Regean (T (f)) S
m
(Regen (eege,(0r.))
2x
where ('(x:) = (fj(z) - (1/2) (x)) + ((n+1(aa)) je[m]] fig(2) > 0, (z) < O,
Equivalently, for f that satisfies, Vx,
Regean (T (f)) S 2K
(RegEz (Φ1))
The inequalities are equalities Va X, vk: [s)), (1) 1
As example, when truncated quadratic loss is used, obtain following upper bound by combining Theorem 6.2 with Theorem 6.7 (Regr, (br)) O (n- Reg ( (f'))
Reg (T (T)( V 2x
(8)
In turn, (eg)( in the RHS of (8)
can be upper bounded as in
Conclusions
The results we presented aim at filling the gap in the statistical analysis of MLC problems and methods. In the non- parametric setting, we provided upper and lower fmite-sample regret bounds for k-NN classifiers in MLC with Hamming loss and Precision @ under suitable smoothness assumptions. In fact, our analysis may be extended to other loss functions, like rank loss (e.g., [II ) and micro- and macro-averaged complex performance measures (e.g., (2)) The crucial point is formulating the right assumptions for those loss functions. The general steps of these analyses will be similar to those contained in this paper, since the Bayes classifiers for these losses are in fact based marginal quantities [23]. As an other important research avenue to explore to make our lower bound more tight. Lower bounds that are given in this paper are tight up to log factors and to weed out this log factor seems challenging in the multi-label setup.
In the parametric setting, we considered general regret reductions from MLC surrogate losses to Hamming loss and Precision@* k, and then we presented regret bounds one can achieve by empirical risk minimization operating on the surrogate losses. One natural question is to understand to what extent these regret bounds (like those contained in Eqs. and are tight, specifically when the approximation error (e()") R is strictly positive.
10

---

References
[1] R. Agrawal, A. Gupta, Y. Prabhu, and M. Varma. Multi-label learning with millions of labels: Recommending advertiser bid phrases for web pages. In WWW, pages 13-24. ACM, 2013.
[2] Jean-Yves Audibert and Alexandre B. Tsybakov. Fast learning rates for plug- in classifiers. Ann. Statist., 35(2):008-63 04 2007.
[3] Peter L Bartlett, Michael I Jordan, and Jon D McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association, 101(473):138-15,, 2006.
[4] J. Bogatinovski, L. Todorovski, S. Džeroski, and D. Kocev. Comprehensive comparative study of multi-label classification methods. Expert Systems with Applications, 203:117-215, 2022.
[5] Kamalika Chaudhuri and Sanjoy Dasgupta. Rates of convergence for nearest neighbor classi- fication. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3437-3445, 2014.
[6] Lena Chekina, Dan Gutfreund, Aryeh Kontorovich, Lior Rokach, and Bracha Shapira. Ex- ploiting label dependencies for improved sample complexity. Machine Learning, 91(1):1 42, 2013.
[7] Evgenii Chzhen. Classification of sparse binary vectors, 2019.
[8] Evgenii Chzhen, Christophe Denis, Mohamed Hebiri, and Joseph Salmon. On the benefits of output sparsity for multi-label classification, 2017.
[9] O. Dekel and O. Shamir.
classification with more classes than examples.
In AISTATS, pages 137-144, 2010.
[10] K. Dembezyíski, W. Waegeman, W. Cheng, and E. Hüllermeier. An analysis of chaining in multi-label classification. ECAI, 2012.
[11] Krzysztof Dembezyúski, Wojciech Kotlowski, and Eyke Hüllermeier. Consistent multilabel ranking through univariate losses. In John Langford and Joelle Pineau, editors, Proceedings of the 29th International Conference on Machine Learning (ICML 2012), pages 1319-1326. Omnipress, 2012.
[12] Krzysztof Dembezyáski, Willem Waegeman, Weiwei Cheng, and Eyke Hüllermeier. On loss minimization and label dependence in multi-label classification. Mach. Learn., 88:5-45, 2012.
[13] J. Deng, S. Satheesh, A. C. Berg, and L. Fei-Fei. Fast and balanced: Efficient label tree learning for large scale object recognition. In NIPS 24, pages 567-575, 2011.
[14] L. Devroye and Györfi. Nonparametric Density Estimation: The L1 View. Wiley, NY, 1985.
[15] L. Devroye, L. Györfi, and Lugosi. Probabilistic Theory of Pattern Recognition. Springer, New York, 1996.
[16] Wei Gao and Zhi-Hua Zhou. On the consistency of multi-label learning, Artificial Intelligence, 199-200:22 44, 2013.
[17] Walter Gerych, Tom Hartvigsen, Luke Buquicchio, Emmanuel Agu, and Elke A. Runden- steiner. Recurrent bayesian classifier chains for exact multi-label classification. In M. Ranzato, A. Beygelzimer, Y. Dauphin, P.S. Liang, and J. Wortman Vaughan, editors, Advances in Neural Information Processing Systems, volume 34, pages 15981-15992, Curran Associates, Inc., 2021.
[18] D. Hsu, S. Kakade, J. Langford, and T. Zhang. Multi-label prediction via compressed sensing. In NIPS, 2009.
[19] Himanshu Jain, Venkatesh Balasubramanian, Bhanu Chunduri, and Manik Varma. Slice: Scalable linear extreme classifiers trained on 100 million labels for related searches. In WSDM '19 February 1 1-15, 2019, Melbourne, VIC, Australia. ACM, February 2019. Best Paper Award at WSDM 19.
11

---

[20] Himanshu Jain, Yashoteja Prabhu, and Manik Varma. Extreme multi-label loss functions for recommendation, tagging, ranking & other missing label applications. In Proceedings of the 22nd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, page New York, NY, USA, 2016. Association for Computing Machinery.
[21] Kalina Jasinska, Krzysztof Dembezyúski, Róbert Busa-Fekete, Karlson Pfannschmidt, Timo Klerx, and Eyke Hüllermeier. Extreme F-measure maximization using sparse probability estimates. In ICML 2016, pages 1435-1444. PMLR, 2016.
[22] Wojciech Kotlowski and Krzysztof Dembezyíski. Surrogate regret bounds for generalized classification performance metrics. Machine Learning Journal, 106:549-572, 2017.
[23] Oluwasanmi Koyejo, Nagarajan Natarajan, Pradeep Ravikumar, and Inderjit Dhillon. Consis- tent multilabel classification. In NIPS 2015, pages 3321-3329, 2015.
[24] S. Koyejo, N. Natarajan, P. Ravikumar, and I. Dhillon. Consistent binary classification with generalized performance metrics. In NIPS 27, pages 21424-7752,00 2014.
[25] Aditya K Menon, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Multilabel reductions: what is my loss optimising? In Advances in Neural Information Processing Systems 32, pages 10699-10610. Curran Associates, Inc., 2019.
[26] Yashoteja Prabhu, Anil Kag, Shrutendra Harsola, Rahul Agrawal, and Manik Varma. Parabel: Partitioned label trees for extreme classification with application dynamic search advertising. In WWW 2018, pages 993-1002. ACM, 2018.
[27] Yashoteja Prabhu and Manik Varma. FastXML: a fast, accurate and stable tree-classifier for extreme multi-label learning. In KDD 2014, pages 263-272. ACM, 2014.
[28] Henry Reeve and Kabán. Classification with unknown class-conditional label noise on non- compact feature spaces. In Alina Beygelzimer and Daniel Hsu, editors, Proceedings of the Thirty-Second Conference on Learning Theory, volume 99 of Proceedings of Machine Learning Research, pages
Phoenix, USA, 25-28 Jun 2019. PMLR.
[29] Henry Reeve and Ata Kaban. Fast rates for kNN classifter robust to unknown asymmetric label noise. In Kamalika Chaudhuri and Ruslan Salakhutdinov, editors, Proceedings of the 36th International Conference on Machine Learning, volume 97 of Proceedings of Machine Learning Research, pages 5401-5409, Long Beach, California, USA, 09-15 Jun 2019. PMLR.
[30] G. Tsoumakas and I. Katakis. Multi-label classification: An overview. Int. J Data. Warehousing and Mining, 2007:1 2007.
[31] W. Waegeman, K. Dembezynski, A. Jachnik, W. Cheng, and E. Hüllermeier. On the bayes- optimality of f-measure maximizers. Journal of Machine Learning Research, 15(1):3333-3388 2014.
[32] Guoqiang Wu and Jun Zhu. Multi-label classification: Do hamming loss and subset accuracy really conflict with each other? In Proceedings of the 34th International Conference on Neural Information Processing Systems, NIPS' Curran Associates Inc., 2020.
[33] Marek Wydmuch, Kalina Jasinska, Mikhail Kuznetsov, Róbert Busa-Fekete, and Krzysztof Dembezyáski. A no-regret generalization of hierarchical softmax extreme multi-label classification. In NeurlPS 2018 pages 6358-6368 2018.
[34] Hsiang-Fu Fu Yu, Prateek Jain, Purushottam Kar, and Inderjit Dhillon. Large-scale multi-label learning with missing labels. In Proceedings of the 31 International Conference on Machine Learning (ICML). Journal of Machine Learning Research, June 2014.
[35] Z.H. Zhang, M.L.; Zhou. MI-knn: A lazy learning approach to multi-label learning. Pattern Recognition, 40(7):2038-200 20408 2007.
12

---

Checklist
1. For all authors...
(a) Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope? [Yes]
(b) Did you describe the limitations of your work? [Yes] Our study is built upon Assump-
tion
4.2 5.I 5.6
Other limitations are described in the conclusions.
(c) Did you discuss any potential negative societal impacts of your work? [No] This is purely theoretic investigation on multi-label classification. We do not foresee any specific negative societal impact.
(d) Have you read the ethics review guidelines and ensured that your paper conforms to them? [Yes]
2. If you are including theoretical results...
(a) Did you state the full set of assumptions of all theoretical results? [Yes] Assumptions
described Assumption 4.1 5.1 5.6
(b) Did you include complete proofs of all theoretical results? [Yes] Yes, all the proof are in the appendix.
3. If you ran experiments..
(a) Did you include the code, data, and instructions needed to reproduce the main ex- perimental results (either in the supplemental material or a URL)? [N/A] No experimential results.
(b) Did you specify all the training details (e.g., data splits, hyperparameters, how they were chosen)? [N/A] No experimential results.
(c) Did you report error bars (c.g., with respect to the random seed after running experi- ments multiple times)? [N/A] No experimential results.
(d) Did you include the total amount of compute and the type of resources used (e.g., type of GPUs, internal cluster, or cloud provider)? [N/A] No experimential results.
4. If you are using existing assets (e.g., code, data, models) or curating/releasing new assets...
(a) your work uses existing assets, did you cite the creators? [N/A] No assets.
(b) Did you mention the license of the assets? [N/A] No assets.
(c) Did you include any new assets either in the supplemental material or as a URL? [N/A]
No assets.
(d) Did you discuss whether and how consent was obtained from people whose data you're using/curating? [N/A] No data provided.
(e) Did you discuss whether the data you using/curating contains personally identifiable information or offensive content? [N/A] No data provided.
5. If you used crowdsourcing or conducted research with human subjects...
(a) Did you include the full text of instructions given participants and screenshots, if applicable? [N/A] No crowsourcing.
(b) Did you describe any potential participant risks, with links to Institutional Review Board (IRB) approvals, applicable? [N/A] No crowsourcing.
(c) Did you include the estimated hourly wage paid to participants and the total amount spent participant compensation? [N/]]
No crowsourcing.
13


Reviews:
[{"review_id": "amHd_DqvoMi", "reviewer": "Reviewer_6DSW", "summary": "This works studies multi-label classification (MLC) problems. Focusing on sparse label regime (in which the number of active labels is small) and an MLC variant of Tsybakov's low-noise condition,  the paper shows upper and lower regret for two typical performance measures in MLC, namely Hamming Loss and Precision@k, in two settings: non-parametric tackled by K-NN classifiers, and empirical risk minimization (ERM) of surrogate loss functions. It is clearly demonstrated in the paper that sparse label regime and low-noise condition, which are two reasonable conditions for practical considerations as well, can improve the regret bounds.", "questions": "In the conclusion section, you said that the analysis can be extended to rank loss, 0/1 loss, and micro/macro F-measures. I don't see how this extension can come easily. As far as I understand, for Hamming Loss and Precision@k, in order to derive Bayes optimal classifiers, it is indeed sufficient to have good estimates of $\\eta_j(x)$ for each label $j$. In some sense, the labels are independent of each other. For other complex performance measures, the relations among different labels are crucial in designing Bayes optimal classifiers. Therefore I don't see how the analysis can be extended easily. Can you elaborate on this point?", "limitations": "One limitation is that the paper does not have experiments. It would be nice to see some empirical verifications of the results on synthetic and real data sets.", "rating": 8, "confidence": 4, "soundness": 4, "presentation": 4, "contribution": 4, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Overall this paper is a very strong piece of theoretical work. It is also well-polished. I really enjoyed reading it. Good work!\n\n**Originality:**\nThis work is novel. The related work is adequately cited and discussed.\n\n**Quality:**\nThe work is sound and rigorous. \n\n**Clarity:**\nThe paper is well-polished. The notations are used consistently. Insights are provided to help readers understand the crux of the theoretical results.\n\n**Significance:**\nThe paper is an important piece of work in advancing the theoretical guarantees in MLC community. Moreover, it is well-presented, so it could have even more influence.\n\n---\nAfter rebuttal: I have read the other reviews and the authors' responses. My rating for this paper has not changed.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0OrFKHIef5H", "reviewer": "Reviewer_ZyU8", "summary": "This paper focus on the regret bounds for multi-label classification. Under a novel low noise assumption and a sparsity constraint on the posterior possibility $\\eta(x)$, the regret upper bounds are provided for surrogate risk minimization w.r.t. Hamming loss and Precision@k. With the sample assumptions and a widely used constraint on smoothness, the high probability bounds are given for a MLC variant of KNN. Regret lower bounds are also provided for KNN to show the tightness of its upper bounds. The given results show that the interplay between low noise assumption and label sparsity can lead to non-trivial improvement on the regret bounds.\n", "questions": "Since $\\nu\\geq 1$ usually holds, I'm not sure if (5) is better than a common regret bound of $O_p(1/\\sqrt{n})$ that is induced without dependence on the low noise and sparsity assumptions.", "limitations": "The authors addressed the limitations of the used assumptions and the absence of regret lower bounds for surrogate risk minimization.", "rating": 7, "confidence": 3, "soundness": 4, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "Strengths:\n1. The assumptions made in this paper are mild and are closely related to the widely accepted ones. \n\n2. The analyses on the MLC variant of KNN are detailed and novel. To the best of my knowledge, there is no work that contributes to the regret bound analysis  of KNN in the regime of MLC. \n\nWeaknesses:\n\nThough the analyses of the MLC variant of KNN are rigorous, the analyzed variant may not be a widely accepted one, which can cause the gap between practice and theory.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}, {"review_id": "0FH1Tme5d0F", "reviewer": "Reviewer_gucg", "summary": "The paper provides regret bounds for large-scale multilabel classification with Hamming loss and precision-at-k, for both nonparametric (k-nn) and parametric estimation. Efficient scaling of the bounds is achieved by making a sparsity assumption on the labels, which combines a hard sparsity constraint (at most s labels with high probability) with a soft sparsity constraint (all other labels at most probability t < 0.5). ", "questions": "I'm not convinced that 5.6 is a realistic condition to impose. The number of relevant labels typically varies per data-point, so it seems very strict to demand that the marginals have a steep change exactly between k and k+1 for most of instances. Do you have an argument why this condition would be fulfilled in real data?\n\nl. 235 announces the introduction of two reductions, but the second does not seem to do so.\n\nAssumption 5.1.\nshould this really be a seminorm, and not an actual norm? A seminorm could just ignore a few of the components (=marginals of labels), so that in these labels there would not need to be any smoothness? Is that deliberate? A norm would also have the nice effect that you need not worry about which norm to use, since this is finite dimensional and any factor could be absorbed by C.\n\n== Supplementary ==\nUnfortunately, I have not yet had the time to go through the proofs in the supplementary. When I'm done with that, I will update the review accordingly.", "limitations": "the main limitation I see is assumption 5.6, which I doubt is realistic.", "rating": 5, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "The paper provides a step in closing the gap in theory between binary classification and large-scale multilabel classification. The authors manage to incorporate the sparsity property in a way that does not require an unrealistically strict sparsity assumption (i.e. at most s labels, ever). The multilabel-generalization of the Tsybakov-condtition also seems sensible to me, but I doubt that the k-vs-k+1 separation condition needed for the precision-at-k result is realistic.\n\nFor the most part, I found it easy enough to follow the writing in the paper. In the regret bounds, the formulas get quite long, so they can become a bit difficult to parse, but I'm not sure if there is anything the authors could have done to improve this.", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"ethics_review_area": ""}, "extra_text": {}}]

Author rebuttal:

Decision: Accept