Title: Cardinality-Aware Set Prediction and Top-$k$ Classification
Venue: NeurIPS 2024
Accepted: yes
Keywords: top-k classification

cardinality-aware algorithms

consistency

cost-sensitive loss

learning theory
Abstract: We present a detailed study of cardinality-aware top-$k$ classification, a novel approach that aims to learn an accurate top-$k$ set predictor while maintaining a low cardinality. We introduce a new target loss function tailored to this setting that accounts for both the classification error and the cardinality of the set predicted. To optimize this loss function, we propose two families of surrogate losses: cost-sensitive comp-sum losses and cost-sensitive constrained losses. Minimizing these loss functions leads to new cardinality-aware algorithms that we describe in detail in the case of both top-$k$ and threshold-based classifiers. We establish $H$-consistency bounds for our cardinality-aware surrogate loss functions, thereby providing a strong theoretical foundation for our algorithms. We report the results of extensive experiments on CIFAR-10, CIFAR-100, ImageNet, and SVHN datasets demonstrating the effectiveness and benefits of our cardinality-aware algorithms.

Cardinality-Aware Set Prediction and Topk Classification

Abstract

We present a detailed study of cardinality-aware topk classification, a novel approach that aims to learn an accurate topk set predictor while maintaining a low cardinality. We introduce a new target loss function tailored to this setting that accounts for both the classification error and the cardinality of the set predicted. To optimize this loss function, we propose two families of surrogate losses: costsensitive comp-sum losses and cost-sensitive constrained losses. Minimizing these loss functions leads to new cardinality-aware algorithms that we describe in detail in the case of both topk and threshold-based classifiers. We establish H -consistency bounds for our cardinality-aware surrogate loss functions, thereby providing a strong theoretical foundation for our algorithms. We report the results of extensive experiments on CIFAR-10, CIFAR-100, ImageNet, and SVHN datasets demonstrating the effectiveness and benefits of our cardinality-aware algorithms.

1 Introduction

Topk classification consists of predicting the k most likely classes for a given input, as opposed to solely predicting the single most likely class. Several compelling reasons support the adoption of this framework. First, it enhances accuracy by allowing the model to consider the top k predictions, accommodating uncertainty and providing a more comprehensive prediction. This is particularly valuable in scenarios where multiple correct answers exist, such as image tagging, where a topk classifier can identify multiple relevant objects in an image. Second, topk classification is applicable in ranking and recommendation tasks such as suggesting the top k most relevant products in ecommerce based on user queries. The confidence scores associated with the top k predictions also serve as a means to estimate the model's uncertainty, which is crucial in applications requiring insight into the model's confidence level.

The predictions of a topk classifier are also useful in several natural settings. For example, ensemble learning can benefit from topk predictions as they can be combined from multiple models, contributing to improved overall performance by introducing a more robust and diverse set of predictions. In addition, topk predictions can serve as input for downstream tasks like natural language generation or dialogue systems, enhancing the performance of these tasks by providing a broader range of potential candidates. Finally, the interpretability of the model's decision-making process is enhanced by examining the top k predicted classes, allowing users to gain insights into the rationale behind the model's predictions.

The appropriate k for a task at hand may be determined by the application itself like a recommendor system always expecting a fixed set size to be returned. For other applications, it may be natural to let the cardinality of the returned set vary with the model's confidence or other properties of the task. Designing effective algorithms with learning guarantees for this setting is our main goal.

In this paper, we introduce the problem of cardinality-aware set prediction, which is to learn an accurate set predictor while maintaining a low cardinality. The core idea is that an effective algorithm should dynamically adjust the cardinality of its prediction sets based on input instances. For topk classifiers, this means selecting a larger k for difficult inputs to ensure high accuracy, while opting for a smaller k for simpler inputs to maintain low cardinality. Similarly, for threshold-based classifiers, a lower threshold can be used for difficult inputs to minimize the risk of misclassification, whereas a higher threshold can be applied to simpler inputs to reduce cardinality.

To tackle this problem, we introduce a novel target loss function which captures both the classification error and the cardinality of a prediction set. Minimizing this target loss function directly is an instance-dependent cost-sensitive learning problem, which is intractable for most hypothesis sets. Instead, we derive two families of general surrogate loss functions that benefit from smooth properties and favorable optimization solutions.

To provide theoretical guarantees for our cardinality-aware topk approach, we first study consistency properties of surrogate loss functions for the general topk problem with a fixed k . Unlike standard classification, the consistency of surrogate loss functions for the topk problem has been relatively unexplored. A crucial property in this context is the asymptotic notion of Bayes-consistency , which has been extensively studied in standard binary and multi-class classification [Zhang, 2004a, Bartlett et al., 2006, Zhang, 2004b, Bartlett and Wegkamp, 2008]. While Bayes-consistency has been explored for various topk surrogate losses [Lapin et al., 2015, 2016, 2018, Yang and Koyejo, 2020, Thilagar et al., 2022], some face limitations. Non-convex 'hinge-like" surrogates [Yang and Koyejo, 2020], surrogates inspired by ranking [Usunier et al., 2009], and polyhedral surrogates [Thilagar et al., 2022] cannot lead to effective algorithms as they cannot be efficiently computed and optimized. Negative results also indicate that several convex "hinge-like" surrogates [Lapin et al., 2015, 2016, 2018] fail to achieve Bayes-consistency [Yang and Koyejo, 2020]. On the positive side, it has been shown that the logistic loss (or cross-entropy loss used with the softmax activation) is a Bayes-consistent loss for topk classification [Lapin et al., 2015, Yang and Koyejo, 2020].

Weshow that, remarkably, several widely used families of surrogate losses used in standard multi-class classification admit H -consistency bounds [Awasthi, Mao, Mohri, and Zhong, 2022a,b, Mao, Mohri, and Zhong, 2023f,b] with respect to the topk loss. These are strong non-asymptotic consistency guarantees that are specific to the actual hypothesis set H adopted, and therefore also imply asymptotic Bayes-consistency. We establish this property for the broad family of comp-sum losses [Mao, Mohri, and Zhong, 2023f], comprised of the composition of a non-decreasing and non-negative function with the sum exponential losses. This includes the logistic loss, the sum-exponential loss, the mean absolute error loss, and the generalized cross-entropy loss. Additionally, we extend these results to constrained losses , a family originally introduced for multi-class SVM [Lee et al., 2004], which includes the constrained exponential, hinge, squared hinge, and ρ -margin losses. The guarantees of H -consistency provide a strong foundation for principled algorithms in topk classification by directly minimizing these surrogate loss functions.

We then leverage these results to derive strong guarantees for the two families of cardinality-aware surrogate losses: cost-sensitive comp-sum and cost-sensitive constrained losses. Both families are obtained by augmenting their topk counterparts [Lapin et al., 2015, 2016, Berrada et al., 2018, Reddi et al., 2019, Yang and Koyejo, 2020, Thilagar et al., 2022] with instance-dependent cost terms. We establish strong H -consistency bounds, implying Bayes-consistency, for both families relative to the cardinality-aware target loss. Our H -consistency bounds for the topk problem are further beneficial here in that the cardinality-aware problem can consist of fixing and selecting from a family topk classifiers-we now know how to effectively learn each topk classifier.

The rest of the paper is organized as follows. In Section 2, we formally introduce the cardinality-aware set prediction problem along with our new families of surrogate loss functions. Section 3 instantiates our algorithms in the case of both topk classifiers and threshold-based classifiers, and Section 4 presents strong theoretical guarantees. In Section 5, as well as in Appendix J and Appendix K, we present experimental results on the CIFAR-10, CIFAR-100, ImageNet, and SVHN datasets, demonstrating the effectiveness of our algorithms.

2 Cardinality-aware set prediction

In this section, we introduce cardinality-aware set prediction, where the goal is to devise algorithms that dynamically adjust the prediction set's size based on the input instance to both achieve high accuracy and maintain a low average cardinality. Specifically, for topk classifiers, our objective is to determine a suitable cardinality k for each input x , with higher values of k for instances that are more difficult to classify.

To address this problem, we first define a cardinality-aware loss function that accounts for both the classification error and the cardinality of the set predicted (Section 2.1). However, minimizing this loss function directly is computationally intractable for non-trivial hypothesis sets. Thus, to optimize it, we introduce two families of surrogate losses: cost-sensitive comp-sum losses (Section 2.2) and cost-sensitive constrained losses (Section 2.3). We will later show that these loss functions benefits from favorable guarantees in terms of H -consistency (Section 4.3).

2.1 Cardinality-aware problem formulation and loss function

The learning setup for cardinality-aware set prediction is as follows.

Problem setup. We denote by X the input space and Y = [ n ]∶ = { 1 , . . . , n } the label space. Let { g k ∶ k ∈ K } denote a collection of given set predictors, induced by a parameterized set predictor g k ∶ X ↦ 2 Y , where each K ⊂ R is a set of indices. This could be a subset of the family of topk classifiers induced by some classifier h , or a family of threshold-based classifiers based on some scoring function s ∶ X × Y ↦ R . In that case, g k ( x ) then comprises the set of y s with a score s ( x, y ) exceeding the threshold τ k defining g k . This formulation covers as a special case standard conformal prediction set predictors [Shafer and Vovk, 2008], as well as set predictors defined as confidence sets described in [Denis and Hebiri, 2017]. We will denote by ∣ g k ( x )∣ the cardinality of the set g k ( x ) predicted by g k for the input x . To simplify the discussion, we will assume that ∣ g k ( x )∣ is an increasing function of k , for any x . For a family of topk classifiers or threshold-based classifiers, this simply means that they are sorted in increasing order of k or decreasing order of the threshold values.

To account for the cost associated with cardinality, we introduce a non-negative and increasing function cost ∶ R + → R + , where cost (∣ g k ( x )∣) represents the cost associated to the cardinality ∣ g k ( x )∣ . Common choices for cost include cost (∣ g k ( x )∣) = ∣ g k ( x )∣ , or a logarithmic function cost (∣ g k ( x )∣) = log (∣ g k ( x )∣) as in our experiments (see Section 5), to moderate the magnitude of the cost relative to the binary classification loss. Our analysis is general and requires no assumption about cost .

Our goal is to learn to assign to each input instance x the most appropriate index k ∈ K to both achieve high accuracy and maintain a low average cardinality.

Cardinality-aware loss function. As in the ordinary multi-class classification problem, we consider a family R of scoring functions r ∶ X × K → R . For any x , r ( x, k ) denotes the score assigned to the label (or index) k ∈ K , given x ∈ X . The label predicted is r ( x ) = argmax k ∈ K r ( x, k ) , with ties broken in favor of the largest index. To account for both classification accuracy and cardinality cost, we define the cardinality-aware loss function for a scoring function r and input-output label pair ( x, y ) ∈ X × Y as a linearized loss of these two criteria:

where the first term is the standard loss for a topk prediction taking the value one when the correct label y is not included in the topk set and zero otherwise, and λ > 0 is a hyperparameter that governs the balance between prioritizing accuracy versus limiting cardinality. The learning problem then consists of using a labeled training sample ( x 1 , y 1 ) , . . . ( x m , y m ) drawn i.i.d. from some (unknown) distribution D to select r ∈ R with a small expected cardinality-aware loss E ( x,y )∼ D [ glyph[lscript] ( r, x, y )] .

The loss function (1) can be equivalently expressed in terms of an instance-dependent cost function c ∶ X × K × Y → R + :

where c ( x, k, y ) = 1 y ∉ g k ( x ) + λ cost (∣ g k ( x )∣) . Minimizing (2) is an instance-dependent cost-sensitive learning problem. However, directly minimizing this target loss is intractable. To optimize this loss function, we introduce two families of surrogate losses in the next sections: cost-sensitive comp-sum losses and cost-sensitive constrained losses. Note that throughout this paper, we will denote all target

(or true) losses on which performance is measured with an glyph[lscript] , while surrogate losses introduced for ease of optimization are denoted by ̃ glyph[lscript] .

2.2 Cost-sensitive comp-sum surrogate losses

Our surrogate cost-sensitive comp-sum, c-comp , losses are defined as follows: for all ( r, x, y ) ∈ R × X × Y , ̃ glyph[lscript] c -comp ( r, x, y ) = ∑ k ∈ K ( 1 -c ( x, k, y )) ̃ glyph[lscript] comp ( r, x, k ) , where the comp-sum loss ̃ glyph[lscript] comp is defined as in [Mao, Mohri, and Zhong, 2023f]. That is, for any r in a hypothesis set R and ( x, y ) ∈ X × Y , ̃ glyph[lscript] comp ( r, x, y ) = Φ (∑ y ′ ≠ y e r ( x,y ′ )-r ( x,y ) ) , where Φ ∶ R + → R + is non-decreasing. See Section 4.2 for more details. For example, when the logistic loss is used, we obtain the cost-sensitive logistic loss:

The negative log-term becomes larger as the score r ( x, k ) increases. Thus, the loss function imposes a greater penalty on higher scores r ( x, k ) through a penalty term ( c ( x, k, y ) -1 ) that depends on the cost assigned to the expert's prediction g k ( x ) .

2.3 Cost-sensitive constrained surrogate losses

Constrained losses are defined as a summation of a function Φ applied to the scores, subject to a constraint, as in [Lee et al., 2004]. For any r ∈ R and ( x, y ) ∈ X × Y , they are expressed as

where Φ ∶ R → R + is non-increasing. See Section 4.2 for a detailed discussion. Inspired by these constrained losses, we introduce a new family of surrogate losses, cost-sensitive constrained ( c-cstnd losses) which are defined, for all ( r, x, y ) ∈ R × X × Y , by ̃ glyph[lscript] c -cstnd ( r, x, y ) = ∑ k ∈ K c ( x, k, y ) Φ (-r ( x, k )) , with the constraint ∑ k ∈ K r ( x, k ) = 0 , where Φ ∶ R → R + is nonincreasing. For example, for Φ ( t ) = e -t , we obtain the cost-sensitive constrained exponential loss:

3 Cardinality-aware algorithms

Minimizing the cost-sensitive surrogate loss functions described in the previous section directly leads to novel cardinality-aware algorithms. In this section, we briefly detail the instantiation of our algorithms in the specific cases of topk classifiers (our main focus) and threshold-based classifiers.

Topk classifiers. Here, the collection of set predictors is a subset of the topk classifiers, defined by g k ( x ) = { h 1 ( x ) , . . . , h k ( x )} , where h 1 ( x ) , . . . , h k ( x ) are the induced topk labels for a classifier h . The cardinality in this case coincides with the index: ∣ g k ( x )∣ = k , for any x ∈ X . The cost is defined as c ( x, k, y ) = 1 y ∉{ h 1 ( x ) ,..., h k ( x )} + λ cost ( k ) , where cost ( k ) can be chosen to be k or log ( k ) . Thus, our cardinality-aware algorithms for topk classification can be described as follows. At training time, we assume access to a sample set {( x i , y i )} m i = 1 and the costs each topk set incurs, { c ( x i , k, y i )} m i = 1 , where k ∈ K , a pre-fixed subset. The goal is to minimize the target cardinality-aware loss function ∑ m i = 1 glyph[lscript] ( r, x i , y i ) = ∑ m i = 1 c ( x i , r ( x i ) , y i ) over a hypothesis set R . Our algorithm consists of minimizing a surrogate loss such as the cost-sensitive logistic loss, defined as ˆ r = argmin r ∈ R ∑ m i = 1 ∑ k ∈ K ( 1 -c ( x i , k, y i )) log (∑ k ′ ∈ K e r ( x,k ′ )-r ( x,k ) ) . At inference time, we use the topˆ r ( x ) set { h 1 ( x ) , . . . , h ˆ r ( x ) ( x )} for prediction, with the accuracy 1 y ∈{ h 1 ( x ) ,..., h ˆ r ( x ) ( x )} and cardinality ˆ r ( x ) for that instance.

In Section 5, we compare the accuracy-versus-cardinality curves of our cardinality-aware algorithms obtained by varying λ with those of topk classifiers, demonstrating the effectiveness of our algorithms. What λ to select for a given application will depend on the desired accuracy. Note that the performance of the algorithm in [Denis and Hebiri, 2017] in this setting is theoretically the same as that of topk

classifiers. The algorithm is designed to maximize accuracy within a constrained cardinality of k , and it always reaches maximal accuracy at the boundary K after the cardinality is constrained to k ≤ K .

Threshold-based classifiers. Here, the set predictor is defined via a set of thresholds τ k : g k ( x ) = { y ∈ Y ∶ s ( x, y ) > τ k } . When the set is empty, we just return argmax y ∈ Y s ( x, y ) by default. The description of the costs and other components of the algorithms is similar to that of topk classifiers. A special case of threshold-based classifier is conformal prediction [Shafer and Vovk, 2008], which is a general framework that provides provably valid confidence intervals for a black-box scoring function. Split conformal prediction guarantees that P ( Y m + 1 ∈ C s,α ( X m + 1 )) ≥ 1 -α for some scoring function s ∶ X × Y → R , where C s,α ( X m + 1 ) = { y ∶ s ( X m + 1 , y ) ≥ ˆ q α } and ˆ q α is the ⌈ α ( m + 1 )⌉/ m empirical quantile of s ( X i , Y i ) over a held-out set {( X i , Y i )} i m = 1 drawn i.i.d. from some distribution D (or just exchangeably). Note, however, that the framework does not supply an effective guarantee on the size of the sets C s,α ( X m + 1 ) .

In Appendix K, we present in detail a series of early experiments for our algorithm used with threshold-based classifiers and include more discussion. Our experiments suggest that, when the training sample is sufficiently large, our algorithm can outperform conformal prediction.

4 Theoretical guarantees

Here, we present theory for our cardinality-aware algorithms. Our analysis builds on theory of topk algorithms, and we start by providing stronger results than previously known for topk surrogates.

4.1 Preliminaries

We denote by D a distribution over X × Y and write p ( x, y ) = D ( Y = y ∣ X = x ) for the conditional probability of Y = y given X = x , and use p ( x ) = ( p ( x, 1 ) , . . . , p ( x, n )) to denote the corresponding conditional probability vector. We denote by glyph[lscript] ∶ H all × X × Y → R a loss function defined for the family of all measurable functions H all . Given a hypothesis set H ⊆ H all , the conditional error of a hypothesis h and the best-in-class conditional error are defined as follows: C glyph[lscript] ( h, x ) = E y ∣ x [ glyph[lscript] ( h, x, y )] = ∑ y ∈ Y p ( x, y ) glyph[lscript] ( h, x, y ) and C ∗ glyph[lscript] ( H , x ) = inf h ∈ H C glyph[lscript] ( h, x ) . Accordingly, the generalization error of a hypothesis h and the best-in-class generalization error are defined by: E glyph[lscript] ( h ) = E ( x,y )∼ D [ glyph[lscript] ( h, x, y )] = E x [ C glyph[lscript] ( h, x )] and E ∗ glyph[lscript] ( H ) = inf h ∈ H E glyph[lscript] ( h ) = inf h ∈ H E x [ C glyph[lscript] ( h, x )] . Given a score vector ( h ( x, 1 ) , . . . , h ( x, n )) generated by hypothesis h , we sort its components in decreasing order and write h k ( x ) to denote the k -th label, that is h ( x, h 1 ( x )) ≥ h ( x, h 2 ( x )) ≥ . . . ≥ h ( x, h n ( x )) . Similarly, for a given conditional probability vector p ( x ) = ( p ( x, 1 ) , . . . , p ( x, n )) , we write p k ( x ) to denote the k -th element in decreasing order, that is p ( x, p 1 ( x )) ≥ p ( x, p 2 ( x )) ≥ . . . ≥ p ( x, p n ( x )) . In the event of a tie for the k -th highest score or conditional probability, the label h k ( x ) or p k ( x ) is selected based on the highest index when considering the natural order of labels.

The target generalization error for topk classification is given by the topk loss, which is denoted by glyph[lscript] k and defined, for any hypothesis h and ( x, y ) ∈ X × Y by

The loss takes value one when the correct label y is not included in the topk predictions made by the hypothesis h , zero otherwise. In the special case where k = 1 , this is precisely the familiar zero-one classification loss. Like the zero-one loss, optimizing the topk loss is NP-hard for common hypothesis sets. Therefore, alternative surrogate losses are typically used to design learning algorithms. A crucial property of these surrogate losses is Bayes-consistency . This requires that, asymptotically, nearly minimizing a surrogate loss over the family of all measurable functions leads to the near minimization of the topk loss over the same family [Steinwart, 2007].

Definition 4.1. A surrogate loss ̃ glyph[lscript] is said to be Bayes-consistent with respect to the topk loss glyph[lscript] k if, for all given sequences of hypotheses { h n } n ∈ N ⊂ H all and any distribution, lim n →+∞ E ̃ glyph[lscript] ( h n ) -E ∗ ̃ glyph[lscript] ( H all ) = 0 implies lim n →+∞ E glyph[lscript] k ( h n ) -E ∗ glyph[lscript] k ( H all ) = 0 .

Bayes-consistency is an asymptotic guarantee and applies only to the family of all measurable functions. Recently, Awasthi, Mao, Mohri, and Zhong [2022a,b] (see also [Awasthi et al., 2021a,b, 2023a,b, Mao et al., 2023c,d,e,a, 2024c,b,a,e,h,i,d,f,g, Mohri et al., 2024]) proposed a stronger consistency guarantee, referred to as H -consistency bounds . These are upper bounds on the target

estimation error in terms of the surrogate estimation error that are non-asymptotic and hypothesis set-specific.

Definition 4.2. Given a hypothesis set H , a surrogate loss ̃ glyph[lscript] is said to admit an H -consistency bound with respect to the topk loss glyph[lscript] k if, for some non-decreasing function f , the following inequality holds for all h ∈ H and for any distribution: f ( E glyph[lscript] k ( h ) -E ∗ glyph[lscript] k ( H )) ≤ E ̃ glyph[lscript] ( h ) -E ∗ ̃ glyph[lscript] ( H ) .

We refer to E glyph[lscript] k ( h ) -E ∗ glyph[lscript] k ( H ) as the target estimation error and E ̃ glyph[lscript] ( h ) -E ∗ ̃ glyph[lscript] ( H ) as the surrogate estimation error. These bounds imply Bayes-consistency when H = H all , by taking the limit.

A key quantity appearing in H -consistency bounds is the minimizability gap , which measures the difference between the best-in-class generalization error and the expectation of the best-inclass conditional error, defined for a given hypothesis set H and a loss function glyph[lscript] by: M glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E x [ C ∗ glyph[lscript] ( H , x )] . As shown by Mao, Mohri, and Zhong [2023f], the minimizability gap is non-negative and is upper bounded by the approximation error A glyph[lscript] ( H ) = E ∗ glyph[lscript] ( H ) -E ∗ glyph[lscript] ( H all ) : 0 ≤ M glyph[lscript] ( H ) ≤ A glyph[lscript] ( H ) . When H = H all or more generally A glyph[lscript] ( H ) = 0 , the minimizability gap vanishes. However, in general, it is non-zero and provides a finer measure than the approximation error. Thus, H -consistency bounds provide a stronger guarantee than the excess error bounds.

4.2 Theoretical guarantees for topk surrogate losses

We study the surrogate loss families of comp-sum losses and constrained losses in multi-class classification, which have been shown in the past to benefit from H -consistency bounds with respect to the zero-one classification loss, that is glyph[lscript] k with k = 1 [Awasthi et al., 2022b, Mao et al., 2023f] (see also [Zheng et al., 2023, Mao et al., 2023b]). We extend these results to topk classification and prove H -consistency bounds for these loss functions with respect to glyph[lscript] k for any 1 ≤ k ≤ n .

Another commonly used family of surrogate losses in multi-class classification is the max losses, which are defined through a convex function, such as the hinge loss function applied to the margin [Crammer and Singer, 2001, Awasthi et al., 2022b]. However, as shown in [Awasthi et al., 2022b], no non-trivial H -consistency guarantee holds for max losses with respect to glyph[lscript] k , even when k = 1 .

We first characterize the best-in-class conditional error and the conditional regret of topk loss, which will be used in the analysis of H -consistency bounds. We denote by S [ k ] = { X ⊂ S ∣ ∣ X ∣ = k } the set of all k -subsets of a set S . We will study any hypothesis set that is regular.

Definition 4.3. Let A ( n, k ) be the set of ordered k -tuples with distinct elements in [ n ] . We say that a hypothesis set H is regular for topk classification , if the topk predictions generated by the hypothesis set cover all possible outcomes: ∀ x ∈ X , {( h 1 ( x ) , . . . , h k ( x ))∶ h ∈ H } = A ( n, k ) .

Common hypothesis sets such as that of linear models or neural networks, or the family of all measurable functions, are all regular for topk classification.

Lemma 4.4. Assume that H is regular. Then, for any h ∈ H and x ∈ X , the best-in-class conditional error and the conditional regret of the topk loss can be expressed as follows:

The proof is included in Appendix A. For k = 1 , the result coincides with the known identities for standard multi-class classification with regular hypothesis sets [Awasthi et al., 2022b, Lemma 3].

As with [Awasthi et al., 2022b, Mao et al., 2023f], in the following sections, we will consider hypothesis sets that are symmetric and complete. This includes the class of linear models and neural networks typically used in practice, as well as the family of all measurable functions. We say that a hypothesis set H is symmetric if it is independent of the ordering of labels. That is, for all y ∈ Y , the scoring function x ↦ h ( x, y ) belongs to some real-valued family of functions F . We say that a hypothesis set is complete if, for all ( x, y ) ∈ X × Y , the set of scores h ( x, y ) can span over the real numbers, that is, { h ( x, y )∶ h ∈ H } = R . Note that any symmetric and complete hypothesis set is regular for topk classification.

Next, we analyze the broad family of comp-sum losses, which includes the commonly used logistic loss (or cross-entropy loss used with the softmax activation) as a special case.

Comp-sum losses are defined as the composition of a function Φ with the sum exponential losses, as in [Mao et al., 2023f]. For any h ∈ H and ( x, y ) ∈ X × Y , they are expressed as

where Φ ∶ R + → R + is non-decreasing. When Φ is chosen as the function t ↦ log ( 1 + t ) , t ↦ t , t ↦ 1 -1 1 + t and t ↦ 1 q ( 1 -( 1 1 + t ) q ) , q ∈ ( 0 , 1 ) , ̃ glyph[lscript] comp ( h, x, y ) coincides with the most commonly used (multinomial) logistic loss, defined as ̃ glyph[lscript] log ( h, x, y ) = log (∑ y ′ ∈ Y e h ( x,y ′ )-h ( x,y ) ) [Verhulst, 1838, 1845, Berkson, 1944, 1951], the sum-exponential loss ̃ glyph[lscript] exp ( h, x, y ) = ∑ y ′ ≠ y e h ( x,y ′ )-h ( x,y ) [Weston and Watkins, 1998, Awasthi et al., 2022b] which is widely used in multi-class boosting [Saberian and Vasconcelos, 2011, Mukherjee and Schapire, 2013, Kuznetsov et al., 2014], the mean absolute error loss ̃ glyph[lscript] mae ( h, x, y ) = 1 - [∑ y ′ ∈ Y e h ( x,y ′ )-h ( x,y ) ] -1 known to be robust to label noise for training neural networks [Ghosh et al., 2017], and the generalized cross-entropy loss ̃ glyph[lscript] gce ( h, x, y ) = 1 q [ 1 -[∑ y ′ ∈ Y e h ( x,y ′ )-h ( x,y ) ] -q ] , q ∈ ( 0 , 1 ) , a generalization of the logistic loss and mean absolute error loss for learning deep neural networks with noisy labels [Zhang and Sabuncu, 2018], respectively. We specifically study these loss functions and show that they benefit from H -consistency bounds with respect to the topk loss.

Theorem 4.5. Assume that H is symmetric and complete. Then, for any 1 ≤ k ≤ n , the following H -consistency bound holds for the comp-sum loss:

In the special case where A ̃ glyph[lscript] comp ( H ) = 0 , for any 1 ≤ k ≤ n , the following upper bound holds:

The proof is included in Appendix B. The second part follows from the fact that when A ̃ glyph[lscript] comp ( H ) = 0 , the minimizability gap M ̃ glyph[lscript] comp ( H ) vanishes. By taking the limit on both sides, Theorem 4.5 implies the H -consistency and Bayes-consistency of comp-sum losses with respect to the topk loss. It further shows that, when the estimation error of ̃ glyph[lscript] comp is reduced to glyph[epsilon1] > 0 , then the estimation error of glyph[lscript] k is upper bounded by kψ -1 ( glyph[epsilon1] ) , which, for a sufficiently small glyph[epsilon1] , is approximately k √ 2 glyph[epsilon1] for ̃ glyph[lscript] log and ̃ glyph[lscript] exp ; knglyph[epsilon1] for ̃ glyph[lscript] mae ; and k √ 2 n q glyph[epsilon1] for ̃ glyph[lscript] gce . Note that different from the other loss functions, the bound for the mean absolute error loss is only linear. The downside of this more favorable linear rate is the dependency on the number of classes and the fact that the mean absolute error loss is harder to optimize [Zhang and Sabuncu, 2018]. The bound for the generalized cross-entropy loss depends on both the number of classes n and the parameter q .

In the proof, we used the fact that the conditional regret of the topk loss is the sum of k differences between two probabilities. We then upper bounded each difference with the conditional regret of the comp-sum loss, using a hypothesis based on the two probabilities. The final bound is derived by summing these differences. In Appendix G, we detail the technical challenges and the novelty.

The key quantities in our H -consistency bounds are the minimizability gaps, which can be upper bounded by the approximation error, or more refined terms, depending on the magnitude of the parameter space, as discussed by Mao et al. [2023f]. As pointed out by these authors, these quantities, along with the functional form, can help compare different comp-sum loss functions. In Appendix C, we further discuss the important role of minimizability gaps under the realizability assumption, and the connection with some negative results of Yang and Koyejo [2020].

Constrained losses are defined as a summation of a function Φ applied to the scores, subject to a constraint, as shown in [Lee et al., 2004]. For any h ∈ H and ( x, y ) ∈ X × Y , they are expressed as

where Φ ∶ R → R + is non-increasing. In Appendix E, we study this family of loss functions and show that several benefit from H -consistency bounds with respect to the topk loss. In Appendix H, we provide generalization bounds for the topk loss in terms of finite samples (Theorems H.1 and H.2).

4.3 Theoretical guarantees for cardinality-aware surrogate losses

The strong theoretical results of the previous sections establish the effectiveness of comp-sum and constrained losses as surrogate losses for the target topk loss for common hypothesis sets used in practice. Building on this foundation, we expand our analysis to their cost-sensitive variants in the study of cardinality-aware set prediction in Section 2. We derive H -consistency bounds for these loss functions, thereby also establishing their Bayes-consistency. To do so, we characterize the conditional regret of the target cardinality-aware loss function in Lemma I.1, which can be found in Appendix I. For this analysis, we will assume, without loss of generality, that the cost c ( x, k, y ) takes values in [ 0 , 1 ] for any ( x, k, y ) ∈ X × K × Y , which can be achieved by normalizing the cost function.

We will use ̃ glyph[lscript] c -log , ̃ glyph[lscript] c -exp , ̃ glyph[lscript] c -gce and ̃ glyph[lscript] c -mae to denote the corresponding cost-sensitive counterparts for ̃ glyph[lscript] log , ̃ glyph[lscript] exp , ̃ glyph[lscript] gce and ̃ glyph[lscript] mae , respectively. Next, we show that these cost-sensitive surrogate loss functions benefit from H -consistency bounds with respect to the target loss glyph[lscript] given in (1).

Theorem 4.6. Assume that R is symmetric and complete. Then, the following bound holds for the cost-sensitive comp-sum loss: for all r ∈ R and for any distribution,

When R = R all , the following holds: E glyph[lscript] ( r ) -E ∗ glyph[lscript] ( R all ) ≤ γ ( E ̃ glyph[lscript] c -comp ( r ) -E ∗ ̃ glyph[lscript] c -comp ( R all )) , where γ ( t ) = 2 √ t when ̃ glyph[lscript] c -comp is either ̃ glyph[lscript] c -log or ̃ glyph[lscript] c -exp ; γ ( t ) = 2 √ ∣ K ∣ q t when ̃ glyph[lscript] c -comp is ̃ glyph[lscript] c -gce ; and γ ( t ) = ∣ K ∣ t when ̃ glyph[lscript] c -comp is ̃ glyph[lscript] c -mae .

The proof is included in Appendix I.1. The second part follows from the fact that when R = R all , all the minimizability gaps vanish. In particular, Theorem 4.6 implies the Bayes-consistency of cost-sensitive comp-sum losses. The bounds for cost-sensitive generalized cross-entropy and mean absolute error loss depend on the number of set predictors, making them less favorable when ∣ K ∣ is large. As pointed out earlier, while the cost-sensitive mean absolute error loss admits a linear rate, it is difficult to optimize even in the standard classification, as reported by Zhang and Sabuncu [2018].

In the proof, we represented the comp-sum loss as a function of the softmax and introduced a softmax-dependent function S µ to upper bound the conditional regret of the target cardinality-aware loss function by that of the cost-sensitive comp-sum loss. This technique is novel and differs from the approach used in the standard scenario (Section 4.2).

We will use ̃ glyph[lscript] cstnd c -exp , ̃ glyph[lscript] c -sq -hinge , ̃ glyph[lscript] c -hinge and ̃ glyph[lscript] c -ρ to denote the corresponding cost-sensitive counterparts for ̃ glyph[lscript] cstnd exp , ̃ glyph[lscript] sq -hinge , ̃ glyph[lscript] hinge and ̃ glyph[lscript] ρ , respectively. Next, we show that these cost-sensitive surrogate losses benefit from H -consistency bounds with respect to the target loss glyph[lscript] given in (1).

Theorem 4.7. Assume that R is symmetric and complete. Then, the following bound holds for the cost-sensitive constrained loss: for all r ∈ R and for any distribution,

When R = R all , the following holds: E glyph[lscript] ( r ) -E ∗ glyph[lscript] ( R all ) ≤ γ ( E ̃ glyph[lscript] c -cstnd ( r ) -E ∗ ̃ glyph[lscript] c -cstnd ( R all )) , where γ ( t ) = 2 √ t when ̃ glyph[lscript] c -cstnd is ̃ glyph[lscript] cstnd c -exp or ̃ glyph[lscript] c -sq -hinge ; γ ( t ) = t when ̃ glyph[lscript] c -cstnd is ̃ glyph[lscript] c -hinge or ̃ glyph[lscript] c -ρ .

The proof is included in Appendix I.2. The second part follows from the fact that when R = R all , all the minimizability gaps vanish. In particular, Theorem 4.7 implies the Bayes-consistency of cost-sensitive constrained losses. Note that while the constrained hinge loss and ρ -margin loss have a more favorable linear rate in the bound, their optimization may be more challenging compared to other smooth loss functions.

2

4

6

8

10

Cardinality

0.78

0.82

0.86

0.90

0.94

0.98

top-

k

classifier

card-aware

2

4

6

8

10

Cardinality

0.75

0.80

0.85

0.90

0.95

top-k classifier

card-aware

1

2

3

4

Cardinality

0.875

0.900

0.925

0.950

0.975

top-

k

classifier

card-aware

1

2

3

4

Cardinality

0.90

0.92

0.94

0.96

0.98

top-

k

classifier

card-aware

CIFAR-100

ImageNet

CIFAR-10

SVHN

1

2

3

4

Cardinality

0.78

0.82

0.86

0.90

0.94

0.98

card-aware, cost(

k

) = log(

k

)

card-aware, cost(

k

) =

k

1

2

3

4

Cardinality

0.75

0.80

0.85

0.90

0.95

card-aware, cost(

k

) = log(

k

)

card-aware, cost(

k

) =

k

1.0

1.5

2.0

2.5

Cardinality

0.875

0.900

0.925

0.950

0.975

card-aware, cost(

k

) = log(

k

)

card-aware, cost(

k

) =

k

1.0

1.5

2.0

2.5

3.0

Cardinality

0.90

0.92

0.94

0.96

0.98

card-aware, cost(

k

) = log(

k

)

card-aware, cost(

k

) =

k

CIFAR-100

ImageNet

CIFAR-10

SVHN

5 Experiments

Here, we report empirical results for our cardinality-aware algorithm and show that it consistently outperforms topk classifiers on benchmark datasets CIFAR-10, CIFAR-100 [Krizhevsky, 2009], SVHN [Netzer et al., 2011] and ImageNet [Deng et al., 2009].

We used the outputs of the second-to-last layer of ResNet [He et al., 2016] as features for the CIFAR10, CIFAR-100 and SVHN datasets. For the ImageNet dataset, we used the CLIP [Radford et al., 2021] model to extract features. We adopted a linear model, trained using multinomial logistic loss, for the classifier h on the extracted features from the datasets. We used a two-hidden-layer feedforward neural network with ReLU activation functions [Nair and Hinton, 2010] for the cardinality selector r . Both the classifier h and the cardinality selector r were trained using the Adam optimizer [Kingma and Ba, 2014], with a learning rate of 1 × 10 -3 , a batch size of 128 , and a weight decay of 1 × 10 -5 .

Figure 1 compares the accuracy versus cardinality curves of the cardinality-aware algorithm with that of topk classifiers induced by h for the various datasets. The accuracy of a topk classifier is measured by E ( x,y )∼ S [ 1 -glyph[lscript] k ( h, x, y )] , that is the fraction of the sample in which the topk predictions include the true label. It naturally grows as the cardinality k increases, as shown in Figure 1. The accuracy of the carnality-aware algorithms is measured by E ( x,y )∼ S [ 1 y ∈{ h 1 ( x ) ,..., hr ( x ) ( x )} ] , that is the fraction of the sample in which the predictions selected by the model r include the true label, and the corresponding cardinality is measured by E ( x,y )∼ S [ r ( x )] , that is the average size of the selected predictions. The cardinality selector r was trained by minimizing the cost-sensitive logistic loss ̃ glyph[lscript] c -log with the cost c ( x, k, y ) defined as glyph[lscript] k ( h, x, y ) + λ log ( k ) and normalized to [ 0 , 1 ] through division by its maximum value over X × K × Y . We allow for topk experts with k ∈ K = { 1 , 2 , 4 , 8 } and vary λ . Starting from high values of λ , as λ decreases in Figure 1, our cardinality-aware algorithm yields solutions with higher average cardinality and increased accuracy. This is because λ controls the trade-off between cardinality and accuracy. The plots end to the right at λ = 0 . 01 .

Figure 1 shows that the cardinality-aware algorithm is superior across the CIFAR-100, ImageNet, CIFAR-10 and SVHN datasets. For a given average cardinality, the cardinality-aware algorithm always achieves higher accuracy than a topk classifier. In other words, to achieve the same level of accuracy, the predictions made by the cardinality-aware algorithm can be significantly smaller in size compared to those made by the corresponding topk classifier. In particular, on the CIFAR-100, CIFAR-10 and SVHN datasets, the cardinality-aware algorithm achieves the same accuracy (98%) as the topk classifier while using roughly only half of the cardinality on average. As with the ImageNet dataset, it achieves the same accuracy (95%) as the topk classifier with only two-thirds of the cardinality. This illustrates the effectiveness of our cardinality-aware algorithm.

Figure 2 presents the comparison of cost (∣ g k ( x )∣)= k and cost (∣ g k ( x )∣)= log k in the same setting (for each dataset, the orange curve in Figure 2 coincides with the orange curve in Figure 1). The

CIFAR-100

CIFAR-10

1

2

4

8

k

0

1000

2000

3000

Count

0.05

0.1

1

2

4

8

k

0

2000

4000

6000

Count

0.05

0.1

True label: cat

True label: frog

True label: deer

True label: airplane

Hard images.

True label: cat

True label: frog

True label: deer

True label: airplane

Easy images.

comparison suggests that the choice between the linear and logarithmic cardinality costs has negligible impact on our algorithm's performance, highlighting its robustness in this regard. We also empirically demonstrate that our algorithm dynamically adjusts the cardinality of prediction sets based on input complexity, selecting larger sets for more challenging inputs to ensure high accuracy and smaller sets for simpler inputs to keep the cardinality low, as illustrated in Figure 3 and Figure 4. We present additional experimental results with different choices of set K in Figure 5 and Figure 6 in Appendix J. Our cardinality-aware algorithm consistently outperforms topk classifiers across all configurations.

6 Conclusion

We introduced a new cardinality-aware set prediction framework for which we proposed two families of surrogate losses with strong H -consistency guarantees: cost-sensitive comp-sum and constrained losses. This leads to principled and practical cardinality-aware algorithms for topk classification, which we showed empirically to be very effective. Additionally, we established a theoretical foundation for topk classification with fixed cardinality k by proving that several common surrogate loss functions, including comp-sum losses and constrained losses in standard classification, admit H -consistency bounds with respect to the topk loss. This provides a theoretical justification for the use of these loss functions in topk classification and opens new avenues for further research in this area.

References

Awasthi, N. Frank, A. Mao, M. Mohri, and Y. Zhong. Calibration and consistency of adversarial surrogate losses. In Advances in Neural Information Processing Systems , pages 9804-9815, 2021a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. A finer calibration analysis for adversarial robustness. arXiv preprint arXiv:2105.01550 , 2021b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. H -consistency bounds for surrogate loss minimizers. In International Conference on Machine Learning , pages 1117-1174, 2022a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Multi-class H -consistency bounds. In Advances in neural information processing systems , pages 782-795, 2022b.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for adversarial robustness. In International Conference on Artificial Intelligence and Statistics , pages 10077-10094, 2023a.

Awasthi, A. Mao, M. Mohri, and Y. Zhong. DC-programming for neural network optimizations. Journal of Global Optimization , 2023b.

L. Bartlett and M. H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research , 9(8), 2008.

L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006.

Berkson. Application of the logistic function to bio-assay. Journal of the American Statistical Association , 39:357--365, 1944.

Berkson. Why I prefer logits to probits. Biometrics , 7(4):327--339, 1951.

Berrada, A. Zisserman, and M. P. Kumar. Smooth loss functions for deep top-k classification. In International Conference on Learning Representations , 2018.

Crammer and Y. Singer. On the algorithmic implementation of multiclass kernel-based vector machines. Journal of machine learning research , 2(Dec):265-292, 2001.

Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and F.-F. Li. Imagenet: A large-scale hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition , pages 248-255. Ieee, 2009.

Denis and M. Hebiri. Confidence sets with expected sizes for multiclass classification. Journal of Machine Learning Research , 18(102):1-28, 2017.

Ghosh, H. Kumar, and P. S. Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI conference on artificial intelligence , 2017.

He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 770-778, 2016.

P. Kingma and J. Ba. Adam: A method for stochastic optimization. arXiv preprint arXiv:1412.6980 , 2014.

Krizhevsky. Learning multiple layers of features from tiny images. Technical report, Toronto University, 2009.

Kuznetsov, M. Mohri, and U. Syed. Multi-class deep boosting. In Advances in Neural Information Processing Systems , pages 2501-2509, 2014.

Lapin, M. Hein, and B. Schiele. Top-k multiclass SVM. In Advances in neural information processing systems , 2015.

Lapin, M. Hein, and B. Schiele. Loss functions for top-k error: Analysis and insights. In Proceedings of the IEEE conference on computer vision and pattern recognition , pages 1468-1477, 2016.

Lapin, M. Hein, and B. Schiele. Analysis and optimization of loss functions for multiclass, top-k, and multilabel classification. IEEE Transactions on Pattern Analysis & Machine Intelligence , 40 (07):1533-1554, 2018.

Lee, Y. Lin, and G. Wahba. Multicategory support vector machines: Theory and application to the classification of microarray data and satellite radiance data. Journal of the American Statistical Association , 99(465):67-81, 2004.

Long and R. Servedio. Consistency versus realizable H-consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013.

Mao, C. Mohri, M. Mohri, and Y. Zhong. Two-stage learning to defer with multiple experts. In Advances in neural information processing systems , 2023a.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds: Characterization and extensions. In Advances in Neural Information Processing Systems , 2023b.

Mao, M. Mohri, and Y. Zhong. H-consistency bounds for pairwise misranking loss surrogates. In International conference on Machine learning , 2023c.

Mao, M. Mohri, and Y. Zhong. Ranking with abstention. In ICML 2023 Workshop The Many Facets of Preference-Based Learning , 2023d.

Mao, M. Mohri, and Y. Zhong. Structured prediction with stronger consistency guarantees. In Advances in Neural Information Processing Systems , 2023e.

Mao, M. Mohri, and Y. Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , 2023f.

Mao, M. Mohri, and Y. Zhong. Principled approaches for learning to defer with multiple experts. In International Symposium on Artificial Intelligence and Mathematics , 2024a.

Mao, M. Mohri, and Y. Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. In International Conference on Algorithmic Learning Theory , 2024b.

Mao, M. Mohri, and Y. Zhong. Theoretically grounded loss functions and algorithms for scorebased multi-class abstention. In International Conference on Artificial Intelligence and Statistics , 2024c.

Mao, M. Mohri, and Y. Zhong. Enhanced H -consistency bounds. arXiv preprint arXiv:2407.13722 , 2024d.

Mao, M. Mohri, and Y. Zhong. H -consistency guarantees for regression. In International Conference on Machine Learning , pages 34712-34737, 2024e.

Mao, M. Mohri, and Y. Zhong. Multi-label learning with stronger consistency guarantees. In Advances in neural information processing systems , 2024f.

Mao, M. Mohri, and Y. Zhong. Realizable H -consistent and Bayes-consistent loss functions for learning to defer. In Advances in neural information processing systems , 2024g.

Mao, M. Mohri, and Y. Zhong. Regression with multi-expert deferral. In International Conference on Machine Learning , pages 34738-34759, 2024h.

Mao, M. Mohri, and Y. Zhong. A universal growth rate for learning with smooth surrogate losses. In Advances in neural information processing systems , 2024i.

Mohri, D. Andor, E. Choi, M. Collins, A. Mao, and Y. Zhong. Learning to reject with a fixed predictor: Application to decontextualization. In International Conference on Learning Representations , 2024.

Mohri, A. Rostamizadeh, and A. Talwalkar. Foundations of machine learning . MIT press, 2018.

Mukherjee and R. E. Schapire. A theory of multiclass boosting. Journal of Machine Learning Research , 2013.

Nair and G. E. Hinton. Rectified linear units improve restricted boltzmann machines. In Proceedings of the 27th international conference on machine learning (ICML-10) , pages 807-814, 2010.

Netzer, T. Wang, A. Coates, A. Bissacco, B. Wu, and A. Y. Ng. Reading digits in natural images with unsupervised feature learning. In Advances in Neural Information Processing Systems , 2011.

Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning , pages 8748-8763. PMLR, 2021.

J. Reddi, S. Kale, F. Yu, D. Holtmann-Rice, J. Chen, and S. Kumar. Stochastic negative mining for learning with large output spaces. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 1940-1949, 2019.

Saberian and N. Vasconcelos. Multiclass boosting: Theory and algorithms. Advances in neural information processing systems , 24, 2011.

Shafer and V. Vovk. A tutorial on conformal prediction. Journal of Machine Learning Research , 9 (3), 2008.

Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007.

Thilagar, R. Frongillo, J. J. Finocchiaro, and E. Goodwill. Consistent polyhedral surrogates for top-k classification and variants. In International Conference on Machine Learning , pages 21329-21359, 2022.

Usunier, D. Buffoni, and P. Gallinari. Ranking with ordered weighted pairwise classification. In International conference on machine learning , pages 1057-1064, 2009.

F. Verhulst. Notice sur la loi que la population suit dans son accroissement. Correspondance mathématique et physique , 10:113--121, 1838.

F. Verhulst. Recherches mathématiques sur la loi d'accroissement de la population. Nouveaux Mémoires de l'Académie Royale des Sciences et Belles-Lettres de Bruxelles , 18:1--42, 1845.

Weston and C. Watkins. Multi-class support vector machines. Technical report, Citeseer, 1998.

Yang and S. Koyejo. On the consistency of top-k surrogate losses. In International Conference on Machine Learning , pages 10727-10735, 2020.

Zhang. Statistical behavior and consistency of classification methods based on convex risk minimization. The Annals of Statistics , 32(1):56-85, 2004a.

Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5(Oct):1225-1251, 2004b.

Zhang and M. Sabuncu. Generalized cross entropy loss for training deep neural networks with noisy labels. In Advances in neural information processing systems , 2018.

Zheng, G. Wu, F. Bao, Y. Cao, C. Li, and J. Zhu. Revisiting discriminative vs. generative classifiers: Theory and implications. In International Conference on Machine Learning , 2023.

Contents of Appendix

A Proof of Lemma 4.4

Lemma 4.4. Assume that H is regular. Then, for any h ∈ H and x ∈ X , the best-in-class conditional error and the conditional regret of the topk loss can be expressed as follows:

Proof. By definition, for any h ∈ H and x ∈ X , the conditional error of topk loss can be written as

By definition of the labels p i ( x ) , which are the most likely topk labels, C glyph[lscript] k ( h, x ) is minimized for h i ( x ) = k min ( x ) , i ∈ [ k ] . Since H is regular, this choice is realizable for some h ∈ H . Thus, we have

Furthermore, the calibration gap can be expressed as

which completes the proof.

B Proofs of H -consistency bounds for comp-sum losses

Theorem 4.5. Assume that H is symmetric and complete. Then, for any 1 ≤ k ≤ n , the following H -consistency bound holds for the comp-sum loss:

In the special case where A ̃ glyph[lscript] comp ( H ) = 0 , for any 1 ≤ k ≤ n , the following upper bound holds:

where ψ ( t ) = 1 -t 2 log ( 1 -t )+ 1 + t 2 log ( 1 + t ) , t ∈ [ 0 , 1 ] when ̃ glyph[lscript] comp is ̃ glyph[lscript] log ; ψ ( t ) = 1 -√ 1 -t 2 , t ∈ [ 0 , 1 ] when ̃ glyph[lscript] comp is ̃ glyph[lscript] exp ; ψ ( t ) = t / n when ̃ glyph[lscript] comp is ̃ glyph[lscript] mae ; and ψ ( t ) = 1 qn q ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ [ ( 1 + t ) 1 1 -q +( 1 -t ) 1 1 -q 2 ] 1 -q -1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ , for all q ∈ ( 0 , 1 ) , t ∈ [ 0 , 1 ] when ̃ glyph[lscript] comp is ̃ glyph[lscript] gce .

Proof. Case I: ̃ glyph[lscript] comp = ̃ glyph[lscript] log . For logistic loss ̃ glyph[lscript] log , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Therefore, for any i ∈ [ k ] , the conditional regret of logistic loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = p ( x, h i ( x )) e h ( x, h i ( x )) -p ( x, p i ( x )) e h ( x, p i ( x )) p ( x, h i ( x ))+ p ( x, p i ( x )) . Plug in µ ∗ , we obtain

where ψ ( t ) = 1 -t 2 log ( 1 -t ) + 1 + t 2 log ( 1 + t ) , t ∈ [ 0 , 1 ] . Therefore, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity of ψ -1 , taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when A ̃ glyph[lscript] log ( H ) = 0 , the minimizability gap M ̃ glyph[lscript] log ( H ) vanishes.

Case II: ̃ glyph[lscript] comp = ̃ glyph[lscript] exp . For sum exponential loss ̃ glyph[lscript] exp , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Therefore, for any i ∈ [ k ] , the conditional regret of sum exponential loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = exp [ h ( x, h i ( x ))] √ p ( x, h i ( x ))-exp [ h ( x, p i ( x ))] √ p ( x, p i ( x )) √ p ( x, h i ( x ))+ √ p ( x, p i ( x )) . Plug in µ ∗ , we obtain

let S i = p ( x, p i ( x )) + p ( x, h i ( x )) and ∆ i = p ( x, p i ( x )) -p ( x, h i ( x )) , we have

(minimum is achieved when S i = 1 )

where ψ ( t ) = 1 -√ 1 -t 2 , t ∈ [ 0 , 1 ] . Therefore, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity of ψ -1 , taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when A ̃ glyph[lscript] exp ( H ) = 0 , the minimizability gap M ̃ glyph[lscript] exp ( H ) vanishes.

Case III: ̃ glyph[lscript] comp = ̃ glyph[lscript] mae . For mean absolute error loss ̃ glyph[lscript] mae , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Therefore, for any i ∈ [ k ] , the conditional regret of mean absolute error loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = -exp [ h ( x, p i ( x )] . Plug in µ ∗ , we obtain

Therefore, the conditional regret of the topk loss can be upper bounded as follows:

Take expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when A ̃ glyph[lscript] mae ( H ) = 0 , the minimizability gap M ̃ glyph[lscript] mae ( H ) vanishes.

Case IV: ̃ glyph[lscript] comp = ̃ glyph[lscript] gce . For generalized cross-entropy loss ̃ glyph[lscript] gce , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Therefore, for any i ∈ [ k ] , the conditional regret of generalized cross-entropy loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = exp [ h ( x, h i ( x ))] p ( x, p i ( x )) 1 q -1 -exp [ h ( x, p i ( x ))] p ( x, h i ( x )) 1 q -1 p ( x, h i ( x )) 1 q -1 + p ( x, p i ( x )) 1 q -1 . Plug in µ ∗ , we obtain

let S i = p ( x, p i ( x )) + p ( x, h i ( x )) and ∆ i = p ( x, p i ( x )) -p ( x, h i ( x )) , we have

(minimum is achieved when S i = 1 )

)))

=

ψ

(

p

(

x,

p

i

(

x

)) -

p

(

x,

h

i

(

x

,

where ψ ( t ) = 1 qn q ⎡ ⎢ ⎢ ⎢ ⎢ ⎣ [ ( 1 + t ) 1 1 -q +( 1 -t ) 1 1 -q 2 ] 1 -q -1 ⎤ ⎥ ⎥ ⎥ ⎥ ⎦ , t ∈ [ 0 , 1 ] . Therefore, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity of ψ -1 , taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when A ̃ glyph[lscript] gce ( H ) = 0 , the minimizability gap M ̃ glyph[lscript] gce ( H ) vanishes.

C Minimizability gaps and realizability

The key quantities in our H -consistency bounds are the minimizability gaps, which can be upper bounded by the approximation error, or more refined terms, depending on the magnitude of the parameter space, as discussed by Mao et al. [2023f]. As pointed out by these authors, these quantities, along with the functional form, can help compare different comp-sum loss functions.

Here, we further discuss the important role of minimizability gaps under the realizability assumption, and the connection with some negative results of Yang and Koyejo [2020].

Definition C.1 ( topk -H -realizability ) . A distribution D over X × Y is topk -H -realizable , if there exists a hypothesis h ∈ H such that P ( x,y )∼ D ( h ( x, y ) > h ( x, h k + 1 ( x ))) = 1 .

This extends the H -realizability definition from standard (top1 ) classification [Long and Servedio, 2013] to topk classification for any k ≥ 1 .

Definition C.2. We say that a hypothesis set H is closed under scaling , if it is a cone, that is for all h ∈ H and β ∈ R + , βh ∈ H .

Definition C.3. We say that a surrogate loss ̃ glyph[lscript] is realizable H -consistent with respect to glyph[lscript] k , if for all k ∈ [ 1 , n ] , and for any sequence of hypotheses { h n } n ∈ N ⊂ H and topk -H -realizable distribution, lim n →+∞ E ̃ glyph[lscript] ( h n ) -E ∗ ̃ glyph[lscript] ( H ) = 0 implies lim n →+∞ E glyph[lscript] k ( h n ) -E ∗ glyph[lscript] k ( H ) = 0 .

When H is closed under scaling, for k = 1 and all comp-sum loss functions glyph[lscript] = ̃ glyph[lscript] log , ̃ glyph[lscript] exp , ̃ glyph[lscript] gce and ̃ glyph[lscript] mae , it can be shown that E ∗ ̃ glyph[lscript] ( H ) = M ̃ glyph[lscript] ( H ) = 0 for any H -realizable distribution. For example, for glyph[lscript] = ̃ glyph[lscript] log , by using the Lebesgue dominated convergence theorem, we have

where h ∗ satisfies P ( x,y )∼ D ( h ∗ ( x, y ) > h ∗ ( x, h 2 ( x ))) = 1 Therefore, Theorem 4.5 implies that all these loss functions are realizable H -consistent with respect to glyph[lscript] 0 -1 ( glyph[lscript] k for k = 1 ) when H is closed under scaling.

Theorem C.4. Assume that H is closed under scaling. Then, ̃ glyph[lscript] log , ̃ glyph[lscript] exp , ̃ glyph[lscript] gce and ̃ glyph[lscript] mae are realizable H -consistent with respect to glyph[lscript] 0 -1 .

The formal proof is presented in Appendix D. However, for k > 1 , since in the realizability assumption, h ( x, y ) is only larger than h ( x, h k + 1 ( x )) and can be smaller than h ( x, h 1 ( x )) , there may exist an H -realizable distribution D such that M ̃ glyph[lscript] log ( H ) > 0 . This explains the inconsistency of the logistic loss on topk separable data with linear predictors, when k = 2 and n > 2 , as shown in [Yang and Koyejo, 2020]. More generally, the exact same example in [Yang and Koyejo, 2020, Proposition 5.1] can be used to show that all the comp-sum losses, ̃ glyph[lscript] log , ̃ glyph[lscript] exp , ̃ glyph[lscript] gce and ̃ glyph[lscript] mae are not realizable H -consistent with respect to glyph[lscript] k . Nevertheless, as previously shown, when the hypothesis set H adopted is sufficiently rich such that M ̃ glyph[lscript] ( H ) = 0 or even A ̃ glyph[lscript] ( H ) = 0 , they are guaranteed to be H -consistent. This is typically the case in practice when using deep neural networks.

D Proofs of realizable H -consistency for comp-sum losses

Theorem C.4. Assume that H is closed under scaling. Then, ̃ glyph[lscript] log , ̃ glyph[lscript] exp , ̃ glyph[lscript] gce and ̃ glyph[lscript] mae are realizable H -consistent with respect to glyph[lscript] 0 -1 .

Proof. Since the distribution is realizable, there exists a hypothesis h ∈ H such that

Therefore, for the logistic loss, by using the Lebesgue dominated convergence theorem,

For the sum exponential loss, by using the Lebesgue dominated convergence theorem,

For the generalized cross entropy loss, by using the Lebesgue dominated convergence theorem,

For the mean absolute error loss, by using the Lebesgue dominated convergence theorem,

Therefore, by Theorem 4.5, the proof is completed.

E H -Consistency bounds for constrained losses

Constrained losses are defined as a summation of a function Φ applied to the scores, subject to a constraint, as shown in [Lee et al., 2004, Awasthi et al., 2022b]. For any h ∈ H and ( x, y ) ∈ X × Y , they are expressed as

with the constraint ∑ y ∈ Y h ( x, y ) = 0 , where Φ ∶ R → R + is non-increasing. When Φ is chosen as the function t ↦ e -t , t ↦ max { 0 , 1 -t } 2 , t ↦ max { 0 , 1 -t } and t ↦ min { max { 0 , 1 -t / ρ } , 1 } , ρ > 0 , ̃ glyph[lscript] cstnd ( h, x, y ) are referred to as the constrained exponential loss ̃ glyph[lscript] cstnd exp ( h, x, y ) = ∑ y ′ ≠ y e h ( x,y ′ ) , the constrained squared hinge loss ̃ glyph[lscript] sq -hinge ( h, x, y ) = ∑ y ′ ≠ y max { 0 , 1 + h ( x, y ′ )} 2 , the constrained hinge loss ̃ glyph[lscript] hinge ( h, x, y ) = ∑ y ′ ≠ y max { 0 , 1 + h ( x, y ′ )} , and the constrained ρ -margin loss ̃ glyph[lscript] ρ ( h, x, y ) = ∑ y ′ ≠ y min { max { 0 , 1 + h ( x, y ′ )/ ρ } , 1 } , respectively [Awasthi et al., 2022b]. We now study these loss functions and show that they benefit from H -consistency bounds with respect to the topk loss.

Theorem E.1. Assume that H is symmetric and complete. Then, for any 1 ≤ k ≤ n , the following H -consistency bound holds for the constrained loss:

In the special case where A ̃ glyph[lscript] cstnd ( H ) = 0 , for any 1 ≤ k ≤ n , the following bound holds:

where γ ( t ) = 2 √ t when ̃ glyph[lscript] cstnd is either ̃ glyph[lscript] cstnd exp or ̃ glyph[lscript] sq -hinge ; γ ( t ) = t when ̃ glyph[lscript] cstnd is either ̃ glyph[lscript] hinge or ̃ glyph[lscript] ρ .

The proof is included in Appendix F. The second part follows from the fact that when the hypothesis set H is sufficiently rich such that A ̃ glyph[lscript] cstnd ( H ) = 0 , we have M ̃ glyph[lscript] cstnd ( H ) = 0 . Therefore, the constrained loss is H -consistent and Bayes-consistent with respect to glyph[lscript] k . If the surrogate estimation error E ̃ glyph[lscript] cstnd ( h )-E ∗ ̃ glyph[lscript] cstnd ( H ) is glyph[epsilon1] , then, the target estimation error satisfies E glyph[lscript] k ( h )-E ∗ glyph[lscript] k ( H ) ≤ kγ ( glyph[epsilon1] ) . Note that the constrained exponential loss and the constrained squared hinge loss both admit a square root H -consistency bound while the bounds for the constrained hinge loss and ρ -margin loss are both linear.

F Proofs of H -consistency bounds for constrained losses

The conditional error for the constrained loss can be expressed as follows:

Theorem E.1. Assume that H is symmetric and complete. Then, for any 1 ≤ k ≤ n , the following H -consistency bound holds for the constrained loss:

In the special case where A ̃ glyph[lscript] cstnd ( H ) = 0 , for any 1 ≤ k ≤ n , the following bound holds:

where γ ( t ) = 2 √ t when ̃ glyph[lscript] cstnd is either ̃ glyph[lscript] cstnd exp or ̃ glyph[lscript] sq -hinge ; γ ( t ) = t when ̃ glyph[lscript] cstnd is either ̃ glyph[lscript] hinge or ̃ glyph[lscript] ρ .

Proof. Case I: ̃ glyph[lscript] cstnd = ̃ glyph[lscript] cstnd exp . For the constrained exponential loss ̃ glyph[lscript] cstnd exp , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Let q ( x, p i ( x )) = 1 -p ( x, p i ( x )) and q ( x, h i ( x )) = 1 -p ( x, h i ( x )) . Therefore, for any i ∈ [ k ] , the conditional regret of constrained exponential loss can be lower bounded as

Therefore, by Lemma 4.4, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when A ̃ glyph[lscript] cstnd exp ( H ) = 0 , we have M ̃ glyph[lscript] cstnd exp ( H ) = 0 .

Case II: ̃ glyph[lscript] cstnd = ̃ glyph[lscript] sq -hinge . For the constrained squared hinge loss ̃ glyph[lscript] sq -hinge , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Let q ( x, p i ( x )) = 1 -p ( x, p i ( x )) and q ( x, h i ( x )) = 1 -p ( x, h i ( x )) . Therefore, for any i ∈ [ k ] , the conditional regret of the constrained squared hinge loss can be lower bounded as

Therefore, by Lemma 4.4, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when the hypothesis set H is sufficiently rich such that A ̃ glyph[lscript] sq -hinge ( H ) = 0 , we have M ̃ glyph[lscript] sq -hinge ( H ) = 0 .

Case III: ̃ glyph[lscript] cstnd = ̃ glyph[lscript] hinge . For the constrained hinge loss ̃ glyph[lscript] hinge , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Let q ( x, p i ( x )) = 1 -p ( x, p i ( x )) and q ( x, h i ( x )) = 1 -p ( x, h i ( x )) . Therefore, for any i ∈ [ k ] , the conditional regret of the constrained hinge loss can be lower-bounded as

Therefore, by Lemma 4.4, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when the hypothesis set H is sufficiently rich such that A ̃ glyph[lscript] hinge ( H ) = 0 , we have M ̃ glyph[lscript] hinge ( H ) = 0 .

Case IV: ̃ glyph[lscript] cstnd = ̃ glyph[lscript] ρ . For the constrained ρ -margin loss ̃ glyph[lscript] ρ , the conditional regret can be written as

where for any i ∈ [ k ] ,

Note that such a choice of h µ,i leads to the following equality holds:

Let q ( x, p i ( x )) = 1 -p ( x, p i ( x )) and q ( x, h i ( x )) = 1 -p ( x, h i ( x )) . Therefore, for any i ∈ [ k ] , the conditional regret of the constrained ρ -margin loss can be lower-bounded as

Therefore, by Lemma 4.4, the conditional regret of the topk loss can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that when the hypothesis set H is sufficiently rich such that A ̃ glyph[lscript] ρ ( H ) = 0 , we have M ̃ glyph[lscript] ρ ( H ) = 0 .

G Technical challenges and novelty in Section 4.2

The technical challenges and novelty of proofs in Section 4.2 lie in the following three aspects:

(1) Conditional regret of the topk loss: This involves a comprehensive analysis of the conditional regret associated with the topk loss, which is significantly more complex than that of the zero-one loss in a standard setting. The conditional regret of the topk loss incorporates both the topk conditional probabilities p i ( x ) , for i = 1 , . . . , k , and the topk scores h i ( x ) , for i = 1 , . . . , k , as characterized in Lemma 4.4.

(2) Relating to the conditional regret of the surrogate loss: To establish H -consistency bounds, it is necessary to upper bound the conditional regret of the topk loss with that of the surrogate loss. This task is particularly challenging in the topk setting due to the intricate nature of the topk loss's conditional regret. A pivotal observation is that the conditional regret of the topk loss can be expressed as the sum of k terms ( p ( x, p i ( x )) -p ( x, h i ( x ))) for i = 1 , . . . , k . Each term ( p ( x, p i ( x )) -p ( x, h i ( x ))) exhibits structural similarities to the conditional regret of the zero-one loss, ( p ( x, p 1 ( x )) -p ( x, h 1 ( x ))) . Consequently, we introduce a series of auxiliary hypotheses h µ,i , each dependent on h i ( x ) and p i ( x ) for i ∈ [ k ] . This approach transforms the challenge of upper bounding the conditional regret of the topk loss into k subproblems, each focusing on upper bounding the term ( p ( x, p i ( x )) -p ( x, h i ( x ))) with the conditional regret of the surrogate loss.

(3) Upper bounding each term ( p ( x, p i ( x )) -p ( x, h i ( x ))) : Following the approach in prior work [Mao et al., 2023f] for top1 classification, we define h µ,i ( x, y ) as:

for the proof of comp-sum losses (Theorem 4.5). The subsequent proof is considered straightforward.

However, for the proof of constrained losses (Theorem E.1), we adopt a different hypothesis formulation for h µ,i ( x, y ) , leveraging the constraint that the scores sum to zero and the specific structure of constrained losses. The hypothesis is defined as:

The remainder of the proof then specifically addresses the peculiarities of constrained losses, which significantly diverges from the previous work.

In summary, aspects (1) and (2) are novel and represent significant advancements that have not been explored previously. For aspect (3), the proof for comp-sum loss closely follows the approach in [Mao et al., 2023f], which appears straightforward due to the innovative ideas presented in aspects (1) and (2). However, the proof for constrained losses significantly deviates from the previous work, particularly in terms of the new auxiliary hypothesis formulation and the specific constrained losses examined.

We would like to further emphasize that these results are significant and useful. They demonstrate that comp-sum losses, which include the cross-entropy loss commonly used in top1 classification, and constrained losses, are H -consistent in topk classification for any k . Notably, the cross-entropy loss is the only Bayes-consistent smooth surrogate loss for topk classification identified to date. Furthermore, the Bayes-consistency of loss functions within the constrained loss family is a novel exploration in the context of topk classification. These findings are pivotal as they highlight two broad families of smooth loss functions that are Bayes-consistent in topk classification. Additionally, they reveal that these families, including the cross-entropy loss, benefit from stronger, non-asymptotic and hypothesis set-specific guaranteesH -consistency bounds-in topk classification.

H Generalization bounds

Given a finite sample S = (( x 1 , y 1 ) , . . . , ( x m , y m )) drawn from D m , let ̂ h S be the minimizer of the empirical loss within H with respect to the topk surrogate loss ̃ glyph[lscript] : ̂ h S = argmin h ∈ H ̂ E ̃ glyph[lscript],S ( h ) = argmin h ∈ H 1 m ∑ m i = 1 ̃ glyph[lscript] ( h, x i , y i ) . Next, we will show that we can use H -consistency bounds for ̃ glyph[lscript] to derive generalization bounds for the topk loss by upper bounding the surrogate estimation error E ̃ glyph[lscript] ( ̂ h S ) -E ∗ ̃ glyph[lscript] ( H ) with the complexity (e.g. the Rademacher complexity) of the family of functions associated with ̃ glyph[lscript] and H : H ̃ glyph[lscript] = {( x, y ) ↦ ̃ glyph[lscript] ( h, x, y )∶ h ∈ H } .

Let R ̃ glyph[lscript] m ( H ) be the Rademacher complexity of H ̃ glyph[lscript] and B ̃ glyph[lscript] an upper bound of the surrogate loss ̃ glyph[lscript] . Then, we obtain the following generalization bounds for the topk loss.

Theorem H.1 ( Generalization bound with comp-sum losses ) . Assume that H is symmetric and complete. Then, for any 1 ≤ k ≤ n , the following topk generalization bound holds for ̂ h S : for any δ > 0 , with probability at least 1 -δ over the draw of an i.i.d sample S of size m :

Proof. By using the standard Rademacher complexity bounds [Mohri et al., 2018], for any δ > 0 , with probability at least 1 -δ , the following holds for all h ∈ H :

Fix glyph[epsilon1] > 0 . By the definition of the infimum, there exists h ∗ ∈ H such that E ̃ glyph[lscript] ( h ∗ ) ≤ E ∗ ̃ glyph[lscript] ( H ) + glyph[epsilon1] . By definition of ̂ h S , we have

Since the inequality holds for all glyph[epsilon1] > 0 , it implies:

Plugging in this inequality in the bounds of Theorem 4.5 completes the proof.

Theorem H.2 ( Generalization bound with constrained losses ) . Assume that H is symmetric and complete. Then, for any 1 ≤ k ≤ n , the following topk generalization bound holds for ̂ h S : for any δ > 0 , with probability at least 1 -δ over the draw of an i.i.d sample S of size m :

where γ ( t ) = 2 √ t when ̃ glyph[lscript] is either ̃ glyph[lscript] cstnd exp or ̃ glyph[lscript] sq -hinge ; γ ( t ) = t when ̃ glyph[lscript] is either ̃ glyph[lscript] hinge or ̃ glyph[lscript] ρ .

Proof. By using the standard Rademacher complexity bounds [Mohri et al., 2018], for any δ > 0 , with probability at least 1 -δ , the following holds for all h ∈ H :

Fix glyph[epsilon1] > 0 . By the definition of the infimum, there exists h ∗ ∈ H such that E ̃ glyph[lscript] ( h ∗ ) ≤ E ∗ ̃ glyph[lscript] ( H ) + glyph[epsilon1] . By definition of ̂ h S , we have

Since the inequality holds for all glyph[epsilon1] > 0 , it implies:

Plugging in this inequality in the bounds of Theorem E.1 completes the proof.

To the best of our knowledge, Theorems H.1 and H.2 provide the first finite-sample guarantees for the estimation error of the minimizer of comp-sum losses and constrained losses, with respect to the topk loss, for any 1 ≤ k ≤ n . The proofs use our H -consistency bounds with respect to the topk loss, as well as standard Rademacher complexity guarantees.

I Proofs of H -consistency bounds for cost-sensitive losses

We first characterize the best-in class conditional error and the conditional regret of the target cardinality aware loss function (2), which will be used in the analysis of H -consistency bounds.

Lemma I.1. Assume that R is symmetric and complete. Then, for any r ∈ K and x ∈ X , the best-in class conditional error and the conditional regret of the target cardinality aware loss function can be expressed as follows:

Proof. By definition, for any r ∈ R and x ∈ X , the conditional error of the target cardinality aware loss function can be written as

Since R is symmetric and complete, we have

Furthermore, the calibration gap can be expressed as

which completes the proof.

I.1 Proof of Theorem 4.6

For convenience, we let c ( x, k, y ) = 1 -c ( x, k, y ) , q ( x, k ) = ∑ y ∈ Y p ( x, y ) c ( x, k, y ) ∈ [ 0 , 1 ] and S ( x, k ) = e r ( x,k ) ∑ k ′ ∈ K e r ( x,k ′ ) . We also let k min ( x ) = argmin k ∈ K ( 1 -q ( x, k )) = argmin k ∈ K ∑ y ∈ Y p ( x, y ) c ( x, k, y ) .

Theorem 4.6. Assume that R is symmetric and complete. Then, the following bound holds for the cost-sensitive comp-sum loss: for all r ∈ R and for any distribution,

When R = R all , the following holds: E glyph[lscript] ( r ) -E ∗ glyph[lscript] ( R all ) ≤ γ ( E ̃ glyph[lscript] c -comp ( r ) -E ∗ ̃ glyph[lscript] c -comp ( R all )) , where γ ( t ) = 2 √ t when ̃ glyph[lscript] c -comp is either ̃ glyph[lscript] c -log or ̃ glyph[lscript] c -exp ; γ ( t ) = 2 √ ∣ K ∣ q t when ̃ glyph[lscript] c -comp is ̃ glyph[lscript] c -gce ; and γ ( t ) = ∣ K ∣ t when ̃ glyph[lscript] c -comp is ̃ glyph[lscript] c -mae .

Proof. Case I: ̃ glyph[lscript] c -comp = ̃ glyph[lscript] c -log . For the cost-sensitive logistic loss ̃ glyph[lscript] c -log , the conditional error can be written as

The conditional regret can be written as

where for any x ∈ X and k ∈ K , S µ ( x, k ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ S ( x, y ) , y ∉ { k min ( x ) , r ( x )} S ( x, k min ( x )) + µ y = r ( x ) S ( x, r ( x )) -µ y = k min ( x ) . Note that such a choice of S µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive logistic loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = q ( x, r ( x )) S ( x, r ( x ))-q ( x,k min ( x )) S ( x,k min ( x )) q ( x,k min ( x ))+ q ( x, r ( x )) . Plug in µ ∗ , we obtain

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -log ( R all ) = 0 .

Case II: ̃ glyph[lscript] c -comp = ̃ glyph[lscript] c -exp . For the cost-sensitive sum exponential loss ̃ glyph[lscript] c -exp , the conditional error can be written as

The conditional regret can be written as

where for any x ∈ X and k ∈ K , S µ ( x, k ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ S ( x, y ) , y ∉ { k min ( x ) , r ( x )} S ( x, k min ( x )) + µ y = r ( x ) S ( x, r ( x )) -µ y = k min ( x ) . Note that such a choice of S µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive sum exponential loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = √ q ( x, r ( x )) S ( x, r ( x ))- √ q ( x,k min ( x )) S ( x,k min ( x )) √ q ( x,k min ( x ))+ √ q ( x, r ( x )) . Plug in µ ∗ , we obtain

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -exp ( R all ) = 0 .

Case III: ̃ glyph[lscript] c -comp = ̃ glyph[lscript] c -gce . For the cost-sensitive generalized cross-entropy loss ̃ glyph[lscript] c -gce , the conditional error can be written as

The conditional regret can be written as

where for any x ∈ X and k ∈ K , S µ ( x, k ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ S ( x, y ) , y ∉ { k min ( x ) , r ( x )} S ( x, k min ( x )) + µ y = r ( x ) S ( x, r ( x )) -µ y = k min ( x ) . Note that such a choice of S µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive generalized cross-entropy loss can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = q ( x, r ( x )) 1 1 -q S ( x, r ( x ))-q ( x,k min ( x )) 1 1 -q S ( x,k min ( x )) q ( x,k min ( x )) 1 1 -q + q ( x, r ( x )) 1 1 -q . Plug in µ ∗ , we obtain

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -gce ( R all ) = 0 .

Case IV: ̃ glyph[lscript] c -comp = ̃ glyph[lscript] c -mae . For the cost-sensitive mean absolute error loss ̃ glyph[lscript] c -mae , the conditional error can be written as

The conditional regret can be written as

where for any x ∈ X and k ∈ K , S µ ( x, k ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ S ( x, y ) , y ∉ { k min ( x ) , r ( x )} S ( x, k min ( x )) + µ y = r ( x ) S ( x, r ( x )) -µ y = k min ( x ) . Note that such a choice of S µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive mean absolute error can be lower bounded as

By the concavity of the function, differentiate with respect to µ , we obtain that the supremum is achieved by µ ∗ = -S ( x, k min ( x )) . Plug in µ ∗ , we obtain

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -mae ( R all ) = 0

I.2 Proof of Theorem 4.7

The conditional error for the cost-sensitive constrained loss can be expressed as follows:

where ̃ q ( x, k ) = ∑ y ∈ Y p ( x, y ) c ( x, k, y ) ∈ [ 0 , 1 ] . Let k min ( x ) = argmin k ∈ K ̃ q ( x, k ) . We denote by Φ exp ∶ t ↦ e -t the exponential loss function, Φ sq -hinge ∶ t ↦ max { 0 , 1 -t } 2 the squared hinge loss function, Φ hinge ∶ t ↦ max { 0 , 1 -t } the hinge loss function, and Φ ρ ∶ t ↦ min { max { 0 , 1 -t / ρ } , 1 } , ρ > 0 the ρ -margin loss function.

Theorem 4.7. Assume that R is symmetric and complete. Then, the following bound holds for the cost-sensitive constrained loss: for all r ∈ R and for any distribution,

When R = R all , the following holds: E glyph[lscript] ( r ) -E ∗ glyph[lscript] ( R all ) ≤ γ ( E ̃ glyph[lscript] c -cstnd ( r ) -E ∗ ̃ glyph[lscript] c -cstnd ( R all )) , where γ ( t ) = 2 √ t when ̃ glyph[lscript] c -cstnd is ̃ glyph[lscript] cstnd c -exp or ̃ glyph[lscript] c -sq -hinge ; γ ( t ) = t when ̃ glyph[lscript] c -cstnd is ̃ glyph[lscript] c -hinge or ̃ glyph[lscript] c -ρ .

Proof. Case I: glyph[lscript] = ̃ glyph[lscript] cstnd c -exp . For the cost-sensitive constrained exponential loss ̃ glyph[lscript] cstnd c -exp , the conditional regret can be written as

where for any k ∈ K , r µ ( x, k ) = ⎧ ⎪ ⎪ ⎪ ⎪ ⎨ ⎪ ⎪ ⎪ ⎪ ⎩ r ( x, y ) , y ∉ { k min ( x ) , r ( x )} r ( x, k min ( x )) + µ y = r ( x ) r ( x, r ( x )) -µ y = k min ( x ) . Note that such a choice of r µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive constrained exponential loss can be lower bounded as

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] cstnd c -exp ( R all ) = 0 .

Case II: glyph[lscript] = ̃ glyph[lscript] c -sq -hinge . For the cost-sensitive constrained squared hinge loss ̃ glyph[lscript] c -sq -hinge , the conditional regret can be written as

where for any k ∈ K ,

Note that such a choice of r µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive constrained squared hinge loss can be lower bounded as to optimize)

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -sq -hinge ( R all ) = 0 .

Case III: glyph[lscript] = ̃ glyph[lscript] c -hinge . For the cost-sensitive constrained hinge loss ̃ glyph[lscript] c -hinge , the conditional regret can be written as

where for any k ∈ K ,

Note that such a choice of r µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive constrained hinge loss can be lower bounded as to optimize)

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -hinge ( R all ) = 0 .

Case IV: glyph[lscript] = ̃ glyph[lscript] c -ρ . For the cost-sensitive constrained ρ -margin loss ̃ glyph[lscript] c -ρ , the conditional regret can be written as

where for any k ∈ K ,

Note that such a choice of r µ leads to the following equality holds:

Therefore, the conditional regret of cost-sensitive constrained ρ -margin loss can be lower bounded as

Therefore, by Lemma I.1, the conditional regret of the target cardinality aware loss function can be upper bounded as follows:

By the concavity, taking expectations on both sides of the preceding equation, we obtain

The second part follows from the fact that M ̃ glyph[lscript] c -ρ ( R all ) = 0 .

2

4

6

8

10

Cardinality

0.78

0.82

0.86

0.90

0.94

0.98

top-

k

classifier

cardinality-aware,

=0.05

cardinality-aware,

=0.1

cardinality-aware,

=0.01

2

4

6

8

10

12

Cardinality

0.75

0.80

0.85

0.90

0.95

top-k classifier

cardinality-aware,

=0.05

cardinality-aware,

=0.1

cardinality-aware,

=0.01

CIFAR-100

ImageNet

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Cardinality

0.88

0.90

0.92

0.94

0.96

0.98

top-

k

classifier

cardinality-aware,

=0.05

cardinality-aware,

=0.1

cardinality-aware,

=0.01

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Cardinality

0.90

0.92

0.94

0.96

0.98

top-

k

classifier

cardinality-aware,

=0.05

cardinality-aware,

=0.1

cardinality-aware,

=0.01

CIFAR-10

SVHN

2

4

6

8

10

Cardinality

0.78

0.82

0.86

0.90

0.94

0.98

top-

k

classifier

cardinality-aware,

(

k

) = log

k

cardinality-aware,

(

k

) =

k

2

4

6

8

10

Cardinality

0.75

0.80

0.85

0.90

0.95

top-k classifier

cardinality-aware,

(

k

) = log

k

cardinality-aware,

(

k

) =

k

CIFAR-100

ImageNet

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Cardinality

0.88

0.90

0.92

0.94

0.96

0.98

top-

k

classifier

cardinality-aware,

(

k

) = log

k

cardinality-aware,

(

k

) =

k

1.0

1.5

2.0

2.5

3.0

3.5

4.0

Cardinality

0.90

0.92

0.94

0.96

0.98

top-

k

classifier

cardinality-aware,

(

k

) = log

k

cardinality-aware,

(

k

) =

k

CIFAR-10

SVHN

J Additional experimental results: topk classifiers

Here, we report additional experimental results with different choices of set K and cost (∣ g k ( x )∣) on benchmark datasets CIFAR-10, CIFAR-100 [Krizhevsky, 2009], SVHN [Netzer et al., 2011], and ImageNet [Deng et al., 2009] and show that our cardinality-aware algorithm consistently outperforms topk classifiers across all configurations.

In Figure 5 and Figure 6, we began with a set K = { 1 } for the loss function and then progressively expanded it by adding choices of larger cardinality, each of which doubles the largest value currently in K . The largest set K for the CIFAR-100 and ImageNet datasets is { 1 , 2 , 4 , 8 , 16 , 32 , 64 } , whereas for the CIFAR-10 and SVHN datasets, it is { 1 , 2 , 4 , 8 } . As the set K expands, there is an increase in

3

4

5

6

7

Cardinality

0.70

0.75

0.80

0.85

0.90

0.95

conformal prediction

card-aware,

m

=500,000

card-aware,

m

=50,000

r

*

both the average cardinality and the accuracy. Figure 5 shows that the accuracy versus cardinality curve of the cardinality-aware algorithm is above that of topk classifiers for various values of λ . Figure 6 presents the comparison of cost (∣ g k ( x )∣) = k and cost (∣ g k ( x )∣) = log k for λ = 0 . 05 . These results demonstrate that different λ and different cost (∣ g k ( x )∣) basically lead to the same curve, which verifies the effectiveness and benefit of our algorithm.

K Additional experimental results: threshold-based classifiers

We first characterize the Bayes predictor r ∗ in this setting. We say that the scenario is deterministic if for all x ∈ X , there exists some true label y ∈ Y such that p ( x, y ) = 1 ; otherwise, we say that the scenario is stochastic. To simplify the discussion, we will assume that ∣ g k ( x )∣ is an increasing function of k , for any x .

Lemma K.1. Consider the deterministic scenario. Assume that λ cost (∣ g k ( x )∣) ≤ 1 for all k and x ∈ X . Then, the Bayes predictor r ∗ for the cardinality-aware loss function glyph[lscript] satisfies: r ∗ ( x ) = argmin k ∶ y ∈ g k ( x ) k , that is the smallest k such that the true label y is in g k ( x ) .

Proof. By the assumption, for k < r ∗ ( x ) , we can write c ( x, r ∗ ( x ) , y ) = λ cost (∣ g r ∗ ( x ) ( x )∣) ≤ 1 ≤ 1 y / ∈ g k ( x ) + λ cost (∣ g k ( x )∣) = c ( x, k, y ) . Furthermore, since ∣ g k ( x )∣ is an increasing function of k , we have c ( x, r ∗ ( x ) , y ) = λ cost (∣ g r ∗ ( x ) ( x )∣) ≤ λ cost (∣ g ′ k ( x )∣) = c ( x, k ′ , y ) for k ′ > r ∗ ( x ) .

Lemma K.2. Consider the stochastic scenario. The Bayes predictor r ∗ for the cardinality-aware loss function glyph[lscript] satisfies:

Proof. The conditional error can be written as follows:

Thus, the Bayes predictor can be characterized as

20

30

40

50

60

Cardinality

0.75

0.80

0.85

0.90

0.95

conformal prediction

cardinality-aware

r

*

300

400

500

600

700

800

Cardinality

0.75

0.80

0.85

0.90

0.95

conformal prediction

cardinality-aware

r

*

CIFAR-100

ImageNet

1.5

2.0

2.5

3.0

3.5

4.0

4.5

Cardinality

0.75

0.80

0.85

0.90

0.95

conformal prediction

cardinality-aware

r

*

2

3

4

Cardinality

0.75

0.80

0.85

0.90

0.95

conformal prediction

cardinality-aware

r

*

CIFAR-10

SVHN

It is clear that Lemma K.2 implies Lemma K.1 when there exists some true y ∈ Y such that p ( x, y ) = 1 and λ cost (∣ g k ( x )∣) ≤ 1 .

We first consider an artificial dataset containing 10 classes. Each class is modeled by a Gaussian distribution in a 100-dimensional space. As in Section 5, we plot the accuracy versus cardinality curve of the cardinality-aware algorithm by varying λ , where the set predictors used are threshold-based classifiers, and compare with that of conformal prediction. In Figure 7, we also indicate the point corresponding to r ∗ . The problem is close to being realizable, as we can train a predictor that performs almost as well as r ∗ on the test set. Thus, the minimizability gaps vanish, and our H -consistency bounds (Theorems 4.6 and 4.7) then suggest that with sufficient training data, we can get close to the optimal solution and therefore outperform conformal prediction. For some tasks, however, the problem is hard, and it appears that a very large training sample would be needed. Figure 7 demonstrates that on the artificial dataset, with training sample size m = 50 , 000 , the performance of our cardinality-aware algorithm is only slightly better than that of conformal prediction. If we increase the training sample size to m = 500 , 000 , then the curve of our algorithm becomes much closer to the optimal point and significantly outperforms conformal prediction.

Additionally, for a weaker scoring function, a smaller training sample suffices in many cases, and our cardinality-aware algorithm can outperform conformal prediction on real datasets as shown in Figure 8.

L Future work

While our framework of cardinality-aware set prediction is very general-applicable to any collection of set predictors (Section 2)-and leads to novel cardinality-aware algorithms (Section 3), benefits from theoretical guarantees with sufficient training data (Section 4), and demonstrates effectiveness and empirical advantages in topk classification (Section 5), the learning problem can be challenging for certain tasks, often requiring a very large training sample (as shown in Appendix K). This underscores the need for a more detailed investigation to enhance our algorithms in these scenarios.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: See lines 81-86 in Section 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: See Section 4, Appendix A, B, C, D, E, F, H, and I.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: See Section 3, Section 5, Appendix J and Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: See Section 5, Appendix J and Appendix K.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: See Section 5, Appendix J and Appendix K.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: See Figure 2, Figure 5, and Figure 6.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: For each model training, we use an Nvidia A100 GPU.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification: The authors have reviewed the NeurIPS Code of Ethics.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: Our work is theoretical in nature and we do not anticipate any immediate negative societal impact.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The paper poses no such risks.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: See Section 5, Appendix J and Appendix K.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA] .

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects.

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
comments: SUMMARY:
This paper proposes a method to handle cardinality-aware top-k classification. It designs an optimization problem where the cardinality of  set selectors is also considered. The authors then generalize it to two types of surrogate losses. Given certain assumptions on the hypothesis set H, the authors theoretically justify their method.

**I must say I am no expert on this domain, so the AC must take my comments wisely.**

STRENGTHS:
This paper tackles a very important problem. And achieves, according to my view, a very nice result. The authors manage to transfer the cardinality-aware term into surrogate terms, and theoretically demonstrate their sub-optimal properties. This paper is also very well written.

WEAKNESSES:
No error bar, very limited evaluation, very strong assumption on the hypothesis set, which is usually not easy to achieve in practice.

QUESTIONS:
In some cases, like those generative modeling questions, the cardinality of predictors may be evaluated by measure, meaning the true answer assembles a region of infinite points. Can your method apply to those scenarios where the cardinality is evaluated by measures?

LIMITATIONS:
As said in weakness.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-09

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper introduces a new loss function for top-k set prediction where k may vary as a function of the input, which the authors call cardinality-aware top-k classification. However, this loss is intractable to optimize in all but trivial cases, so the authors introduce surrogate loss functions for learning these cardinality-aware set predictors, and provide a theoretical analysis of these surrogate losses. The paper also explores the proposed algorithms empirically, on benchmark computer vision datasets.\n\nSTRENGTHS:\n- The contribution seems novel -- the authors identify a new type of prediction problem in which top-k sets of input-varying size can be predicted. \n- The introduction of two new surrogate loss functions for optimizing the proposed (but intractable) cardinality-aware top-k loss seems to be a good contribution. \n- Empirically, the authors show on benchmark datasets that by giving up control over specific cardinalities, their top-k predictors can outperform existing top-k predictors that prescribe specific cardinalities a priori.\n\nWEAKNESSES:\n- It isn't immediately clear why the cost-constrained surrogate loss is tractable to optimize, while the original (non-surrogate) loss is intractable. Please clarify this in the text. \n- (clarity of the abstract) The abstract should elaborate on H-consistency and the types of losses being introduced, or discuss these at a higher level. Without knowing what these are prior to reading the abstract, the significance of these is unclear. \n- The usage of the term \"accuracy\" in Figure 1 is strange, as the actual \"accuracy\" definitions vary slightly between the two methods being compared. Additionally, the expected cardinality is used, which makes the comparison strange. Furthermore, this comparison on four benchmark computer vision datasets makes up the bulk of the experimental results. The experimental results section could be additionally strengthened by analyzing other aspects of the predictors generated by the authors' framework, such as the distribution of cardinalities, whether or not harder examples indeed correspond to higher cardinality, the top-k accuracy scores of worst-case cardinalities and best-case cardinalities, and so on. \n- The value of Figure 2 is unclear. What is the significance of making the cost function linear?\n\nQUESTIONS:\n- Out of curiosity, have the authors considered simpler techniques for cardinality-varying top-k prediction, such as setting a total probability threshold on prediction probabilities, and returns the top-k that maximizes the probability, subject to the threshold? At least empirically, it seems like simpler techniques such as this one should be included as a baseline. \n- Is it in fact the case that for \"harder\" examples, the top-k cardinality of the predictions are larger in this framework? Have the authors explored this empirically? Examples of this would strengthen the argument for this new type of prediction.\n\nLIMITATIONS:\nThe authors discuss limitations.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-15"}

comments: SUMMARY:
The paper introduced a new cardinality-aware set prediction algorithm with cost-sensitive comp-sum and constrained surrogate losses. Additionally, the paper established theoretical guarantees for top-k classification with fixed cardinality k using the H-consistency bounds. Finally, experiments on linear classifiers show the superiority of their algorithms.

STRENGTHS:
1. The paper is presented clearly and is easy to follow.
2. The paper considers the set prediction, which is a novel setting in recent years.
3. The paper proposes new algorithms with rigorous guarantees and good performance on various datasets.

WEAKNESSES:
1. Since the theoretical results include the neural network class as a special case, the experiments should additionally consider training whole neural networks on the mentioned datasets (at least CIFAR), which is more important in practice.

QUESTIONS:
1. Can we estimate the $M$ in H-consistency bounds?
2. Authors claim the framework includes the standard conformal prediction, so do some relationships exist between the H-consistency bounds and coverage bounds?

LIMITATIONS:
The authors have adequately discussed the limitations.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-21