Title: Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel–Young Losses
Venue: NeurIPS 2025
Accepted: yes
Keywords: Surrogate regret bounds

Excess risk bounds

Fenchel-Young losses
Abstract: Surrogate regret bounds, also known as excess risk bounds,
bridge the gap between the convergence rates of surrogate and target losses. The regret transfer is lossless if the surrogate regret bound is linear.
While convex smooth surrogate losses are appealing in particular due to the efficient estimation and optimization, 
the existence of a trade-off between the loss smoothness and linear regret bound has been believed in the community.
Under this scenario, the better optimization and estimation properties of convex smooth surrogate losses may inevitably deteriorate after undergoing the regret transfer onto a target loss.
We overcome this dilemma for arbitrary discrete target losses
by constructing a convex smooth surrogate loss,
which entails a linear surrogate regret bound composed with a tailored
prediction link. 
The construction is based on Fenchel--Young losses generated by the *convolutional negentropy*,
which are equivalent to the infimal convolution of a generalized negentropy and the target Bayes risk.
Consequently, the infimal convolution enables us to derive a smooth loss while maintaining the surrogate regret bound linear.
We additionally benefit from the infimal convolution to have a consistent estimator of the underlying class probability.
Our results are overall a novel demonstration of how convex analysis penetrates into optimization and statistical efficiency in risk minimization.

Establishing Linear Surrogate Regret Bounds for Convex Smooth Losses via Convolutional Fenchel-Young Losses

Abstract

Surrogate regret bounds, also known as excess risk bounds, bridge the gap between the convergence rates of surrogate and target losses. The regret transfer is lossless if the surrogate regret bound is linear. While convex smooth surrogate losses are appealing in particular due to the efficient estimation and optimization, the existence of a trade-off between the loss smoothness and linear regret bound has been believed in the community. Under this scenario, the better optimization and estimation properties of convex smooth surrogate losses may inevitably deteriorate after undergoing the regret transfer onto a target loss. We overcome this dilemma for arbitrary discrete target losses by constructing a convex smooth surrogate loss, which entails a linear surrogate regret bound composed with a tailored prediction link. The construction is based on Fenchel-Young losses generated by the convolutional negentropy , which are equivalent to the infimal convolution of a generalized negentropy and the target Bayes risk. Consequently, the infimal convolution enables us to derive a smooth loss while maintaining the surrogate regret bound linear. We additionally benefit from the infimal convolution to have a consistent estimator of the underlying class probability. Our results are overall a novel demonstration of how convex analysis penetrates into optimization and statistical efficiency in risk minimization.

1 Introduction

The risk of a machine learning model is often measured by the expectation of a target loss ℓ that quantifies the error between the model prediction t and a natural label y . However, minimizing the target risk over a dataset is often computationally hard because a target prediction problem is usually discretely structured, including multiclass, multilabel, topk prediction problems. For this reason, a tractable surrogate risk induced by a surrogate loss L ( θ , y ) serves as an essential proxy with a score θ ∈ R d . The resulting surrogate optimization is no longer discretely constrained.

An ideal surrogate loss should be convex, smooth (or entailing a Lipschitz continuous gradient), and calibrated toward a given target prediction problem. Convexity and smoothness have been fundamental both in optimization and statistical estimation-indeed, classical optimization theory reveals that convex smooth functions can be optimized with first-order methods more efficiently than

3

non-smooth functions [67, 83]. In addition, several studies have demonstrated that the convexity and smoothness can further enhance fast rates in the risk estimation [87, 98]. Meanwhile, calibration is regarded as a minimal requirement on surrogate losses, ensuring that the surrogate risk minimization leads to the target risk minimization [12, 88]. 2 To establish a relationship between surrogate and target risks, surrogate regret bounds play a crucial role. Therein the regret (or the suboptimality) of a target loss is controlled by that of a surrogate loss through a non-decreasing rate function ψ : R ≥ 0 → R ≥ 0 . A surrogate regret bound can be informally written as follows: for any score vector θ ∈ R d and a class distribution η ,

where φ is a prediction link that converts a score vector θ into a target prediction φ ( θ ) . If the regret rate function is linear ψ ( r ) = O ( r ) , which is the best possible (data-independent) regret rate, 3 then the optimization and estimation errors of the surrogate loss are optimally translated to the target loss. For example, under the binary classification target loss, Bartlett et al. [12] demonstrated that the linear regret rate is possible with the hinge loss. Later, symmetric losses [24] and polyhedral losses [39] were shown to yield the linear regret rate. Unfortunately, these loss functions lack either convexity or smoothness, which deteriorates the optimization and estimation errors of the surrogate regret, even if they enjoy linear regret bounds. By contrast, a square-root regret rate ψ ( r ) = O ( √ r ) is common for convex smooth surrogate losses, such as the logistic, exponential, and squared losses [12, 69, 39]. These losses typically enjoy better optimization and estimation properties, yet suffer from larger target regrets due to the suboptimal regret rate. Therefore, it has been open to develop a convex smooth surrogate loss without sacrificing the linear regret rate.

Notwithstanding, the previous literature in this line has implied that such an ideal surrogate loss may be inconceivable. Mahdavi et al. [50] considered this question with an interpolated loss between the hinge (non-smooth) and logistic (smooth) losses and showed that we inevitably face the tradeoff between generalization and optimization unless strong distributional assumptions are imposed. Further, Frongillo and Waggoner [39] proved that locally smooth and strongly convex losses must suffer from a square-root regret rate at least. Ramaswamy et al. [76] blame convex smooth losses for redundantly modeling continuous class distributions, which is unnecessary if our goal is merely to solve discrete target problems.

In this paper, we demonstrate that this seemingly impossible trade-off can be overcome for arbitrary discrete target losses. Specifically, we build a convex smooth surrogate loss built upon the framework of Fenchel-Young losses [16]-a framework to generate a loss function from a generalized negentropy and its conjugate. In a nutshell, our main results are summarized as follows:

Theorem 1 (Informal version of Theorem 15) For any discrete target loss ℓ , there exist a convex smooth surrogate loss L (defined over a score θ ∈ R d ) and prediction link φ such that the surrogate regret bound (1) holds with some linear rate ψ ( r ) = O ( r ) .

This existence is proved constructively, which provides a systematic framework on the construction of convex smooth surrogate losses and their corresponding prediction links with linear surrogate regret bounds. Our high-level construction, which significantly leverages convex analysis and is detailed in Section 3.1, proceeds as follows. First, a user chooses a strongly convex negentropy with some regularity conditions, such as the Shannon negentropy. We encode the structure of a target loss ℓ into the chosen base negentropy by adding the negative Bayes risk of ℓ . This additivity eventually translates into the infimal convolution in the induced Fenchel-Young loss, which we call the convolutional Fenchel-Young loss. The convolutional Fenchel-Young loss is endowed with the convexity and smoothness arising from the base negentropy, shown in Section 3.2. Then we can obtain a prediction link via the infimal convolution. Paired with the convolutional Fenchel-Young loss, it admits a linear surrogate regret bound, demonstrated in Section 3.3.

In addition, we improve the multiplicative term in the initial linear surrogate regret bound in Section 3.4 to make the bound tighter, which exploits the low-rank structure of a target loss ℓ . As a by-product of the loss smoothness, we can provide a Fisher-consistent probability estimator of the underlying probability in Section 3.5. Finally, Section 4 instantiates the framework of the convolutional Fenchel-Young loss for the multiclass classification problem, highlighting the efficient computation

of the prediction link. More examples of target prediction problems, such as classification with rejection and multilabel ranking, are available in Appendix D.

1.1 Related Works

Surrogate losses for general discrete prediction problems. A discrete prediction problem aims to predict t that minimizes a discrete target loss ℓ ( t, y ) over the class distribution, and the study of convex surrogates for ℓ is of significant interest. Extensive research has been conducted on surrogate losses for specific discrete tasks, including but not limited to classification [12, 101, 89, 93, 85, 66, 64, 81], topk [95, 90], and multilabel learning [40, 100, 54, 49, 65].

In contrast to ad-hoc approaches, recent studies have advanced the principled design of calibrated convex surrogate losses for general discrete prediction problems, without imposing restrictions on the target losses. In Ramaswamy and Agarwal [73] and Ramaswamy et al. [75], surrogate losses based on the squared loss and error correcting output codes are proposed, respectively, which embed the structure of a target problem into a surrogate loss and are shown to be calibrated with the corresponding target losses. The design of calibrated surrogate losses has also been extensively studied in the context of structured prediction [29, 74, 26, 27, 70, 68, 69, 23], where the structural/lowrank properties of target losses are exploited to construct more efficient surrogates. As opposed to the smooth losses utilized in the works above, polyhedral losses [34, 35, 37] have attracted attention recently, which provides a systematic framework for efficiently constructing calibrated yet non-smooth convex losses based on a discrete target loss.

Surrogate regret bounds. Surrogate regret bounds have been well studied for margin losses under binary and multiclass classification [101, 12, 89, 84, 56, 86], where the target loss is the 0-1 loss. Similarly, proper losses [21, 42, 78, 3, 45, 63] have been shown to provide surrogate regret bounds w.r.t. the L 1 distance between the estimated and true class probabilities [77, 8, 9], which facilitates analyses for downstream tasks. For structured prediction, surrogate regret bounds are sometimes called comparison inequalities [26, 70, 68, 14], which are typically of square-root type. Notably, Frongillo and Waggoner [39] demonstrated that polyhedral losses exhibit linear surrogate regret bounds for general discrete prediction problems, covering various piecewise linear non-smooth losses [95, 76] as special cases. However, they lack the smoothness. To the contrary, Mao et al. [53] and Mao et al. [52] propose smooth losses with linear regret rates but lacking convexity.

Whereas a growing line of research has focused on H -consistency to analyze how the restriction to a hypothesis space H affects consistency [48, 99, 7], we implicitly suppose that the hypothesis space is all measurable functions because the analysis is often more transparent and it is reasonable to suppose that the hypothesis space is sufficiently expressive under the overparametrization regime. The extension to H -consistency is rather straightforward by integrating the minimizability gap [52, 55, 54, 57, 51]. 4

2 Preliminaries

Let [ d ] := { 1 , 2 , · · · , d } and [ [ · ] ] is the Iverson bracket. The p -norm is denoted by ∥ · ∥ p , which we assume to be the 2 -norm unless otherwise noted. Let R := R ∪ {∞} be the extended real-line and ∆ d := { η ∈ R d ≥ 0 : ∥ η ∥ 1 = 1 } the d -simplex. For a set S ⊆ R d , int( S ) and relint( S ) are its interior and relative interior, respectively, and conv( S ) is its convex hull. The indicator function is denoted by I S : R d → { 0 , + ∞} , where I S ( θ ) = 0 if θ ∈ S and + ∞ otherwise. For a function f : R d → R , dom( f ) := { θ ∈ R d : f ( θ ) < + ∞} is its effective domain. A function f is extended to be set-valued with slight abuse of notation by f ( S ) := { f ( s ) : s ∈ S} . The Fenchel conjugate of Ω is Ω ∗ ( θ ) := sup p ∈ dom(Ω) { θ ⊤ p -Ω( p ) } . The identity matrix is I . The canonical basis of the Euclidean space is denoted by { e i } , where the dimensionality depends on the contexts.

2.1 Discrete Prediction Problems and Target Losses

Let Y = [ K ] be the finite class space. The class distribution on Y is η ∈ ∆ K such that class Y = y has probability η y . A discrete prediction problem aims to find a target prediction t from the finite

prediction space ̂ Y := [ N ] for each η by minimizing a discrete target loss ℓ : ̂ Y × Y → R over y ∼ η . The averaged target loss is called the target risk: R ℓ ( t, η ) := E y ∼ η [ ℓ ( t, y )] = ⟨ η , ℓ ( t ) ⟩ , where ℓ ( t ) ∈ R K is the loss vector such that ℓ ( t ) y = ℓ ( t, y ) for each y ∈ Y . The Bayes risk of ℓ at η is R ℓ ( η ) := min t ∈ ̂ Y R ℓ ( t, η ) , which is a concave function of η . The suboptimality of a prediction t w.r.t. the target loss ℓ over a class distribution η is characterized by the target regret , which is the gap between its risk and the Bayes risk.

Definition 2 (Target regret) Given a discrete target loss ℓ , the target regret of a prediction t w.r.t. a class probability η ∈ ∆ K is defined as follows:

Equivalent lower-dimensional decomposition. By encoding every possible class label y ∈ Y into the canonical basis e y ∈ R K , we can express any target loss in the following form:

An equivalent but more efficient decomposition of (3), known as Structure Encoding Loss Functions (SELF), was introduced in Ciliberto et al. [26], which allows lower-dimensional label encodings. It has been widely used to construct efficient loss functions, and further generalized via Affine Decomposition in Blondel [14, (12)]. We also adopt a general form to represent discrete target losses.

Definition 3 ( ( ρ , ℓ ρ ) -decomposition) For a discrete target loss ℓ : ̂ Y × Y → R , its ( ρ , ℓ ρ ) -decomposition is given as follows:

where ρ : Y → R d is a label encoding function that maps discrete labels into the d -dimensional Euclidean space, ℓ ρ : ̂ Y → R d is the corresponding loss encoding function, and c : Y → R is the remainder independent of prediction t .

This loss decomposition contains the Affine Decomposition [14, (12)]. Any discrete target loss admits such a decomposition via the trivial choice ρ ( y ) = e y , ℓ ρ ( t ) = ℓ ( t ) , and c = 0 with d = K , which immediately recovers (3). With a properly chosen ρ , the encoded cardinality d can often be greatly smaller than K , particularly in the context of structured prediction. For example, consider multilabel classification, where Y = ̂ Y = [ K ] , K = 2 d , and each y ∈ [ K ] corresponds to a unique binary vector ν ( y ) ∈ { 0 , 1 } d representing a set of binary labels. Suppose our target loss is the Hamming loss ℓ H ( t, y ) := 1 ⊤ ( ν ( t ) + ν ( y )) -2 ⟨ ν ( t ) , ν ( y ) ⟩ , which is the Hamming distance between ν ( t ) and ν ( y ) . This entails the decomposition ρ ( y ) = 1 -2 ν ( y ) , ℓ ρ ( t ) = ν ( t ) , and c ( y ) = 1 ⊤ ν ( y ) . Here, d = log 2 K significantly reduces the dimensionality from the cardinality of Y . We refer reader to Blondel [14, Appendix A] and Appendix D for more examples of discrete prediction problems. The cardinality d of label encoding ρ is closely related to surrogate regret bounds later in Section 3.4.

2.2 Surrogate Losses and Surrogate Regret Bounds

Unlike discrete target losses assessing a discrete prediction t ∈ ̂ Y , a surrogate loss L : R d ×Y → R receives a continuous score θ ∈ R d . Then a prediction link φ : R d → ̂ Y is used to transform a score θ to a prediction t . Its risk and Bayes risk are defined as R L ( θ , η ) := E y ∼ η [ L ( θ , y )] = ⟨ η , L ( θ ) ⟩ and R L ( η ) := inf θ ∈ R d R L ( θ , η ) , respectively, where L ( θ ) := [ L ( θ , y )] K y =1 . The regret of a surrogate loss is defined as the gap between the risk of a score θ and the Bayes risk, similar to the target regret.

Definition 4 (Surrogate regret) Given a surrogate loss L , the surrogate regret of score θ w.r.t. a class probability η ∈ ∆ K is defined as follows:

For a desirable surrogate loss, the convergence of its surrogate regret will dominate the target regret. That is, for a fixed class distribution η , a convergent θ such that Regret L ( θ , η ) → 0 should imply Regret ℓ ( φ ( θ ) , η ) → 0 . This convergence relationship indicates that the target regret minimization can be achieved by the minimization of the surrogate regret adopted with an appropriate prediction link. A surrogate regret bound offers a quantitative characterization of this relationship through a regret rate function ψ , which is the key focus of this work.

Definition 5 (Surrogate regret bound) A surrogate loss L and prediction link φ entail a surrogate regret bound w.r.t. target ℓ with a non-decreasing regret rate function ψ : R ≥ 0 → R ≥ 0 satisfying ψ (0) = 0 if the following inequality holds:

While a linear regret rate ψ ( r ) = O ( r ) is the best possible, previously know linear-rate losses are either non-smooth or non-convex, including the (non-smooth) hinge loss [12] and (non-convex) sigmoid loss [24]. These loss functions may face challenges in optimization and estimation [87]. In this work, we aim to develop a framework that facilitates the use of convex smooth surrogates with linear surrogate regret bounds.

2.3 Fenchel-Young Loss

In this work, we build upon Fenchel-Young losses [16, 18, 59] to construct convex smooth surrogates equipped with linear regret rate functions. Let us review Fenchel-Young losses first.

Definition 6 (Fenchel-Young loss) For Ω : R d → R , the associated Fenchel-Young loss L Ω : dom(Ω ∗ ) × dom(Ω) → R ≥ 0 is defined as follows:

Similar constructions can also be found in Duchi et al. [33], Agarwal et al. [2], which focus on multiclass classification. The function Ω is often regarded as a generalized negentropy, which equals the negative of the Bayes risk of its induced Fenchel-Young loss. Fenchel-Young losses often impose additional requirements on the domains of Ω and Ω ∗ , tailored to specific target problems, and we will highlight them when necessary.

Fenchel-Young losses possess favorable properties: they are inherently convex (in score θ ) by definition. Moreover, we can use the map ∇ Ω ∗ ( θ ) to obtain the mean E y ∼ η [ ρ ( y )] , which is the class distribution η under multiclass classification with ρ ( y ) = e y ; or linear properties in general [1]. Many common losses are encompassed in Fenchel-Young losses, including the cross-entropy loss, squared loss, and Crammer-Singer loss [31]. For example, Ω( p ) = 1 2 ∥ p ∥ 2 + I ∆ K ( p ) generates the sparsemax loss, which induces the sparsemax as a Fisher-consistent estimator of η [58].

Connections to information geometry. While Fenchel-Young losses are systematically defined under the convex-analytic formulation, it also has a longstanding history in information geometry, particularly through its connection to the Bregman divergence and related generalizations [72]. In Blondel et al. [16], it is noted that a Fenchel-Young loss can be interpreted as the mixed-type Bregman divergence [4]. When Ω is of Legendre-type, Ω can yield the dual coordinate system and the Fenchel-Young loss is further equivalent to the canonical divergence generated by Ω [5, Eq. (3.44)].

3 Convex Smooth Surrogates with Linear Surrogate Regret Bounds

In this section, we first show our construction of surrogate losses and prediction links. After showing that this loss is convex and smooth with an appropriately chosen base negentropy (Theorem 11), we move on to the main result of this paper, linear surrogate regret bounds with convex smooth surrogate losses (Theorem 13). We further discuss the computational aspects and improve the regret bound constant. The missing proofs are deferred to Appendix B.

3.1 Construction

We design a convex smooth surrogate loss built upon the framework of Fenchel-Young losses. To make its surrogate regret bound linear, we craft a negentropy by delicately leveraging the structure of a target prediction problem. Denote by T the polyhedral convex function T ( p ) = -min t ∈ ̂ Y ⟨ p , ℓ ρ ( t ) ⟩ . Then, we have the following definition.

Definition 7 (Convolutional negentropy) Suppose ( ρ , ℓ ρ ) -decomposition for a target loss ℓ . For a negentropy Ω : R d → R , its convolutional negentropy is defined as follows:

While commonly used negentropy in Fenchel-Young losses, such as the Shannon negentropy and norm negentropy [19], are unstructured toward a target loss, the convolutional negentropy Ω T encodes a target loss ℓ explicitly into the base negentropy Ω via T . This polyhedral convex function T is an affinely transformed negative Bayes risk of a target loss ℓ when p ∈ conv { ρ ( Y ) } . Indeed, with the linear property p = E y ∼ η [ ρ ( y )] , i.e., expectation of ρ ( y ) w.r.t. class probability η :

The base negentropy Ω is up to our choice. Before deriving the conjugate of Ω T , we make the following requirement on Ω throughout this work for a well-behaved convolutional negentropy.

Condition 1 Ω is proper convex and lower-semicontinuous (l.s.c.), and satisfies conv( ρ ( Y )) ⊆ dom(Ω) and dom(Ω ∗ ) = R d .

It is a mild requirement. Indeed, proper convexity and lower-semicontinuity merely aim to avoid pathologies, and the domain assumptions are met by a differentiable and finite Ω over the valid prediction space conv( ρ ( Y )) . The assumption on Ω ∗ is crucial for the induced loss to have unconstrained domain R d . Then we show an explicit form of the conjugated convolutional negentropy.

Lemma 8 (Conjugate of Ω T ) Suppose Condition 1 holds, then Ω ∗ T is proper convex and l.s.c. with dom(Ω ∗ T ) = R d , and can be expressed as follows:

where L ρ ∈ R d × N is the loss matrix with the t -th column being ℓ ρ ( t ) ∈ R d for t ∈ ̂ Y . In addition, there always exists π ∈ ∆ N that achieves the infimum of (10) for any θ ∈ R d .

The conjugated form (10) owes to the infimal convolution between the base negentropy Ω and the target Bayes risk T . We coined the name of the convolutional negentropy inspired by this structure. Intuitively, the conjugated convolutional negentropy Ω ∗ T can be viewed as a 'perturbed' version of the conjugated base negentropy Ω ∗ toward the direction of the target loss matrix L ρ . This result facilitates a more concrete formulation of the Fenchel-Young loss generated by Ω T , which we refer to as the convolutional Fenchel-Young loss-the central focus of this work.

Definition 9 (Convolutional Fenchel-Young loss) Suppose that a discrete target loss ℓ enjoys ( ρ , ℓ ρ ) -decomposition. For a negentropy Ω : R d → R satisfying Condition 1, the convolutional Fenchel-Young loss L Ω T : R d ×Y → R induced by the convolutional negentropy Ω T is defined as follows:

Note that L Ω T ( θ , · ) is deliberately constrained to the class space Y instead of the general domain dom(Ω T ) , to align with the surrogate loss form introduced in Section 2.2.

Given a surrogate loss, we need to specify a prediction link. For general discrete prediction problems where Y ̸ = ̂ Y , a prediction link yields a discrete prediction in ̂ Y from a score θ ∈ R d obtained through minimizing the surrogate loss. Thus, a loss and link are the two sides of the same coin. Unlike the standard argmax link in multiclass classification [89, 94], we create an alternative argmax-like prediction link based on the minimizer of (10).

Definition 10 ( π -argmax link) Let Π : R d → 2 ∆ N be a set-valued map defined as follows:

Let π : R d → ∆ N be a selector of Π such that π ( θ ) ∈ Π( θ ) for all θ ∈ R d . 5 Then the π -argmax link φ : R d → ̂ Y is defined as

where the tie can be broken arbitrarily.

The existence of such links is guaranteed as long as Π( θ ) is non-empty for every θ ∈ R d , which is guaranteed by Lemma 8. While the standard argmax link returns a prediction t ∈ ̂ Y with the maximum score θ t , the π -argmax link returns t with the maximum π t . This probabilistic quantity π ∈ Π( θ ) defined via (10) distorts the original score θ by leveraging the target loss structure L ρ .

3.2 Convexity and Smoothness

Before discussing surrogate regret bounds, we verify that convolutional Fenchel-Young losses are indeed convex and smooth, implied by the conjugacy between smoothness and strong convexity [44].

Corollary 11 (Convexity and smoothness of L Ω T ) Suppose that a discrete target loss ℓ enjoys ( ρ , ℓ ρ ) -decomposition. For a base negentropy Ω , we additionally suppose that Condition 1 is satisfied. If Ω is strictly convex on dom(Ω) , L Ω T ( · , y ) is convex and differentiable over R d for any y ∈ Y . If Ω is additionally strongly convex on dom(Ω) , L Ω T ( · , y ) is smooth over R d for any y ∈ Y .

There are several exemplar base negentropies fulfilling the conditions of Corollary 11. For example, the squared norm Ω( p ) = 1 2 ∥ p ∥ 2 is strongly convex with the self-conjugate Ω ∗ ( θ ) = 1 2 ∥ θ ∥ 2 . In this case, Ω is effective over the entire R d and hence includes conv( ρ ( Y )) , satisfying Condition 1. In light of the target-loss decomposition (3), we have conv( ρ ( Y )) = ∆ K . In this case, the Shannon negentropy Ω( p ) = ⟨ p , ln p ⟩ + I ∆ K ( p ) is strongly convex on ∆ K with its conjugate Ω ∗ ( θ ) = ln ⟨ exp( θ ) , 1 ⟩ satisfying dom(Ω ∗ ) = R K , where ln and exp are element-wise. Hence, both base negentropy yield convex and smooth L Ω T .

Note that L Ω T is not locally strongly convex at every point. We can see this by noting that T is not differentiable and neither is Ω T , which implies that Ω ∗ T is not strictly convex [80, Theorem 11.13]. This is important for establishing linear surrogate regret bounds (in Section 3.3) because the known square-root regret rate lower bound considers locally strongly convex surrogate losses [39, Theorem 2].

While the loss L Ω T is now ensured to be convex and smooth, its gradient calculation, which is the basis for gradient-based optimization [17, 15], remains non-trivial. This is because the gradient of min π ∈ ∆ N Ω ∗ ( θ + L ρ π ) contained in (11) cannot be written analytically in general. We show that its gradient calculation reduces to computing Π( θ ) through the following variant of envelope theorems.

Lemma 12 (Envelope theorem) Suppose Condition 1 holds and Ω is strictly convex on dom(Ω) . For any θ ∈ R d and π ∈ Π( θ ) , we have

Deviating from the standard envelope theorems [15, 13], we carefully addresses the non-uniqueness of the minimizers. Given this, the gradient of the convolutional Fenchel-Young loss (11) is accessed via ∇ θ L Ω T ( θ , y ) = ∇ Ω ∗ ( θ + L ρ π ) -ρ ( y ) . At the core of its proof, the differentiability of Ω ∗ T (indirectly obtained via Theorem 11) is vital.

3.3 Linear Surrogate Regret Bounds

Now we exhibit linear surrogate regret bounds with the convolutional Fenchel-Young loss.

Theorem 13 (Linear surrogate regret bound) Consider a target loss with ( ρ , ℓ ρ ) -decomposition. For a negentropy Ω : R d → R , suppose Condition 1 holds. For any π -argmax link φ , ( L Ω T , φ ) admits the following surrogate regret bound:

The regret bound in Theorem 13 indeed has the linear rate ψ ( r ) = Nr in the form (1), while L Ω T can be naturally made convex and smooth as discussed in Section 3.2. This is a remarkable consequence because many authors have previously implied the impossibility to overcome the barrier of the square-root regret rate with convex smooth surrogate losses [50, 76, 39, 8]. The crux of this success lies in the infimal convolution structure in (10), which enables us to additively decompose the conjugate Ω ∗ T , exploited below. The constant N will be improved in Section 3.4.

From now on we sketch the regret bound proof. The following lemma is a cornerstone herein.

Lemma 14 (Lower bound of surrogate regret) Assume the same set of conditions as in Theorem 13. For any θ ∈ R d and π ∈ Π( θ ) , we have the following inequality:

Its proof, formally given in Appendix B, hinges on the following additive regret decomposition:

which directly implies the lower bound (15) because the Fenchel-Young loss L Ω is non-negative. Note that this additive decomposition is indispensable for the linear lower bound (15). This becomes possible just because we create the convolutional negentropy Ω T based on the additive form in (8), which yields the additive form θ + L ρ π in the conjugate expression (10). All of these are thanks to the structure of the infimal convolution, and Theorem 13 immediately follows with rescaling.

Finally, we achieve the main result by combining Corollary 11 and Theorem 13. We constructively prove the existence of convex smooth linear-regret surrogate losses, by noting that both strongly convex Ω satisfying Condition 1 and π -argmax link do exist.

Theorem 15 (Main result) Consider a target loss ℓ with ( ρ , ℓ ρ ) -decomposition. For a negentropy Ω : R d → R satisfying Condition 1, suppose that Ω is strongly convex on dom(Ω) . Then the convolutional Fenchel-Young loss L Ω T ( · , y ) is convex, differentiable, and smooth over R d for any y ∈ Y . Moreover, with π -argmax link φ , ( L Ω T , φ ) enjoys the linear surrogate regret bound (14) .

3.4 Improving Constant of Linear Surrogate Regret Bounds

The constant N in the linear surrogate regret bound in (14) can be prohibitively large, leading to a vacuous bound. This issue is significant in structured prediction as studied in Osokin et al. [70]. For example, in multilabel classification with the Hamming loss, N = 2 d is the number of all the potential binary label predictions that increases exponentially with d , the number of binary labels. In topk classification, N = ( K k ) is the number of all possible sizek subsets of the class space. Thankfully, the geometry property of the problem (10) provides a promising scheme for reducing the dependency on prediction number N , which induces the following improved link.

Corollary 16 (Improved surrogate regret bound) Suppose Condition 1 holds. There exists ˜ π : R d → ∆ N such that ˜ π ( θ ) ∈ Π( θ ) and ∥ ˜ π ( θ ) ∥ 0 ≤ affdim( L ρ ) + 1 for any θ ∈ R d . Moreover, with the induced ˜ π -argmax link φ ˜ π : R d → ∆ N , ( L Ω T , φ ˜ π ) admits the following surrogate regret bound:

where affdim( L ρ ) is the dimension of the affine hull of the column vectors of L ρ .

Note that we have affdim( L ρ ) ≤ min { N,d } , which indicates that the dependency on N can be largely reduced when d ≪ N . For example, in topk classification, ρ ( y ) = e y is used with d = K , which is much smaller than N = ( K k ) . In multilabel classification with Hamming loss, d = log 2 N is logarithmically smaller than N when binary encoding ρ ( y ) = ν ( y ) is used (see Section 2.1).

3.5 Bonus: Fisher-consistent Probability Estimator

We discuss a benefit of convex smooth surrogate losses beyond discrete prediction problems. Oftentimes people are interested in a probability estimator over possible prediction outcomes, recovering from surrogate risk minimizers, as in classification with rejection [25, 11, 28, 30, 22, 55, 36]. Herein non-smooth surrogate losses have been unfavored because of lacking reasonable probability estimators [60, 24]. Ramaswamy et al. [76] conjectures the incompatibility of probability estimation with linear regret bounds. By contrast, we give a Fisher-consistent estimator of the linear property E y ∼ η [ ρ ( y )] for convolutional Fenchel-Young losses without sacrificing the linear regret bound.

Theorem 17 (Fisher-consistent probability estimator) Suppose Condition 1 holds and Ω is strictly convex on dom(Ω) . For any η ∈ relint(∆ K ) , the surrogate risk R L Ω T ( · , η ) is minimized at θ ∗ ∈ R d such that ∇ Ω ∗ ( θ ∗ + L ρ π ) = E y ∼ η [ ρ ( y )] for any π ∈ Π( θ ∗ ) .

Algorithm 1 Exact Solution of (17) in O ( K ln K )

1: Sort θ ∈ R K such that θ (1) ≥ · · · ≥ θ ( K ) .

2: n ← max { k ∈ [ K ] : 1 + kθ ( k ) > ∑ k i =1 θ ( i ) } .

4: π log ( θ ) i ← max { θ i -τ ( θ ) , 0 } .

3: τ ( θ ) ← ∑ n i =1 θ ( i ) -1 n .

As a result, the empirical minimizers w.r.t. L Ω T are Fisher-consistent estimators of the linear property E y ∼ η [ ρ ( y )] . According to the result above, we can first solve Π( θ ) in (12) and select π ∈ Π( θ ) . Then, ∇ Ω ∗ ( θ + L ρ π ) is a rational estimate of the linear property E y ∼ η [ ρ ( y )] . When we follow the trivial loss decomposition (3) with ρ ( y ) = e y , the estimand is E y ∼ η [ ρ ( y )] = η ∈ ∆ K . For this reason, we say ∇ Ω ∗ ( θ ∗ + L ρ π ) is a 'probability' estimator.

For example, let us recap the encoding of multilabel classification in Section 2.1, where y ∈ Y is a multilabel among all 2 d possible combinations of d binary labels. A multilabel y is encoded by ν ( y ) ∈ { 0 , 1 } d , and the Hamming loss is decomposed with ρ ( y ) = 1 -2 ν ( y ) . Here E y ∼ η [ ν ( y )] reads ¯ ν = [Prob( ν i ( y ) = 1)] d i =1 , whose i -th element indicates the likelihood of the binary class i being positive. Through the relation ρ = 1 -2 ν , the probability estimator ∇ Ω ∗ ( θ + L ρ π ) is capable of recovering ¯ ν by [1 -∇ Ω ∗ ( θ + L ρ π )] / 2 .

4 Example: Multiclass Classification

̸

We demonstrate convolutional Fenchel-Young losses for multiclass classification. and further examples of prediction problems can be found in Appendices D and E, including detailed computational complexity analyses and visualizations of the associated binary classification losses. In multiclass classification, the class space and the prediction space are the same: Y = ̂ Y = [ K ] . For an input with class probability η ∈ ∆ K , the goal is to predict the most likely class t ∈ argmax t ∈Y η t . The target loss is the 0-1 loss ℓ 01 ( t, y ) = [ [ t = y ] ] .

Firstly, we adopt the decomposition (3) for this task, that is, ℓ ρ ( t ) = [ ℓ 01 ( t, 1) , · · · , ℓ 01 ( t, K )] = 1 -e t and ρ ( y ) = e y , which corresponds to the one-hot encoding commonly used in this task. In this case, N = d = K , and L ρ = 11 ⊤ -I .

Next, we move on to the convolutional negentropy Ω T . For the choice of Ω , we use the Shannon negentropy, which induces the celebrated cross-entropy loss in the original Fenchel-Young loss framework [16, Table 1]. Its conjugate is the log-sum-exp function Ω ∗ ( θ ) = ln ⟨ exp( θ ) , 1 ⟩ . Then we can calculate the conjugate of Ω T based on Lemma 8:

Since Ω is proper convex and l.s.c. with conv( ρ ( Y )) = ∆ K = dom(Ω) and dom(Ω ∗ ) = R K , Ω satisfies Condition 1. Furthermore, since Ω is strongly convex on ∆ K , we can finally derive the convolutional Fenchel-Young loss (11) by nothing Ω T ( ρ ( y )) = 0 for all y ∈ Y , as follows:

which is convex and smooth in θ thanks to Theorem 11. While shares the form of cross-entropy loss L Ω ( θ , y ) = ln ⟨ exp( θ ) , 1 ⟩ -θ y , it further incorporates an additional bounded perturbation.

Solving the minimization problem (17) is important from the computational aspects, including the gradient calculation of L Ω T and accessing to the probability estimator given by Theorem 17. We provide Algorithm 1 to solve (17) with O ( K ln K ) time.

Lemma 18 For any θ ∈ R K , the problem (17) has a unique minimizer π log ( θ ) ∈ Π( θ ) , which can be obtained in O ( K ln K ) time by Algorithm 1.

The proof is deferred to Appendix C.1, following from the KKT conditions. Eventually we have

Algorithm 1 comes with a significant resemblance with the sparsemax [58, Algorithm 1], which is determined by the similar structure shared by (17) and Euclidean projection problem [58, (2)]. Since Π( θ ) is a singleton, we have the unique π log -argmax link φ π log (Definition 10). While we need to solve problem (17) for every θ to have access to ∇ θ L Ω T ( θ , y ) , the prediction by the π log -argmax link is much cheaper, without requiring Algorithm 1, which indicates that we can simply use the class label with the largest score max y ∈ [ K ] θ y in test time. The proof can be found in Appendix C.2.

Proposition 19 A prediction link φ is the π log -argmax link if and only if φ ( θ ) ∈ argmax t ∈Y θ t .

Remark 20 Under classification, a surrogate loss with a regret bound is classification-calibrated, which indicates that the surrogate risk minimization eventually leads to the Bayes-optimal classifier. In literature, Blondel [14] and Wang and Scott [94] have investigated sufficient conditions for Fenchel-Young losses to be classification-calibrated. Therein the base negentropy is assumed to be of Legendre-type or twice differentiable. Our convolutional Fenchel-Young losses are interesting because we do not require these conditions to yield both the smoothness and a regret bound. Thus it remains open to relax these existing sufficient conditions for Fenchel-Young losses further.

5 Discussion

Convex non-smooth linear regret losses. Compared to existing convex non-smooth surrogates with linear regret, e.g., polyhedral losses [37], our smooth convex surrogate has several advantages. First, smoothness enables more efficient optimization, as supported by results in both deterministic and stochastic optimization regimes [67, 83] while it is left open to exploit specific structures arising from polyhedral losses to achieve faster optimization. In addition, the smoothness can lead to optimistic ERM rates of estimation error [87], offering better estimation in easier tasks. Another appealing aspect is that our loss admits a consistent probability estimator (Theorem 17), which can be valuable for downstream tasks such as uncertainty quantification and calibration.

Efficient gradient calculation. In general discrete prediction problems, we need to solve the minimization (10) to take a gradient of L Ω T , which is potentially demanding over a high-dimensional domain ∆ N . To have access to the gradient, we can alternatively solve

To see this, let us denote the minimizer of (19) by ν ∗ . Then (19) is an equivalent optimization problem to (10) because there exists π ∈ Π( θ ) such that ν ∗ = L ρ π . Eventually ∇ Ω ∗ ( θ + ν ∗ ) serves as an alternative to the gradient formula (13). Thus we can reduce the dimensionality of the optimization problem. For example, in multilabel classification (Section 2.1), V = [0 , 1] d has a logarithmically smaller optimization dimensionality than N = 2 d . We can solve (19) with this box constraint efficiently by using the standard L-BFGS solver [47].

Randomized prediction link. Recall that the π -argmax link φ (Definition 10) deterministically outputs in the probability simplex ∆ N . Instead we can define a randomized link ˜ φ such that Pr[ ˜ φ ( θ ) = t ] = π t ( θ ) . This yields a better regret bound by Lemma 14, as follows:

where the expectation is taken over the randomness of ˜ φ . Although we adopt the expected target regret differently from Theorem 13, the constant of regret bounds is strikingly improved from N to 1 , which is dimension-free. Sakaue et al. [82] observes a similar regret improvement by a randomized link for online structured prediction with Fenchel-Young losses.

6 Conclusion

In this work, we construct convex and smooth surrogate losses with linear surrogate regret bounds by leveraging Fenchel-Young losses and infimal convolution. Our results demonstrate that convexity, smoothness, and linear surrogate regret are compatible for arbitrary discrete prediction problems. Moreover, our loss naturally admits consistent probability estimator, bridging the gap between linear regret and estimation. We illustrate the broad applicability of our approach through examples in multiclass classification, classification with rejection, and multilabel ranking. Overall, this study highlights the utility of convex analysis as a principled tool for designing surrogate losses.

References

Jacob D. Abernethy and Rafael Frongillo. A characterization of scoring rules for linear properties. In Conference on Learning Theory , page 27.1, 2012. Cited on page: 5

Alekh Agarwal, Sham Kakade, Nikos Karampatziakis, Le Song, and Gregory Valiant. Least squares revisited: Scalable approaches for multi-class prediction. In International Conference on Machine Learning , pages 541-549, 2014. Cited on page: 5

Shivani Agarwal. Surrogate regret bounds for bipartite ranking via strongly proper losses. Journal of Machine Learning Research , 15(1):1653-1674, 2014. Cited on page: 3

Shun-ichi Amari. Information Geometry and Its Applications , volume 194. Springer, 2016. Cited on page: 5

Shun-ichi Amari and Hiroshi Nagaoka. Methods of Information Geometry , volume 191. American Mathematical Soc., 2000. Cited on page: 5

Pranjal Awasthi, Natalie Frank, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Calibration and consistency of adversarial surrogate losses. Advances in Neural Information Processing Systems , 34:9804-9815, 2021. Cited on page: 25

Pranjal Awasthi, Anqi Mao, Mehryar Mohri, and Yutao Zhong. Multi-class H -consistency bounds. Advances in Neural Information Processing Systems , 35:782-795, 2022. Cited on page: 3

Han Bao. Proper losses, moduli of convexity, and surrogate regret bounds. In Conference on Learning Theory , pages 525-547, 2023. Cited on pages: 3, 7

Han Bao and Asuka Takatsu. Proper losses regret at least 1/2-order. arXiv preprint arXiv:2407.10417 , 2024. Cited on page: 3

Han Bao, Clayton Scott, and Masashi Sugiyama. Calibrated surrogate losses for adversarially robust classification. In Conference on Learning Theory , pages 408-451, 2020. Cited on page: 25

Peter L. Bartlett and Marten H. Wegkamp. Classification with a reject option using a hinge loss. Journal of Machine Learning Research , 9(59):1823-1840, 2008. Cited on page: 8

Peter L. Bartlett, Michael I. Jordan, and Jon D. McAuliffe. Convexity, classification, and risk bounds. Journal of the American Statistical Association , 101(473):138-156, 2006. Cited on pages: 2, 3, and 5

Dimitri Bertsekas. Nonlinear Programming , volume 4. Athena Scientific, 2016. Cited on page: 7

Mathieu Blondel. Structured prediction with projection oracles. Advances in Neural Information Processing Systems , 32:12145-12156, 2019. Cited on pages: 3, 4, and 10

Mathieu Blondel and Vincent Roulet. The elements of differentiable programming. arXiv preprint arXiv:2403.14606 , 2024. Cited on page: 7

Mathieu Blondel, André F. T. Martins, and Vlad Niculae. Learning with Fenchel-Young losses. Journal of Machine Learning Research , 21(35):1-69, 2020. Cited on pages: 2, 5, and 9

Mathieu Blondel, Quentin Berthet, Marco Cuturi, Roy Frostig, Stephan Hoyer, Felipe LlinaresLópez, Fabian Pedregosa, and Jean-Philippe Vert. Efficient and modular implicit differentiation. Advances in Neural Information Processing Systems , 35:5230-5242, 2022. Cited on page: 7

Mathieu Blondel, Felipe Llinares-López, Robert Dadashi, Léonard Hussenot, and Matthieu Geist. Learning energy networks with generalized Fenchel-Young losses. Advances in Neural Information Processing Systems , 35:12516-12528, 2022. Cited on page: 5

Dick E. Boekee and Jan C. A. van der Lubbe. The R -norm information measure. Information and Control , 45(2):136-155, 1980. Cited on page: 6

Jonathan M. Borwein and Adrian S. Lewis. Convex Analysis and Nonlinear Optimization: Theory and Examples . CMS Books in Mathematics. Springer New York, 2005. Cited on page: 25

Andreas Buja, Werner Stuetzle, and Yi Shen. Loss functions for binary class probability estimation and classification: Structure and applications. Technical report, University of Pennsylvania, 2005. Cited on page: 3

Yuzhou Cao, Tianchi Cai, Lei Feng, Lihong Gu, Jinjie Gu, Bo An, Gang Niu, and Masashi Sugiyama. Generalizing consistent multi-class classification with rejection to be compatible with arbitrary losses. Advances in Neural Information Processing Systems , 35:521-534, 2022. Cited on page: 8

Yuzhou Cao, Lei Feng, and Bo An. Consistent hierarchical classification with a generalized metric. In International Conference on Artificial Intelligence and Statistics , pages 4825-4833, 2024. Cited on page: 3

Nontawat Charoenphakdee, Jongyeong Lee, and Masashi Sugiyama. On symmetric losses for learning from corrupted labels. In International Conference on Machine Learning , pages 961-970, 2019. Cited on pages: 2, 5, and 8

Chao-Kong Chow. On optimum recognition error and reject tradeoff. IEEE Transactions on Information Theory , 16(1):41-46, 1970. Cited on pages: 8, 30

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A consistent regularization approach for structured prediction. Advances in Neural Information Processing Systems , 29:4412-4420, 2016. Cited on pages: 3, 4

Carlo Ciliberto, Lorenzo Rosasco, and Alessandro Rudi. A general framework for consistent structured prediction with implicit loss embeddings. Journal of Machine Learning Research , 21(98):1-67, 2020. Cited on page: 3

Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Learning with rejection. In International Conference on Algorithmic Learning Theory , pages 67-82, 2016. Cited on page: 8

Corinna Cortes, Vitaly Kuznetsov, Mehryar Mohri, and Scott Yang. Structured prediction theory based on factor graph complexity. Advances in Neural Information Processing Systems , 29:2514-2522, 2016. Cited on page: 3

Corinna Cortes, Giulia DeSalvo, and Mehryar Mohri. Theory and algorithms for learning with rejection in binary classification. Annals of Mathematics and Artificial Intelligence , 92(2): 277-315, 2023. Cited on page: 8

Koby Crammer and Yoram Singer. On the algorithmic implementation of multiclass kernelbased vector machines. Journal of Machine Learning Research , 2:265-292, 2001. Cited on page: 5

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A largescale hierarchical image database. In 2009 IEEE Conference on Computer Vision and Pattern Recognition , pages 248-255, 2009. Cited on page: 37

John Duchi, Khashayar Khosravi, and Feng Ruan. Multiclass classification, information, divergence and surrogate risk. The Annals of Statistics , 46(6B):3246-3275, 2018. Cited on page: 5

Jessica Finocchiaro, Rafael Frongillo, and Bo Waggoner. An embedding framework for consistent polyhedral surrogates. Advances in Neural Information Processing systems , 32: 10781-10791, 2019. Cited on page: 3

Jessie Finocchiaro, Rafael Frongillo, and Bo Waggoner. Embedding dimension of polyhedral losses. In Conference on Learning Theory , pages 1558-1585, 2020. Cited on page: 3

Jessie Finocchiaro, Rafael Frongillo, and Enrique Nueve. The structured abstain problem and the Lovász hinge. In Conference on Learning Theory , pages 3718-3740, 2022. Cited on page: 8

Jessie Finocchiaro, Rafael Frongillo, and Bo Waggoner. An embedding framework for the design and analysis of consistent polyhedral surrogates. Journal of Machine Learning Research , 25(63):1-60, 2024. Cited on pages: 3, 10

Dean P. Foster and Rakesh V. Vohra. Asymptotic calibration. Biometrika , 85(2):379-390, 1998. Cited on page: 2

Rafael Frongillo and Bo Waggoner. Surrogate regret bounds for polyhedral losses. Advances in Neural Information Processing Systems , 35:21569-21580, 2021. Cited on pages: 2, 3, 7, 25, and 37

Wei Gao and Zhi-Hua Zhou. On the consistency of multi-label learning. In Conference on Learning Theory , pages 341-358, 2011. Cited on page: 3

Aritra Ghosh, Himanshu Kumar, and P. Shanti Sastry. Robust loss functions under label noise for deep neural networks. In Proceedings of the AAAI Conference on Artificial intelligence , volume 31, 2017. Cited on page: 25

Tilmann Gneiting and Adrian E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American Statistical Association , 102(477):359-378, 2007. Cited on page: 3

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition , pages 770-778, 2016. Cited on page: 37

Sham Kakade, Shai Shalev-Shwartz, and Ambuj Tewari. On the duality of strong convexity and strong smoothness: Learning applications and matrix regularization. Technical report, Toyota Technological Institute at Chicago, 2009. Cited on page: 7

Wojciech Kotłowski and Krzysztof Dembczy´ nski. Surrogate regret bounds for generalized classification performance metrics. In Asian Conference on Machine Learning , pages 301-316, 2016. Cited on page: 3

Alex Krizhevsky and Geoffrey Hinton. Learning multiple layers of features from tiny images. 2009. Cited on page: 38

Dong C. Liu and Jorge Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical Programming , 45(1):503-528, 1989. Cited on page: 10

Phil Long and Rocco Servedio. Consistency versus realizable H -consistency for multiclass classification. In International Conference on Machine Learning , pages 801-809, 2013. Cited on page: 3

Michal Lukasik, Lin Chen, Harikrishna Narasimhan, Aditya Krishna Menon, Wittawat Jitkrittum, Felix X Yu, Sashank J Reddi, Gang Fu, Mohammadhossein Bateni, and Sanjiv Kumar. Bipartite ranking from multiple labels: On loss versus label aggregation. arXiv preprint arXiv:2504.11284 , 2025. Cited on page: 3

Mehrdad Mahdavi, Lijun Zhang, and Rong Jin. Binary excess risk for smooth convex surrogates. arXiv preprint arXiv:1402.1792 , 2014. Cited on pages: 2, 7

Anqi Mao, Mehryar Mohri, and Yutao Zhong. H -consistency bounds: Characterization and extensions. Advances in Neural Information Processing Systems , 36:4470-4508, 2023. Cited on page: 3

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Cross-entropy loss functions: Theoretical analysis and applications. In International Conference on Machine Learning , pages 2380323828, 2023. Cited on pages: 3, 25

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Structured prediction with stronger consistency guarantees. Advances in Neural Information Processing Systems , 36:46903-46937, 2023. Cited on pages: 3, 25

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Multi-label learning with stronger consistency guarantees. Advances in neural information processing systems , 37:2378-2406, 2024. Cited on page: 3

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Predictor-rejector multi-class abstention: Theoretical analysis and algorithms. In International Conference on Algorithmic Learning Theory , pages 822-867, 2024. Cited on pages: 3, 8

Anqi Mao, Mehryar Mohri, and Yutao Zhong. A universal growth rate for learning with smooth surrogate losses. Advances in Neural Information Processing Systems , 37:41670-41708, 2024. Cited on page: 3

Anqi Mao, Mehryar Mohri, and Yutao Zhong. Enhanced H -consistency bounds. In International Conference on Algorithmic Learning Theory , pages 772-813, 2025. Cited on page: 3

André F. T. Martins and Ramon Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International Conference on Machine Learning , pages 1614-1623, 2016. Cited on pages: 5, 10

André F. T. Martins, Marcos Treviso, António Farinhas, Pedro M. Q. Aguiar, Mário A. T. Figueiredo, Mathieu Blondel, and Vlad Niculae. Sparse continuous distributions and FenchelYoung losses. Journal of Machine Learning Research , 23(257):1-74, 2020. Cited on page: 5

Hamed Masnadi-Shirazi and Nuno Vasconcelos. On the design of loss functions for classification: Theory, robustness to outliers, and SavageBoost. Advances in Neural Information Processing Systems , 21:1049-1056, 2008. Cited on page: 8

Nishant A. Mehta and Robert C. Williamson. From stochastic mixability to fast rates. Advances in Neural Information Processing Systems , 27:1197-1205, 2014. Cited on page: 25

Shahar Mendelson. Lower bounds for the empirical minimization algorithm. IEEE Transactions on Information Theory , 54(8):3797-3803, 2008. Cited on page: 25

Aditya Krishna Menon and Robert C. Williamson. Bayes-optimal scorers for bipartite ranking. In Conference on Learning Theory , pages 68-106, 2014. Cited on page: 3

Aditya Krishna Menon, Harikrishna Narasimhan, Shivani Agarwal, and Sanjay Chawla. On the statistical consistency of algorithms for binary classification under class imbalance. In International Conference on Machine Learning , pages 603-611, 2013. Cited on page: 3

Aditya Krishna Menon, Ankit Singh Rawat, Sashank Reddi, and Sanjiv Kumar. Multilabel reductions: What is my loss optimising? Advances in Neural Information Processing Systems , 32:10600-10611, 2019. Cited on page: 3

Harikrishna Narasimhan, Harish G. Ramaswamy, Shiv Kumar Tavker, Drona Khurana, Praneeth Netrapalli, and Shivani Agarwal. Consistent multiclass algorithms for complex metrics and constraints. Journal of Machine Learning Research , 25(367):1-81, 2024. Cited on page: 3

Yurii Nesterov. A method for solving the convex programming problem with convergence rate O (1 /k 2 ) . Dokl Akad Nauk SSSR , 269:543, 1983. Cited on pages: 2, 10

Alex Nowak, Francis Bach, and Alessandro Rudi. Sharp analysis of learning with discrete losses. In International Conference on Artificial Intelligence and Statistics , pages 1920-1929, 2019. Cited on page: 3

Alex Nowak, Francis Bach, and Alessandro Rudi. A general theory for structured prediction with smooth convex surrogates. arXiv preprint arXiv:1902.01958 , 2019. Cited on pages: 2, 3

Anton Osokin, Francis Bach, and Simon Lacoste-Julien. On structured prediction theory with calibrated convex surrogate losses. Advances in Neural Information Processing Systems , 30: 302-313, 2017. Cited on pages: 3, 8

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, Gregory Chanan, Trevor Killeen, Zeming Lin, Natalia Gimelshein, Luca Antiga, Alban Desmaison, Andreas Köpf, Edward Z. Yang, Zachary DeVito, Martin Raison, Alykhan Tejani, Sasank Chilamkurthy, Benoit Steiner, Lu Fang, Junjie Bai, and Soumith Chintala. PyTorch: An imperative style, high-performance deep learning library. Advances in Neural Information Processing Systems , 32:8024-8035, 2019. Cited on page: 37

Seta Rakotomandimby, Jean-Philippe Chancelier, Michel De Lara, and Mathieu Blondel. Learning with Fitzpatrick losses. Advances in Neural Information Processing Systems , 37: 79381-79409, 2024. Cited on page: 5

Harish G. Ramaswamy and Shivani Agarwal. Classification calibration dimension for general multiclass losses. Advances in Neural Information Processing Systems , 25:2078-2086, 2012. Cited on pages: 3, 28

Harish G. Ramaswamy, Shivani Agarwal, and Ambuj Tewari. Convex calibrated surrogates for low-rank loss matrices with applications to subset ranking losses. Advances in Neural Information Processing Systems , 26:1475-1483, 2013. Cited on page: 3

Harish G. Ramaswamy, Balaji S. Babu, Shivani Agarwal, and Robert C. Williamson. On the consistency of output code based learning algorithms for multiclass learning problems. In Conference on Learning Theory , pages 885-902, 2014. Cited on page: 3

Harish G. Ramaswamy, Ambuj Tewari, and Shivani Agarwal. Consistent algorithms for multiclass classification with an abstain option. Electronic Journal of Statistics , 12:530-554, 2018. Cited on pages: 2, 3, 7, and 8

Mark D. Reid and Robert C. Williamson. Surrogate regret bounds for proper losses. In International Conference on Machine Learning , pages 897-904, 2009. Cited on page: 3

Mark D. Reid and Robert C. Williamson. Composite binary losses. Journal of Machine Learning Research , 11:2387-2422, 2010. Cited on page: 3

Ralph T. Rockafellar. Convex Analysis . Princeton University Press, 1997. Cited on pages: 25, 26, and 28

Ralph T. Rockafellar and Roger J.-B. R. Wets. Variational Analysis , volume 317. Springer Science & Business Media, 2009. Cited on pages: 7, 26, 27, and 28

Vincent Roulet, Tianlin Liu, Nino Vieillard, Michael E. Sander, and Mathieu Blondel. Loss functions and operators generated by f -divergences. In International Conference on Machine Learning , volume 267, pages 52110-52138, 2025. Cited on page: 3

Shinsaku Sakaue, Han Bao, Taira Tsuchiya, and Taihei Oki. Online structured prediction with Fenchel-Young losses and improved surrogate regret for online multiclass classification with logistic loss. In Conference on Learning Theory , pages 4458-4486, 2024. Cited on page: 10

Mark Schmidt, Nicolas Le Roux, and Francis Bach. Minimizing finite sums with the stochastic average gradient. Mathematical Programming , 162:83-112, 2017. Cited on pages: 2, 10

Clayton Scott. Surrogate losses and regret bounds for cost-sensitive classification with exampledependent costs. In International Conference on Machine Learning , pages 153-160, 2011. Cited on page: 3

Clayton Scott. Calibrated asymmetric surrogate losses. Electronic Journal of Statistics , 6: 958-992, 2012. Cited on page: 3

Sanket Shah, Milind Tambe, and Jessie Finocchiaro. Analyzing cost-sensitive surrogate losses via H -calibration. arXiv preprint arXiv:2502.19522 , 2025. Cited on page: 3

Nathan Srebro, Karthik Sridharan, and Ambuj Tewari. Smoothness, low noise and fast rates. Advances in Neural Information Processing Systems , 24:2199-2207, 2010. Cited on pages: 2, 5, 10, and 25

Ingo Steinwart. How to compare different loss functions and their risks. Constructive Approximation , 26(2):225-287, 2007. Cited on page: 2

Ambuj Tewari and Peter L. Bartlett. On the consistency of multiclass classification methods. Journal of Machine Learning Research , 8(36):1007-1025, 2007. Cited on pages: 3, 6

Anish Thilagar, Rafael Frongillo, Jessica Finocchiaro, and Emma Goodwill. Consistent polyhedral surrogates for top-k classification and variants. In International Conference on Machine Learning , pages 21329-21359, 2022. Cited on page: 3

Tim van Erven, Peter D. Grünwald, Mark D. Reid, and Robert C. Williamson. Mixability in statistical learning. Advances in Neural Information Processing Systems , 26:1700-1708, 2012. Cited on page: 25

Tim van Erven, Peter D. Grünwald, Nishant A. Mehta, Mark D. Reid, and Robert C. Williamson. Fast rates in statistical and online learning. Journal of Machine Learning Research , 16:1793-1861, 2015. Cited on page: 25

Yutong Wang and Clayton Scott. On classification-calibration of gamma-phi losses. In Conference on Learning Theory , pages 4929-4951, 2023. Cited on page: 3

Yutong Wang and Clayton Scott. Unified binary and multiclass margin-based classification. Journal of Machine Learning Research , 25(143):1-51, 2024. Cited on pages: 6, 10

Forest Yang and Sanmi Koyejo. On the consistency of top-k surrogate losses. In International Conference on Machine Learning , volume 119, pages 10727-10735, 2020. Cited on page: 3

Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In British Machine Vision Conference , 2016. Cited on page: 38

Jingwei Zhang, Tongliang Liu, and Dacheng Tao. On the rates of convergence from surrogate risk minimizers to the Bayes optimal classifier. IEEE Transactions on Neural Networks and Learning Systems , 33(10):5766-5774, 2021. Cited on page: 2

Lijun Zhang, Tianbao Yang, and Rong Jin. Empirical risk minimization for stochastic convex optimization: O (1 /n ) - and O (1 /n 2 ) -type of risk bounds. In Conference on Learning Theory , pages 1954-1979, 2017. Cited on page: 2

Mingyuan Zhang and Shivani Agarwal. Bayes consistency vs. H -consistency: The interplay between surrogate loss functions and the scoring function class. Advances in Neural Information Processing Systems , 33:16927-16936, 2020. Cited on page: 3

Mingyuan Zhang, Harish G. Ramaswamy, and Shivani Agarwal. Convex calibrated surrogates for the multi-label F-measure. In International Conference on Machine Learning , pages 11246-11255, 2020. Cited on page: 3

Tong Zhang. Statistical analysis of some multi-category large margin classification methods. Journal of Machine Learning Research , 5:1225-1251, 2004. Cited on page: 3

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: The abstract and Theorem 1 in the introduction (Section 1) reflect our scope (surrogate regret bound of losses) and contribution (a family of convex smooth surrogates with linear surrogate regret bounds).

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: The limitation of this work is discussed in Appendix A.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory assumptions and proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: Assumptions are clearly stated in each theoretical claim. All theoretical claims are provided with proofs.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental result reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: The experimental setting, including the data augmentation, optimizer, model architecture, and associated computational cost.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [No]

Justification: Our experiments use public datasets, the official PyTorch training script, and detailed formulations of both the loss and solver, making the work fully reproducible without releasing extra code or data.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( https://nips.cc/ public/guides/CodeSubmissionPolicy ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( https: //nips.cc/public/guides/CodeSubmissionPolicy ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental setting/details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The experimental settings are provided in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment statistical significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: For ImageNet experiments, we perform a 5% t-test for comparison. For classification with rejection results, no comparison is involved, and we report only the averaged performance.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments compute resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The detailed information on computer resources is provided in Appendix F.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code of ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics https://neurips.cc/public/EthicsGuidelines ?

Answer: [Yes]

Justification:

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: This theoretical work aims to advance machine learning by informing the design and use of surrogate losses for discrete prediction. While it may influence downstream research and applications, we do not identify any specific risks that warrant emphasis.

Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, paperswithcode.com/datasets has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and research with human subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional review board (IRB) approvals or equivalent for research with human subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

16. Declaration of LLM usage

Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology, scientific rigorousness, or originality of the research, declaration is not required.

Answer: [NA]

Justification:

Guidelines:

The answer NA means that the core method development in this research does not involve LLMs as any important, original, or non-standard components.

Please refer to our LLM policy ( https://neurips.cc/Conferences/2025/ LLM ) for what should or should not be described.

A Additional Discussions on Related Losses

Fast rate losses. Let us briefly discuss the target risk estimation error rates induced by different surrogate losses. Suppose that a surrogate risk can be estimated with the estimation error upper bound ϵ est and the surrogate regret rate is ψ . Then, the target risk estimation error is of order ψ ( ϵ est ) . Hence, we need to take care of both ψ and ϵ est to discuss the target risk estimation.

On the one hand, loss functions entailing either strongly convex or exponentially concave are known to achieve fast estimation error rates with ERM (e.g., O (1 /n ) ) in standard parametric setting, where the hypothesis class is of finite dimension [87, 91, 61, 92]. However, Frongillo and Waggoner [39, Theorem 4] reveals that typical fast rate losses suffer at least from square-root regret rates, and thus their corresponding target risk estimation error bounds are O (1 / √ n ) at least. On the other hand, convolutional Fenchel-Young losses (which is convex smooth but not strongly convex) yield the estimation error upper bounds of order O (1 / √ n ) under the same setting, while the final target risk upper bounds are in order of O (1 / √ n ) thanks to the linear regret rate function. As a result, convolutional Fenchel-Young losses achieve target regret convergence rates that are comparable to those of existing fast-rate losses.

In more general nonparametric settings, where fast rates are often hard to achieve without additional assumptions [62], convolutional Fenchel-Young losses achieve a target risk estimation error bound of order O (1 / √ n ) . In contrast, strongly convex surrogate losses typically achieve a slower target risk estimation error rate than O (1 / √ n ) because the general nonparametric estimation error is O (1 / √ n ) but the regret rate are slower than ψ ( r ) = O ( r ) .

While we do not intend to argue that the convolutional Fenchel-Young loss is always better, we would like to highlight that it may be a good alternative when the parametric conditions for fast rates cannot be readily justified. It also remains an open and worthwhile question whether fast rates can be obtained for our loss under additional assumptions, such as low-noise or margin conditions.

Smooth non-convex linear regret losses. While our focus is on convex surrogates due to their favorable optimization and statistical properties, we note that certain smooth but non-convex surrogates, such as the mean absolute error (MAE) [41, 52] and structured comp-sum losses with MAE [53], also achieve linear regret. These methods, while typically non-convex and not equipped with probability estimator, offer valuable advantages in other aspects, such as robustness to label noise, H -consistency, and potential benefits under adversarial conditions [10, 6], where non-convexity can play a meaningful role.

B Deferred Proofs in Section 3

Recalling the definition of T ( p ) = -min t ∈ ̂ Y ⟨ p , ℓ ρ ( t ) ⟩ , which is the negative of the affinely transformed target Bayes risk in (9). It can be inferred that it is proper convex and l.s.c., and we have Ω T = Ω + T .

B.1 Proof of Lemma 8

Proof. Since Ω and T are proper convex and l.s.c., so are Ω T and thus Ω ∗ T . Then we prove (10) using the infimal convolution. First, by noting that T is nothing else but the support function of the closed convex set conv( {-ℓ ρ ( t ) } t ∈ ̂ Y ) , we can express T ∗ as follows:

where we use the conjugacy relationship between a support function and indicator function of a closed convex set conv( {-ℓ ρ ( t ) } t ∈ ̂ Y ) [79, Section 13]. According to Condition 1 and the definition of T , we see that both Ω and T are proper convex and T is continuous on dom(Ω) . Then we use the infimal convolution [20] to derive the conjugate of Ω T :

then we have

Next we show that the infimum is indeed achieved by some π ∈ ∆ N . Note that for any θ ∈ R d , the set Θ := { θ + L ρ π : π ∈ ∆ N } is compact and non-empty, and that Ω ∗ is l.s.c. on Θ since dom(Ω ∗ ) = R d . Then the infimum of Ω ∗ ( θ ) is achieved by some θ ′′ ∈ Θ , and there must exist π ∈ ∆ N such that θ + L ρ π = θ ′′ by the definition of Θ , which complete the proof of the existence of the minimizer.

Finally we see that for any θ ∈ R d , there exists π ∈ ∆ N such that Ω ∗ T ( θ ) = Ω ∗ ( θ + L ρ π ) , which is smaller than + ∞ since dom(Ω ∗ ) = R d and θ + L ρ π ∈ R d . This indicates that dom(Ω ∗ T ) = R d because θ is chosen arbitrarily.

B.2 Proof of Theorem 11

Proof. In the convolutional Fenchel-Young loss (11), the term Ω T ( ρ ( y )) -⟨ θ , ρ ( y ) ⟩ is linear. Hence it suffices to prove the convexity and smoothness of the conjugate min π ∈ ∆ N Ω ∗ ( θ + L ρ π ) .

When Ω is strictly convex, Ω T is also strictly convex since T is convex. Since the strict convexity holds on dom(Ω T ) , Ω T is further essentially strictly convex, that is, strictly convex on every convex subset of dom( ∂ Ω T ) [79, p253]. According to Rockafellar [79, Theorem 26.3], Ω ∗ T is essentially smooth, that is, differentiable throughout non-empty int(dom(Ω ∗ T )) with ∥∇ Ω ∗ T ( θ ) ∥ diverging to + ∞ when θ approaches a boundary point of int(dom(Ω ∗ T )) , which immediately indicates the differentiability on R d .

When Ω is further strongly convex, so is Ω T . According to Condition 1 and Lemma 8 , Ω T is proper convex and l.s.c., and thus the biconjugate Ω ∗∗ T matches Ω T by the Fenchel-Moreau theorem. According to Rockafellar and Wets [80, Proposition 12.60], Ω ∗ T is smooth since its conjugate Ω ∗∗ T = Ω T is strongly convex.

B.3 Proof of Lemma 12

Proof. According to Condition 1, the strict convexity of Ω , and the proof of Theorem 11, both Ω ∗ T and Ω ∗ are differentiable on R d . In addition, we have dom(Ω) = dom(Ω T ) according to (8).

Note that Ω ∗ ( θ + L ρ π ) is convex in π . Then its first-order optimal condition for any π ∈ Π( θ ) reads

First, for any π ∈ Π( θ ) , we choose

where (A) owes to the optimality of π ∈ Π( θ ) , (B) holds by the definition of t , and (C) holds by the definition of the convolutional negentropy Ω T (8). Since π ∈ Π( θ ) , we have Ω ∗ ( θ + L ρ π ) = Ω ∗ T ( θ ) according to Lemma 8. Thus the above inequality is indeed an identity. In particular, (D) becomes an identity, which implies that the supremum of sup p ∈ dom(Ω T ) θ ⊤ p -Ω T ( p ) is achieved at ∇ Ω ∗ ( θ + L ρ π ) . Since the supremum is also achieved at ∇ Ω ∗ T ( θ ) and the maximizer is unique according to Rockafellar and Wets [80, Proposition 11.3] and the differentiability of Ω ∗ T , we have

Since π ∈ Π( θ ) and θ ∈ R d are chosen arbitrarily, this concludes the proof.

B.4 Proof of Theorem 13

Proof. Fix any θ ∈ R d and choose any π ∈ ∆ N out of Π( θ ) . Then we have

B.5 Proof of Lemma 14

Proof. First, we derive the Bayes risk of the convolutional Fenchel-Young loss L Ω T as follows:

where the Fenchel-Moreau theorem is applied to proper convex and l.s.c. Ω ∗ T (by Lemma 8) at the last identity. Then we can get the following regret lower bound for any π ∈ Π( θ ) :

where (A) holds according to the explicit form of Ω ∗ T in (10) and the definition of Π( θ ) , (B) owes to Fenchel-Young inequality, and (C) holds according to the definition of ( ρ , ℓ ρ ) -decomposition in (4). This concludes the proof.

B.6 Proof of Corollary 16

Proof. Fix an arbitrary θ ∈ R d . Since Π( θ ) is non-empty, we can select π from Π( θ ) . Note that L ρ π ∈ conv( { ℓ ρ ( t ) } t ∈ ̂ Y ) . According to Carathéodory's theorem, there exists ˜ π such that ∥ ˜ π ∥ 0 ≤ affdim( L ρ ) +1 and L ρ ˜ π = L ρ π is also in Π( θ ) . Since θ is chosen arbitrarily, we can then construct a single-valued function ˜ π : R d → ∆ N such that ˜ π ( θ ) ∈ Π( θ ) and ∥ ˜ π ( θ ) ∥ 0 ≤ affdim( L ρ ) + 1 . Noting that max t ∈ ̂ Y ˜ π t ( θ ) > 1 / [affdim( L ρ ) + 1] , we can complete the rest of the proof similarly to Theorem 13.

Remark 21 Interestingly, Ramaswamy and Agarwal [73] also used the affine dimension of the loss matrix to study the dimension of convex surrogates, suggesting that this concept plays a important role in loss function design and deserves more attention.

B.7 Proof of Theorem 17

Proof. First of all, we prove that relint(conv( ρ ( Y ))) is in the image of ∇ Ω ∗ T by contradiction. According to Rockafellar [79, Theorem 23.4], we have that ∂ Ω T ( p ) is non-empty for all p ∈ relint(dom(Ω T )) . Now suppose there exists p ∈ relint(dom(Ω T )) such that p = ∇ Ω ∗ T ( θ ) for any θ ∈ R d . Since ∂ Ω T ( p ) is non-empty, there exists θ ′ ∈ R d such that θ ′ = ∇ Ω T ( p ) . By Rockafellar and Wets [80, Proposition 11.3] (applied on proper l.s.c. Ω T ), we have p = ∇ Ω ∗ T ( θ ′ ) , which contradicts the assumption p = ∇ Ω ∗ T ( θ ′ ) . Thus we have verified that relint(dom(Ω T )) is in the image of ∇ Ω ∗ T . This additionally implies that relint(conv( ρ ( Y ))) ⊆ relint(dom(Ω T )) is also in the image of ∇ Ω ∗ T because of Condition 1.

̸

Now we fix η ∈ relint(∆ K ) and note that

is differentiable and convex in θ . If θ ∗ ∈ R d is the minimizer of R L Ω T ( · , η ) , its optimality condition indicates

When η ∈ relint(∆ K ) , we have E y ∼ η [ ρ ( y )] ∈ relint(conv( ρ ( Y ))) , and thus θ ∗ satisfying the above optimality condition does exist. Finally, we conclude the proof by combining Lemmas 8 and 12.

C Deferred Proofs in Section 4

C.1 Proof of Lemma 18

Proof. Denote by V θ ( π ) := ∑ K i =1 e θ i +1 -π i . Since ln( · ) is strictly increasing on R > 0 , V θ ( π ) shares the same minimizer as ln (∑ K i =1 e θ i +1 -π i ) . The Lagrangian F of the minimization problem V θ for π ∈ ∆ K is written as follows:

where α 1 , . . . , α K ≥ 0 and β are the Lagrangian multipliers. Since V θ ( π ) is convex and differentiable, and the feasible region ∆ K is convex, the KKT conditions are necessary and sufficient for the optimality of π ∗ ∈ Π( θ ) , which requires that there exists ( α ∗ , β ∗ ) satisfying

The conditions (21) and (22) indicates that π ∗ y > 0 = ⇒ α ∗ y = 0 . Then according to (20), we have

̸

Meanwhile, for π ∗ y = 0 , (22) indicates that α ∗ y ≥ 0 , which further indicates

Then (23) and (24) can be rewritten as follows:

According to (21) and ln β ∗ -1 ∈ R , the KKT conditions can be further simplified into the existence of τ ∗ ∈ R such that the following conditions simultaneously hold:

Denote by f ( τ ) := ∑ K y =1 max { θ y -τ, 0 } . Then f is continuous on R , and moreover, it is strictly decreasing on ( -∞ , θ (1) ] and equals 0 for τ ≥ θ (1) because of the following expression:

Then n defined in line 2 of Algorithm 1 exists since f ( θ (1) ) = 0 < 1 . We also have that f ( θ ( n ) ) = ∑ n i =1 ( θ ( i ) -θ ( n ) ) < 1 . When n < K , we have that f ( θ ( n +1) ) = ∑ n +1 i =1 ( θ ( i ) -θ ( n +1) ) = ∑ n i =1 ( θ ( i ) -θ ( n +1) ) ≥ 1 by definition. Then we see

When n = K , we have that ∑ K i =1 θ ( i ) -Kθ ( k ) < 1 , that is,

and

Then

Combining these two cases, we have

which is what line 3 of Algorithm 1 returns. Since f ( τ ) is strictly decreasing on ( -∞ , θ (1) ] , τ ∗ uniquely satisfies f ( τ ∗ ) = 1 .

Since ∑ k i =1 θ ( i ) can be calculated cumulatively in O ( K ) time and the quick sort runs in O ( K log K ) time, Algorithm 1 runs in O ( K log K ) time in total. Furthermore, since τ ∗ is unique and π ∗ y = max { θ y -τ ∗ , 0 } , we ensure that Π( θ ) is a singleton, which concludes the proof.

C.2 Proof of Proposition 19

Proof. By Definition 10, the statement is equivalent to

According to Lemma 18 and Algorithm 1, there exists τ ∈ R such that π log ( θ ) i = max { θ i -τ, 0 } . We also have that max t θ t > τ by contradiction: if max t θ t ≤ τ , π log ( θ ) = 0 holds, which contradicts π log ( θ ) ∈ ∆ K .

Denote the set argmax t ∈Y θ t by I . For any i ∈ I , we have π log ( θ ) i = max { θ i -τ, 0 } = max t θ t -τ > 0 . For any j ̸∈ I ,

If θ j > τ : π log ( θ ) j = max { θ j -τ, 0 } = θ j -τ < max t θ t -τ ,

If θ j ≤ τ : π log ( θ ) j = max { θ j -τ, 0 } = 0 < max t θ t -τ .

These imply that for any i ∈ I , π log ( θ ) i ≥ π log ( θ ) j and the equality holds if and only if j ∈ I , which concludes the proof.

D Additional Examples

In this section, we further provide more examples of target losses to demonstrate that we can generate convex smooth surrogate losses for a wide range of target prediction problems. The generated convolutional Fenchel-Young losses are automatically guaranteed to entail linear surrogate regret bounds thanks to Theorem 13.

D.1 Multiclass Classification with Rejection

Problem setup. In multiclass classification with rejection [25], the class space is Y = [ K ] , and the prediction space ̂ Y = [ K +1] , which is augmented by a rejection option K +1 . We focus on the case that rejection cost c is in [0 , 0 . 5) here. For an input instance with class probability η ∈ ∆ K , the goal is to predict the most likely class label t ∈ argmax t ∈Y η t if max y ∈Y η y > 1 -c for the predetermined cost c , and refrain from predicting otherwise. The standard target loss is the 0-1c loss:

̸

that is, the prediction suffers from the ordinary classification error if it is a wrong class label, and suffers from an intermediate error c if it chooses to refrain from prediction.

̸

We adopt the decomposition (3) for this task: ℓ ρ ( t ) = [ ℓ 01 c ( t, 1) , · · · , ℓ 01 ( t, K )] = 1 -e t if t = K +1 , and ℓ ρ ( t ) = c 1 if t = K +1 . We choose the label encoding function ρ ( y ) ... as in the multiclass classification task. In this case, N = K +1 and d = K , and L ρ = [ 11 ⊤ -I, c 1 ] , where 1 is K -dimensional.

Loss formulation and calculation. In this case, we consider the Shannon negentropy Ω as in Section 4. Based on the discussion above and Lemma 8, the conjugated convolutional negentropy Ω ∗ T can be written as follows: 6

and we can then get the corresponding convolutional Fenchel-Young loss as follows, by noting Ω T ( ρ ( y )) = 0 for all y ∈ Y :

To compute L Ω T , we need to know the minimizer π in (26). We show its closed form below.

Lemma 22 For any θ ∈ R K , the problem (26) has a unique minimizer π ∗ ( θ ) ∈ Π( θ ) , which can be obtained in O ( K ) time. Denote by y ∗ an arbitrary element in argmax y ′ ∈ [ K ] θ y ′ , 7 then the minimizer

can be written as follows:

Proof. First, the Lagrangian of the minimization problem (26) is written as follows:

where α 1 , . . . , α K +1 ≥ 0 and β ∈ R are the Lagrangian multipliers. Since the log-sum-exp term is convex and differentiable and the feasible region ∆ K +1 is convex, the KKT conditions are necessary and sufficient for the optimality of π ∗ ∈ Π( θ ) , which requires that there exists ( α ∗ , β ∗ ) satisfying

We continue the proof with the following four observations:

Observation 1. By noting 1 -c > 0 . 5 , we have

Observation 2. From the KKT conditions, we have

̸

To see this, let us integrate the above left-hand side with (28) and (29):

̸

̸

which indicates that α ∗ K +1 > α ∗ y ∗ ≥ 0 , and thus π ∗ K +1 = 0 due to (30) and (31). On the other hand, for any y ∈ [ K ] \ { y ∗ } , we have

̸

and thus α ∗ y > α ∗ K +1 > 0 , which indicates that π ∗ y = 0 except for y = y ∗ due to (30) and (31). Thus, we verify π ∗ = e y ∗ .

Observation 3. From the KKT conditions, we have

To see this, we show the contraposition of Observation 3. Suppose there exists y ∈ [ K ] such that π ∗ y > 0 . Then, we have α ∗ y = 0 due to (31), and consequently

Then, for any y ′ ∈ [ K ] \ { y } , this inequality implies

which indicates that α ∗ y ′ > 0 and π ∗ y ′ = 0 due to (30) and (31). Then, we have

̸

where we used (32) at the last line. This inequality further implies

̸

Thus, the contraposition of Observation 3 is shown.

Observation 4. From the KKT conditions again, we have

̸

where γ ( θ ) = -ln ( ∑ K i =1 exp( θ y ) exp( θ y ∗ ) -1 ) -ln ( 1 -c c ) . To see this, we carefully examine π ∗ . First, if there exists different y ′ , y ′′ ∈ [ K ] with π y ′ , π y ′′ > 0 , we have

̸

̸

which is impossible since

̸

This contradiction indicates that there exists at most one y ∈ [ K ] such that π ∗ y > 0 . Second, we show y ∈ [ K ] with π ∗ y > 0 must be y = y ∗ . To see this, suppose there exists y ∈ [ K ] such that π ∗ y > 0 but y = y ∗ , then π ∗ y ′ = 0 for any y ′ ∈ [ K ] \ { y } and we have

̸

However, this contradicts the following inequality:

̸

Thus, we have verified that π ∗ y > 0 is possible with y = y ∗ or y = K +1 only. From this, we have

fulfill the KKT conditions under the assumption of Observation 4. By using (33), we have

̸

By elementary algebra and plugging in α ∗ = 0 given by (34), we can solve it with respect to π ∗ y ∗ as follows:

From now on, we show that

By rearranging the assumption of Observation 4, we have

from which we can see 0 ≤ π ∗ y ∗ = γ ( θ ) ≤ 1 . All in all, ( π ∗ , α ∗ , β ∗ ) in (34) fulfill the KKT conditions, and thus Observation 4 is verified.

By combining Observations 2, 3, and 4, we have shown the closed form of π ∗ ( θ ) . Furthermore, Observation 1 guarantees the uniqueness of π ∗ ( θ ) .

Lastly, it is easy to see that π ∗ ( θ ) can be computed in linear time because both of the following terms

̸

can be computed in linear time.

D.2 Multilabel Learning with Precision@ k

Problem setup. In multilabel learning, the target prediction space Y = [2 d ] is the collection of indices of all possible combinations of d binary labels with |Y| = K = 2 d . Precision@ k is a common performance metric for multilabel ranking. We consider that the prediction space ̂ Y = [ ( d k ) ] is the collection of indices of all possible sizek subsets of multilabels with | ̂ Y| = N = ( d k ) . In addition, we encode the label y ∈ Y into ρ ( y ) ∈ { 0 , 1 } d and the prediction t ∈ ̂ Y into µ ( t ) ∈ { 0 , 1 } d , where { µ ( t ) } N t =1 is the collection of all distinct permutations of ω = e 1 + · · · + e k . Then, the target loss of Precision@ k is defined as follows:

This is the portion of binary labels with value 0 in the topk list.

We adopt the decomposition (3) by using the aforementioned ρ ( y ) , ℓ ρ ( t ) = -µ ( t ) k , and c ( y ) = 1 for all y ∈ Y .

Loss formulation and calculation. For Precision@ k , we consider the base negentropy

Then, we have and

We can simplify it into the following problem:

where V := { v ∈ R d : v i ∈ [0 , 1] , ∥ v ∥ 1 = k } . Then, the convolutional Fenchel-Young loss can be written as follows:

To calculate the gradient of L Ω T ( · , y ) , we only need the solution v ∗ in the optimization problem (37), which can be obtained efficiently.

Algorithm 2 Exact Solution of (37) in O ( d ln d )

1: Set f ( λ ) = ∑ d i =1 max { 0 , min { θ i -λ, 1 }}

2: Sort ˜ θ = [ θ ; θ -1 ] ∈ R 2 d such that ˜ θ (1) ≥ · · · ≥ ˜ θ (2 d ) .

3: n ← max { n ∈ [2 d ] : f ( ˜ θ ( n ) ) < k } .

4: λ ∗ ← ˜ θ ( n ) + k -f ( ˜ θ ( n ) ) f ( ˜ θ ( n +1) ) -f ( ˜ θ ( n ) ) ( ˜ θ ( n +1) -˜ θ ( n ) ) .

5: v ∗ i ← max { 0 , min { θ i -λ ∗ , 1 }} .

Lemma 23 For any θ ∈ R d , the minimization problem (37) has a unique minimizer v ∗ , which can be obtained in O ( d log d ) time with Algorithm 2.

Proof. The uniqueness can be seen by noting that the objective function is strictly convex and its domain V is compact and convex.

Let us analyze the Lagrangian of the minimization problem (37), which can be written as follows:

where α ∈ R d ≥ 0 , β ∈ R d ≥ 0 , and γ ∈ R are the Lagrangian multipliers. Then, we have the following KKT conditions:

where the inequalities in (40) are element-wise.

First, we show that the KKT conditions are fulfilled by v ∗ i = max { 0 , min { θ i -λ ∗ , 1 }} for each i ∈ [ d ] , where λ ∗ := ln( γ ∗ / (1 -γ ∗ )) . Fixing i ∈ [ d ] , we divide into cases. If v ∗ i > θ i -λ ∗ , we have

which implies β ∗ i -α ∗ i < 0 by (39). Since β ∗ i > 0 by (40), we have α ∗ i > 0 , which implies v ∗ i = 0 due to (41). If v ∗ i < θ i -λ ∗ , we can show v ∗ i = 1 similarly. Thus, we have

If v ∗ i > θ i -λ ∗ , then v ∗ i = 0 ;

If v ∗ i < θ i -λ ∗ , then v ∗ i = 1 .

Moreover, let us divide into cases on θ i -λ ∗ . If θ i -λ ∗ < 0 , we must have v ∗ i > θ i -λ ∗ since v i ∈ [0 , 1] (due to the feasibility (40)), which yields v ∗ i = 0 . If θ i -λ ∗ > 1 , we have v ∗ i = 1 similarly. If 0 ≤ θ i -λ ∗ ≤ 1 , we have v ∗ i = θ i -λ ∗ otherwise it ends up with contradiction-say, supposing v ∗ i > θ i -λ ∗ , then we have 0 = v ∗ i > θ i -λ ∗ , which contradicts the premise θ i -λ ∗ ∈ [0 , 1] . Combining all above, we have verified that

fulfills the KKT conditions.

Next, we show that Algorithm 2 returns this v ∗ . By noting the constraint ∑ d i =1 v ∗ i = k in the feasibility conditions (40), we have

Thus, we need to solve f ( λ ∗ ) = k to obtain v ∗ . Let us sort the elements of ˜ θ := [ θ ; θ -1 ] ∈ R 2 d such that ˜ θ (1) ≥ ··· ≥ ˜ θ [2 d ] . The solution to f ( λ ∗ ) = k uniquely exists since we have

4

2

0

2

4

0

1

2

3

4

5

Loss value

Conv FY

Logistic

(a) Loss curves for y = +1 .

4

2

0

2

4

0.0

0.2

0.4

0.6

0.8

1.0

Probability Estimate

Sigmoid

Proposed

(b) Probability estimators

f ( ˜ θ (1) ) = 0 < k , f ( ˜ θ [2 d ] ) = d > k , and f is continuous and strictly decreasing on [ ˜ θ [2 d ] , ˜ θ (1) ] . From the strict decreasing nature, it can also be inferred that n in Step 3 of Algorithm 2 is the largest possible n ∈ [2 d ] that f ( ˜ θ ( n ) ) < k and f ( ˜ θ ( n +1) ) ≥ k , which indicates that the solution is in ( ˜ θ n +1 , ˜ θ ( n ) ] . Furthermore, f is linear on each segment [ ˜ θ [ i +1] , ˜ θ ( i ) ] . This indicates that for the point λ ∗ ∈ [ ˜ θ ( n +1) , ˜ θ ( n ) ] , we have

which yields Step 4 in Algorithm 2.

The time complexity of sorting ˜ θ is O ( d log d ) . Note that the computation of f ( z ) is in O ( d ) and n can be found using binary search due to monotonicity of f in O (log d ) steps, we can conclude that Step 3 is in O ( d log d ) .

E A Special Case: Visualization on Binary Classification

̸

In this section, we visualize the graphs of convolutional Fenchel-Young losses and the corresponding probability estimator (provided in Theorem 17) for binary classification to get more intuition. For binary classification, we adopt the class space Y = ̂ Y = {-1 , +1 } , and the target loss ℓ 01 ( y, t ) = [ [ y = t ] ] . We use the following decomposition of the target loss:

With this decomposition, surrogate losses can operate on a univariate prediction, which is convenient for the visualization purpose.

We compare convolutional Fenchel-Young losses with the standard Fenchel-Young losses generated by the binary Shannon negentropy:

where 1 2 + p =: η +1 is the positive class probability and 1 2 -p =: η -1 is the negative class probability. The corresponding convolutional negentropy is as follows:

where max { p, -p } = 1 2 -min { η +1 , η -1 } is the negative Bayes 0-1 risk of class probability ( η +1 , η -1 ) = ( 1 2 + p, 1 2 -p ) . Then, the conjugate of convolutional negentropy can be written

as follows:

Correspondingly, the convolutional Fenchel-Young loss is

We can explicitly write down the gradient of the conjugated entropy as follows:

which is the estimator of η +1 ρ (+1) + η -1 ρ ( -1) = η +1 -η -1 2 = η +1 -1+ η +1 2 = η +1 -1 2 by Theorem 17. Finally, we can use the link function ∇ θ Ω ∗ T ( θ ) + 1 2 as the estimator of η +1 .

We show the convolutional Fenchel-Young loss and logistic loss (which is the standard FenchelYoung loss generated by the binary Shannon negentropy) in Figure 1. It can be seen that the convolutional Fenchel-Young loss is linear in the shaded region θ ∈ [ -1 , 1] , while resembling the logistic loss outside of this region. Compared with the sigmoid function used for probability estimation with logistic loss, the link function induced by the convolutional Fenchel-Young loss also generates a valid probability estimate in [0 , 1] for any θ ∈ R , with θ ∈ [ -1 , 1] generates constant value 0 . 5 as the estimate.

Further discussion. In case of binary classification, Figure 1 (a) nicely illustrates that the convolutional Fenchel-Young loss linearly 'extends' the logistic loss at θ = 0 , which is the boundary point for binary classification. This illustration is possible because the above formulation for binary classification operates on the univariate score θ ∈ R . The linear extension at the boundary point aligns with Frongillo and Waggoner [39], which shows the square-root regret lower bound by assuming that a surrogate loss is locally strongly convex around the boundary points. In general prediction tasks, it is not straightforward to overcome the square-root regret lower bound by such a linear extension because the boundary points for a high-dimensional prediction task can be infinitely many. By contrast, convolutional Fenchel-Young losses provide a general recipe to get linear surrogate regret bound via infimal convolution.

F Additional Empirical Results

F.1 Multiclass Classification

To provide empirical validation of our proposed results, we use the loss introduced in Section 4 as an example and evaluate its performance on the ImageNet-1k dataset [32] using the ResNet-50 architecture [43].

Experimental Setup. We follow the default configuration of the PyTorch [71] ImageNet training script. Specifically, we use stochastic gradient descent (SGD) with a momentum of 0 . 9 , training for 120 epochs with a mini-batch size of 256 . The initial learning rate is set to 0 . 1 and divided by 10 every 30 epochs. For comparison, we also report results obtained using the standard cross-entropy loss under the same configuration. All experiments are conducted on 8 GeForce RTX 3090 GPUs, and we report the average validation accuracy over 3 independent runs.

Results and Discussion. As shown in Table 1, our proposed loss slightly outperforms the standard cross-entropy loss in terms of validation accuracy under a 5% t-test,. This improvement is achieved without any additional hyperparameter tuning, demonstrating the potential of our approach. Meanwhile, the computation time per epoch remains comparable between the two methods, which indicates the efficiency of the proposed loss. These results confirm the compatibility of our loss with both mini-batch and distributed optimization settings.

F.2 Classification with Rejection

We further evaluate the proposed loss (27) under the classification with rejection (CwR) setting, where the rejection cost is fixed at c = 0 . 05 . The experiments are conducted on the CIFAR-10 and CIFAR-100 datasets [46]. We adopt the WideResNet-28 architecture [96] with a widen factor of 4 and a hidden dimension of 50 as the backbone network for all experiments.

Experimental Setup. We train each model for 120 epochs on 8 GeForce RTX 3090 GPUs with a per-GPU batch size of 128. The optimizer is SGD with a momentum of 0.9, weight decay of 5 e -4 , and an initial learning rate of 0.1. The learning rate is scheduled by cosine annealing.

For data augmentation, each image is first converted to a tensor, then padded by 4 pixels on each side using reflection padding, followed by random cropping to 32 × 32 and random horizontal flipping. The images are finally normalized using the dataset-specific mean and standard deviation. We report the average system accuracy (1-averaged 0-1-c loss) and acceptance rate over 5 runs. Meanwhile, we also provide the result of ordinary classification using the similar loss (18) we proposed.

Results and Discussion. As shown in Table 2, both CIFAR-10 and CIFAR-100 experiments demonstrate that our proposed loss successfully performs classification with rejection, achieving higher system accuracy while maintaining a reasonable acceptance rate. This indicates that the model learns to reject uncertain samples effectively without sacrificing overall performance. In addition, the average computation time per epoch is even lower than that of ordinary classification, which we attribute to the O ( K ) complexity of gradient computation for the proposed formulation, compared with the O ( K log K ) cost in standard classification. These results confirm the efficiency and practicality of our loss for reliable decision-making under uncertainty.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies *surrogate regret bounds* in the context of multiclass prediction problems. Given a *target loss* $\\ell: \\mathcal{Y} \\to R$, a *prediction link* $\\varphi: R^{\\varrho} \\to \\mathcal{Y}$ and a class distribution $\\eta \\in \\Delta_{N}$, the authors show that there is a surrogate loss $L$ satisfying the following regret bound:\n\n$$\\mathtt{Regret}(\\ell \\circ \\varphi; \\eta) \\lesssim \\mathtt{Regret}(L; \\eta).$$\n\nThe construction is via the framework of so-called *Fenchel-Young* losses. Given a CCP regularizer $\\Omega$,\n\n$$L_{\\Omega}(\\mu, \\theta) = \\Omega(\\mu) + \\Omega^*(\\theta) - \\langle \\mu, \\theta \\rangle \\geq 0$$\n\nis the associated Fenchel-Young loss. The paper shows how to adapt a given regularizer $\\Omega$ to a target loss $\\ell$ so that:\n- the induced Fenchel-Young loss satisfies a linear surrogate bound;\n- the induced Fenchel-Young loss is convex and differentiable (or even smooth when $\\Omega$ is strongly convex).\n\nIn particular, it is well-known in the literature that smooth losses achieve fast excess loss bounds for ERM (Srebro et al., NeurIPS 2010), but transferring these guarantees to the target loss was thought to suffer from square-root regret bounds.\n\nTo achieve this, the authors introduce a so-called *convolutional negentropy*, which is a sum of a base negentropy $\\Omega$ and an affinely-transformed version of the Bayes risk. Under this choice, the surrogate regret can be written as the sum of the Fenchel-Young loss risk and a convex combination of the *target* regret, which yields a linear upper bound on the surrogate loss after rearranging. The authors present a strengthened version of the upper bound, where the constant only depends on the affine dimension of the matrix of concatenated loss vectors.\n\nSTRENGTHS AND WEAKNESSES:\n**Note**: The literature on surrogate regret bounds is far from my area of research. As such, I primarily reviewed this paper for correctness. \n\nStrengths:\n- The problem and proposed solution are clearly stated.\n- The contribution appears to be novel (based on my limited knowledge of the literature).\n\nWeaknesses:\n- Notation can be confusing at times.\n- I was expecting a discussion of optimization tradeoffs. My understanding is that while polyhedral losses lacking smoothness may not achieve fast rates for excess risk, their structure should allow for faster optimization (except maybe in the stochastic optimization setting, assuming no interpolation).\n\nQUESTIONS:\nMy main question was already posed in the \"strengths and weaknesses\" section. Other minor questions / suggestions:\n\n- My understanding is that $T$ is used to \"couple\" the target loss $\\ell$ with the base negentropy $\\Omega$. Are there other ways of systematically producing such couplings? If so, do some of these lead to less tractable losses?\n- p. 3, line 120: \"$\\mathbf{dom}(\\Omega)$\" should be $\\mathbf{dom}(f)$.\n- Definition 10: $\\pi$ simultaneously denotes a selector function and vector in the simplex. The authors may want to consider alternative notation for the selector to avoid confusion.\n- Similarly, $\\rho$ and $\\varrho$ are used with very different meanings (label encoding vs. dimension of the encoding).\n\nLIMITATIONS:\nSee my comment under Strengths and Weaknesses.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper demonstrates that the believed tradeoff between smoothness and linear regret rate of a surrogate loss is not true. This paper shows that for an arbitrary discrete target loss, there exists a smooth surrogate loss that facilitates linear regret. This result is presented constructively, allowing for its use. Beyond this core result, this paper presents an extended tightness result and demonstrates the proposed construction for multi-class classification.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths\n1. Answers decisively that a belief that a tradeoff of linear regret and smoothness is not true for convex surrogates. This is a novel and insightful result in the convex surrogate literature.\n2. Furthers the established literature regarding Fenchal-young losses and their applications. \n3. The results are practically constructive (which is not always the case in the convex surrogate literature), allowing for the use of these novel insights. \n4. The paper is well-written, given the technical density behind many of the ideas being used.\n\nWeaknesses\n\nAlthough the results are clearly presented, rigorously proven, novel, and valuable, I find the underlying inspiration and intuition that facilitated said results to be opaque. From an educational standpoint, I would have found it intriguing to get insight into how this very, very specific approach was conceived while other researchers considered this tradeoff infeasible.\n\nQUESTIONS:\n1. Could you comment on what inspired examining Convolutional Fenchel–Young Losses as a candidate loss which is smooth that satisfies linear regret rates?\n2. To help with the presentation, have you considered adding any plots which might give insight into the geometry?\n3. Section 3.6 talks about a Fisher-consistent estimator, could you add to the appendix a formal definition?\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-02"}

comments: SUMMARY:
This paper presents a theoretical contribution to machine learning by constructing convex smooth surrogate losses that achieve linear surrogate regret bounds for discrete prediction problems. This provides a theoretical contribution a long-standing belief in the community that there exists a fundamental trade-off between smoothness and linear regret bounds.

STRENGTHS AND WEAKNESSES:
The construction of convolutional Fenchel-Young losses through infimal convolution is elegant and seems per se theoretically sound. The key innovation lies in encoding the target loss structure into the base negentropy via the polyhedral function $T(\mathbf{p})$, which enables the additive decomposition crucial for achieving linear regret bounds.

While this is primarily a theoretical paper, the complete absence of experiments is notable. Even simple synthetic experiments comparing convergence rates of the proposed losses against existing smooth losses (logistic, squared) and non-smooth losses (hinge) would strengthen the practical relevance, and validate the results from an empirical point of view. 

The computational complexity of the approach is not discussed, at least not in a way that would clarify how expensive the approach is. In particular, in relation to standard loss computations.

QUESTIONS:
Could you provide experimental evaluations? If not, could you explain why?

What are reasonable values for the constants that imply computational complexity?

Could you characterize problems where your approach would be inferior to existing methods? E.g., when $N$ is small or when computational resources are limited?

How do these losses behave under the stochastic setting of SGD with mini-batches?

LIMITATIONS:
The lack of empirical evaluation very much weakens the paper.
TITLE: Official Review
CLARITY: 2
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-07-12

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nPrevious work has shown that while convex and smooth surrogate losses are effective for estimation and optimization, they often face a trade-off between smoothness and achieving linear regret bounds. This paper overcomes that trade-off by introducing a novel surrogate loss that is both convex and smooth, yet still admits linear surrogate regret bounds. A convolutional negentropy is constructed by encoding the structure of the target loss into a chosen base negentropy, resulting in a convolutional Fenchel–Young loss that inherits both convexity and smoothness. Crucially, the derived prediction link, when paired with this loss, enables a linear surrogate regret bound and provides a consistent probability estimator.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\n- Proposes a novel surrogate loss that is both convex and smooth while preserving linear surrogate regret bounds.\n\n- The writing and presentation are clear and well-structured.\n\nWeaknesses:\n\n- The paper would be stronger with the inclusion of empirical results to support the theoretical claims.\n\nQUESTIONS:\n1. While the theoretical contributions are compelling, I believe the paper’s strengths would be significantly enhanced with empirical validation. For instance, the authors could evaluate the proposed surrogate loss on toy datasets or simple multiclass classification benchmarks with the algorithm presented in Section 4. This would help illustrate the practical impact of the loss and clarify its advantages over existing approaches.\n\n2. I found the definition and role of the term $T(p)$ in Equation (8) somewhat confusing. In line 69, the authors state: *“We encode the structure of a target loss $\\ell$ into the chosen base negentropy by adding the negative Bayes risk of $\\ell$”*, and in lines 210–211, they mention: *“This polyhedral convex function $T$ is an affinely transformed Bayes risk of a target loss $\\ell$”*. While Equation (9) shows that $T $ is an affinely transformed Bayes risk when $p = \\mathbb{E}_{y \\sim \\eta}[\\rho(y)] $, this does not seem to align with Equation (11), where $ p = \\rho(y) $. Could the authors clarify this discrepancy and more precisely define the conditions under which the interpretation of $ T(p) $ holds?\n\n3. Minor points:  \n   - Lines 256 and 297: The term *“Theorem”* should be replaced with *“Corollary”*.  \n   - Line 282: “The regret bound (13)” should be rewritten as “The regret bound in Theorem (13)” or “The regret bound in Eq. (14)”.  \n   - Equation (18): The notation $ \\theta_i $ should be corrected to $\\theta_y$.\n\nLIMITATIONS:\nPlease see the above questions.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-13"}