Title: Learning with Fitzpatrick Losses
Venue: NeurIPS 2024
Accepted: yes
Keywords: loss functions

convex analysis

monotone operators
Abstract: Fenchel-Young losses are a family of loss functions, encompassing the squared,
logistic and sparsemax losses, among others. They are convex w.r.t. the model
output and the target, separately. Each Fenchel-Young loss is implicitly associated
with a link function, that maps model outputs to predictions. For instance, the
logistic loss is associated with the soft argmax link function. Can we build new
loss functions associated with the same link function as Fenchel-Young losses?
In this paper, we introduce Fitzpatrick losses, a new family of separately convex
loss functions based on the Fitzpatrick function. A well-known theoretical tool in
maximal monotone operator theory, the Fitzpatrick function naturally leads to a
refined Fenchel-Young inequality, making Fitzpatrick losses tighter than Fenchel-
Young losses, while maintaining the same link function for prediction. As an
example, we introduce the Fitzpatrick logistic loss and the Fitzpatrick sparsemax
loss, counterparts of the logistic and the sparsemax losses. This yields two new
tighter losses associated with the soft argmax and the sparse argmax, two of the
most ubiquitous output layers used in machine learning. We study in details the
properties of Fitzpatrick losses and, in particular, we show that they can be seen as
Fenchel-Young losses using a modified, target-dependent generating function. We
demonstrate the effectiveness of Fitzpatrick losses for label proportion estimation.

Learning with Fitzpatrick Losses

Abstract

Fenchel-Young losses are a family of loss functions, encompassing the squared, logistic and sparsemax losses, among others. They are convex w.r.t. the model output and the target, separately. Each Fenchel-Young loss is implicitly associated with a link function, that maps model outputs to predictions. For instance, the logistic loss is associated with the soft argmax link function. Can we build new loss functions associated with the same link function as Fenchel-Young losses? In this paper, we introduce Fitzpatrick losses, a new family of separately convex loss functions based on the Fitzpatrick function. A well-known theoretical tool in maximal monotone operator theory, the Fitzpatrick function naturally leads to a refined Fenchel-Young inequality, making Fitzpatrick losses tighter than FenchelYoung losses, while maintaining the same link function for prediction. As an example, we introduce the Fitzpatrick logistic loss and the Fitzpatrick sparsemax loss, counterparts of the logistic and the sparsemax losses. This yields two new tighter losses associated with the soft argmax and the sparse argmax, two of the most ubiquitous output layers used in machine learning. We study in details the properties of Fitzpatrick losses and, in particular, we show that they can be seen as Fenchel-Young losses using a modified, target-dependent generating function. We demonstrate the effectiveness of Fitzpatrick losses for label proportion estimation.

1 Introduction

Loss functions are a cornerstone of statistics and machine learning: they measure the difference, or 'loss,' between a ground-truth target and a model prediction. As such, they have attracted a wealth of research. Proper losses (a.k.a. proper scoring rules) [17, 16] measure the discrepancy between a target distribution and a probability forecast. They are essentially primal-primal Bregman divergences, with both the target and the prediction belonging to the same primal space. They are typically explicitly composed with a link function [26, 30], in order to map the model output to a prediction. A disadvantage of this explicit composition is that it often makes the resulting composite loss function non-convex. A related family of loss functions are Fenchel-Young losses [7, 8], which encompass many commonly-used loss functions in machine learning including the squared, logistic, sparsemax and perceptron losses. Fenchel-Young losses can be seen as primal-dual Bregman divergences [1], with the target belonging to the primal space and the model output belonging to the dual space. In contrast to proper losses, each Fenchel-Young loss is implicitly associated with a given link function, mapping the dual-space model output to a primal-space prediction (for instance, the soft argmax is the link function associated with the logistic loss). This crucial difference makes Fenchel-Young losses always convex w.r.t. the model output and w.r.t. the target, separately. Can we build new (separately) convex losses associated with the same link function as Fenchel-Young losses?

3

2

1

0

1

2

3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Logistic loss

Fitzpatrick logistic loss

3

2

1

0

1

2

3

0.0

0.5

1.0

1.5

2.0

2.5

3.0

Sparsemax loss

Fitzpatrick sparsemax loss

s

s

In this paper, we introduce Fitzpatrick losses, a new family of primal-dual (separately) convex loss functions. Our proposal builds upon the Fitzpatrick function, a well-known theoretical object in maximal monotone operator theory [15, 11, 2]. So far, the Fitzpatrick function had been used as a theoretical tool to represent maximal monotone operators [28] and to construct Bregman-like primal-primal divergences [10], but it had not been used to construct primal-dual loss functions for machine learning, as we do. Crucially, the Fitzpatrick function naturally leads to a refined Fenchel-Young inequality, making Fitzpatrick losses tighter than Fenchel-Young losses. Yet, their predictions are produced using the same link function, suggesting that we can use Fitzpatrick losses as a tighter replacement for the corresponding Fenchel-Young losses (Figure 1). We make the following contributions.

After reviewing some background, we introduce Fitzpatrick losses. They can be thought as a tighter version of Fenchel-Young losses, that use the same link function.

Weinstantiate two new loss functions in this family: the Fitzpatrick logistic loss and the Fitzpatrick sparsemax loss. They are the counterparts of the logistic and sparsemax losses, two instances of Fenchel-Young losses. We therefore obtain two new tighter losses for the soft argmax and the sparse argmax, two of the most popular output layers in machine learning.

We study in detail the properties of Fitzpatrick losses. We show that Fitzpatrick losses are equivalent to Fenchel-Young losses with a modified, target-dependent generating function.

We demonstrate the effectiveness of Fitzpatrick losses for probabilistic classification on 11 datasets.

2 Background

2.1 Convex analysis

We denote the non-negative real numbers by R + := [0 , + ∞ ) , the positive real numbers by R ++ := (0 , + ∞ ) , and the extended real numbers by R := R ∪ {-∞ , + ∞} . We suppose given a nonzero natural number k . We denote the probability simplex by △ k := { p ∈ R k + ∣ ∣ ∑ k i =1 p i = 1 } . We denote the indicator function of a set C ⊂ R k by ι C ( y ) = 0 if y ∈ C , + ∞ otherwise. We denote the effective domain of a function Ω: R k → R by domΩ := { y ∈ R k ∣ ∣ Ω( y ) < + ∞} . A function Ω: R k → R is said to be proper if it never takes the value -∞ and if domΩ = ∅ . We denote the Euclidean projection onto a nonempty closed convex set C ⊂ R k by P C ( θ ) , the unique solution y ′ ∈ C of the minimization problem min y ′ ∈C ∥ y ′ -θ ∥ 2 2 .

̸

For a function Ω : R k → R , its subdifferential ∂ Ω ⊂ R k × R k is defined by

When Ω is convex and differentiable, the subdifferential is a singleton and we have ∂ Ω( y ′ ) = {∇ Ω( y ′ ) } . The normal cone to a nonempty closed convex set C ⊂ R k at y ′ is defined by

if y ′ ∈ C and N C ( y ′ ) = ∅ if y ′ ̸∈ C . The Fenchel conjugate Ω ∗ : R k → R of a function Ω : R k → R is defined by

From standard results in convex analysis [27, Proposition 11.3], when Ω : R k → R is a proper convex l.s.c. (lower semicontinuous) function,

When the argmax is unique, it is equal to ∇ Ω ∗ ( θ ) . We define the generalized Bregman divergence [18] D Ω : R k × R k → R + generated by a proper convex l.s.c. function Ω : R k → R , by with the convention + ∞ +( -∞ ) = + ∞ (we comment on this convention in Appendix C). When Ω is differentiable, Equation (1) gives the classical Bregman divergence

Both y and y ′ belong to the primal space .

2.2 Fenchel-Young losses

Definition and properties

The Fenchel-Young loss [8] L Ω ⊕ Ω ∗ : R k × R k → R generated by a proper convex l.s.c. function Ω is

As its name indicates, it is grounded in the Fenchel-Young inequality

The Fenchel-Young loss enjoys many desirable properties, notably it is non-negative and it is separately convex in y and θ . The Fenchel-Young loss can be seen as a primal-dual Bregman divergence [1, 8], where y belongs to the primal space and θ belongs to the dual space.

Link functions

To map an element θ of a dual space to an element y of a primal space, we define the link function (potentially set-valued) associated with the loss L by

Given a proper convex function Ω , the associated Fenchel-Young loss L Ω ⊕ Ω ∗ produces the canonical link function ∂ Ω ∗ , since

In particular when Ω is strictly convex, and thus Ω ∗ is differentiable according to [27, Theorem 11.13], the Fenchel-Young loss satisfies the identity of indiscernibles

In the remainder of this paper, we will use the notation ̂ y Ω ( θ ) for the canonical link function ∂ Ω ∗ ( θ ) . When the function Ω ∗ is differentiable, ̂ y Ω ( θ ) will denote ∇ Ω ∗ ( θ ) . Since Ω ∗ is convex, ̂ y Ω is monotone (see §2.3). As shown in [8], the monotonicity implies that θ and ̂ y Ω ( θ ) are sorted the same way, i.e., θ i > θ j = ⇒ ̂ y Ω ( θ ) i ≥ ̂ y Ω ( θ ) j . Link functions also play an important role in the loss subgradients (and in the loss gradient when it is differentiable), as we have

Examples of Fenchel-Young loss instances and their associated link function

We give a few examples of Fenchel-Young losses. With the squared 2 -norm, Ω( y ′ ) = 1 2 ∥ y ′ ∥ 2 2 , we obtain the squared loss

With the indicator of a nonempty closed convex set C , Ω( y ′ ) = ι C ( y ′ ) , we obtain the perceptron loss and the argmax link

With the squared 2 -norm restricted to some nonempty closed convex set C , Ω( y ′ ) = 1 2 ∥ y ′ ∥ 2 2 + ι C ( y ′ ) , we obtain the sparseMAP loss [24]

and the link is the Euclidean projection onto C ,

When the set is C = △ k , we obtain the sparsemax loss [21] and the sparse argmax link ̂ y Ω ( θ ) = P △ k ( θ ) (also known as sparsemax), which is known to produce sparse probability distributions.

With the Shannon negentropy restricted to the probability simplex, Ω( y ′ ) := ⟨ y ′ , log y ′ ⟩ + ι △ k ( y ′ ) , we obtain the logistic loss

and the soft argmax link (also known as softmax)

2.3 Maximal monotone operators and the Fitzpatrick function

An operator A , that is, a subset A ⊂ R k × R k , is called monotone if for all ( y, θ ) ∈ A and all ( y ′ , θ ′ ) ∈ A , we have

We overload the notation to denote A ( y ) := { θ ∈ R k ∣ ∣ ( y, θ ) ∈ A } . A monotone operator A is said to be maximal if there does not exist ( y, θ ) ̸∈ A such that A ∪ { ( y, θ ) } is still monotone. It is well-known that the subdifferential ∂ Ω of a proper convex l.s.c. function Ω is maximal monotone. For more details on monotone operators, see [3, 28].

A well-known object in monotone operator theory, the Fitzpatrick function associated with a maximal monotone operator A [15, 11, 2], denoted F [ A ] : R k × R k → R , is defined by

In particular, with A = ∂ Ω , we have

and the identity link

The Fitzpatrick function was studied in depth in [2]. In particular, it is (jointly) convex and satisfies

We introduce the operator y ⋆ F [ ∂ Ω] ⊂ ( R k × R k ) × R k , associated to the Fitzpatrick function F [ ∂ Ω] :

As proven for Item 4 in Proposition 1, we have y ⋆ F [ ∂ Ω] ( y, θ ) ⊂ ∂ θ F [ ∂ Ω]( y, θ ) . For the rest of the paper, in case that the argmax in (4) is a singleton { y ⋆ } , we will write y ⋆ F [ ∂ Ω] ( y, θ ) := y ⋆ . The Fitzpatrick function F [ ∂ Ω]( y, θ ) and ( Ω ⊕ Ω ∗ ) ( y, θ ) = Ω( y )+Ω ∗ ( θ ) play a similar role but the latter function is separable in y and θ , while the former is not . In particular this makes the subdifferential ∂ θ F [ ∂ Ω]( y, θ ) depend on both y and θ , while ∂ θ (Ω ⊕ Ω ∗ )( y, θ ) = ∂ Ω ∗ ( θ ) depends only on θ .

The Fitzpatrick function was used in [10] to theoretically study primal-primal Bregman-like divergences. As discussed in more detail in Section 3.4, using these divergences for machine learning would require us to compose them with an explicit link function, which would typically break convexity. In the next section, we introduce new primal-dual losses based on the Fitzpatrick function.

3 Fitzpatrick losses

3.1 Definition and properties

Inspired by the inequality in (3), which we can view as a refined Fenchel-Young inequality, we introduce Fitzpatrick losses, a new family of loss functions generated by a convex l.s.c. function Ω .

Definition 1 Fitzpatrick loss

Let Ω : R k → R be a proper convex l.s.c. function. When y ∈ domΩ and θ ∈ R k , we define the Fitzpatrick loss L F [ ∂ Ω] : R k × R k → R generated by Ω as

When y ̸∈ domΩ , L F [ ∂ Ω] ( y, θ ) := + ∞ .

Fitzpatrick losses enjoy similar properties as Fenchel-Young losses, while being tighter than FenchelYoung losses.

Proposition 1 Properties of Fitzpatrick losses

Non-negativity: for all ( y, θ ) ∈ R k × R k , L F [ ∂ Ω] ( y, θ ) ≥ 0 .

Same link function: L Ω ⊕ Ω ∗ ( y, θ ) = L F [ ∂ Ω] ( y, θ ) = 0 ⇐⇒ y ∈ ̂ y Ω ( θ ) .

Separable convexity: L F [ ∂ Ω] ( y, θ ) is separately convex.

(Sub-)Gradient: ∂ θ L F [ ∂ Ω] ( y, θ ) ⊃ y ⋆ F [ ∂ Ω] ( y, θ ) -y where y ⋆ F [ ∂ Ω] ( y, θ ) is given by (4) .

Tighter inequality: for all ( y, θ ) ∈ R k , 0 ≤ L F [ ∂ Ω] ( y, θ ) ≤ L Ω ⊕ Ω ∗ ( y, θ ) .

A proof is given in Appendix B.2. Because the Fitzpatrick loss and the Fenchel-Young loss generated by the same Ω have the same link function ̂ y Ω , they share the same minimizers w.r.t. θ for y fixed. However, the Fitzpatrick loss is always a lower bound of the corresponding Fenchel-Young loss. Moreover, they have different gradients w.r.t. θ : ∂ θ L Ω ⊕ Ω ∗ ( y, θ ) = ̂ y Ω ( θ ) -y vs. ∂ θ L F [ ∂ Ω] ( y, θ ) ⊃ y ⋆ F [ ∂ Ω] ( y, θ ) -y . It is worth noticing that y ⋆ F [ ∂ Ω] ( y, θ ) depends on both y and θ , contrary to ̂ y Ω ( θ ) .

When Ω is a twice differentiable function on its domain (which is for instance the case of the squared 2 -norm or the negentropy), we next show that Fitzpatrick losses enjoy a particularly simple expression and become a squared Mahalanobis-like distance.

Proposition 2 Expressions of F [ ∂ Ω]( y, θ ) and L F [ ∂ Ω] ( y, θ ) when Ω is twice differentiable

Let Ω : R k → R be a convex function such that dom Ω is an open set. Let us assume that Ω is twice differentiable. Then, for any y ∈ domΩ and for any y ⋆ ∈ y ⋆ F [ ∂ Ω] ( y, θ ) , as defined in (4) , we have that

and

A proof is given in B.3. When Ω is constrained (i.e., when it contains an indicator function), we show in §3.5 that the above expression becomes a lower bound.

3.2 Examples

We now present the Fitzpatrick loss counterparts of various Fenchel-Young losses.

Squared loss.

Proposition 3 Squared loss as a Fitzpatrick loss

When Ω( y ′ ) = 1 2 ∥ y ′ ∥ 2 2 , we have for all y ∈ R k and θ ∈ R k

A proof is given in Appendix B.4. Therefore, the Fenchel-Young and Fitzpatrick losses generated by Ω coincide, but up to a factor 1 2 .

Perceptron loss.

Proposition 4 Perceptron loss as a Fitzpatrick loss

When Ω( y ′ ) = ι C ( y ′ ) , where C is a nonempty closed convex set, we have for all y ∈ C and θ ∈ R k

A proof is given in Appendix B.5. Therefore, the Fenchel-Young and Fitzpatrick losses generated by Ω exactly coincide in this case.

Fitzpatrick sparseMAP and Fitzpatrick sparsemax losses. As our first example where FenchelYoung and Fitzpatrick losses substantially differ, we introduce the Fitzpatrick sparseMAP loss, which is the Fitzpatrick counterpart of the sparseMAP loss [24].

Proposition 5 Fitzpatrick sparseMAP loss

When Ω( y ′ ) = 1 2 ∥ y ′ ∥ 2 2 + ι C ( y ′ ) , where C is a nonempty closed convex set, we have for all y ∈ C and θ ∈ R k where we used y ⋆ as a shorthand for

A proof is given in Appendix B.6. As a special case, when C = △ k , we call the obtained loss the Fitzpatrick sparsemax loss , as it is the counterpart of the sparsemax loss [21]. Like the sparseMAP and sparsemax losses, these new losses rely on the Euclidean projection as a core building block. The Euclidean projection onto the probability simplex △ k can be computed exactly in O ( k ) expected time and O ( k log k ) worst-case time [9, 22, 14, 12].

Fitzpatrick logistic loss. We now derive the Fitzpatrick counterpart of the logistic loss. Before stating the next proposition, we recall the definition of the Lambert function [13] W : R + → R + . The function W is the inverse of the function f : R + → R + where f ( w ) = w exp( w ) for all w ∈ R + .

Proposition 6 Fitzpatrick logistic loss

where we used y as a shorthand for y F [ ∂ Ω] ( y, θ ) defined by

A proof and the value of λ ⋆ = λ ⋆ ( y, θ ) ∈ R are given in Appendix B.7. To obtain λ ⋆ ( y, θ ) , we need to solve a one-dimensional root equation, which can be done using, for instance, a bisection.

3.3 Relation with Fenchel-Young losses

On first sight, Fitzpatrick losses and Fenchel-Young losses appear quite different. In the next proposition, we show that the Fitzpatrick loss generated by Ω is in fact equal to the Fenchel-Young loss generated by the modified, target-dependent function

where D Ω is the generalized Bregman divergence defined in (1). In particular, Lemma 1 in the appendix shows that if Ω = Ψ + ι C , where C is a nonempty closed convex set, then Ω y ( y ′ ) = Ψ y ( y ′ ) + ι C ( y ′ ) , where Ψ y ( y ′ ) := Ψ( y ′ ) + D Ψ ( y, y ′ ) .

Proposition 7 Characterization of F [ ∂ Ω] , L F [ ∂ Ω] and y ⋆ F [ ∂ Ω] using Ω y

Let Ω : R k → R be a proper convex l.s.c. function. Then, for all y ∈ domΩ and all θ ∈ R k ,

This characterization of the Fitzpatrick function F [ ∂ Ω] is also new to our knowledge. A proof is given in Appendix B.8. Proposition 7 is very useful, as it means that Fitzpatrick losses inherit from all the known properties of Fenchel-Young losses, analyzed in prior works [8, 6]. In particular, FenchelYoung losses are smooth (i.e., with Lipschitz gradients) when Ω is strongly convex. We therefore immediately obtain that Fitzpatrick losses are smooth in their second argument θ if Ω is strongly convex and D Ω is convex in its second argument, which is the case when Ω( y ′ ) = 1 2 ∥ y ′ ∥ 2 2 and Ω( y ′ ) = ⟨ y ′ , log y ′ ⟩ . Therefore, the Fitzpatrick sparsemax and logistic losses are smooth. However in the general case, this does not hold, as D Ω is usually not convex in its second argument. Proposition 7 also provides a mean to compute Fitzpatrick losses and their gradient. Finally, it suggests a very natural geometric interpretation of Fitzpatrick losses, as presented in Figure 2.

3.4 Relation with generalized Bregman divergences

As we stated before, the generalized Bregman divergence D Ω ( y, y ′ ) in (1) is a primal-primal divergence, as both y and y ′ belong to the same primal space. In contrast, Fenchel-Young losses

Ω

y

Ω

y

Ω(

y

) = Ω

y

(

y

)

〈

y, θ

〉 -

Ω

∗

(

θ

)

〈

y, θ

〉 -

Ω

∗

y

(

θ

)

L

F

[

∂

Ω]

(

y, θ

)

=

L

Ω

y

⊕

Ω

∗

y

(

y, θ

)

L

Ω

⊕

Ω

∗

(

y, θ

)

-

Ω

∗

(

θ

)

-

Ω

∗

y

(

θ

)

L Ω ⊕ Ω ∗ ( y, θ ) are primal-dual, since y belongs to the primal space and θ belongs to the dual space. Both can however be related for any proper l.s.c. convex function Ω , since, for any y ′ such that ∂ Ω( y ′ ) = ∅ , we have

̸

where, in the penultimate line, we have used that Ω ∗ ( θ ′ ) = ⟨ y ′ , θ ′ ⟩ -Ω( y ′ ) , as θ ′ ∈ ∂ Ω( y ′ ) . This equality remains true when ∂ Ω( y ′ ) = ∅ , by convention inf ∅ = + ∞ and by definition of D Ω ( y, y ′ ) in (1). This identity suggests that we can create Bregman-like primal-primal divergences by replacing Ω ⊕ Ω ∗ with F [ ∂ Ω] ,

This recovers one of the two Bregman-like divergences proposed in [10], the other one replacing the inf above by a sup . As stated in [10], F [ ∂ Ω] and Ω ⊕ Ω ∗ are representations of ∂ Ω .

In order to use a primal-primal divergence as a loss, we need to explicitly compose it with a link function, such as ̂ y Ω ( θ ) = ∇ Ω ∗ ( θ ) . Unfortunately, D Ω ( y, ̂ y Ω ( θ )) or D F [ ∂ Ω] ( y, ̂ y Ω ( θ )) are typically non convex functions of θ , while Fenchel-Young and Fitzpatrick losses are always convex in θ . In addition, differentiating through ̂ y Ω ( θ ) typically requires implicit differentiation [19, 5], while Fenchel-Young and Fitzpatrick losses enjoy easy-to-compute gradients, thanks to Danskin's theorem.

3.5 Lower bound on Fitzpatrick losses

If Ω = Ψ + ι C , where Ψ : R k → R is a proper convex Legendre-type function and C ⊆ domΨ is a nonempty closed convex set, then it was shown in [8, Proposition 3] that Fenchel-Young losses

satisfy the lower bound

with equality if C = domΨ . We now show that a similar result holds for Fitzpatrick losses. Similarly as before, we define Ψ y ( y ′ ) := Ψ( y ′ ) + D Ψ ( y, y ′ ) .

Proposition 8 Lower bound on Fitzpatrick losses

Let Ω = Ψ + ι C , where Ψ : R k → R is a proper convex Legendre-type function, as defined in [8, Definition 3], and C ⊆ domΨ is a nonempty closed convex set. We remind that Ω y ( y ′ ) := Ω( y ′ ) + D Ω ( y, y ′ ) . Let us assume that Ω ∗ y is differentiable. Then,

with equality if domΨ = C , where we used y ⋆ as a shorthand for ∇ Ω ∗ y ( θ ) .

A proof is given in Appendix B.9. If Ψ y is µ -strongly convex, we obtain µ 2 ∥ y -y ⋆ ∥ 2 2 ≤ D Ψ y ( y, y ⋆ ) .

4 Experiments

Experimental setup. We follow exactly the same experimental setup as in [7, 8]. We consider a dataset of n pairs ( x i , y i ) of feature vectors x i ∈ R d and label proportions y i ∈ △ k , where d is the number of features and k is the number of classes. At inference time, given an unknown input vector x ∈ R d , our goal is to estimate a vector of label proportions ̂ y ∈ △ k . A model is specified by a matrix W ∈ R k × d and a convex l.s.c. function Ω : R k → R . Predictions are then produced by the generalized linear model x ↦→ ̂ y Ω ( Wx ) . At training time, we estimate the matrix W ∈ R k × d by minimizing the convex objective

where L ∈ { L Ω ⊕ Ω ∗ , L F [ ∂ Ω] } . We focus on the (Fitzpatrick) sparsemax and the (Fitzpatrick) logistic losses. We optimize (5) using the L-BFGS algorithm [20]. The gradient of the Fenchel-Young loss is given in (2), while the gradient of the Fitzpatrick loss is given in Proposition 1, Item 4. Experiments were conducted on a Intel Xeon E5-2667 clocked at 3.30GHz with 192 GB of RAM running on Linux. Our implementation relies on the SciPy [29] and scikit-learn [25] libraries.

We ran experiments on 11 standard multi-label benchmark datasets 1 (see Table 2 in Appendix A for statistics on the datasets). For all datasets, we removed samples with no label, normalized samples to have zero mean unit variance, and normalized labels to lie in the probability simplex. In (5), we chose the hyperparameter λ ∈ { 10 -4 , 10 -3 , . . . , 10 4 } against the validation set. We report test set mean squared error (also known as Brier score in a probabilistic forecasting context) in Table 1.

Results. We found that the logistic loss and the Fitzpatrick logistic loss are comparable on most datasets, with the logistic loss significantly winning on 2 datasets and the Fitzpatrick logistic loss significantly winning on 2 datasets, out of 11 . Since the Fitzpatrick logistic loss is slightly more computationally demanding, requiring to solve a root equation while the logistic loss does not, we believe that the logistic loss remains the best choice when we wish to use the softargmax as link function ̂ y Ω .

Similarly, we found that the sparsemax loss and the Fitzpatrick sparsemax loss are comparable on most datasets, with the sparsemax loss significantly winning on only 1 dataset out of 11 and the Fitzpatrick loss significantly winning on 2 datasets out of 11 . Since the two losses both use the Euclidean projection onto the simplex P △ k as their link function ̂ y Ω , we conclude that the Fitzpatrick sparsemax loss is a serious contender to the sparsemax loss, especially when predicting sparse label proportions is important.

5 Conclusion

We proposed to leverage the Fitzpatrick function, a theoretical tool from monotone operator theory, to build a new family of primal-dual separately convex loss functions for machine learning. We reinterpreted Fitzpatrick losses as lower bounds of Fenchel-Young losses that maintains the same link function. Our paper therefore challenges the idea that there can only be one loss function, convex in each argument, associated with a certain link function. For instance, we created the Fitzpatrick logistic and sparsemax losses, that are associated with the soft argmax and sparse argmax links, traditionally associated with the logistic and sparsemax losses, respectively. We believe that even more loss functions with the same link can be created, which calls for a systematic study of their properties and respective benefits. In future work, we intend to study calibration guarantees for Fitzpatrick losses, to test new link functions from maximal monotone operator theory and to implement more efficient training for the Fitzpatrick logistic loss.

References

S.-i. Amari. Information geometry and its applications , volume 194. Springer, 2016.

H. Bauschke, D. McLaren, and H. Sendov. Fitzpatrick functions: Inequalities, examples, and remarks on a problem by S. Fitzpatrick. Journal of Convex Analysis , 13, 07 2005.

H. H. Bauschke and P. L. Combettes. Convex analysis and monotone operator theory in Hilbert spaces . CMS Books in Mathematics/Ouvrages de Mathématiques de la SMC. Springer-Verlag, New York, second edition, 2017.

D. P. Bertsekas. Nonlinear programming. Journal of the Operational Research Society , 48(3):334-334, 1997.

M. Blondel, Q. Berthet, M. Cuturi, R. Frostig, S. Hoyer, F. Llinares-Lopez, F. Pedregosa, and J.-P. Vert. Efficient and modular implicit differentiation. In S. Koyejo, S. Mohamed, A. Agarwal, D. Belgrave, K. Cho, and A. Oh, editors, Advances in Neural Information Processing Systems , volume 35, pages 5230-5242. Curran Associates, Inc., 2022.

M. Blondel, F. Llinares-López, R. Dadashi, L. Hussenot, and M. Geist. Learning energy networks with generalized Fenchel-Young losses. Advances in Neural Information Processing Systems , 35:12516-12528, 2022.

M. Blondel, A. Martins, and V. Niculae. Learning classifiers with Fenchel-Young losses: Generalized entropies, margins, and algorithms. In The 22nd International Conference on Artificial Intelligence and Statistics , pages 606-615. PMLR, 2019.

M. Blondel, A. F. Martins, and V. Niculae. Learning with Fenchel-Young losses. Journal of Machine Learning Research , 21(35):1-69, 2020.

P. Brucker. An O ( n ) algorithm for quadratic knapsack problems. Operations Research Letters , 3(3):163-166, 1984.

R. S. Burachik and J. E. Martínez-Legaz. On Bregman-type distances for convex functions and maximally monotone operators. Set-Valued and Variational Analysis , 26:369-384, 2018.

R. S. Burachik and B. F. Svaiter. Maximal monotone operators, convex functions and a special family of enlargements. Set-Valued Analysis , 10:297-316, 2002.

L. Condat. Fast projection onto the simplex and the ℓ 1 ball. Mathematical Programming , 158(1-2):575-585, 2016.

R. M. Corless, G. H. Gonnet, D. E. G. Hare, D. J. Jeffrey, and D. E. Knuth. On the Lambert W function. Advances in Computational Mathematics , 5(1):329-359, Dec 1996.

J. C. Duchi, S. Shalev-Shwartz, Y. Singer, and T. Chandra. Efficient projections onto the ℓ 1 -ball for learning in high dimensions. In Proc. of ICML , 2008.

S. Fitzpatrick. Representing monotone operators by convex functions. In Workshop/Miniconference on Functional Analysis and Optimization , volume 20, pages 59-66. Australian National University, Mathematical Sciences Institute, 1988.

T. Gneiting and A. E. Raftery. Strictly proper scoring rules, prediction, and estimation. Journal of the American statistical Association , 102(477):359-378, 2007.

P. D. Grünwald and A. P. Dawid. Game theory, maximum entropy, minimum discrepancy and robust Bayesian decision theory. The Annals of Statistics , 32(4):1367-1433, 2004.

K. C. Kiwiel. Proximal minimization methods with generalized Bregman functions. SIAM journal on control and optimization , 35(4):1142-1168, 1997.

S. G. Krantz and H. R. Parks. The implicit function theorem: history, theory, and applications . Springer Science & Business Media, 2002.

D. C. Liu and J. Nocedal. On the limited memory BFGS method for large scale optimization. Mathematical programming , 45(1):503-528, 1989.

A. Martins and R. Astudillo. From softmax to sparsemax: A sparse model of attention and multi-label classification. In International conference on machine learning , pages 1614-1623. PMLR, 2016.

C. Michelot. A finite algorithm for finding the projection of a point onto the canonical simplex of R n . Journal of Optimization Theory and Applications , 50(1):195-200, 1986.

J. J. Moreau. Inf-convolution, sous-additivité, convexité des fonctions numériques. J. Math. Pures Appl. (9) , 49:109-154, 1970.

V. Niculae, A. Martins, M. Blondel, and C. Cardie. Sparsemap: Differentiable sparse structured inference. In International Conference on Machine Learning , pages 3799-3808. PMLR, 2018.

F. Pedregosa, G. Varoquaux, A. Gramfort, V. Michel, B. Thirion, O. Grisel, M. Blondel, P. Prettenhofer, R. Weiss, V. Dubourg, et al. Scikit-learn: Machine learning in Python. the Journal of machine Learning research , 12:2825-2830, 2011.

M. D. Reid and R. C. Williamson. Composite binary losses. The Journal of Machine Learning Research , 11:2387-2422, 2010.

R. T. Rockafellar and R. J.-B. Wets. Variational Analysis . sv, Berlin, 1998.

E. K. Ryu and W. Yin. Large-scale convex optimization: algorithms & analyses via monotone operators . Cambridge University Press, 2022.

P. Virtanen, R. Gommers, T. E. Oliphant, M. Haberland, T. Reddy, D. Cournapeau, E. Burovski, P. Peterson, W. Weckesser, J. Bright, et al. Scipy 1.0: fundamental algorithms for scientific computing in Python. Nature methods , 17(3):261-272, 2020.

R. C. Williamson, E. Vernet, and M. D. Reid. Composite multiclass losses. Journal of Machine Learning Research , 17(222):1-52, 2016.

A Datasets statistics

B Proofs

B.1 Lemmas

̸

̸

Let Ω = Ψ + ι C , where Ψ : R k → R is proper convex l.s.c. and C ⊆ domΨ is a nonempty closed convex set such that ri C ∩ ri dom Ψ = ∅ , where ri C is the relative interior of C . Then, for all y, y ′ ∈ R k

Proof.

As C , domΨ ⊂ R k and ri C ∩ ri dom Ψ = ∅ , we can apply [3, Proposition 6.19] and [3, Theorem 16.47] to write ∂ Ω( y ′ ) = ∂ Ψ( y ′ ) + ∂ι C ( y ′ ) .

Thus, we have

Lemma 2 Generalized Bregman divergence of indicator function of some nonempty closed convex set set C ⊂ R k

Proof. Using (1) and the fact that ∂ι C = N C [3, Example 16.13], we obtain that

When y ′ ∈ C and y ∈ C ,

When y ′ ∈ C and y / ∈ C , D ι C ( y, y ′ ) = + ∞ , as + ∞ +( -∞ ) = + ∞ in the definition of the Bregman divergence. Therefore, when y ′ ∈ C , we get that D ι C ( y, y ′ ) = ι C ( y ) .

When y ′ ̸∈ C , N C ( y ′ ) = ∅ . Again, in the definition of the Bregman divergence, + ∞ +( -∞ ) = + ∞ and we use the convention sup ∅ = -∞ .

Let Ψ : R k → R be proper, convex and twice differentiable on the interior of its domain. For all y ∈ R k , let Ψ y := Ψ + D Ψ ( y, · ) . Then, for all y, y ′ , y ′′ ∈ int dom Ψ ,

and, in particular, for all y, y ′ ∈ int dom Ψ

Proof. For all y, y ′ ∈ int dom Ψ , as Ψ is convex and differentiable on int dom Ψ , we have that

and therefore, as Ψ is twice differentiable on int dom Ψ , we get that

Therefore, for all y, y ′ , y ′′ ∈ domΨ ,

and in particular using the just last established equality with the triplet ( y, y, y ′ ) , we have for all y, y ′ ∈ domΨ ,

Let α ∈ R . Let Ψ( y ′ ) := ∑ k i =1 y ′ i log y ′ i -α ∑ k i =1 y ′ i be defined for y ′ ∈ R k + . Then, for y, y ′ ∈ R k + ,

which does not depend on α .

Proof. First, if y ′ ∈ R k ++ , Ψ is differentiable at y ′ and ∇ Ψ( y ′ ) i = log y i +1 -α . Thus, ∂ Ψ( y ′ ) = {∇ Ψ( y ′ ) } and

Second, if we prove that ∂ Ψ( y ′ ) = ∅ when there exists a component i such that y ′ i = 0 , we can conclude the proof, as sup ∅ = -∞ by convention. Indeed, Let us assume that y ′ i = 0 . Suppose that θ ′ ∈ ∂ Ψ( y ′ ) . Then, by definition of subgradients,

We consider ε > 0 and choose y ′′ = y ′ + εe i , where e i is the i -th canonical base vector. Thus, we obtain

as y i = 0 and 0 log 0 = 0 by convention. By noticing that lim ε → 0 + ( ε log ε -αε ) /ε = -∞ , we get a contradiction, which concludes the proof.

Lemma 5 Subdifferential inclusion for sup function

Let Y ⊂ R k be some set. Let { f y } y ∈ Y be a family of finite functions f y : R k → R . We define the function g : R k → R by g ( θ ) = sup y ∈ Y f y ( θ ) , for any θ ∈ R k . Then, we have

for all θ ∈ R k and for all ˜ y ∈ argmax y ∈ Y f y ( θ ) .

Proof. Left as an exercise.

Lemma 6 Value and gradient of Ψ ∗ y

Let Ψ : R k → R be a proper strictly convex function. Let us assume that D Ψ ( y, y ′ ) is convex w.r.t. y ′ , for all y ∈ domΨ . We remind that Ψ y ( y ′ ) = Ψ( y ′ ) + D Ψ ( y, y ′ ) . Then, for all θ ∈ R k and for all ˜ y ∈ argmax y ′ ∈ domΨ ⟨ y ′ , θ ⟩ + ⟨ y -y ′ , ∇ Ψ( y ′ ) ⟩ ,

Furthermore, if the function Ψ is twice differentiable, we have

Proof. As Ψ ≤ Ψ y , we have domΨ y ⊂ domΨ . Thus, we get

As Ψ is strictly convex and D Ψ ( y, y ′ ) is convex w.r.t. y ′ , we have that Ψ y is strictly convex. Thus, according to [27, Theorem 11.13], Ψ ∗ y is differentiable. Thus,

using Lemma 5 for the inclusion.

In the case that Ψ is twice differentiable, setting the gradient of the inner function to zero concludes the proof.

Lemma 7 Gradient of Ψ ∗ y , squared norm case

Let Ψ( y ′ ) := 1 2 ∥ y ′ ∥ 2 2 and Ψ y defined as in Lemma 6. Then, the gradient of Ψ ∗ y is given by

Proof. Let us notice that

̸

and that Ψ is strictly convex function. By strong convexity of y ′ ↦→ ∥ y ′ ∥ 2 , we deduce that argmax y ′ ∈ domΨ ⟨ y ′ , θ ⟩ + ⟨ y -y ′ , ∇ Ψ( y ′ ) ⟩ = ∅ . Furthermore, as Ψ is twice differentiable and ∇ 2 Ψ( y ′ ) = I and domΨ = R k , we have that { y ′ ∈ domΨ ∣ ∣ ∇ 2 Ψ( y ′ )( y ′ -y ) = θ -∇ Ψ( y ′ ) } is a singleton. Thus, we use Lemma 6 with ∇ Ψ( y ′ ) = y ′ and ∇ 2 Ψ( y ′ ) = I , we obtain that ∇ Ψ ∗ y ( θ ) is the solution w.r.t. y ′ of the equation y ′ -y = θ -y ′ . Rearranging the terms concludes the proof.

Before stating the next lemma, we recall the definition of the Lambert function [13] W : R + → R + . The function W is the inverse of the function f : R + → R + where f ( w ) = w exp( w ) for all w ∈ R + .

Lemma 8 Gradient of Ψ ∗ y , negentropy case

Let Ψ( y ′ ) := ∑ k i =1 y ′ i log y ′ i -α ∑ k i =1 y ′ i be defined for y ′ ∈ R k + . Then,

Proof. Following the same arguments as in the proof of Lemma 7, we use Lemma 6. We know that ˜ y is the solution of ∇ 2 Ψ(˜ y )(˜ y -y ) = θ -∇ Ψ(˜ y ) . Using ∇ Ψ(˜ y ) = log ˜ y +1 -α and ∇ 2 Ψ(˜ y ) = 1 / ˜ y (where logarithm and division are performed element-wise), we obtain for all i ∈ { 1 , . . . , k }

When y i = 0 , we immediately have ˜ y i = exp( θ i -2 + α ) . When y i > 0 , after rearranging, we obtain

hence the result.

Lemma 9 Gradient of Ω ∗ y

Let Ψ : R k → R be a proper strictly convex l.s.c. function and assume that, for all y ∈ R k , the function D Ψ ( y, · ) is convex. Let Ω y = Ψ y + ι C with Ψ y defined as in Lemma 6 and assume that C is a nonempty compact convex set included in domΨ . Then ∇ Ω ∗ y ( θ ) is the unique element of the set

Proof. The result follows from Danskin's theorem [4, Proposition B.25].

Lemma 10 Dual of simplex-constrained conjugate

Let Ψ : R k → R be a proper strictly convex function. Then,

and for any τ ⋆ ∈ argmin τ ∈ R τ +(Ψ+ ι R k + ) ∗ ( θ -τ 1 ) , we get

Proof. Wehave by the definition of the Fenchel conjugate that (Ψ+ ι △ k ) ∗ ( θ ) = max y ′ ∈△ k ( ⟨ y ′ , θ ⟩-Ψ( y ′ ) ) . As the unit simplex △ k is a compact set and as (Ψ+ ι △ k ) ∗ is differentiable (by [27, Theorem 11.13] given that Ψ is strictly convex and △ k is convex ), we apply Danskin's theorem [4, Proposition B.25] and get {∇ (Ψ + ι △ k ) ∗ ( θ ) } = argmax y ′ ∈△ k ( ⟨ y ′ , θ ⟩ -Ψ( y ′ ) ) .

Furthermore, we rewrite (Ψ + ι △ k ) ∗ in the following dual minimization problem

Again, as Ψ is strictly convex, (Ψ + ι R k + ) ∗ is differentiable [27, Theorem 11.13]. After some elementary calculus on subdifferentials, we conclude by applying primal and dual forms of Fermat's rule [27, Theorem 11.39, Item (d)]: for any τ ⋆ ∈ argmin τ ∈ R τ + (Ψ + ι R k + ) ∗ ( θ -τ 1 ) , we have (Ψ + ι R k ) ∗ ( θ τ ⋆ 1 ) argmax y k ( y ′ , θ Ψ( y ′ ) ) .

Lemma 11 Gradient of Ω ∗ y , negentropy, constrained to the simplex

Let Ψ( y ′ ) = ⟨ y ′ , log y ′ ⟩ , if y ′ ∈ R k + , + ∞ otherwise. Then, for all y ∈ R k + , the gradient of Ω y = Ψ y + ι △ k (as defined in Lemma 9) is given by

where λ ⋆ is the unique solution of the equation

Proof. From Lemma 9 and Lemma 10, since domΨ y = R k + , we have

As τ ↦→ τ +Ψ ∗ y ( θ -τ 1 ) is a convex function, Setting the gradient of the inner function to zero, we get

Using Lemma 8, we obtain that τ ⋆ satisfies

By monotonicity in τ ⋆ , we conclude that τ ⋆ exists and is unique. Using the change of variable τ ⋆ = λ ⋆ +2 concludes the proof.

B.2 Proof of Proposition 1 (Properties of Fitzpatrick losses)

Apart from differentiability, the proofs follow from the study of Fitzpatrick functions found in [15, 2, 28]. We include the proofs for completeness.

Link function and non-negativity. We recall that

From the monotonicity of ∂ Ω , we have that if ( y, θ ) ∈ ∂ Ω and ( y ′ , θ ′ ) ∈ ∂ Ω , then ⟨ y ′ -y, θ ′ -θ ⟩ ≥ 0 . Therefore, for all ( y, θ ) ∈ ∂ Ω ,

with the infimum being attained at ( y ′ , θ ′ ) = ( y, θ ) . This proves the link function.

From the maximality of ∂ Ω , if ( y, θ ) ̸∈ ∂ Ω , there exists ( y ′ , θ ′ ) ∈ ∂ Ω such that ⟨ y ′ -y, θ ′ -θ ⟩ < 0 . Therefore, for all ( y, θ ) ̸∈ ∂ Ω ,

for any solution τ ⋆ of

This proves the non-negativity.

Separable convexity. We recall that

where

The function ( y, θ ) ↦→⟨ y ′ , θ ⟩ + ⟨ y, θ ′ ⟩ - ⟨ y ′ , θ ′ ⟩ is (jointly) convex in ( y, θ ) for all ( y ′ , θ ′ ) . Since the supremum preserves convexity, F [ ∂ Ω]( y, θ ) is (jointly) convex in ( y, θ ) . The function ⟨ y, θ ⟩ is convex in y and θ but not (jointly) convex in ( y, θ ) . Therefore, L F [ ∂ Ω] ( y, θ ) is separately convex in y and θ .

Subgradient We are going to prove that, for any y, θ ∈ R k and for any y ⋆ ∈ y ⋆ F [ ∂ Ω] ( y, θ ) as defined in (4), we have y ⋆ -y ∈ ∂ θ L F [ ∂ Ω] ( y, θ ) .

Indeed, by Definition 1, we have that

Furthermore, ∂f y ′ ( θ ) = { y ′ -y } , where f y ′ ( θ ) = ⟨ y ′ , θ ⟩ + sup θ ′ ∈ ∂ Ω( y ′ ) ⟨ y -y ′ , θ ′ ⟩ . Thus, by applying Lemma 5, we get for any y ⋆ ∈ argmax y ′ ∈ domΩ f y ′ ( θ ) = argmax y ′ ∈ domΩ [ ⟨ y ′ -y, θ ⟩ +sup θ ′ ∈ ∂ Ω( y ′ ) ⟨ y -y ′ , θ ′ ⟩ ] , which yields the result by definition of y ⋆ F [ ∂ Ω] ( y, θ ) in (4).

Differentiability. Since the function Ω is strictly convex and y ′ ↦→ D Ω ( y, y ′ ) is convex, then Ω y ( y ′ ) = Ω( y ′ ) + D Ω ( y, y ′ ) is strictly convex in y ′ . From the duality between strict convexity and differentiability, Ω ∗ y ( θ ) is differentiable in θ .

Tighter inequality. Using

and

we get for any ( y ′ , θ ′ ) ∈ ∂ Ω ,

Therefore

B.3 Proof of Proposition 2 (Expression of Fitzpatrick loss when Ω is twice differentiable)

We recall that

Since Ω is differentiable, we have ∂ Ω( y ′ ) = {∇ Ω( y ′ ) } and therefore θ ′ = ∇ Ω( y ′ ) , which gives

Setting the gradient of the inner function w.r.t. y ′ to zero, we get, for any y ⋆ ∈ y ⋆ F [ ∂ Ω] ( y, θ ) as defined in (4),

Using the y ′ = y ⋆ and θ ′ = ∇ Ω( y ⋆ ) in Definition 1, we then obtain

B.4 Proof of Proposition 3 (squared loss)

Using Proposition 2 with ∇ Ω( y ′ ) = y ′ and ∇ 2 Ω( y ′ ) = I , we obtain

We therefore obtain

B.5 Proof of Proposition 4 (perceptron loss)

A proof of the Fitzpatrick function for this case was given in [2, Example 3.1]. We include a proof for completeness. Since Ω = ι C , we have domΩ = C . Therefore, for all y ∈ C and θ ∈ R k ,

where in the third line we used that ι C ( y ) = ι C ( y ′ ) = 0 and where in the last line we used Lemma 2. Therefore, for all y ∈ R k and θ ∈ R k ,

B.6 Proof of Proposition 5 (Fitzpatrick sparseMAP loss)

A proof of the Fitzpatrick function for this case was given in [2, Example 3.13]. We provide an alternative proof.

From Proposition 7, we know that for any y ∈ domΩ(= C here )

where

Using conjugate calculus, we obtain

From Proposition 7, the supremum w.r.t. y ′ is achieved at y ⋆ = ∇ Ω ∗ (( y + θ ) / 2) = P C (( y + θ ) / 2) . We therefore obtain

Therefore,

B.7 Proof of Proposition 6 (Fitzpatrick logistic loss)

Differentiability w.r.t. θ and formula of gradient. According to Proposition 7, we have

Thus the differentiability w.r.t. θ of L F [ ∂ Ω] ( y, θ ) follows from the differentiability of θ ↦→ Ω ∗ y ( θ ) . Lemma 11 yields the differentiability of Ω ∗ y ( θ ) and a formula for its gradient y ⋆ F [ ∂ Ω] ( y, θ ) := ∇ Ω ∗ y ( θ )

It follows that ∇ θ L F [ ∂ Ω] ( y, θ ) = y ⋆ F [ ∂ Ω] ( y, θ ) -y .

Formula of the Fitzpatrick logistic loss. We use y ⋆ as a shorthand for y ⋆ F [ ∂ Ω] ( y, θ ) . As we know that Ω ∗ y ( θ ) = ⟨ y ⋆ , θ ⟩ -Ω y ( y ⋆ ) , we use again Proposition 7 to get as Ω y ( y ′ ) = Ω( y )+ D Ω ( y, y ′ ) and in particular Ω y ( y ) = Ω( y ) . Furthermore, as y ⋆ ∈ △ k ∩ R k ++ , Ω is differentiable at y ⋆ and D Ω ( y, y ⋆ ) = Ω( y ) -Ω( y ⋆ ) -⟨ y -y ⋆ , ∇ Ω( y ⋆ ) ⟩ , where ∇ Ω( y ⋆ ) = log y ⋆ +1 . Thus

Bisection formula for λ ⋆ and bounds. We also get from Lemma 11 a bisection formula for λ ⋆ , which is a shorthand for λ ⋆ F [ ∂ Ω] ( y, θ ) .

We focus here on a lower bound and an upper bound for λ ⋆ ∈ R . Let us prove that

where ℓ 0 ( y ) = Card( j : y j = 0) .

For the lower bound, we use the concavity [13] of the Lambert function W , which implies 1 W ( y i e λ ⋆ -θ i ) ≥ 1 y i e λ ⋆ -θ i . Thus, which in turn implies

and yields the lower bound.

For the upper bound, the function g ( λ ) = e -λ ∑ i : y i =1 e θ i + ∑ i : y i > 0 y i W ( y i e λ -θ i ) is continuous and decreasing (as it is a positive combination of decreasing functions) and g ( -∞ ) = + ∞ . Thus if we find a λ such that g ( λ ) < 1 , we know that λ ⋆ ≤ λ .

We deal with each term of g ( λ ) separately. If λ ∈ R satisfies

̸

then

Thus, all λ satisfying the following inequalities are upper bounds of λ ⋆

As W is monotone and W -1 ( t ) = t e t , we get

Thus taking λ = max { log 2+log ∑ k i : y i =0 e θ i , max i : y i > 0 log 2+log ℓ 0 ( y )+ θ i +2 ℓ 0 ( y ) y i } yields an upper bound of λ ⋆ .

B.8 Proof of Proposition 7 (characterization of F [ ∂ Ω] using D Ω )

The supremum above is achieved at some y ′ ∈ ∂ Ω ∗ y ( θ ) .

When Ω = Ψ + ι C , where C ⊆ domΨ is a nonempty closed convex set, using Lemmas 1 and 2, we have for all y ∈ C

B.9 Proof of Proposition 8 (lower bound)

It was shown in [8, Proposition 3] that if f = g + ι C , where the function g is Legendre-type with C ⊆ domΨ a nonempty closed convex set, then for all y ∈ C and θ ∈ R k , with equality if C = dom g . Using g = Ψ y , f = Ω y = Ψ y + ι C , y ⋆ = ∇ Ω ∗ y ( θ ) = y ⋆ F [ ∂ Ω] ( y, θ ) , and Lemma 3, we therefore obtain

If Ψ y is µ -strongly convex and D Ψ is convex in its second argument, then Ψ y is µ -strongly convex as well. Therefore, we also have

C Comment on the inf-addition convention + ∞ +( -∞ ) = + ∞

The question of adding conflicting infinities has been thoroughly studied by Moreau in [23]. In this paper, Moreau introduced two additions for R ∪ {-∞ , + ∞} . One of Moreau's two conventions for summing infinities can be found under the name of inf-addition in [27, Chapter 1]. The other convention is named sup-addition.

Using the inf-addition on R ∪{-∞ , + ∞} allows, for instance, the equivalence of min x ∈ C f ( x ) and min x ∈ R n f ( x ) + ι C ( x ) . The same goes for the sup-addition but for maximization problems.

We used the inf-addition convention for the definition of the generalized Bregman divergence in (1). As the generalized Bregman divergence is to be thought of as a generalization of a distance, it is generally a quantity that will be minimized. Therefore, the inf-addition is a good fit for the definition of the generalized Bregman divergence.

In the proof of Proposition 7 in B.8, the choice of inf-addition is corroborated by the fact that we want to calculate, for some fixed y ∈ R k , ( Ω( · ) + D Ω ( y, · ) ) ( θ ) = sup y ′ ⟨ y ′ , θ ⟩ -( Ω( y ′ ) + D Ω ( y, y ′ ) ) , ∀ y ′ ∈ R k .

The minus sign transforms the inf-addition into the sup-addition when distributed. If we had chosen the sup-addition in the definition of the Bregman divergence D Ω , we would here calculate a supremum of an inf-addition, thus entering unknown territory.

NeurIPS Paper Checklist

1. Claims

Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?

Answer: [Yes]

Justification: As stated in the abstract and in the introduction, we have defined Fitzpatrick losses in Definition 1 and studied their properties in Proposition 1. We instantiated the Fitzpatrick sparseMAP and the Fitzpatrick logistic loss in Proposition 5 and Proposition 6; we have studied the properties of Fitzpatrick losses in Proposition 1 and Proposition 7; we have gathered the results of the label proportion estimation in Table 1.

Guidelines:

The answer NA means that the abstract and introduction do not include the claims made in the paper.

The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.

The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.

It is fine to include aspirational goals as motivation as long as it is clear that these goals are not attained by the paper.

2. Limitations

Question: Does the paper discuss the limitations of the work performed by the authors?

Answer: [Yes]

Justification: We discuss the computational limitations of the Fitzpatrick logistic loss. As stated in Proposition 6 and in Section 4, the computation of the loss value and gradient involves solving a root equation.

Guidelines:

The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.

The authors are encouraged to create a separate "Limitations" section in their paper.

The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.

The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on a few datasets or with a few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.

The authors should reflect on the factors that influence the performance of the approach. For example, a facial recognition algorithm may perform poorly when image resolution is low or images are taken in low lighting. Or a speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.

The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.

If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.

While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an important role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.

3. Theory Assumptions and Proofs

Question: For each theoretical result, does the paper provide the full set of assumptions and a complete (and correct) proof?

Answer: [Yes]

Justification: All results in the paper are proved in the appendix. We strived to make the proofs self-contained.

Guidelines:

The answer NA means that the paper does not include theoretical results.

All the theorems, formulas, and proofs in the paper should be numbered and crossreferenced.

All assumptions should be clearly stated or referenced in the statement of any theorems.

The proofs can either appear in the main paper or the supplemental material, but if they appear in the supplemental material, the authors are encouraged to provide a short proof sketch to provide intuition.

Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.

Theorems and Lemmas that the proof relies upon should be properly referenced.

4. Experimental Result Reproducibility

Question: Does the paper fully disclose all the information needed to reproduce the main experimental results of the paper to the extent that it affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided or not)?

Answer: [Yes]

Justification: In Section 4, we describe the training setup, the linear model we use for predictions and give a link to access the datasets we use. We also indicate that we use the L-BFGS algorithm for training. Furthermore, Proposition 5 and Proposition 6 yield formulae for the gradients of the new losses we introduce.

Guidelines:

The answer NA means that the paper does not include experiments.

If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.

If the contribution is a dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.

Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way to accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g., in the case of a large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.

While NeurIPS does not require releasing code, the conference does require all submissions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example

(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.

(b) If the contribution is primarily a new model architecture, the paper should describe the architecture clearly and fully.

(c) If the contribution is a new model (e.g., a large language model), then there should either be a way to access this model for reproducing the results or a way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).

(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.

5. Open access to data and code

Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?

Answer: [Yes]

Justification: We provided a link for the used datasets in the experiments. The code will be opened upon release. For now, we provide instructions to reproduce the main experimental results. These instructions involved basic supervised learning tools such as L-BFGS algorithm, linear prediction model and standard cross-validation for the hyperparameter.

Guidelines:

The answer NA means that paper does not include experiments requiring code.

Please see the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿✴✴♥✐♣/a115✳❝❝✴ ♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

While we encourage the release of code and data, we understand that this might not be possible, so 'No' is an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for a new open-source benchmark).

The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines ( ❤/a116/a116♣/a115✿ ✴✴♥✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❣✉✐❞❡/a115✴❈♦❞❡❙✉❜♠✐/a115/a115✐♦♥/a80♦❧✐❝② ) for more details.

The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.

The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only a subset of experiments are reproducible, they should state which ones are omitted from the script and why.

At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).

Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted.

6. Experimental Setting/Details

Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?

Answer: [Yes]

Justification: The selection of hyperparameter are discussed in section 4. We use predetermined train-test splits that come with the datasets.

Guidelines:

The answer NA means that the paper does not include experiments.

The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them.

The full details can be provided either with the code, in appendix, or as supplemental material.

7. Experiment Statistical Significance

Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?

Answer: [Yes]

Justification: The split between training sets and test sets are fixed in advance so there is no variability when conducting the label proportion estimation tests. Furthermore, as we minimize convex losses, the convergence of the minimization algorithm is independent of the starting point.

Guidelines:

The answer NA means that the paper does not include experiments.

The authors should answer "Yes" if the results are accompanied by error bars, confidence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).

The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)

The assumptions made should be given (e.g., Normally distributed errors).

It should be clear whether the error bar is the standard deviation or the standard error of the mean.

It is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have a 96% CI, if the hypothesis of Normality of errors is not verified.

For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out of range (e.g. negative error rates).

If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.

8. Experiments Compute Resources

Question: For each experiment, does the paper provide sufficient information on the computer resources (type of compute workers, memory, time of execution) needed to reproduce the experiments?

Answer: [Yes]

Justification: The characteristics of the computer we used to conduct the tests. On the largest dataset, our experiment only takes a couple of hours to run.

Guidelines:

The answer NA means that the paper does not include experiments.

The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.

The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.

The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).

9. Code Of Ethics

Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics ❤/a116/a116♣/a115✿✴✴♥❡✉/a114✐♣/a115✳❝❝✴♣✉❜❧✐❝✴❊/a116❤✐❝/a115●✉✐❞❡❧✐♥❡/a115 ?

Answer: [Yes]

Justification: The research conducted does not involve human subjects. We did not create new datasets and used standard open access datasets for multiclassification. After reviewing the 'societal impact and potential harmful consequences' from the Code of Ethics, we conclude that the research conduct in the paper does not pose a risk of harmful consequences.

Guidelines:

The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.

If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.

The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction).

10. Broader Impacts

Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?

Answer: [NA]

Justification: The research conducted in this paper is theoretical and has no societal impact. Guidelines:

The answer NA means that there is no societal impact of the work performed.

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.

Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.

The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.

The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.

If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).

11. Safeguards

Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?

Answer: [NA]

Justification: The datasets used in the tests are publicly available. No new prediction model has been released in this paper.

Guidelines:

The answer NA means that the paper poses no such risks.

Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.

Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.

We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort.

12. Licenses for existing assets

Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?

Answer: [Yes]

Justification: In Section 4, the libraries used for the implementation of the numerical experiments are credited and cited.

Guidelines:

The answer NA means that the paper does not use existing assets.

The authors should cite the original paper that produced the code package or dataset.

The authors should state which version of the asset is used and, if possible, include a URL.

The name of the license (e.g., CC-BY 4.0) should be included for each asset.

For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.

If assets are released, the license, copyright information, and terms of use in the package should be provided. For popular datasets, ♣❛♣❡/a114/a115✇✐/a116❤❝♦❞❡✳❝♦♠✴❞❛/a116❛/a115❡/a116/a115 has curated licenses for some datasets. Their licensing guide can help determine the license of a dataset.

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if it has changed) should be provided.

If this information is not available online, the authors are encouraged to reach out to the asset's creators.

13. New Assets

Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?

Answer: [NA]

Justification: The paper does not release new assets.

Guidelines:

The answer NA means that the paper does not release new assets.

Researchers should communicate the details of the dataset/code/model as part of their submissions via structured templates. This includes details about training, license, limitations, etc.

The paper should discuss whether and how consent was obtained from people whose asset is used.

At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.

14. Crowdsourcing and Research with Human Subjects

Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Including this information in the supplemental material is fine, but if the main contribution of the paper involves human subjects, then as much detail as possible should be included in the main paper.

According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.

15. Institutional Review Board (IRB) Approvals or Equivalent for Research with Human Subjects

Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?

Answer: [NA]

Justification: The paper does not involve crowdsourcing nor research with human subjects. Guidelines:

The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.

Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurIPS Code of Ethics and the guidelines for their institution.

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes the Fitzpatrick loss, a tighter version of the Fenchel-Young loss. Fitzpatrick losses are defined based on the monotone operator theory. In particular, the Fitzpatrick function satisfies a tighter inequality than the Fenchel-Young inequality, leading to the tighter loss function. The authors elucidate the fundamental properties of the Fitzpatrick loss and present concrete examples, demonstrating that Fitzpatrick sparsemax/logistic losses indeed differ from the Fenchel-Young counterparts. The authors also show that the Fitzpatrick loss can be seen as the Fenchel-Young loss generated by target-dependent regularizes and that it is connected to generalized Bregman divergences, while the primal-dual nature of the Fitzpatrick loss crucially makes it convex. A lower bound on the Fitzpatrick loss is also presented, which is analogous to the Fenchel-Young counterpart. Finally, experiments compare Fitzpatrick and Fenchel-Young losses on classification tasks.\n\nSTRENGTHS:\n1. The paper presents a novel framework for designing loss functions from link functions. Prior to the work, the Fenchel-Young loss is the only such framework. The current work opens a new direction for designing various other loss functions with rigorous foundations of monotone operator theory. I believe this is of great benefit to the machine-learning community. \n2. The paper discusses the properties of the Fitzpatrick loss and its connection to other notions (Fenchel-Young losses and generalized Bregman divergences) in detail. This is helpful for readers who are familiar with either of the existing notions.\n\nWEAKNESSES:\nA weakness (not so serious) is that the experimental results are not very compelling. Still, I find the slight superiority of Fitzpatrick-sparsemax to Sparsemax noteworthy as their computation costs are almost the same.\n\nQUESTIONS:\n1. A special form of target-dependent regularizers is used for constructing the cost-sensitive Fenchel-Young loss ([Blondel et al. 2020, Section 3.4](https://jmlr.csail.mit.edu/papers/v21/19-021.html)). Does the Fitzpatrick loss have any relation with the cost-sensitive Fenchel-Young loss?\n2. Is it possible to establish calibration of target and Fitzpatrick excess risks, as in [Blondel (2019, Proposition 5)](https://proceedings.neurips.cc/paper_files/paper/2019/hash/7990ec44fcf3d7a0e5a2add28362213c-Abstract.html)? The lower bound in Proposition 8 seems useful for this. \n3. Regarding lines 164-165, is there a canonical class of convex functions $\\Omega$ such that $D\\_\\Omega$ is convex in the second argument?\n4. In the experiments, predictions are produced by $\\hat y\\_\\Omega(Wx)$, and I can understand this since the Fitzpatrick loss has $\\hat y_\\Omega$ as the link function as in Proposition 1. On the other hand, given the discussion in Section 3.3, $y^*\\_{F[\\partial\\Omega]}(y, \\theta)$ also appears to be a Fitzpatrick counterpart of $\\hat y\\_\\Omega(\\theta)$, which, however, cannot be used in the test phase since ground truth $y$ is not available. So, when $W$ is learned with the Fitzpatrick loss, aren't there other possible methods for converting $Wx$ to a prediction other than $\\hat y\\_\\Omega(Wx)$ (without knowing $y$)?\n\n#### **Minor comments**\n- l. 84. \"$\\hat y\\_\\Omega$ is monotone...\" Does this \"monotone\" refer to the one in Section 2.3? If so, it is not yet defined there.\n- Eq. (4). Enclosing the terms after \"argmax\" in parentheses will make it easier to read.\n- l. 174. that that.\n- Proposition 8. Reminding that $\\Psi_y$ is defined as with $\\Omega_y$ in Section 3.3 is helpful; alternatively, it may be better to define $\\Omega_y$ as a general notation in Section 3.3 or earlier.\n\nLIMITATIONS:\nThe authors have discussed the limitation regarding computation costs.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-06-14"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article presents a refreshing take on losses used in ML, with solid motivational examples and a several of interesting and beautiful results in pure convex analysis.\n\nSTRENGTHS:\nThe Fitzpatrick function is a well-known tool for achieving very pure-theoretical results like Minty's theorem in convex analysis. However, before reading this article, I was under the impression that the Fitzpatrick function was just a theoretical tool with limited applications in-practice...\n\nUntil now! This paper takes a beautiful tool from pure convex analysis, spruces up some very interesting and new results, and presents it as a new application in ML with some very solid justification concerning sharper inequalities in comparison to the classical class of Fenchel-Young inequalities.\n\nWEAKNESSES:\nMy main regret is that, due to the reviewing load, I was not able to fully check all of the proofs presented in the appendix. The claims *in the article* are reasonable, interesting, well-presented, and thorough; however, I am quite concerned that there may be some (unintentionally) omitted hypothesis, or holes in the proofs in the appendix (e.g., please also see my statement about Compactness of the domain in \"limitations\" section), because this often happens in ML conference articles. I wish that an appendix counted as the \"reviewed\" part of the article, but alas I simply do not have the time to check every detail. I am going to suggest that this paper is accepted, but I suggest that the authors please take care to revise based on the following minor points.\n\n\nMinor comments:\n- Citations appear to be out of numerical order when they are cited\n- Line 30: In what sense is \"proper\" meant? All losses I'm aware of\n  are proper functions (using the convex analysis definition).\n- Line 56: Is R_+ numbers which are strictly positive, or just\n  positive?\n- Line 59: The set must be nonempty for the projection to be defined.\n- Line 62: This identity only holds when \\Omega is differentiable  *and convex*! Otherwise, this definition of the subdifferential produces the emptyset in areas of concavity.\n- Line 66: Please provide a citation.\n- Line 68: Does this function need to be Legendre?\n- Line 101: Monotone is defined here, but mentioned earlier in the\n  article; would be nice to have the definition at the beginning.\n- Definition 1: Within the text of the definition, \"dom Omega\"  appears before Omega is introduced.\n- A bit more discussion would be nice on specifically which problems (4) can be calculated.\n- Line 132: \"Above expression\" referring to \"\\nabla^2 Omega\" is ambiguously defined, since in the constrained case Omega is not differentiable.\n\nQUESTIONS:\n- Line 69: As far as I've seen in most convex analysis books and papers, this convention is a bit nonstandard; typically, \"+infinity +  (-infinity)\" is *undefined*. Please comment on which results in this article break if the quantity \"+infinity + (-infinity)\" is undefined. I.e., if this is \"not a valid move\", which of your results still hold? It is very important to clarify which results hold under varying algebraic models of convex analysis.\n\n- Line 157 / Prop. 7: the relationship between the claim in the equation following Line 157 and Proposition 7 is unclear. How does the claim in line 157 follow from Proposition 7?\n\nLIMITATIONS:\nThe mathematically precise statements (propositions, lemmas, theorems/etc.) should state **all** of their assumptions. In particular, the authors do not repeat that a compact domain of the objective function is required for several of their results. (Unless I'm missing something and compactness is not actually required?)\n\nThe authors do a great job of explaining the results of their experiments. More numerics is always a plus, but since the article has so many strong theoretical contributions, I am quite happy with this as-is.", "TITLE": "Official Review", "RECOMMENDATION": 9, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-06-23"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper proposes a class of loss functions called Fitzpatrick loss, based on the Fitzpatrick function known in maximal monotone operator theory. It can be shown that building a loss function with the proposed method, can give a convex loss function that lower bounds the Fenchel-Young loss under the same choice of link function for prediction. Extensive analysis provides in the paper also shows the relationship between Fitzpatrick loss and Fenchel-Young loss. Finally, experiments were provided to validate the usefulness of Fitzpatrick loss.\n\nSTRENGTHS:\n1. The proposed method is theoretically sound and novel to the best of my knowledge. This give rises to a way to construct a convex loss function systematically that lower bounds the family of Fenchel-young loss under the same link function for prediction.\n2. Not only the new class of loss function is proposed, but the characterization of the relationship between the proposed class of loss functions and existing loss was also provided to understand the research in this direction better.\n3. The paper is well-written overall and is friendly for also new people in the field who is not familiar with Fenchel-young loss function.\n\nWEAKNESSES:\nAlthough it is nice to also provide the experiments in the paper, I found that the experiment failed to motivate the usefulness of Fitzpatrick loss family in the sense that the performance is not preferable overall. I think no so many insights were provided when Fitzpatrick loss is more useful than existing losses in the literature. The proposed loss is also more computationally demanding. Perhaps different kinds of experiments would give more values to the paper. From the current experiments, I think we still need lots of work to study when to use Fitzpatrick loss in practice.\n\nMy current score is Weak accept (6) as I put importance to the theory to advance the understanding of Fenchel-young loss and Fitspatrick loss function quite high and believe these are the main contributions of this paper, rather than the experimental result.\n\nQUESTIONS:\n1. Since Fitzpatrick loss lower bounds the Fenchel-young loss for the same link function, is it possible to find some use-cases that the proposed loss (e.g., Fitzpatrick-sparsemax) is preferable? For example, could it be more useful when we want to have sparse probability output and we have to learn under the case where there are outliers/noisy data?\n2. It is said that we can provide a relationship between Fitzpatrick loss and Fenchel-young loss by using \"target-dependent function\" in Section 3.3 (Prop. 7). I think the result is very interesting and I have two questions regarding this finding. \n- 2.1 Since we need target-dependent function for $\\Omega_y$ to make the relationship holds, does the original Fenchel-Young loss supports target-dependent function case? If so, then this should be no problem and we can state that all properties of Fenchel-young loss inherits to Fitzpatrick loss.\n- 2.2 Is it safe to say that Fitzpatrick loss is a special case of Fenchel-Young loss? Perhaps not?\n- 2.3 Can we also say that any Fenchel-young loss can be rewritten in a form of Fitzpatrick loss generated by different $\\Omega$?\n3. In the conclusion section, there was a discussion about \"there can only be one loss function associated with a certain link function\". I am not sure about this statement for the definition of loss function here. Is it about a \"proper composite\" loss function, a \"classification-calibrated\" loss function, or a convex + (proper composite/classification-calibrated) loss function? Maybe we can be more explicit here otherwise one might think about any function or small ad-hoc modification using the same link and use it as a loss function with not much problems. \n\n\nMinor comment\n1. Typo: Line 99: also know as -> also known as\n2. I feel it is a bit strange to say One loss is tighter than another loss since it is closer to zero everywhere.\n\nLIMITATIONS:\nThe discussion was sufficient. The authors noted the challenge of computational problem and appropriately discussed the experimental results of the proposed method.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2024-07-10"}

comments: SUMMARY:
The paper describes an alternative approach to constructing loss functions. The authors propose a Fitzpatrick loss and compare it to Fenchel-Young loss and Bregman divergence. The work enumerates the basic properties of the introduced loss and compares it to existing loss functions by numerical simulations.

STRENGTHS:
The overview of Fitzpatrick loss is nice. For the reader not acknowledged with the topic it is easy to get the main concepts. The authors also give examples and compare Fitzpatrick loss with Fenchel-Young loss.

WEAKNESSES:
I think the paper needs more thorough numerical evaluation due to the format of the conference. Although the properties of Fitzpatrick loss are given, the logical question is, why to use it instead of known loss functions for machine learning? At the moment, the authors presented comparison for classification problem over several datasets. Without broader experimental study, the manuscript seems more like a mathematical note. Therefore, I think that additional experiments will strengthen the paper.

Moreover, the datasets are for classification, but the results are compared in terms of mean squared error. It seems more logical to compare using some metric for classification, i.e. accuracy, f1-score etc.

QUESTIONS:
The paper claims that the Fitzpatrick losses are tighter than Fenchel-Young. Can something be said from the statistical point of view? Does using new loss lead to better generalization, for example.

LIMITATIONS:
The paper does not have negative societal impact.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2024-07-13

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper proposes a family of losses based on the notion of a Fitzpatrick function, which takes the role of composable subdifferentials in Fenchel-Young losses. The resulting family of losses is parallel to Fenchel-Young losses in the sense that each Fenchel-Young loss has a shared link function with a certain Fitzpatrick loss, but the losses themselves differ. It is shown that these losses posses some desirable properties such as convexity, and that they are based on a type of duality that yields a smaller duality gap than the one based on the Fenchel-Young inequality. Representative Fitzpatrick losses, such as logistic and sparsemax losses are developed as examples of useful members of the family. \n\nExperiments show that the losses are mostly comparable in performance to their Fenchel-Young parallels, with the exception of the sparsemax loss that might be slightly improved in its Fitzpatrick version.\n\nSTRENGTHS:\nThe approach is original and imports ideas from operator theory into ML. Theory is laid out neatly and in an easy to understand manner. Loss functions are clearly an important part of training any ML model, hence improvements in them can be very significant even if they are small, and I assume that further progress beyond this paper can be significant in practice as well.\n\nWEAKNESSES:\nThe empirical utility of the developed losses in not entirely clear, since they do not offer a significant advantage over standard losses, and in some cases (e.g. Fitzpatrick sigmoid) may be more computationally demanding.\nIn other losses such as the squared loss and hinge loss, the parallel Fitzpatrick losses are practically the same. Hence, the contribution here is mainly in exploring further definitions of losses that are associated with familiar link functions, but attach to them a loss function that is not the standard one that stems from the Fenchel-Young family.\n\nI am not an expert on this topic, so while I'm unsure whether the paper is interesting for its sub-community within ML, or for the wider NeurIPS audience (mainly because readers put a lot of emphasis on empirical results), I found it interesting enough to read hence I will give a borderline acceptance rating.\n\nQUESTIONS:\nNo significant questions arise from reading the paper. It might be nice if the authors could comment on what potential they see for future research on this topic, and how\n\nLIMITATIONS:\nThe authors mention some limitations such as somewhat increased computation time, and acknowledge the mixed empirical results. I find this to be an adequate treatment of limitations.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2024-07-13"}