Learning with Fitzpatrick Losses

Authors: seta  rakotomandimby
jean-philippe  chancelier
michel  de lara
mathieu  blondel
Venue: NeurIPS_cc_2024_Conference
Published: 2024-05-13 18:16:40

Abstract: Fenchel-Young losses are a family of convex loss functions,
encompassing the squared, logistic and sparsemax losses, among others.
Each Fenchel-Young loss is implicitly associated with a link function, for
mapping model outputs to predictions. For instance, the logistic loss is
associated with the soft argmax link function. Can we build new loss functions
associated with the same link function as Fenchel-Young losses?
In this paper, we introduce Fitzpatrick losses, a new family of convex loss
functions based on the Fitzpatrick function. A well-known theoretical tool in
maximal monotone operator theory, the Fitzpatrick function naturally leads to a
refined Fenchel-Young inequality, making Fitzpatrick losses tighter than
Fenchel-Young losses, while maintaining the same link
function for prediction.  
As an example, we introduce the Fitzpatrick logistic loss and the
Fitzpatrick sparsemax loss, counterparts of the logistic and the sparsemax
losses. This yields two
new tighter losses associated with the soft argmax and the sparse argmax,
two of the most ubiquitous output layers used in machine learning. We study in
details the properties of Fitzpatrick losses and in particular, we show that
they can be seen as Fenchel-Young losses using a modified, target-dependent
generating function. We demonstrate the effectiveness of Fitzpatrick losses for
label proportion estimation.

Main paper: 
 introduction: 
 Loss functions are a cornerstone of statistics and machine learning: they measure the difference, or "loss," between a ground-truth target and a model prediction. As such, they have attracted a wealth of research. Proper losses (a.k.a. proper scoring rules) [17, 16] measure the discrepancy between a target distribution and a probability forecast. They are essentially primal-primal Bregman divergences, with both the target and the prediction belonging to the same primal space. They are typically explicitly composed with a link function [26, 30] , in order to map the model output to a prediction. A disadvantage of this explicit composition is that it often makes the resulting composite loss function nonconvex. A related family of loss functions are Fenchel-Young losses [6, 7] , which encompass many commonly-used loss functions in machine learning including the squared, logistic, sparsemax and perceptron losses. Fenchel-Young losses can be seen as primal-dual Bregman divergences [1] , with the target belonging to the primal space and the model output belonging to the dual space. In contrast to proper losses, each Fenchel-Young loss is implicitly associated with a given link function, mapping the dual-space model output to a primal-space prediction (for instance, the soft argmax is the link function associated with the logistic loss). This crucial difference makes Fenchel-Young losses always convex. Can we build new convex losses associated with the same link function as Fenchel-Young losses?38th Conference on Neural Information Processing Systems (NeurIPS 2024). In this paper, we introduce Fitzpatrick losses, a new family of primal-dual convex loss functions. Our proposal builds upon the Fitzpatrick function, a well-known theoretical object in maximal monotone operator theory [15, 11, 2] . So far, the Fitzpatrick function had been used as a theoretical tool to represent maximal monotone operators [28] and to construct Bregman-like primal-primal divergences [10] , but it had not been used to construct primal-dual loss functions for machine learning, as we do. Crucially, the Fitzpatrick function naturally leads to a refined Fenchel-Young inequality, making Fitzpatrick losses tighter than Fenchel-Young losses. Yet, their predictions are produced using the same link function, suggesting that we can use Fitzpatrick losses as a tighter replacement for the corresponding Fenchel-Young losses (Figure 1 ). We make the following contributions.• After reviewing some background, we introduce Fitzpatrick losses. They can be thought as a tighter version of Fenchel-Young losses, that use the same link function.• We instantiate two new loss functions in this family: the Fitzpatrick logistic loss and the Fitzpatrick sparsemax loss. They are the counterparts of the logistic and sparsemax losses, two instances of Fenchel-Young losses. We therefore obtain two new tighter losses for the soft argmax and the sparse argmax, two of the most popular output layers in machine learning.• We study in detail the properties of Fitzpatrick losses. We show that Fitzpatrick losses are equivalent to Fenchel-Young losses with a modified, target-dependent generating function.• We demonstrate the effectiveness of Fitzpatrick losses for probabilistic classification on 11 datasets.2 Background 
convex analysis: 
 We define [k] := {1, . . . , k}, the non-negative real numbers by R + := [0, +∞) and the positive real numbers by R ++ := (0, +∞). We denote the probability simplex by △ k := {p ∈ R k + : k i=1 p i = 1} the extended reals by R := R ∪ {-∞, +∞}. We denote the indicator function of a set C by ι C (y) = 0 if y ∈ C, +∞ otherwise. We denote the effective domain of a function Ω : R k → R by dom Ω := {y ∈ R k : Ω(y) < +∞}. A function Ω : R k → R is said to be proper if it never takes the value -∞ and that dom Ω ̸ = ∅. We denote the Euclidean projection onto a nonempty closed convex set C by P C (θ) = argmin y∈C ∥y -θ∥ 2 2 .For a function Ω : R k → R, its subdifferential ∂Ω (possibly empty) is defined by(y ′ , θ ′ ) ∈ ∂Ω ⇐⇒ θ ′ ∈ ∂Ω(y ′ ) ⇐⇒ Ω(y) ≥ Ω(y ′ ) + ⟨y -y ′ , θ ′ ⟩, ∀y.When Ω is convex and differentiable, the subdifferential is a singleton and we have ∂Ω(y ′ ) = {∇Ω(y ′ )}. The normal cone to a set C at y ′ is defined byθ ′ ∈ N C (y ′ ) ⇐⇒ ⟨y -y ′ , θ ′ ⟩ ≤ 0 ∀y ∈ C if y ′ ∈ C and N C (y ′ ) = ∅ if y ′ ̸ ∈ C. The Fenchel conjugate Ω * : R k → R of a function Ω : R k → R is defined by Ω * (θ) := sup y ′ ∈R k ⟨y ′ , θ⟩ -Ω(y ′ ).From standard results in convex analysis [27, Proposition 11.3] , when Ω : R k → R is a proper convex l.s.c. (lower semicontinuous) function,∂Ω * (θ) = argmax y ′ ∈R k ⟨y ′ , θ⟩ -Ω(y ′ ).When the argmax is unique, it is equal to ∇Ω * (θ). We define the generalized Bregman divergence [18] EQUATIONwith the convention +∞ + (-∞) = +∞ (we comment on this convention in Appendix C). When Ω is differentiable, it recovers the classical Bregman divergenceD Ω (y, y ′ ) := Ω(y) -Ω(y ′ ) -⟨y -y ′ , ∇Ω(y ′ )⟩.Both y and y ′ belong to the primal space.The Fenchel-Young loss [7] As its name indicates, it is grounded in the Fenchel-Young inequality ⟨y, θ⟩ ≤ Ω(y) + Ω * (θ) ∀y, θ ∈ R k .L Ω⊕Ω * : R k × R k → RThe Fenchel-Young loss enjoys many desirable properties, notably it is non-negative and it is convex in y and θ separately. The Fenchel-Young loss can be seen as a primal-dual Bregman divergence [1, 7] , where y belongs to the primal space and θ belongs to the dual space.To map an element θ of a dual space to an element y of a primal space, we define the link function (potentially set-valued) associated with the loss L byθ → {y : L(y, θ) = 0}.Given a proper convex function Ω, the associated Fenchel-Young loss L Ω⊕Ω * produces the canonical link function ∂Ω * , sinceL Ω⊕Ω * (y, θ) = 0 ⇐⇒ y ∈ ∂Ω * (θ).In particular when Ω is strictly convex, the Fenchel-Young loss is positive, meaning that it satisfies the identity of indiscerniblesL Ω⊕Ω * (y, θ) = 0 ⇐⇒ y = ∇Ω * (θ).In the remainder of this paper, we will use the notation y Ω (θ) for the canonical link function ∂Ω * (θ).When the function Ω * is differentiable, y Ω (θ) will denote ∇Ω * (θ). Since Ω * is convex, y Ω is monotone (see Section 2.3). As shown in [7] , the monotonicity implies that θ and y Ω (θ) are sorted the same way, i.e., θ i > θ j =⇒ y Ω (θ) i ≥ y Ω (θ) j . Link functions also play an important role in the loss gradient, as we haveEQUATIONExamples of Fenchel-Young loss instances and their associated link functionWe give a few examples of Fenchel-Young losses. With the squared 2-norm, Ω(y ′ ) = 1 2 ∥y ′ ∥ 2 2 , we obtain the squared lossL Ω⊕Ω * (y, θ) = L squared (y, θ) := 1 2 ∥y -θ∥ 2 2and the identity link y Ω (θ) = θ.With the indicator of a convex set C, Ω(y ′ ) = ι C (y ′ ), we obtain the perceptron lossL Ω⊕Ω * (y, θ) = L perceptron (y, θ) := max y ′ ∈C⟨y ′ , θ⟩ -⟨y, θ⟩.and the argmax link y Ω (θ) = argmax y∈C ⟨y, θ⟩.With the squared 2-norm restricted to some convex set C, Ω(y ′ ) = 1 2 ∥y ′ ∥ 2 2 + ι C (y ′ ), we obtain the sparseMAP loss [24] L Ω⊕Ω * (y, θ) = L sparseMAP (y, θ) := 1 2 ∥y -θ∥ 2 2 - 1 2 ∥P C (y) -θ∥ 2 2 .The link is the Euclidean projection onto C,y Ω (θ) = P C (θ).When the set is C = △ k , we obtain the sparsemax loss [21] and the sparsemax link y Ω (θ) = P △ k (θ), which is known to produce sparse probability distributions. With the Shannon negentropy restricted to the probability simplex, Ω(y ′ ) := ⟨y ′ , log y ′ ⟩ + ι △ k (y ′ ), we obtain the logistic lossL Ω⊕Ω * (y, θ) = L logistic (y, θ) := log k i=1 exp(θ i ) + ⟨y, log y⟩ -⟨y, θ⟩,and the soft argmax link (also known as softmax)y Ω (θ) = softargmax(θ) := exp(θ)/ k i=1 exp(θ i ).An operator A, that is, a subset A ⊂ R k × R k , is called monotone if for all (y, θ) ∈ A and all (y ′ , θ ′ ) ∈ A, we have ⟨y ′ -y, θ ′ -θ⟩ ≥ 0.We overload the notation to denote A(y) := {θ : (y, θ) ∈ A}. A monotone operator A is said to be maximal if there does not exist (y, θ) ̸ ∈ A such that A ∪ {(y, θ)} is still monotone. It is well-known that the subdifferential ∂Ω of a proper convex l.s.c. function Ω is maximal monotone. For more details on monotone operators, see [3, 28] .A well-known object in monotone operator theory, the Fitzpatrick function associated with a monotone operator A [15, 11, 2] , denotedF [A] : R k × R k → R, is defined by F [A](y, θ) := sup (y ′ ,θ ′ )∈A ⟨y -y ′ , θ ′ ⟩ + ⟨y ′ , θ⟩.In particular, with A = ∂Ω, we haveF [∂Ω](y, θ) = sup (y ′ ,θ ′ )∈∂Ω ⟨y -y ′ , θ ′ ⟩ + ⟨y ′ , θ⟩ = sup y ′ ∈dom Ω ⟨y ′ , θ⟩ + sup θ ′ ∈∂Ω(y ′ )⟨yy ′ , θ ′ ⟩ .The Fitzpatrick function was studied in depth in [2] . In particular, it is jointly convex and satisfiesEQUATIONFrom Danskin's theorem, when dom Ω is compact, we also haveEQUATIONThe Fitzpatrick function F [∂Ω](y, θ) and Ω ⊕ Ω * (y, θ) = Ω(y) + Ω * (θ) play a similar role but the latter is separable in y and θ, while the former is not. In particular this makes the subdifferential ∂ θ F [∂Ω](y, θ) depend on both y and θ, while ∂ θ (Ω ⊕ Ω * )(y, θ) = ∂Ω * (θ) depends only on θ.The Fitzpatrick function was used in [10] to theoretically study primal-primal Bregman-like divergences. As discussed in more detail in Section 3.4, using these divergences for machine learning would require us to compose them with an explicit link function, which would typically break convexity. In the next section, we introduce new primal-dual losses based on the Fitzpatrick function.3 Fitzpatrick losses 
definition and properties: 
 Inspired by the inequality in (3), which we can view as a refined Fenchel-Young inequality, we introduce Fitzpatrick losses, a new family of loss functions generated by a convex l.s.c. function Ω.Let Ω : R k → R be a proper convex l.s.c. function. When y ∈ dom Ω and θ ∈ R k , we define the Fitzpatrick lossL F [∂Ω] : R k × R k → R generated by Ω as L F [∂Ω] (y, θ) := F [∂Ω](y, θ) -⟨y, θ⟩ = sup (y ′ ,θ ′ )∈∂Ω ⟨y -y ′ , θ ′ ⟩ + ⟨y ′ , θ⟩ -⟨y, θ⟩ = sup (y ′ ,θ ′ )∈∂Ω ⟨y ′ -y, θ -θ ′ ⟩.When y ̸ ∈ dom Ω, L F [∂Ω] (y, θ) := +∞.Fitzpatrick losses enjoy similar properties as Fenchel-Young losses, while being tighter than Fenchel-Young losses.Proposition 1 Properties of Fitzpatrick losses 1. Non-negativity: for all (y, θ) ∈ R k × R k , L F [∂Ω] (y, θ) ≥ 0.2. Same link function:L Ω⊕Ω * (y, θ) = L F [∂Ω] (y, θ) = 0 ⇐⇒ y = y Ω (θ).3. Convexity: L F [∂Ω] (y, θ) is convex in y and θ separately.4. (Sub-)Gradient:∂ θ L F [∂Ω] (y, θ) = y ⋆ F [∂Ω](y, θ)-y where y ⋆ F [∂Ω] (y, θ) is given by (4).We now present the Fitzpatrick loss counterparts of various Fenchel-Young losses.Proposition 3 Squared loss as a Fitzpatrick lossWhen Ω(y′ ) = 1 2 ∥y ′ ∥ 2 2 , we have for all y ∈ R k and θ ∈ R k L F [∂Ω] (y, θ) = 1 4 ∥y -θ∥ 2 2 = 1 2 L squared (y, θ).A proof is given in Appendix B.4. Therefore, the Fenchel-Young and Fitzpatrick losses generated by Ω coincide, up to a factor 1 2 . Perceptron loss.When Ω(y ′ ) = ι C (y ′ ), where C is a closed convex set, we have for all y ∈ C and θ∈ R k L F [∂Ω] (y, θ) = L perceptron (y, θ) = max y ′ ∈C ⟨y ′ , θ⟩ -⟨y, θ⟩.A proof is given in Appendix B.5. Therefore, the Fenchel-Young and Fitzpatrick losses generated by Ω exactly coincide in this case.Fitzpatrick sparseMAP and Fitzpatrick sparsemax losses. As our first example where Fenchel-Young and Fitzpatrick losses substantially differ, we introduce the Fitzpatrick sparseMAP loss, which is the Fitzpatrick counterpart of the sparseMAP loss [24] .Proposition 5 Fitzpatrick sparseMAP loss When Ω(y ′ ) = 1 2 ∥y ′ ∥ 2 2 + ι C (y ′ ), where C is a closed convex set, we have for all y ∈ C and θ ∈ R k L F [∂Ω] (y, θ) = 2Ω * ((y + θ)/2) -⟨y, θ⟩ = ⟨y ⋆ -y, θ -y ⋆ ⟩where we used y ⋆ as a shorthand fory ⋆ F [∂Ω] (y, θ) = ∇Ω * ((y + θ)/2) = P C ((y + θ)/2).A proof is given in Appendix B.6. As a special case, when C = △ k , we call the obtained loss the Fitzpatrick sparsemax loss, as it is the counterpart of the sparsemax loss [21] . Like the sparseMAP and sparsemax losses, these new losses rely on the Euclidean projection as a core building block. The Euclidean projection onto the probability simplex △ k can be computed exactly in O(k) expected time and O(k log k) worst-case time [9, 22, 14, 12] .Fitzpatrick logistic loss. We now derive the Fitzpatrick couterpart of the logistic loss. Before stating the next proposition, we recall the definition of the Lambert W function [13] . For z ≥ 0,W (z) is the inverse of the function f (w) = w exp(w), that is, W (z) = f -1 (z) = w. Proposition 6 Fitzpatrick logistic loss When Ω(y ′ ) = ⟨y ′ , log y ′ ⟩ + ι △ k (y ′ ), we have for all y ∈ △ k and θ ∈ R k L F [∂Ω] (y, θ) = ⟨y ⋆ -y, θ -log y ⋆ -1⟩where we used y ⋆ as a shorthand for y ⋆ F [∂Ω] (y, θ) defined byy ⋆ F [∂Ω] (y, θ) i = e -λ ⋆ e θi , if y i = 0, yi W (yie λ ⋆ -θ i ) , if y i > 0.A proof and the value of λ ⋆ = λ ⋆ (y, θ) ∈ R are given in Appendix B.7. To obtain λ ⋆ (y, θ), we need to solve a one-dimensional root equation, which can be done using for instance a bisection.On first sight, Fitzpatrick losses and Fenchel-Young losses appear quite different. In the next proposition, we show that the Fitzpatrick loss generated by Ω is in fact equal to the Fenchel-Young loss generated by the modified, target-dependent functionΩ y (y ′ ) := Ω(y ′ ) + D Ω (y, y ′ ),where D Ω is the generalized Bregman divergence defined in (1) . In particular, Lemma 1 in the appendix shows that ifΩ = Ψ + ι C , then Ω y (y ′ ) = Ψ y (y ′ ) + ι C (y ′ ) = Ψ(y ′ ) + D Ψ (y, y ′ ) + ι C (y ′ ). Proposition 7 Characterization of F [∂Ω], L F [∂Ω] and y ⋆ F [∂Ω] using Ω yLet Ω : R k → R be a proper convex l.s.c. function. Then, for all y ∈ dom Ω and all θ ∈ R k ,F [∂Ω](y, θ) = Ω y (y) + Ω * y (θ) L F [∂Ω] (y, θ) = L Ωy⊕Ω * y (y, θ) y ⋆ F [∂Ω] (y, θ) = y Ωy (θ).is also new to our knowledge. A proof is given in Appendix B.8. Proposition 7 is very useful, as it means that Fitzpatrick losses inherit from all the known properties of Fenchel-Young losses, analyzed in prior works [7, 5] . In particular, Fenchel-Young losses are smooth (i.e., with Lipschitz gradients) when Ω is strongly convex. We therefore immediately obtain that Fitzpatrick losses are smooth if Ω is strongly convex and D Ω is convex in its second argument, which is the case when Ω(y ′ ) = 1 2 ∥y ′ ∥ 2 2 and Ω(y ′ ) = ⟨y ′ , log y ′ ⟩. Therefore, the Fitzpatrick sparsemax and logistic losses are smooth. However in the general case, this does not hold, as D Ω is usually not convex in its second argument. Proposition 7 also provides a mean to compute Fitzpatrick losses and their gradient. Finally, it suggests a very natural geometric interpretation of Fitzpatrick losses, as presented in Figure 2 .As we stated before, the generalized Bregman divergence D Ω (y, y ′ ) in ( 1) is a primal-primal divergence, as both y and y ′ belong to the same primal space. In contrast, Fenchel-Young losses L Ω⊕Ω * (y, θ) are primal-dual, since y belongs to the primal space and θ belongs to the dual space.Ω y Ω y Ω(y) = Ω y (y) y, θ -Ω * (θ) y, θ -Ω * y (θ) L F [∂Ω] (y, θ) = L Ωy⊕Ω * y (y, θ) L Ω⊕Ω * (y, θ) -Ω * (θ) -Ω * y (θ) Figure 2: Geometric interpretation, with Ω(y ′ ) = 1 2 ∥y ′ ∥ 2 2 .The Fenchel-Young loss L Ω⊕Ω * (y, θ) is the gap (depicted with a double-headed arrow) between Ω(y) and ⟨y, θ⟩ -Ω * (θ), the value at y of the tangent with slope θ and intercept -Ω * (θ). As per Proposition 7, the Fitzpatrick loss L F [∂Ω] (y, θ) is equal to L Ωy⊕Ω * y (y, θ) and is therefore equal to the gap between Ω y (y) = Ω(y) and ⟨y, θ⟩ -Ω * y (θ), the value at y of the tangent with slope θ and intercept -Ω * y (θ). Since Ω y (y ′ ) = Ω(y ′ ) + D Ω (y, y ′ ), we have that Ω y (y ′ ) ≥ Ω(y ′ ), with equality when y = y ′ . We therefore have Ω * y (θ) ≤ Ω * (θ), implying that the Fitzpatrick loss is a lower bound of the Fenchel-Young loss.Both can however be related, since infθ ′ ∈∂Ω(y ′ ) L Ω⊕Ω * (y, θ ′ ) = inf θ ′ ∈∂Ω(y ′ ) Ω(y) + Ω * (θ ′ ) -⟨y, θ ′ ⟩ = Ω(y) + inf θ ′ ∈∂Ω(y ′ ) Ω * (θ ′ ) -⟨y, θ ′ ⟩ = Ω(y) -sup θ ′ ∈∂Ω(y ′ ) -Ω * (θ ′ ) + ⟨y, θ ′ ⟩ = Ω(y) -Ω(y ′ ) -sup θ ′ ∈∂Ω(y ′ ) ⟨y -y ′ , θ ′ ⟩ = D Ω (y, y ′ )where, in the penultimate line, we have used that Ω * (θ ′ ) = ⟨y ′ , θ ′ ⟩ -Ω(y ′ ), as θ ′ ∈ ∂Ω(y ′ ). This identity suggests that we can create Bregman-like primal-primal divergences by replacingΩ ⊕ Ω * with F [∂Ω], D F [∂Ω] (y, y ′ ) := inf θ ′ ∈∂Ω(y ′ ) L F [∂Ω] (y, θ ′ ) = inf θ ′ ∈∂Ω(y ′ ) F [∂Ω](y, θ ′ ) -⟨y, θ ′ ⟩.This recovers one of the two Bregman-like divergences proposed in [10] , the other one replacing the inf above by a sup. As stated in [10] , F [∂Ω] and Ω ⊕ Ω * are representations of ∂Ω.In order to use a primal-primal divergence as a loss, we need to explicitly compose it with a link function, such as y Ω (θ) = ∇Ω * (θ). Unfortunately, D Ω (y, y Ω (θ)) or D F [∂Ω] (y, y Ω (θ)) are typically nonconvex functions of θ, while Fenchel-Young and Fitzpatrick losses are always convex. In addition, differentiating through y Ω (θ) typically requires implicit differentiation [19, 4] , while Fenchel-Young and Fitzpatrick losses enjoy easy-to-compute gradients, thanks to Danskin's theorem.If Ω = Ψ + ι C , where Ψ : R k → R is a proper convex Legendre-type function and C ⊆ dom Ψ, then it was shown in [7, Proposition 3] that Fenchel-Young losses satisfy the lower boundD Ψ (y, y) ≤ L Ω⊕Ω * (y, θ),with equality if C = dom Ψ, where we used y as a shorthand for y Ω (θ). We now show that a similar result holds for Fitzpatrick losses. Similarly as before, we define Ψ y (y ′ ) := Ψ(y ′ ) + D Ψ (y, y ′ ).Let Ω = Ψ + ι C , where Ψ : R k → R is a proper convex Legendre-type function and C ⊆ dom Ψ.Then,D Ψy (y, y ⋆ ) = ⟨y -y ⋆ , ∇ 2 Ψ(y ⋆ )(y -y ⋆ )⟩ ≤ L F [∂Ω] (y, θ),with equality if dom Ψ = C, where we used y ⋆ as a shorthand for y ⋆ F [∂Ω] (y, θ).A proof is given in Appendix B.9. If Ψ y is µ-strongly convex, we obtain µ 2 ∥yy ⋆ ∥ 2 2 ≤ D Ψy (y, y ⋆ ). 
experiments: 
 Experimental setup. We follow exactly the same experimental setup as in [6, 7] . We consider a dataset of n pairs (x i , y i ) of feature vector x i ∈ R d and label proportions y i ∈ △ k ,EQUATIONwhereL ∈ L Ω⊕Ω * , L F [∂Ω]. We focus on the (Fitzpatrick) sparsemax and the (Fitzpatrick) logistic losses. We optimize (5) using the L-BFGS algorithm [20] . The gradient of the Fenchel-Young loss is given in (2), while the gradient of the Fitzpatrick loss is given in Proposition 1, item 4. Experiments were conducted on a Intel Xeon E5-2667 clocked at 3.30GHz with 192 GB of RAM running on Linux. Our implementation relies on the SciPy [29] and scikit-learn [25] libraries.We ran experiments on 11 standard multi-label benchmark datasets1 (see Table 2 in Appendix A for statistics on the datasets). For all datasets, we removed samples with no label, normalized samples to have zero mean unit variance, and normalized labels to lie in the probability simplex. We chose the hyperparameter λ ∈ {10 -4 , 10 -3 , . . . , 10 4 } against the validation set. We report test set mean squared error (also known as Brier score in a probabilistic forecasting context) in Table 1 .Results. We found that the logistic loss and the Fitzpatrick logistic loss are comparable on most datasets, with the logistic loss significantly winning on 2 datasets and the Fitzpatrick logistic loss significantly winning on 2 datasets, out of 11. Since the Fitzpatrick logistic loss is slightly more computationally demanding, requiring to solve a root equation while the logistic loss does not, we believe that the logistic loss remains the best choice when we wish to use the softargmax as link function y Ω .Similarly, we found that the sparsemax loss and the Fitzpatrick sparsemax loss are comparable on most datasets, with the sparsemax loss significantly winning on only 1 dataset out of 11 and the Fitzpatrick loss significantly winning on 2 datasets out of 11. Since the two losses both use the Euclidean projection onto the simplex P △ k as their link function y Ω , we conclude that the Fitzpatrick sparsemax loss is a serious contender to the sparsemax loss, especially when predicting sparse label proportions is important. 
tighter inequality:: 
 for all (y, θ) ∈ R k , 0 ≤ L F [∂Ω] (y, θ) ≤ L Ω⊕Ω * (y, θ).A proof is given in Appendix B.2. Because the Fitzpatrick loss and the Fenchel-Young loss generated by the same Ω have the same link function y Ω , they share the same minimizers w.r.t. θ for y fixed. However, the Fitzpatrick loss is always a lower bound of the corresponding Fenchel-Young loss. Moreover, they have different gradients w.r.t. θ:∂ θ L Ω⊕Ω * (y, θ) = y Ω (θ) -y vs. ∂ θ L F [∂Ω] (y, θ) = y ⋆ F [∂Ω] (y, θ) -y.It is worth noticing that y ⋆ F [∂Ω] (y, θ) depends on both y and θ, contrary to y Ω (θ). When Ω is a twice differentiable function on its domain (which is for instance the case of the squared 2-norm or the negentropy), we next show that Fitzpatrick losses enjoy a particularly simple expression and become a squared Mahalanobis-like distance.Proposition 2 Expressions of F [∂Ω](y, θ) and L F [∂Ω] (y, θ) when Ω is twice differentiable Let Ω : R k → R be a convex function such that dom Ω is an open set. Let us assume that Ω is twice differentiable. Then, for any y ∈ dom Ω, we have thatF [∂Ω](y, θ) = ⟨y, ∇Ω(y ⋆ )⟩ + ⟨y ⋆ , θ⟩ -⟨y ⋆ , ∇Ω(y ⋆ )⟩ L F [∂Ω] (y, θ) = ⟨y ⋆ -y, θ -∇Ω(y ⋆ )⟩ = ⟨y ⋆ -y, ∇ 2 Ω(y ⋆ )(y ⋆ -y)⟩ where y ⋆ = y ⋆ F [∂Ω] (y, θ) is a solution w.r.t. y ′ ∈ int dom Ω of ∇ 2 Ω(y ′ )(y ′ -y) = θ -∇Ω(y ′ ).A proof is given in B.3. When Ω is constrained (i.e., when it contains an indicator function), we show in Section 3.5 that the above expression becomes a lower bound.We proposed to leverage the Fitzpatrick function, a theoretical tool from monotone operator theory, to build a new family of primal-dual convex loss functions for machine learning. We reinterpreted Fitzpatrick losses as lower bounds of Fenchel-Young losses that maintains the same link function.Our paper therefore challenges the idea that there can only be one loss function, convex in each argument, associated with a certain link function. For instance, we created the 1 : Test performance comparison between the sparsemax loss, the logistic loss and their Fitzpatrick counterparts on the task of label proportion estimation, with regularization parameter λ tuned against the validation set. For each dataset, label proportion errors are measured using the mean squared error (MSE). We use bold if the error is at least 0.005 lower than its counterpart. and sparsemax losses, that are associated with the soft argmax and sparse argmax links, traditionally associated with the logistic and sparsemax losses, respectively. We believe that even more loss functions with the same link can be created, which calls for a systematic study of their properties and respective benefits. In future work, we intend to study calibration guarantees for Fitzpatrick losses, to test new link functions from the maximal monotone operator theory and to implement more efficient training for the Fitzpatrick logistic loss. ∈ R k D Ω (y, y ′ ) = D Ψ (y, y ′ ) + D ι C (y, y ′ ).As C, dom Ψ ⊂ R k and riC ∩ ri dom Ψ ̸ = ∅, we can apply [3, Proposition 6.19 ] and [3, Theorem 16 .47] to write ∂Ω(y ′ ) = ∂Ψ(y ′ ) + ∂ι C (y ′ ).D Ω (y, y ′ ) := Ω(y) -Ω(y ′ )supθ ′ ∈∂Ω(y ′ ) ⟨y -y ′ , θ ′ ⟩ = Ψ(y) + ι C (y) -Ψ(y ′ ) -ι C (y ′ ) - sup θ ′ ∈∂Ψ(y ′ )+∂ι C (y ′ ) ⟨y -y ′ , θ ′ ⟩ = Ψ(y) -Ψ(y ′ ) -sup θ ′ ∈∂Ψ(y ′ ) ⟨y -y ′ , θ ′ ⟩ + ι C (y) -ι C (y ′ ) -sup θ ′ ∂ι C (y ′ ) ⟨y -y ′ , θ ′ ⟩ = D Ψ (y, y ′ ) + D ι C (y, y ′ ). Lemma 2 Generalized Bregman divergence of indicator function of some set C ⊂ R k D ι C (y, y ′ ) = ι C (y) if y ′ ∈ C ∞ if y ′ ̸ ∈ C = ι C (y) + ι C (y ′ ).Proof. Using (1) and the fact that ∂ι C = N C [3, Example 16.13], we obtain thatD ι C (y, y ′ ) := ι C (y) -ι C (y ′ ) -sup θ ′ ∈N C (y ′ ) ⟨y -y ′ , θ ′ ⟩.When y ′ ∈ C and y ∈ C, supθ ′ ∈N C (y ′ ) ⟨y -y ′ , θ ′ ⟩ = sup θ ′ ∈R k ⟨z-y ′ ,θ ′ ⟩≤0 ∀z∈C ⟨y -y ′ , θ ′ ⟩ = 0.When y ′ ∈ C and y / ∈ C, D ι C (y, y ′ ) = +∞, as +∞ + (-∞) = +∞ in the definition of the Bregman divergence. Therefore, when y ′ ∈ C D ι C (y, y ′ ) = ι C (y).When y ′ ̸ ∈ C, N C (y ′ ) = ∅. Again, in the definition of the Bregman divergence, +∞ + (-∞) = +∞ and we use the convention sup ∅ = -∞.Let Ψ : R k → R be proper, convex and twice differentiable on the interior of its domain. For all y ∈ R k , let Ψ y := Ψ + D Ψ (y, •). Then, for all y, y ′ , y ′′ ∈ int dom Ψ, D Ψy (y ′′ , y ′ ) = D Ψ (y, y ′′ ) -D Ψ (y, y ′ ) + D Ψ (y ′′ , y ′ ) + ⟨y ′′y ′ , ∇ 2 Ψ(y ′ )(yy ′ )⟩ = ⟨y ′′y, ∇Ψ(y ′′ )⟩ -⟨y ′′y, ∇Ψ(y ′ )⟩ + ⟨y ′′y ′ , ∇ 2 Ψ(y ′ )(yy ′ )⟩ and in particular for all y, y ′ ∈ int dom Ψ D Ψy (y, y ′ ) = ⟨yy ′ , ∇ 2 Ψ(y ′ )(yy ′ )⟩.Proof. For all y, y ′ ∈ int dom Ψ, as Ψ is convex and differentiable on int dom Ψ, we have thatΨ y (y ′ ) = Ψ(y ′ ) + D Ψ (y, y ′ ) = Ψ(y ′ ) + Ψ(y) -Ψ(y ′ ) -⟨y -y ′ , ∇Ψ(y ′ )⟩ = Ψ(y) -⟨y -y ′ , ∇Ψ(y ′ )⟩.and therefore, as Ψ is twice differentiable on int dom Ψ, we get that∇Ψ y (y ′ ) = ∇ 2 Ψ(y ′ )(y ′ -y) + ∇Ψ(y ′ ).Therefore, for all y, y ′ , y ′′ ∈ dom Ψ,D Ψy (y ′′ , y ′ ) = Ψ y (y ′′ ) -Ψ y (y ′ ) -⟨y ′′ -y ′ , ∇Ψ y (y ′ )⟩ = Ψ(y ′′ ) + D Ψ (y, y ′′ ) -Ψ(y ′ ) -D Ψ (y, y ′ ) -⟨y ′′ -y ′ , ∇ 2 Ψ(y ′ )(y ′ -y)⟩ -⟨y ′′ -y ′ , ∇Ψ(y ′ )⟩ = D Ψ (y, y ′′ ) -D Ψ (y, y ′ ) + D Ψ (y ′′ , y ′ ) + ⟨y ′′ -y ′ , ∇ 2 Ψ(y ′ )(y -y ′ )⟩ = ⟨y ′′ -y, ∇Ψ(y ′′ )⟩ -⟨y ′′ -y, ∇Ψ(y ′ )⟩ + ⟨y ′′ -y ′ , ∇ 2 Ψ(y ′ )(y -y ′ )⟩and in particular using the just last established equality with the triplet (y, y, y ′ ), we have for all y, y ′ ∈ dom Ψ, D Ψy (y, y ′ ) = ⟨yy, ∇Ψ(y)⟩ -⟨yy, ∇Ψ(y ′ )⟩ + ⟨yy ′ , ∇ 2 Ψ(y ′ )(yy ′ )⟩ = ⟨yy ′ , ∇ 2 Ψ(y ′ )(yy ′ )⟩.Let α ∈ R. Let Ψ(y ′ ) := k i=1 y ′ i log y ′ i -α k i=1 y ′ i be defined for y ′ ∈ R k + . Then, for y, y ′ ∈ R k + , D Ψ (y, y ′ ) = k i=1 y i log y i y ′ i - k i=1 (y i -y ′ i ) + ι R k ++ (y ′ ). Proof. First, if y ′ ∈ R k ++ , Ψ is differentiable at y ′ and ∇Ψ(y ′ ) i = log y i + 1 -α. Thus, ∂Ψ(y ′ ) = {∇Ψ(y ′ )} and D Ψ (y, y ′ ) = Ψ(y) -Ψ(y ′ ) -sup θ ′ ∈∂Ψ(y ′ ) ⟨y -y ′ , θ ′ ⟩, = Ψ(y) -Ψ(y ′ ) -⟨y -y ′ , ∇Ψ(y ′ )⟩, = k i=1 y i log y i y ′ i - k i=1 (y i -y ′ i ).Second, if we prove that ∂Ψ(y ′ ) = ∅ when there exists a component i such that y ′ i = 0, we can conclude the proof, as sup ∅ = -∞ by convention. Indeed, Let us assume that y ′ i = 0. Suppose that θ ′ ∈ ∂Ψ(y ′ ). Then, by definition of subgradients,⟨y ′′ -y ′ , θ ′ ⟩ + Ψ(y ′ ) ≤ Ψ(y ′′ ), ∀y ′′ ∈ R k ++ .We consider ε > 0 and choose y ′′ = y ′ + εe i , where e i is the i-th canonical base vector. Thus, we obtainεθ i ≤ Ψ(y ′ + εe i ) -Ψ(y ′ ), = k j=1 y ′ j log y ′ j + ε log ε -α k j=1 y ′ j -αε - k j=1 y ′ j log y ′ j -α k j=1 y ′ j , = ε log ε -αε,as y i = 0 and 0 log 0 = 0 by convention. By noticing that lim ε→0 + ε log εαε /ε = -∞, we get a contradiction, which concludes the proof.Lemma 5 Value and gradient of Ψ * y Let Ψ : R k → R be a proper strictly convex twice differentiable function. Let us assume that D Ψ (y, y ′ ) is convex w.r.t. y ′ , for all y ∈ dom Ψ. We remind that Ψ y (y ′ ) = Ψ(y ′ ) + D Ψ (y, y ′ ).Then, for all θ ∈ R k ,Ψ * y (θ) = ⟨ỹ, θ⟩ -Ψ(y) + ⟨y -ỹ, ∇Ψ(ỹ)⟩ ∇Ψ * y (θ) = ỹwhere ỹ is the unique element of the setargmax y ′ ∈dom Ψ ⟨y ′ , θ⟩ + ⟨y -y ′ , ∇Ψ(y ′ )⟩ = y ′ ∇ 2 Ψ(y ′ )(y ′ -y) = θ -∇Ψ(y ′ ) .Proof. As Ψ ≤ Ψ y , we have dom Ψ y ⊂ dom Ψ. Thus, we getΨ * y (θ) = sup y ′ ∈dom Ψ ⟨y ′ , θ⟩ -Ψ y (y ′ ) = sup y ′ ∈dom Ψ ⟨y ′ , θ⟩ -(Ψ(y ′ ) + Ψ(y) -Ψ(y ′ ) -⟨y -y ′ , ∇Ψ(y ′ )⟩) = sup y ′ ∈dom Ψ⟨y ′ , θ⟩ -Ψ(y) + ⟨yy ′ , ∇Ψ(y ′ )⟩.As Ψ is strictly convex and D Ψ (y, y ′ ) is convex w.r.t. y ′ , we have that Ψ y is strictly convex. Thus, according to [8, Theorem 4.2.5], Ψ * y is differentiable. Using Danskin's theorem, {∇Ψ * y (θ)} = argmaxy ′ ∈dom Ψ ⟨y ′ , θ⟩ -Ψ(y) + ⟨y -y ′ , ∇Ψ(y ′ )⟩ = argmax y ′ ∈dom Ψ ⟨y ′ , θ⟩ + ⟨y -y ′ , ∇Ψ(y ′ )⟩.Setting the gradient of the inner function to zero concludes the proof.Lemma 6 Gradient of Ψ * y , squared norm case Let Ψ(y ′ ) := 1 2 ∥y ′ ∥ 2 2 and Ψ y defined as in Lemma 5. Then, the gradient of Ψ * y is given by∇Ψ * y (θ) = y + θ 2 .Proof. Using Lemma 5 with ∇Ψ(y ′ ) = y ′ and ∇ 2 Ψ(y ′ ) = I, we obtain that ∇Ψ * y (θ) is the solution w.r.t. y ′ of the equation y ′y = θy ′ . Rearranging the terms concludes the proof.Before stating the next lemma, we recall the definition of the Lambert function [13] W : R + → R + . The function W is the inverse of the function f : R + → R + where f (w) = w exp(w) for all w ∈ R + .Lemma 7 Gradient of Ψ * y , negentropy case Let Ψ(y ′ ) := k i=1 y ′ i log y ′ i -α k i=1 y ′ i be defined for y ′ ∈ R k + . Then, ∇Ψ * y (θ) i = e θi-2+α , if y i = 0 yi W (yie -(θ i -2+α) ) , if y i > 0.Proof. Using Lemma 5, we know that ỹ is the solution of ∇ 2 Ψ(ỹ)(ỹy) = θ -∇Ψ(ỹ). Using ∇Ψ(ỹ) = log ỹ + 1α and ∇ 2 Ψ(ỹ) = 1/ỹ (where logarithm and division are performed elementwise), we obtain for all i ∈ [k](ỹ i -y i )/ỹ i = θ i -log ỹi -1 + α ⇐⇒ 1 -y i /ỹ i = θ i -log ỹi -1 + α.When y i = 0, we immediatly have ỹi = exp(θ i -2 + α). When y i > 0, after rearranging, we obtainy i ỹi exp y i ỹi = y i exp(-(θ i -2 + α)) ⇐⇒ y i ỹi = W (y i exp(-(θ i -2 + α))),hence the result. Proof. The result again follows from Danskin's theorem.Let Ψ : R k → R be a proper strictly convex function. Then,(Ψ + ι △ k ) * (θ) = min τ ∈R τ + (Ψ + ι R k + ) * (θ -τ 1). and ∇(Ψ + ι △ k ) * (θ) = ∇(Ψ + ι R k + ) * (θ -τ ⋆ 1), where τ ⋆ denotes an optimal dual variable (not necessarily unique).Proof.(Ψ + ι △ k ) * (θ) = max y ′ ∈△ k ⟨y ′ , θ⟩ -Ψ(y ′ ) = max y ′ ∈R k + min τ ∈R ⟨y ′ , θ⟩ -Ψ(y ′ ) -τ (⟨y ′ , 1⟩ -1) = min τ ∈R τ + max y ′ ∈R k + ⟨y ′ , θ -τ 1⟩ -Ψ(y ′ ) = min τ ∈R τ + (Ψ + ι R k + ) * (θ -τ 1), where we used that, as the constraints of belonging to the simplex are affine, they are qualified and we can invert the max and the min. [8, Theorem 4.2.5 ]. We conclude using Danskin's theorem.As Ψ is strictly convex, (Ψ + ι △ k ) * and (Ψ + ι R k + ) * are differentiableLemma 10 Gradient of Ω * y , negentropy, constrained to the simplex Let Ψ(y ′ ) = ⟨y ′ , log y ′ ⟩, if y ′ ∈ R k + , +∞ otherwise. Then, for all y ∈ R k + , the gradient of Ω y = Ψ y + ι △ k (as defined in Lemma 8) is given by∇Ω * y (θ) i = e -λ ⋆ e θi if y i = 0, yi W (yie λ ⋆ -θ i ) if y i > 0.where λ ⋆ is the unique solution of the equation e -λ ⋆ i:yi=0 e θi + i:yi>0y i W (y i e -(θi-λ ⋆ ) ) = 1.Proof. From Lemma 8 and Lemma 9, since dom Ψ y = R k + , we havey ⋆ = ∇Ω * y (θ) = ∇Ψ * y (θ -τ ⋆ 1) where τ ⋆ is a solution of min τ ∈R τ + Ψ * y (θ -τ 1). Setting the gradient of the inner function to zero, we get ⟨∇Ψ * y (θτ ⋆ 1), 1⟩ = 1. Using Lemma 7, we obtain that τ ⋆ satisfies e -τ ⋆ -2 i:yi=0e θi + i:yi>0 y i W (y i e -(θi-τ ⋆ -2) ) = 1.By monotonicity in τ ⋆ , we conclude that τ ⋆ is unique. Using the change of variable τ ⋆ = λ ⋆ + 2 concludes the proof.Apart from differentiability, the proofs follow from the study of Fitzpatrick functions found in [15, 2, 28] . We include the proofs for completeness.Link function and non-negativity. We recall thatL F [∂Ω] (y, θ) = sup (y ′ ,θ ′ )∈∂Ω ⟨y ′ -y, θ -θ ′ ⟩ = -inf (y ′ ,θ ′ )∈∂Ω ⟨y ′ -y, θ ′ -θ⟩.From the monotonicity of ∂Ω, we have that if (y, θ) ∈ ∂Ω and (y ′ , θ ′ ) ∈ ∂Ω, then ⟨y ′ -y, θ ′ -θ⟩ ≥ 0.Therefore, for all (y, θ) ∈ ∂Ω, inf(y ′ ,θ ′ )∈∂Ω ⟨y ′ -y, θ ′ -θ⟩ = 0,with the infimum being attained at (y ′ , θ ′ ) = (y, θ). This proves the link function.From the maximality of ∂Ω, if (y, θ) ̸ ∈ ∂Ω, there exists (y ′ , θ ′ ) ∈ ∂Ω such that ⟨y ′y, θ ′ -θ⟩ < 0.Therefore, for all (y, θ) ̸ ∈ ∂Ω, inf(y ′ ,θ ′ )∈∂Ω ⟨y ′ -y, θ ′ -θ⟩ < 0.This proves the non-negativity.Convexity. We recall thatL F [∂Ω] (y, θ) = F [∂Ω](y, θ) -⟨y, θ⟩ where F [∂Ω](y, θ) = sup (y ′ ,θ ′ )∈∂Ω ⟨y -y ′ , θ ′ ⟩ + ⟨y ′ , θ⟩ = sup (y ′ ,θ ′ )∈∂Ω ⟨y ′ , θ⟩ + ⟨y, θ ′ ⟩ -⟨y ′ , θ ′ ⟩.The function (y, θ) → ⟨y ′ , θ⟩ + ⟨y, θ ′ ⟩ -⟨y ′ , θ ′ ⟩ is jointly convex in (y, θ) for all (y ′ , θ ′ ). Since the supremum preserves convexity, F [∂Ω](y, θ) is jointly convex in (y, θ). The function ⟨y, θ⟩ is separately convex / concave in y and θ but not jointly convex / concave in (y, θ). Therefore, L F [∂Ω] (y, θ) is separately convex in y and θ.Differentiability. Since Ω(y ′ ) is strictly convex and y ′ → D Ω (y, y ′ ) is convex, Ω y (y ′ ) = Ω(y ′ ) + D Ω (y, y ′ ) is strictly convex in y ′ . From the duality between strict convexity and differentiability, Ω * y (θ) is differentiable in θ.Using∂Ω = {(y ′ , θ ′ ) : Ω(y) ≥ Ω(y ′ ) + ⟨y -y ′ , θ ′ ⟩ ∀y} and Ω * (θ) = sup y ′ ∈R k ⟨y ′ , θ⟩ -Ω(y ′ ),we get for any (y ′ , θ ′ ) ∈ ∂Ω,⟨y -y ′ , θ ′ ⟩ + ⟨y ′ , θ⟩ ≤ Ω(y) -Ω(y ′ ) + ⟨y ′ , θ⟩ ≤ Ω(y) + Ω * (θ). Therefore F [∂Ω](y, θ) = sup (y ′ ,θ ′ )∈∂Ω ⟨y -y ′ , θ ′ ⟩ + ⟨y ′ , θ⟩ ≤ Ω(y) + Ω * (θ).We recall thatF [∂Ω](y, θ) = sup (y ′ ,θ ′ )∈∂Ω ⟨y, θ ′ ⟩ + ⟨y ′ , θ⟩ -⟨y ′ , θ ′ ⟩ = sup y ′ ∈dom Ω ⟨y ′ , θ⟩ + sup θ ′ ∈∂Ω(y ′ ) ⟨y, θ ′ ⟩ -⟨y ′ , θ ′ ⟩.Since Ω is differentiable, we have ∂Ω(y ′ ) = {∇Ω(y ′ )} and therefore θ ′ = ∇Ω(y ′ ), which givesF [∂Ω](y, θ) = sup y ′ ∈R k ⟨y, ∇Ω(y ′ )⟩ + ⟨y ′ , θ⟩ -⟨y ′ , ∇Ω(y ′ )⟩.Setting the gradient of the inner function w.r.t. y ′ to zero, we get∇ 2 Ω(y ′ )y + θ -∇Ω(y ′ ) -∇ 2 Ω(y ′ )y ′ = 0.Using the y ′ = y ⋆ and θ ′ = ∇Ω(y ′ ) in Definition 1, we then obtainL F [∂Ω] (y, θ) = ⟨y ′ -y, θ -θ ′ ⟩ = ⟨y ′ -y, θ -∇Ω(y ′ )⟩ = ⟨y ′ -y, ∇ 2 Ω(y ′ )(y ′ -y)⟩.Using Proposition 2 with ∇Ω(y ′ ) = y ′ and ∇ 2 Ω(y ′ ) = I, we obtainy + θ -2y ′ = 0 ⇐⇒ y ′ = y + θ 2 .We therefore obtainL F [∂Ω] (y, θ) = y + θ 2 -y, θ - y + θ 2 = θ -y 2 , θ -y 2 = 1 4 ∥y -θ∥ 2 2 .A proof of the Fitzpatrick function for this case was given in [2, Example 3.1]. We include a proof for completeness. Since Ω = ι C , we have dom Ω = C. Therefore, for all y ∈ C and θ ∈ R k ,F [∂Ω](y, θ) = sup y ′ ∈dom Ω ⟨y ′ , θ⟩ + sup θ ′ ∈∂Ω(y ′ ) ⟨y -y ′ , θ ′ ⟩ = sup y ′ ∈C ⟨y ′ , θ⟩ -ι C (y) -ι C (y ′ ) -sup θ ′ ∈∂ι C (y ′ ) ⟨y -y ′ , θ ′ ⟩ = sup y ′ ∈C ⟨y ′ , θ⟩ -D ι C (y, y ′ ) = sup y ′ ∈C ⟨y ′ , θ⟩,where in the third line we used that ι C (y) = ι C (y ′ ) = 0 and where in the last line we used Lemma 2.Therefore, for all y ∈ R k and θ ∈ R k ,F [∂Ω](y, θ) = sup y ′ ∈C ⟨y ′ , θ⟩ + ι C (y) = ι C (y) + ι * C (θ).A proof of the Fitzpatrick function for this case was given in [2, Example 3.13]. We provide an alternative proof.From Proposition 7, we know thatF [∂Ω](y, θ) = Ω y (y) + Ω * y (θ) = Ω(y) + Ω * y (θ), where Ω y (y ′ ) = 1 2 ∥y ′ ∥ 2 2 + 1 2 ∥y -y ′ ∥ 2 2 + ι C (y ′ ) = ∥y ′ ∥ 2 2 + 1 2 ∥y∥ 2 2 -⟨y, y ′ ⟩ + ι C (y ′ ) = 2Ω(y ′ ) + Ω(y) -⟨y, y ′ ⟩.Using conjugate calculus, we obtainΩ * y (θ) = 2Ω * y + θ 2 -Ω(y).Therefore,F [∂Ω](y, θ) = 2Ω * y + θ 2 .From Proposition 7, the supremum w.r.t. y ′ is achieved at y ⋆ = ∇Ω * ((y + θ)/2) = P C ((y + θ)/2).We therefore obtain L F [∂Ω] (y, θ) = ⟨y ⋆ -y, θ -y ⋆ ⟩.y ⋆ F [∂Ω] (y, θ) i = e -λ ⋆ e θi , if y i = 0, yi W (yie λ ⋆ -θ i ) , if y i > 0. It follows that ∇ θ L F [∂Ω] (y, θ) = y ⋆ F [∂Ω] (y, θ) -y.Formula of the Fitzpatrick logistic loss. We use y ⋆ as a shorthand for y ⋆ F [∂Ω] (y, θ). As we know that Ω * y (θ) = ⟨y ⋆ , θ⟩ -Ω y (y ⋆ ), we use again Proposition 7 to getL F [∂Ω] (y, θ) = Ω y (y) + ⟨y ⋆ , θ⟩ -Ω y (y ⋆ ) -⟨y, θ⟩ = Ω(y) -(Ω(y ⋆ ) + D Ω (y, y ⋆ )) + ⟨y ⋆ -y, θ⟩as Ω y (y ′ ) = Ω(y)+D Ω (y, y ′ ) and in particular Ω y (y) = Ω(y). Furthermore, as y ⋆ ∈ △ k ∩R k ++ , Ω is differentiable at y ⋆ and D Ω (y, y ⋆ ) = Ω(y)-Ω(y ⋆ )-⟨y -y ⋆ , ∇Ω(y ⋆ )⟩, where ∇Ω(y ⋆ ) = log y ⋆ +1. ThusL F [∂Ω] (y, θ) = ⟨y -y ⋆ , ∇Ω(y ⋆ )⟩ + ⟨y ⋆ -y, θ⟩ = ⟨y ⋆ -y, θ -log y ⋆ -1⟩.Bisection formula for λ ⋆ and bounds. We also get from Lemma 10 a bisection formula for λ ⋆ , which is a shorthand for λ ⋆ F [∂Ω] (y, θ).e -λ ⋆ i:yi=0e θi + i:yi>0y i W (y i e -(θi-λ ⋆ ) ) = 1.We focus here on a lower bound and an upper bound for λ ⋆ ∈ R. Let us prove thatlog k i=1 e θi ≤ λ ⋆ ≤ log 2 + max log k i:yi=0e θi , log ℓ 0 (y) + max i:yi>0θ i + 2ℓ 0 (y)y i ,where ℓ 0 (y) = Card(j : y j ̸ = 0).For the lower bound, we use the concavity of the Lambert function W , which implies1 W (yie λ ⋆ -θ i ) ≥ 1 yie λ ⋆ -θ i . Thus, 1 ≥ e -λ ⋆ i:yi=0e θi + i:yi>0 y i y i e λ ⋆ -θi , which in turn implies e λ ⋆ ≥ i:yi=0e θi + i:yi>0e θi and yields the lower bound.For the upper bound, the function g(λ) = e -λ i:yi=1 e θi + i:yi>0 yi W (yie λ -θ i ) is continuous and decreasing (as it is a positive combination of decreasing functions) and g(-∞) = +∞. Thus if we find a λ such that g(λ) < 1, we know that λ ⋆ ≤ λ.We deal with each term of g(λ) separately. If λ ∈ R satisfies e -λ i:yi=0e θi ≤ 1 2 max i:yi>0 y i W (y i e λ-θi ) ≤ 1 2ℓ 0 (y) , then g(λ) = e -λ i:yi=0 e θi ≤1/2 + i:yi>0 y i W (y i e λ -θi ) ≤1/ 2ℓ0(y) ≤ 1.Thus, all λ satisfying the following inequalties are upper bounds of λ ⋆ 2 i:yi=0 e θi ≤ e λ 2ℓ 0 (y)y i ≤ W (y i e λ-θi ), ∀i : y i > 0.As W is monotone and W -1 (t) = te t , we get log 2 + log i:yi=0 e θi ≤ λ 2ℓ 0 (y)e 2ℓ0(y)yi ≤ e λ ⋆ -θi , ∀i : y i > 0.Thus taking λ = max log 2 + log Let (y, θ) ∈ dom Ω × R k . We haveEQUATION= Ω y (y) + Ω * y (θ).The supremum above is achieved at y ′ ∈ ∂Ω * y (θ) = y ⋆ F [∂Ω] (y, θ). When Ω = Ψ + ι C , where C ⊆ dom Ψ, using Lemma 1 and 2, we have for all y ∈ C Ω y (y ′ ) = Ψ(y ′ ) + D Ψ (y, y ′ ) + ι C (y ′ ).It was shown in [7, Proposition 3] that if f = g + ι C , where g is Legendre type with C ⊆ dom Ψ, then for all y ∈ C and θ ∈ R k ,0 ≤ D g (y, ∇f * (θ)) ≤ L f ⊕f * (y, θ), with equality if C = dom g. Using g = Ψ y , f = Ω y = Ψ y + ι C , y ⋆ = ∇Ω * y (θ) = y ⋆ F [∂Ω](y, θ), and Lemma 3, we therefore obtainD Ψy (y, y ⋆ ) = ⟨y -y ⋆ , ∇ 2 Ψ(y ⋆ )(y -y ⋆ )⟩ ≤ L Ωy⊕Ω * y (y, θ) = L F [∂Ω] (y, θ).If Ψ y is µ-strongly convex and D Ψ is convex in its second argument, then Ψ y is µ-strongly convex as well. Therefore, we also have µ 2 ∥yy ⋆ ∥ 2 2 ≤ D Ψy (y, y ⋆ ).The question of adding conflicting infinites has been thoroughly studied by Moreau in [23] . In this paper, Moreau introduced two additions for R ∪ {-∞, +∞}. One of Moreau's two conventions for summing the infinites can be found under the name of inf-addition in [27, Chapter 1] . The other convention is named sup-addition.Using the inf-addition on R ∪ {-∞, +∞} allows, for instance, the equivalence of min x∈C f (x) and min x∈R n f (x) + ι C (x). The same goes for the sup-addition but for maximization problems.We used the inf-addition convention for the definition of the generalized Bregman divergence in (1) .As the generalized Bregman divergence is to be thought of as a generalization of a distance, it is generally a quantity that will be minimized. Therefore, the inf-addition is a good fit for the definition of the generalized Bregman divergence.In the proof of Proposition 7 in B.8, the choice of inf-addition is corroborated by the fact that we want to calculate, for some fixed y ∈ R k , Ω(•) + D Ω (y, •) (θ) = sup y ′ ⟨y ′ , θ⟩ -Ω(y ′ ) + D Ω (y, y ′ ) , ∀y ′ ∈ R k .The minus sign transforms the inf-addition into the sup-addition when distributed. If we had chosen the sup-addition in the definition of the Bregman divergence D Ω , we would here calculate a supremum of an inf-addition, thus entering unknown territory.(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility.In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers to have some path to reproducing or verifying the results.Question: Does the paper provide open access to the data and code, with sufficient instructions to faithfully reproduce the main experimental results, as described in supplemental material?Answer: [Yes] Justification: We provided a link for the used datasets in the experiments. The code will be opened upon release. For now, we provide instructions to reproduce the main experimental results. These instructions involved basic supervised learning tools such as L-BFGS algorithm, linear prediction model and standard cross-validation for the hyperparameter.Guidelines:• The answer NA means that paper does not include experiments requiring code. • Providing as much information as possible in supplemental material (appended to the paper) is recommended, but including URLs to data and code is permitted. 
experimental setting/details: 
 Question: Does the paper specify all the training and test details (e.g., data splits, hyperparameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?Answer: [Yes]Justification: The selection of hyperparameter are discussed in section 4. We use predetermined train-test splits that come with the datasets.Guidelines:• The answer NA means that the paper does not include experiments.• The experimental setting should be presented in the core of the paper to a level of detail that is necessary to appreciate the results and make sense of them. • The full details can be provided either with the code, in appendix, or as supplemental material. 
experiment statistical significance: 
 Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?Answer: [Yes] Justification: The split between training sets and test sets are fixed in advance so there is no variability when conducting the label proportion estimation tests. Furthermore, as we minimize convex losses, the convergence of the minimization algorithm is independent of the starting point. Guidelines:• The answer NA means that the paper does not include experiments. Justification: The research conducted does not involve human subjects. We did not create new datasets and used standard open access datasets for multiclassification. After reviewing the "societal impact and potential harmful consequences" from the Code of Ethics, we conclude that the research conduct in the paper does not pose a risk of harmful consequences. Guidelines:• The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.• If the authors answer No, they should explain the special circumstances that require a deviation from the Code of Ethics.• The authors should make sure to preserve anonymity (e.g., if there is a special consideration due to laws or regulations in their jurisdiction). 10. Broader Impacts Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed? Answer: [NA] Justification: The research conducted in this paper is theoretical and has no societal impact. Guidelines:• The answer NA means that there is no societal impact of the work performed.• If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact. • Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations. • The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is a direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster. • The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology. • If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML). 
safeguards: 
 Question: Does the paper describe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)? Answer: [NA] Justification: The datasets used in the tests are publicly available. No new prediction model has been released in this paper. Guidelines:• The answer NA means that the paper poses no such risks.• Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters. • Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images. • We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make a best faith effort. 12. Licenses for existing assets Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected? Answer: [Yes] The datasets can be downloaded from http://mulan.sourceforge.net/datasets-mlc.html and https://www.csie.ntu.edu.tw/~cjlin/libsvmtools/datasets/. 
 
 Appendix:  


Decision: Overall, all reviewers found the main theoretical contribution of this paper (the introduction of the convex Fitzpatrick loss functions) to be compelling and well explained. It is thus an easy accept. Most reviewers found empirical motivation a bit lacking, since the new loss functions do not seem to lead to any especially large performance gains in common settings. However, this was mitigated by the strong theoretical contributions of the work. Hopefully the paper will spur further investigation on the empirical side of things.

Reviews: review: paper_summary: paper_summary: The paper proposes a family of losses based on the notion of a Fitzpatrick function, which takes the role of composable subdifferentials in Fenchel-Young losses. The resulting family of losses is parallel to Fenchel-Young losses in the sense that each Fenchel-Young loss has a shared link function with a certain Fitzpatrick loss, but the losses themselves differ. It is shown that these losses posses some desirable properties such as convexity, and that they are based on a type of duality that yields a smaller duality gap than the one based on the Fenchel-Young inequality. Representative Fitzpatrick losses, such as logistic and sparsemax losses are developed as examples of useful members of the family. 

Experiments show that the losses are mostly comparable in performance to their Fenchel-Young parallels, with the exception of the sparsemax loss that might be slightly improved in its Fitzpatrick version.
strength_weakness: strength_weakness: The empirical utility of the developed losses in not entirely clear, since they do not offer a significant advantage over standard losses, and in some cases (e.g. Fitzpatrick sigmoid) may be more computationally demanding.
In other losses such as the squared loss and hinge loss, the parallel Fitzpatrick losses are practically the same. Hence, the contribution here is mainly in exploring further definitions of losses that are associated with familiar link functions, but attach to them a loss function that is not the standard one that stems from the Fenchel-Young family.

I am not an expert on this topic, so while I'm unsure whether the paper is interesting for its sub-community within ML, or for the wider NeurIPS audience (mainly because readers put a lot of emphasis on empirical results), I found it interesting enough to read hence I will give a borderline acceptance rating.
questions: questions: No significant questions arise from reading the paper. It might be nice if the authors could comment on what potential they see for future research on this topic, and how
limitations: limitations: The authors mention some limitations such as somewhat increased computation time, and acknowledge the mixed empirical results. I find this to be an adequate treatment of limitations.
paper_summary: paper_summary: The paper describes an alternative approach to constructing loss functions. The authors propose a Fitzpatrick loss and compare it to Fenchel-Young loss and Bregman divergence. The work enumerates the basic properties of the introduced loss and compares it to existing loss functions by numerical simulations.
strength_weakness: strength_weakness: I think the paper needs more thorough numerical evaluation due to the format of the conference. Although the properties of Fitzpatrick loss are given, the logical question is, why to use it instead of known loss functions for machine learning? At the moment, the authors presented comparison for classification problem over several datasets. Without broader experimental study, the manuscript seems more like a mathematical note. Therefore, I think that additional experiments will strengthen the paper.

Moreover, the datasets are for classification, but the results are compared in terms of mean squared error. It seems more logical to compare using some metric for classification, i.e. accuracy, f1-score etc.
questions: questions: The paper claims that the Fitzpatrick losses are tighter than Fenchel-Young. Can something be said from the statistical point of view? Does using new loss lead to better generalization, for example.
limitations: limitations: The paper does not have negative societal impact.
paper_summary: paper_summary: This paper proposes a class of loss functions called Fitzpatrick loss, based on the Fitzpatrick function known in maximal monotone operator theory. It can be shown that building a loss function with the proposed method, can give a convex loss function that lower bounds the Fenchel-Young loss under the same choice of link function for prediction. Extensive analysis provides in the paper also shows the relationship between Fitzpatrick loss and Fenchel-Young loss. Finally, experiments were provided to validate the usefulness of Fitzpatrick loss.
strength_weakness: strength_weakness: Although it is nice to also provide the experiments in the paper, I found that the experiment failed to motivate the usefulness of Fitzpatrick loss family in the sense that the performance is not preferable overall. I think no so many insights were provided when Fitzpatrick loss is more useful than existing losses in the literature. The proposed loss is also more computationally demanding. Perhaps different kinds of experiments would give more values to the paper. From the current experiments, I think we still need lots of work to study when to use Fitzpatrick loss in practice.

My current score is Weak accept (6) as I put importance to the theory to advance the understanding of Fenchel-young loss and Fitspatrick loss function quite high and believe these are the main contributions of this paper, rather than the experimental result.
questions: questions: 1. Since Fitzpatrick loss lower bounds the Fenchel-young loss for the same link function, is it possible to find some use-cases that the proposed loss (e.g., Fitzpatrick-sparsemax) is preferable? For example, could it be more useful when we want to have sparse probability output and we have to learn under the case where there are outliers/noisy data?
2. It is said that we can provide a relationship between Fitzpatrick loss and Fenchel-young loss by using "target-dependent function" in Section 3.3 (Prop. 7). I think the result is very interesting and I have two questions regarding this finding. 
- 2.1 Since we need target-dependent function for $\Omega_y$ to make the relationship holds, does the original Fenchel-Young loss supports target-dependent function case? If so, then this should be no problem and we can state that all properties of Fenchel-young loss inherits to Fitzpatrick loss.
- 2.2 Is it safe to say that Fitzpatrick loss is a special case of Fenchel-Young loss? Perhaps not?
- 2.3 Can we also say that any Fenchel-young loss can be rewritten in a form of Fitzpatrick loss generated by different $\Omega$?
3. In the conclusion section, there was a discussion about "there can only be one loss function associated with a certain link function". I am not sure about this statement for the definition of loss function here. Is it about a "proper composite" loss function, a "classification-calibrated" loss function, or a convex + (proper composite/classification-calibrated) loss function? Maybe we can be more explicit here otherwise one might think about any function or small ad-hoc modification using the same link and use it as a loss function with not much problems. 


Minor comment
1. Typo: Line 99: also know as -> also known as
2. I feel it is a bit strange to say One loss is tighter than another loss since it is closer to zero everywhere.
limitations: limitations: The discussion was sufficient. The authors noted the challenge of computational problem and appropriately discussed the experimental results of the proposed method.
paper_summary: paper_summary: This article presents a refreshing take on losses used in ML, with solid motivational examples and a several of interesting and beautiful results in pure convex analysis.
strength_weakness: strength_weakness: My main regret is that, due to the reviewing load, I was not able to fully check all of the proofs presented in the appendix. The claims *in the article* are reasonable, interesting, well-presented, and thorough; however, I am quite concerned that there may be some (unintentionally) omitted hypothesis, or holes in the proofs in the appendix (e.g., please also see my statement about Compactness of the domain in "limitations" section), because this often happens in ML conference articles. I wish that an appendix counted as the "reviewed" part of the article, but alas I simply do not have the time to check every detail. I am going to suggest that this paper is accepted, but I suggest that the authors please take care to revise based on the following minor points.


Minor comments:
- Citations appear to be out of numerical order when they are cited
- Line 30: In what sense is "proper" meant? All losses I'm aware of
  are proper functions (using the convex analysis definition).
- Line 56: Is R_+ numbers which are strictly positive, or just
  positive?
- Line 59: The set must be nonempty for the projection to be defined.
- Line 62: This identity only holds when \Omega is differentiable  *and convex*! Otherwise, this definition of the subdifferential produces the emptyset in areas of concavity.
- Line 66: Please provide a citation.
- Line 68: Does this function need to be Legendre?
- Line 101: Monotone is defined here, but mentioned earlier in the
  article; would be nice to have the definition at the beginning.
- Definition 1: Within the text of the definition, "dom Omega"  appears before Omega is introduced.
- A bit more discussion would be nice on specifically which problems (4) can be calculated.
- Line 132: "Above expression" referring to "\nabla^2 Omega" is ambiguously defined, since in the constrained case Omega is not differentiable.
questions: questions: - Line 69: As far as I've seen in most convex analysis books and papers, this convention is a bit nonstandard; typically, "+infinity +  (-infinity)" is *undefined*. Please comment on which results in this article break if the quantity "+infinity + (-infinity)" is undefined. I.e., if this is "not a valid move", which of your results still hold? It is very important to clarify which results hold under varying algebraic models of convex analysis.

- Line 157 / Prop. 7: the relationship between the claim in the equation following Line 157 and Proposition 7 is unclear. How does the claim in line 157 follow from Proposition 7?
limitations: limitations: The mathematically precise statements (propositions, lemmas, theorems/etc.) should state **all** of their assumptions. In particular, the authors do not repeat that a compact domain of the objective function is required for several of their results. (Unless I'm missing something and compactness is not actually required?)

The authors do a great job of explaining the results of their experiments. More numerics is always a plus, but since the article has so many strong theoretical contributions, I am quite happy with this as-is.
paper_summary: paper_summary: The paper proposes the Fitzpatrick loss, a tighter version of the Fenchel-Young loss. Fitzpatrick losses are defined based on the monotone operator theory. In particular, the Fitzpatrick function satisfies a tighter inequality than the Fenchel-Young inequality, leading to the tighter loss function. The authors elucidate the fundamental properties of the Fitzpatrick loss and present concrete examples, demonstrating that Fitzpatrick sparsemax/logistic losses indeed differ from the Fenchel-Young counterparts. The authors also show that the Fitzpatrick loss can be seen as the Fenchel-Young loss generated by target-dependent regularizes and that it is connected to generalized Bregman divergences, while the primal-dual nature of the Fitzpatrick loss crucially makes it convex. A lower bound on the Fitzpatrick loss is also presented, which is analogous to the Fenchel-Young counterpart. Finally, experiments compare Fitzpatrick and Fenchel-Young losses on classification tasks.
strength_weakness: strength_weakness: A weakness (not so serious) is that the experimental results are not very compelling. Still, I find the slight superiority of Fitzpatrick-sparsemax to Sparsemax noteworthy as their computation costs are almost the same.
questions: questions: 1. A special form of target-dependent regularizers is used for constructing the cost-sensitive Fenchel-Young loss ([Blondel et al. 2020, Section 3.4](https://jmlr.csail.mit.edu/papers/v21/19-021.html)). Does the Fitzpatrick loss have any relation with the cost-sensitive Fenchel-Young loss?
2. Is it possible to establish calibration of target and Fitzpatrick excess risks, as in [Blondel (2019, Proposition 5)](https://proceedings.neurips.cc/paper_files/paper/2019/hash/7990ec44fcf3d7a0e5a2add28362213c-Abstract.html)? The lower bound in Proposition 8 seems useful for this. 
3. Regarding lines 164-165, is there a canonical class of convex functions $\Omega$ such that $D\_\Omega$ is convex in the second argument?
4. In the experiments, predictions are produced by $\hat y\_\Omega(Wx)$, and I can understand this since the Fitzpatrick loss has $\hat y_\Omega$ as the link function as in Proposition 1. On the other hand, given the discussion in Section 3.3, $y^*\_{F[\partial\Omega]}(y, \theta)$ also appears to be a Fitzpatrick counterpart of $\hat y\_\Omega(\theta)$, which, however, cannot be used in the test phase since ground truth $y$ is not available. So, when $W$ is learned with the Fitzpatrick loss, aren't there other possible methods for converting $Wx$ to a prediction other than $\hat y\_\Omega(Wx)$ (without knowing $y$)?

#### **Minor comments**
- l. 84. "$\hat y\_\Omega$ is monotone..." Does this "monotone" refer to the one in Section 2.3? If so, it is not yet defined there.
- Eq. (4). Enclosing the terms after "argmax" in parentheses will make it easier to read.
- l. 174. that that.
- Proposition 8. Reminding that $\Psi_y$ is defined as with $\Omega_y$ in Section 3.3 is helpful; alternatively, it may be better to define $\Omega_y$ as a general notation in Section 3.3 or earlier.
limitations: limitations: The authors have discussed the limitation regarding computation costs.

Comments: title: Thank you for the author feedback
Post Rebuttal Update
Answer to Rebuttal
comment: I thank the authors very much for their responses and clarifications.

I appreciate the authors' motivation for using this model of defining $+\infty - (+\infty)$ as $+\infty$. I am aware of the Moreau article; however, other more modern convex analysis books (e.g., *Convex Analysis and Monotone Operator Theory in Hilbert Spaces* by Bauschke and Combettes) treat $+\infty - (+\infty)$ as an undefined term. This is an important distinction, since convex analysts certainly still use both notions in modern work. While I am still overall quite impressed with the article, I would like to see more emphasis on this distinction in the final version.

Overall, it was a pleasure to review this article, and I look forward to seeing its final form in the future. I thank the authors for their time.
Thank you for the detailed answers to my questions. I retain my score of 7 (Accept).
I have read other reviews as well as the author rebuttal. Thank you very much for the detailed feedback. 

Thank you for clarifying my concerns and my potential misunderstanding. I agree with other reviewers and the paper that this paper provides a theoretical foundation for the Fitzpatrick loss which is novel to the best of my knowledge. Even though experimental results are not very strong and thus more investigation of empirical performance is needed, it can be studied in the future for some specific applications of interest. 

In this current form, I think the paper is worth an acceptance. As a result, I raised the score to 7 (Accept).
Thank you for the response.

I understand that the point of the paper is not to introduce "better" losses than FY, and appreciate the paper for the results it presents. Therefore I will keep my recommendation to accept the paper.
Dear Authors,

Thank you for addressing my comments. I decided to raise my score by 1 point.