Loss Functions and Operators Generated by f-Divergences
Authors: Vincent Roulet
Tianlin Liu
Nino Vieillard
Michael Eli Sander
Mathieu Blondel
Venue: ICML 2025 poster (2025)
Keywords: loss functions
f-divergences
entropies
Fenchel conjugates
Primary area: optimization
TL;DR: We propose to generate loss function from f-divergences and do experiments on language modeling tasks.

Abstract: The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for multiclass classification. It is also the loss function of choice for next-token prediction in language modeling. 
It is associated with the Kullback-Leibler (KL) divergence and the softargmax operator. 

In this work, we propose to construct new convex loss functions based on $f$-divergences.  Our loss functions generalize the logistic loss in two directions: i) by replacing the KL divergence with $f$-divergences and ii) by allowing non-uniform reference measures. 
We instantiate our framework for numerous $f$-divergences, recovering existing losses and creating new ones.

By analogy with the logistic loss, the loss function generated by an $f$-divergence is associated with an operator, that we dub $f$-softargmax. We derive a novel parallelizable bisection algorithm for computing the $f$-softargmax associated with any $f$-divergence.

On the empirical side, one of the goals of this paper is to determine the effectiveness of loss functions beyond the classical cross-entropy in a language model setting, including on pre-training, post-training (SFT) and distillation. We show that the loss function generated by the $\alpha$-divergence (which is equivalent to Tsallis $\alpha$-negentropy in the case of unit reference measures) with $\alpha=1.5$ performs well across several tasks.

Loss Functions and Operators Generated by f-Divergence
Vincent Roulet Tianlin Liu
Nino Vieillard
Michaël E. Sander' Mathieu Blondel
Abstract
The classical softargmax is known to optimize a trade-off between the expected value of the logits and the Kullback- Leibler (KI.) divergence with a uniform measure. This per- spective is also adopted in the space of sequences, replacing the uniform measure with a reference measure, in reinforce- ment learning from human feedback (RLHF) (Christiano et al., 2017) and in direct preference optimization (DPO) (Rafailov et al., 2024). Recent works extended this per- spective to general f-divergences (Go et al., 2023; Wang et al., 2024). However, these works naively compose an f-divergence with the (classical) softargmax, which does not result in convex loss function.
The logistic loss (a.k.a. cross-entropy loss) is one of the most popular loss functions used for mul- ticlass classification. It is also the loss function of choice for next-token prediction in language modeling. It is associated with the Kullback- Leibler (KL) divergence and the softargmax oper- ator. In this work, we build upon Fenchel- Young losses to construct convex loss functions gener- ated from f f-divergences. Our loss functions gen- eralize the logistic loss in two directions: i) by replacing the KL divergence with f-divergences and ii) by allowing non-uniform reference mea- sures. We instantiate our framework for numerous f-divergences, recovering existing losses and cre- ating new ones. By analogy with the logistic loss, the loss function generated by an f-divergence is associated with an operator, that we dub f- softargmax. We derive novel parallelizable bi- section algorithm for computing the /ssoftargmax associated with any f-divergence. On the empiri- cal side, one of the goals of this paper to deter- mine the effectiveness of loss functions beyond the classical cross-entropy in a language model setting, including on pre-training, post-training (SFT) and distillation. We show that the loss function generated by the a-divergence (which is equivalent to Tsallis a-negentropy in the case of unit reference measures) with a = 1.5 performs well across several tasks.
In this work, building upon Senchel-Young losses (Blondel et al., 2020), we propose to construct new convex losses based on f-divergences. Our losses generalize the logistic loss in two directions: i) by replacing the KL divergence with f f-divergences and ii) by allowing non-uniform refer- ence measures (class prior probabilities). Our loss construc- tion generalizes the sparsemax (Martins & Astudillo, 2016) and entmax (Peters et al., 2019) losses, and allows to create entirely new losses: see Table 1. In addition, each loss generated by an f-divergence is associated with a new operator for turning logits into probabilities, that we dub f /-softargmax.
On the empirical side, of the goals of this paper is to determine the effectiveness of loss functions beyond the clas- sical cross-entropy in language model setting, including on pre-training, post-training (SFT) and distillation.
To summarize, we make the following contributions.
We propose to use f-divergence regularization to gen- erate loss functions. Each loss function is associated with new operator, that dub f-softargmax. We instantiate this framework for numerous f f-divergences, recovering existing losses and creating new ones.
1. Introduction
The logistic loss, a.k.a. cross-entropy loss, is widely used for multiclass classification. It is associated with the sof- targmax (a.k.a. softmax) operator, which turns logits into class probabilities. The logistic loss and the softargmax operators are also frequently used in the space of tokens for language model pre-training, post-training and distillation.
We derive a novel bisection algorithm for computing the f T-softargmax associated with any f-divergence. Our algorithm parallelizes well on modern hardware.
We demonstrate loss functions on image classifica- tion, language model post-training and distillation.
'Google DeepMind. Correspondence to: Mathicu Blondel, Vincent Roulet <mblondel cocondddl@@gogl.coooo google.com, vroulet @ww.@ggoooggo........
Proceedings of the ndnd International Conference Machine Learning. Vancouver, Canada. PMLR 267, 2025. Copyright 2025 by the author(s).
Notation. Throughout this paper, l denotes the num- ber of classes in classification problem or the vocab-

---

Loss Functions and Operators Generated by /-Divergences
ulary size in language modeling. We denote the set [k: = (1, (I,...,Rt) k). We denote the probability simplex by
and the sigmoid
= = Ep R$: (p. = 13. We denote the domain of f by (f) = fu: -00 < f()) < co). We denote the con- vex conjugate of f by f", where (v) == sup,. we f(u). We denote by 0 = (x) the logits produced by a net- work h with parameters w E W for the input æ E X. We denote hard (one-hot) labels as y E {ei,..., ex). which used for classification. More generally, we denote soft labels as A which are useful for learning from label proportions, as in distillation.
sigmoid(()
|softargmax(0)1
1 + exp(-0)
The two operators are again related by
softplus' sigmoid.
The binary logistic loss between the logit 0 E R and the ground-truth label E [0, 1] is then
((0.v) = softplus(0) (0) + (1 y) log(1 - -1) + y log y 0y.
(4)
Beyond binary classification, binary logistic loss can be used for pairwise ranking if we define 0 == 0; 0j. where 0, and 0 are the scores of items i E [A] and j E [k]] respec- tively. The probability that item i E [k] is ranked higher than item j E [Kt] according to the model is then
2. Background
2.1. Logistic loss
Given logits 0 E IRt,, we define the softargmax as
exp(0.)
exp(O,) (softargmax(O);; 2/ exp(O),)
sigmoid(0, 03) exp(0.) + exp(0,)
jE E [K]. (1)
This called the Bradley-Terry model (1952).
Note that (1) is commonly known in the literature as "soft- max," which is a misnomer as is a smooth approximation of the argmax function (Blondel & Roulet, 2024). We in-
2.3. Fenchel-Young losses
stead define the softmax as
It is well-known that the softmax and softargmax can be written from a variational perspective as
k
softmax (0) = log Σ exp(0)),
softmax (0) maa (p. 0) (p. log p)
PEA*
softargmax (0) argmax (p. 0) (p. p).
which is smooth approximation of the maximum function. The softargmax and softmax are related by
PEA*
This suggests that can create more general softmax and softargmax operators if we replace Shannon's neg- ative entropy log by more general regularization Ω: A + R, namely,
Vsofumax = softargmax.
The logistic loss (a.k.a. cross-entropy loss) between logits Oc I and ground-truth y E A is often defined as
softmaxp (0) = max (p. 0) - 1(())
DEL
Ev; Uj log ((sottaggmax(0),,) softmax(O)) (0,
softargmax (0) =
argmaax(p 0) -((p).
PEA#
The logistic loss is equivalent up to constant to the KL divergence between the target and the softargmax output,
By analogy with (2), we define the Fenchel-Yove loss (Blondel et al., 2020) generated by as
((0 y) = kT softargmax(()))
tn (, y) == soltmax. (0. y) + P(y)
(5)
softmax (0) - (0, y) + (y. log log
(2)
2 Bn softargmax, (0)).
(6)
Adding the constant term (y, log y) ensures that the loss is non-negative even when using soft labels (the constant term is zero with hard labels). The loss e is convex in O, and its gradient w.r.t. O E R is
where Bg is the Bregman divergence generated by Ω. The loss in (5) is always convex w.r.t. 0 unlike the loss in (6). Among many other desirable properties, if Ω is strictly con- vex, Fenchel- Young losses satisfy
0 y) = softargmax(O) y.
(3)
(n v) = 0 + softargmax,
and the gradient w.r.t. R is
2.2. Binary logistic loss
Vela = softargmax,,
In the special case = 2, letting 0 == (0, 8) and y ==
In the binary classification setting, we can similarly replace sigmoid with sigmoidg in (4), as well as the regularization term (1 y) log(1 - y) + y log y with 22(1 - y. v.,)).
(1 - y. 1) for 0 E IR and y E [0, 1). we obtain the softplus
softplus(0) == log(1 + exp(O)) = softmax (f)
2

---

Loss Functions and Operators Generated by f-Divergences
3.2. f-ettropies
Table By using f-divergences as regularization, we can gen- eralize existing loss functions (logistic, sparsemax, entmax) to non-uniform reference measure q and we can construet several new loss functions. Reverse f-divergences (not listed below), such as the reverse KL, can also be used to generate alternative loss functions. The f-softargmax operators associated with the Chi- square divergence and the a-divergence for a > 1 can produce probability distributions with sparse support. When using the unit positive measure q = 1, we obtain f-entropies, which recover ex- isting known entropies (Shannon, Gini, Tsallis) up to constant.
As we emphasized, our proposed framework can take into account discrete reference measure q R4, which in- tuitively contains the prior class weights. When such measure is not available, we can simply choose the unit positive measure q = 1. In this case, we recover negative /- entropies (Cichocki & Amari, 2010), a.k.a. f-negentropies. They are defined as
Divergence
Entropy
Loss
Sparse
lr(p) = Σ f(p)) 1)
Kullback-Leine
Shannon
Logistic
No
Chi-square
Gini
Sparsemax
Yes
By choosing f. we recover (up to constant) numerous existing negentropies. When f(u) u log u, which is the generating function of the KL divergence, we recover the Shannon negentropy.
a-divergence
Tsallis
Entmax
0 > 1
Jensen-Shannon
No
Squared Hellinger
No
12( = (p, logp).
When f(u) = (u² which the generating function of the Chi-square divergence, we recover the Gini negentropy. used for the sparsemax loss (Martins & Astudillo, 2016),
3. Generating losses from f-divergences
In this paper, we propose to study Fenchel-Young losses and associated operators when the regularizer is defined
(p((p) 2 (11p112 1)
(p(p: g)
(8)
(we use = for equality up to a constant). More generally, with f(u) (a 11-- -1)-((1))) which is the generating func- tion of a-divergences (using Cichocki & Amari (2010)'s definition), we recover the Tsallis negentropy,
where D, is a f-divergence and q E ! is a reference measure, which contains the prior class weights. When q is not available, we can simply use q = 1, in which case we will obtain negative f-entropies, as explained below.
a(a-1)
22 - (p. log, (p)) = a(a -
(Ipll 1),
3.1. 3.1. f-divergences
used by the entmax loss (Peters et al., 2019). The Tsallis negentropy itself is very general, it recovers the Shannon negentropy when + 1 and the Gini negentropy when = 2. In our experiments, we will demonstrate good results with the choice a = 1.5, which can be thought as a middle ground between Shannon entropy (used by the logistic loss) and Gini entropy (used by the sparsemax loss).
Let f: 1 + IR a convex function such that f(1) = and f(0)- lim fuu). The f-divergence between two discrete positive measures p I& and q E Rt (Rényi, 1961; Csiszár, 1967; Ali & Silvey, 1966) is then
Dy(p.g)= = Σ f(py/1,9 (f(pg)).))
Unexplored entropy instances. Beyond existing en- tropies, some choices of lead to entropies that had not been considered before, such as the squared Hellinger, illus- trated in Figure 1. We leave a formal axiomatic analysis of such new entropies to future work, and refer the interested reader to, e.g., Ben- Bassat (1978); Friedland et al. (2013); Sharma & Sazim (2021); Sbert et al. (2024).
where f and division are applied element-wise. An f- divergence is always non-negative and jointly convex in and q. Many existing divergences can be written in f- divergence form: see Table 1 and Appendix B.
Divergence reversal. Since an f-divergence is not neces- sarily symmetric in p and q, it may seem that setting the reference measure q as right argument in (8) is arbitrary. However, if we define g(u) = u (L/uu, we obtain the re- verse divergence D,. D, q) = (y p). Therefore, for any f-divergence, we can always construct the corre- sponding reverse divergence, without loss of generality.
Effective domain. We point out, however, that some f- entropies are only well-defined on the relative interior of the probability simplex, if lim f'(u) -oo. This is for instance the case of the t f-entropies associated with the reverse KL. and Jeffrey divergences. This means that the loss functions generated by these choices only work with strictly positive soft labels.
3

---

Loss Functions and Operators Generated by /Divergeencss
Kullback Leibler (equiv. a-div, a=1.0)
Shannon (KllbbckkLLebler)
P1
P2
Pa
(Chi-square)
Tsallis (a- deviiereeece, a
Jensen-Shannon
Squared Hellinger
0.0
0.0
0.0
-2.5
2.5
-2.5
0.0
2.5
-2.5
0.
2.5
0.6
a-div, a= 1.5
2.5
2.5
0.2
0.6 π
-2.5
-2.5
-2.5
9
9
8]
Figure 1. Illustration of f-entropies (p) for = (1 π) and varying [0, 1]. We add constant f(0) ensure non- negativity of the f-entropies.
Chi-Square (equiv. to a-div, a=2.0)
Pi
P2
Ps
2.5
2.5
2.5
0.0
6
0.0
0.0
3.3. -softmaxx and f-softargmax
-2.5
-2.5
-25
Overloading the notation, we define the /-softmax as
-2.5
0.0
2.5
-2.5
0.0
2,5
-2.5
0.0
2,5
0
softmax (0; q) = max (p, 0) (y ER A
(9)
0.0
1.0
Value
and the f-softargmax as softargmax, q)
argmax(p, 4)(pp A
pa) = softargmax (01, (0,, 02, 0; when varying 01, 1.02 E R for three possible divergences and with
Figure 2. Illustration (pI, Pa.
PEA
(10)
q == 11, More illustrations are given in Appendix B.
Compared to classical softmax and softargmax, our oper- ators use function f and include an additional reference measure q as argument. When (w) = u log u and 1, we recover the classical softmax and softargmax.
and the f f-sigmoid
sigmoid,
argmax - D) E [ 1]. #E(U.1]
Scaling the divergence by a temperature parameter B can easily be done. Indeed, for any
The effect of the prior class weights g E IR3 on the shape of the sigmoid is illustrated in Figure in Appendix B. The f-softplus and f-iigmoid can be used, not only for binary classification, but also for pairwise ranking. by analogy with Section 2.2. Similarly the f-softmax and f-softargmax, we easily scale Dy by a temperature parameter 0.
softmax,3/ (0; q) = 3softmax, (0/3: softargmax, By (0: (intargmax,(0//353
Sparse distributions. As summarized in Table 1, the f- softargmax associated with the Chi-square and a diver- gences for can produce probability distributions with sparse support, meaning that some classes have exactly zero probability according the model. As will be clear from Proposition 1, the /-softargmax can be sparse when 0 E dom(f') + lim, f'( > -00.
3.5. Loss function
obtain loss function associated with the f-softargmax, we instantiate the Fenchel-Young loss defined in (5) with the regularization defined in (8) define
(6 y: q) softmax b (0: ④) +2y q) (0,
softmax,(0:q) + Dy(y. (0, y)
3.4. f-softplus and f-sigmoid
(11)
As special cases of -softmax and /-softargmax, by defining = (0, 0), = (1 - π) and = (go. q1), we obtain the f-softplus
where O E RR are the logits, y E A is the ground-truth and E Rt is reference measure, which contains the prior class weights. This loss inherits from all the desir- able properties of Fenchel-Youmg losses. In particular, is convex w.r.t. O and differentiable everywhere
(0::)lus((0:g)) == max nl - Dy((1 n, ,, ER xE(0,1]

---

Loss Functions and Operators Generated by f-Divergences
is strietly convex. On the other hand, it is not necessarily convex w.r.t. q, since softmax, involves the maximum over collection of concave functions of q. Because y A and not just y E fei..... ) this loss can also be used for learning from label proportions, as is useful in distillation for example.
reference measures q- Our proof technique different: it uses Fenchel duality opposed Lagrange duality and it rigorously accounts for the domain of f', which is one of the technical difficulties for supporting general f-divergences. Proposition 1 is a generalization of (Wang et al., 2024, The- orem 1) to the case E dom( (f)), that is, to f-softargmax operators with sparse output. A Newton algorithm was pro- posed in (Terjék, 2021) for computing the same operator. However, that algorithm was used for regularized optimal transport, not for creating loss functions. In addition, it assumes that (ʃ + 6R,)* is twice differentiable, which is not the case when 0 E dom(f"), and does not enjoy conver- gence guarantees, as a line search was not used. See also the discussion in (Belousov & Peters, 2017) on the implications of 0 of dom(f").
Choosing q. One distinctive feature of our losses com- pared to usual instances of Fenchel-Young losses is the possibility to adjust the reference measure q. In the ab- sence of prior knowledge, we can simply use q 1/k: This recovers Fenchel-Young losses generated by /-negentropies. If class weights are available, we can use this prior knowledge as q.
or
3.6. Computation
On first sight, it is not obvious how to solve the variational problems (9) and (10) involved in computing the f-softmax and f-softargmax, respectively. We need an algorithm that works for any valid choice of f and parallelizes well, as we often need to compute the operators on batch of logits ......., Oi. where 0,,- hw (x:;))
Implementation. In practice, we solve the root equation in (14) by bisection (Algorithm 1). This algorithm has several advantages: it is simple, parallelizes well on GPU and TPU (we often need to compute the f-softargmax of batch) and achieves an error on iteration t of (Tmax Tmin 3/2'. That is, the error exponentially decreases with the number of iterations. We show in Appendix A.4 that the overhead of our proposed algorithm is negligible compared to a classical softargmax.
In this section, introduce novel generic algorithm for computing the f-softmax and f-softargmax given access to f and f", where f is the convex conjugate of f. The latter is usually available in closed form; we give numerous examples in Appendix B.
For convenience, with a slight abuse of notation, we de- fine the shorthand S. = (s*Y. We denote f"(0)= 1im fx) which may be finite or -o0.
Algorithm 1 Computing f-softmax and a f-softargmax
Inputs: logits 0 E R² prior q E 1R4, tolerance e >
[p(r)]; = (max - T,
( (()))) [k]
Proposition 1. Reduction to root finding
Q(T) = (p((), 1)
Let f be strictly convex and differentiable function such that (0, +00) C dom f". Then, for any 0 E R*
jr E argmax,ee 0j
Tmin 0j. (((1//
Tmax +- 0f- 8 1/ (1/
softmax,(0: 9) - + Σ 95 (max -
('0)))
T T (Tmin + Tmaxa 1/2
(12)
while [Ø(T] >
(0; 93 03 (max(00 - ,
if (((T)) <
Tmax t T
(13)
else
where 7* is the unique solution of
Tmin T
T + (Tmin + Tmax))22 )/2 Outputs: softargmax, (0: q) = p(T) softmax,(0:0) ~ E q; (max(0,- - ((0)))
Σ (mxxoo - T, -, ('0))) 1, (14)
on 7 E [Tmin; m. where, for jr E argmax 0z.
Tani == 0j- (1(1///-)
Closed forms in the binary case. In the special case k = 2 (binary classification), we can often derive closed- form solutions for the f-sottllus and the f-sigmoid operator. For completeness, we derive the expressions for numerous cases in Appendix C.3.
Tiax = 0,- - 8 /1/ ///
A proof is given in Appendix C.4. Proposition is a gener- alization of (Blondel al., 2020, Proposition 9) to arbitrary
5

---

Loss Functions and Operators Generated by f-Divergences
3.7. Differentiation
Table 2. ImageNet classification results.
Differentiating through f-softmax and loss. In order to differentiate the f-softmax, simply use Danskin's theorem to obtain
Divergence
Accuracy (%)
Kullback-
76.87
Chi-square
76.06
V fax = =p" == softargmax, V qasoftmax = D .)).
a-divergence (ar 1.5)
77.56
Jensen-Shannon
72.24
Squared Hellinger
72.81
As a result, the loss gradients are
Vol,(0, y: q) p" -y y= = softargmax (0: y Vgl, y: VyD,(y.q) V (p")))
we use linear warmup that achieves peak learning rate of 0.2; we then use cosine annealing to reduce the learning rate to O. The weight decay is set to be 10 The batch size is 512.
The first equation is in complete analogy with (3). The second equation is the difference of the f-divergence gra- dients evaluated at the ground-trum -ruth y and the prediction p'. Differentiating through the f-softplus operator and its associated loss function is similar.
Table shows the validation accuracy of training ResNet50 using different f-divergence generated losses; boldface in- dicates the highest accuracy. Recall that the KL generated loss is equivalent to the standard cross-entropy loss; there- fore, using the KL generated loss should match the result of prior work (He et al., 2016). Our experiments confirm this: the KL loss reaches 76.87% % accuracy, consistent with previous benchmarks (Appendix A.1). Perhaps surprisingly, find that the a-divergence loss function surpasses the KI loss in validation accuracy. This improvement is achieved without adjusting any hyperparameters and therefore is no- table. In addition, a = 1.5 seems to be optimal within a-divergences as shown in Figure 5, where we see that vali- dation accuracy maximal for a = 1.5 among 11 values for a E [1, 2]. However, the Jensen Jensen-Shannon, Squared Hellinger, and Chi-square (Pearson divergence) generated loss functions perform worse than the KI. divergence in this experiment.
Differentiating through f /-softargmax. When the goal to use the f-softargmax as the operator associated with our loss functions, we do not need to differentiate through the f-softargmax. As explained above, thanks to Danskin's s theorem, differentiating through the f-softmax is sufficient. When the goal is to use the /-softargmax as an attention mechanism, however, we do need to differentiate through the f fssoftargmax. This is more challenging as, from Propo- sition 1, we need to differentiate through the solution of root equation. Under assumptions on we can apply the implicit function theorem (Krantz& & Parks, 2002) through the root equation's 's solution. This can be implemented us- ing automatic implicit differentiation (Blondel et al., 2022). Importantly, implicit differentiation does not require solv- ing a costly linear system here. Since the root equation is one-dimensional, simple division is sufficient.
4.2. Language modeling
We compare the performance of different. f-divergence- based loss functions for training language models (LMs), To provide thorough evaluation, we consider three com- mon LM training strategies: (i) pretraining, (ii) supervised tuninuni (SFT), and (iii) distillation fine-tuning All three approaches optimize a next-token prediction loss:
4. Experiments
To evaluate different f-divergence generated losses, we ap- ply them to tasks of different data modalities, including image classification (Section 1) and text generation (Sec- tion 4.2). These experiments also cover different training strategies: from scratch, finetuning, and distillation.
W
Ei
(x Bee), )())):
(15)
4.1. ImageNet classification
where hw the LM that outputs logits with parameters w, ly] is the length of the text sequence y. and the reference measure q 1 is unitary prior; ef. Equation (11). Pretrain- ing and SFT use one-hot vectors d(v) E {eI,.... ex) as the targets; distillation uses soft labels o(3)) E A obtained as the next-token probability of a teacher model.
We apply different /ddivergence generated losses to train ResNet50 model (He et al., 2016) on the ImageNet-2012 dataset (Russakovsky et al., 2015). The ImageNet dataset contains 1.28 million training images and 50,000 validation images, belonging to one of 1,000 classes. The ResNet50 model is a standard choice for ImageNet.
We use an SGD optimizer with 0.9 momentum to train the ResNet50 model for 90 epochs. During the initial epochs,
Pretraining with f-divergence generated losses. We used the same pretraining method as the NanoDO models
6

---

Loss Functions and Operators Generated by -Divergences
(Liu et al., 2024; Wortsman et al., 2024), set of well-tuned decoder-only transformer models trained on the public C4 dataset (Raffel et al., 2020). Specifically, we follow the training set up of the 1.2B parameter model from Wortsman et al. (2024); we use 250B tokens of C4 to train the model, so that the model is Chinchilla optimal (Hoffmann et al.,
SFT
Distillation
15
Rouge-2 Scores
2022). Our pretraining results (see Appendix A.2) echo our main findings from the ImageNet experiments in Section 4.1. Specifically, losses based on a-divergence, with 1.5, performed best for predicting the next word, slightly outperforming the standard KI. divergence loss.
10
5
0
Learning q. Since our loss functions take a reference distribution q and we can compute gradients with respect to q, natural idea is to try to learn it from data. As negative result, using the NanoDO implementation above on the C4 dataset, we found that learning does not seem to help improve performance compared to q = 1, both when using the KL divergence and when using the a = 1.5 divergence.
is
rC5
kl
alphal.5
Figure 3. Comparing the effect of different f-divergence gener- ated losses used for SFT and distillation training of LMs. The f-divergence generated losses are lensen-Shannon (js), reverse Chi-square (res), Kullback Bailbacck-Lebberr (kl), Chi-square (cs), and a- divergence with a = 1.5 (alphal.5)
SFT and distillation with f-divergence generated losses. Instead of pretraining LMs from scratch, practitioners often fine-tune readily available, open-weight models pretrained with standard cross-entropy. This prompts question: Can we effectively fine-tune these cross-entropy pretrained mod- els using different f f-divergence generated losses? To ex- plore this, we evaluated f-divergence generated losses with two common finetuning methods, SFT and distillation, on a text summarization task (Narayan et al., 201 8). For SFT, we use pretrained T5-base model (Raffel et al., 2020) with 250M parameters and train it on the XSum dataset (Narayan et al., 2018) with next-token prediction loss (15). Tra- ditionally, the cross-entropy loss is used as the next-token prediction loss, which is equivalent KI.-divergence. In our experiments, however, we evaluate more general f- divergence generated losses.
tion (hatched). Distillation consistently outperformed SFT. Critically, Chi-squared and a-divergence (a 1.5) losses outperformed KL divergence for both training paradigms.
Comparing /-softargmax variants for decoding. In ad- dition to the loss function used at the training of LMs, an- other key factor that influences the performance of LMs is how one decode responses from them. While the training of LM uses fixed f fdiiergeence loss, when decoding, one can choose different f-softargmax variants to turn LM log- its into next-token probabilities for sampling. In our prior experiments of SFT and distillation (Figure 3), at decod- ing time we used the f-softargmax corresponding to each f /-divergence loss function used at training time. But this raises a question: Do the performance differences primar- ily come from the f-divergence loss used during training or from the usage of different f-softargmax at decoding? To study this, we perform SFT with different f-divergence generated losses as before, but then decode from the trained LM using either the associated f-softargmax or the classical (KL-based) softargmax. Figure 4 shows the results. De- coding with the two types of f-softargmax variants yields nearly identical results. Indeed, note that the two bars of KL divergence are slightly different only due to the effect of random sampling: the observed performance differences in the case of other divergences are comparable to those seen in the KL case. This suggests that the f-divergence used during training primarily contributes to the performance differences of models.
Distillation uses a large teacher model's class probabilities as soft labels to train a smaller student model (Hinton et al., 2015). In the LM context, such soft labels are the teacher model's next-token probabilities, which can be more effec- tive than one-hot labels as they provide richer information about the likely next tokens. We use T5-XL model (800M parameters) as the teacher and a T5-base model (250M pa- rameters) as the student; prior to distillation, both models were SFT'ed on XSum as in Agarwal et al. (2024). We then fit the student model with the loss (15), where (by:) in the loss are the soft labels produced by the teacher model.
For both SFT and distillation, we adopted the XSum set- tings from Agarwal et al. (2024, Appendix A.3). Following Chowdhery et al. (2023) and Agarwal et al. (2024), we evalu- ated summarization quality using the ROUGE-2 score (Lin, 2004) with temperature-baat based sampling (temperature=1). Figure 3 shows results for SFT (unhatched) and distilla-
7

---

Loss Functions and Operators Generated by /DDivergences
5. Related work
Decode with the KL-softargmax Decode with the associated f-softargmax
Loss functions based on f-divergences. Nguyen et al. (2009) studied surrogate losses for binary classification based on f-divergences. However, this approach is straightforward to generalize the multiclass setting, as it requires multi-distribution extensions of f-divergences StarciaGGarcia & Williamson, 2012; Duchi et al., 2018), Recently, f-divergences have been used for distillation in combination with a classical softargmax (Agarwal et al., 2024). In our notation, this defines a loss - D (y softargmax(O)), However, this does result in convex loss in 9. Novello & Tonello (2024) developed deep learning objectives based the variational formu- lation of f-divergences. Sbert et al. (2024); Sharma & Sazim (2021); Aminian et al. (2024) also defined entropies from f-divergence to measure uncertainty, homogeneity or to build losses robust to label errors as also done by Zhu et (2023). f-divergences have also been used to design GANs (Nowozin al., 2016) or more generally provide alternate learning frameworks (Li & Turner, 2016).
15
Rouge-2 Scores
10
TCS
alphal.5
Figure Decoding with the classical (KI--based) softargmax per- forms similarly to decoding with f-softargmax associated with the loss used for training. This suggests that the choice of f f-divergence loss during training, not the decoding method, primarily drives performance differences.
Belousov (2017) briefly studied the idea of generating Breg- man divergences from f-divergences. However, Bregman divergences need to be explicitly composed with a soft- argmax and this composition again usually not convex. In contrast, in our approach, each loss 0 (0, y: q) convex and is implicitly associated with the corresponding softargmax, operator.
4.3. Summary of empirical findings
We summarize the main take-aways from our experiments.
We found that the loss function generated by the a- divergence with a 1.5 works well across all tasks we tried. While equivalent to Tsallis negentropy (a 1.5) with uniform reference measure (Blondel et al., 2020; Peters et al., 2019), its effectiveness across several lan- guage modeling tasks is novel. Its good performance could intuitively come from the fact that it is middle ground between the logistic loss (a = 1) and the sparse- max loss 2). Figure 5 confirms that accuracy is maximized around 1.5.
LM alignment with f-divergences. In RLHF, given a learned reward model X - TR, the optimal policy
p p=(+x=) = argmax Symplr(z) (x, y)] KL(p.g(-1az) PEAl>
We successfully fine-tuned LMs with various f- divergence generated losses, despite the fact the pretrain- ing was carried out using the cross-entrops loss. This enables direct application our f-divergence generated losses to pretrained, open-weight LMs.
where æ E X' is given prompt and q(-(z) e A is reference conditional distribution, usually pre-trained model. This approach can be generalized to (Go et al., 2023). In our notation, this can be written as
f f-divergences
We show that, during text generation, standard soft- argmax yielded good performance is fine-tuned with /-diverreence generated losses. This is surprising because (7) tells us that we should in principle use the t f-softargmax operator associated with the loss. This empirical finding opens up the possibility to use our losses without changing the inference code, which convenient when working with open-weight LMs.
p" (+(x) (oltargmax,(r(ee)))) (r(1)))
when the model
To avoid learning a separate reward model altogether, we can use direct preference optimization (DPO) (Rafailov et al., 2024). which was generalized f-divergences (Wang et al., 2024). Proposition can be seen as generalization of (Wang et al., 2024, Theorem 1) that supports dom(f'). In this paper, our focus was on the SFT and distillation steps, which usually precede RLHF Unlike these works, we propose to use the f-softargmax operator and the corresponding Senchel-Young loss for pretraining, SET and distillation in the space of tokens, making SGDd-based optimization easy.
Loss functions generated by f-divergences not consid- ered before (Jensen-Shannon, squared Hellinger) did not lead to better accuracy. The KL and a 1.5 generated losses consistently achieved the best results across tasks.
8

---

Loss Functions and Operators Generated by /Diveergences
6. Conclusion
Blondel, M., Martins, A. F,, and Niculae, V. Learning with Fenchel-Young losses. Journal of Machine Learning Research, 21(35):1-69) 2020.
We proposed to use f-divergences as regularization for gen- erating loss functions and associated operators (f-softmax, f-softargmax, f-softplus and f-sigmoid). Our proposal es- tablishes a link between the logistie, sparsemax and entmax loss functions and the KI, Chi-square and a divergences, respectively. Thanks to this perspective, our proposal gen- eralizes these losses to non-uniform class weights. Other choices of f-divergences allowed us to create entirely new loss functions. Overall, we found that the loss function generated by the a 1.5 divergence worked comparably or better to the cross-entropy loss all tasks we tried.
Blondel, M., Berthet, Q. Cuturi, M., Frostig, R., Hoyer, S., Llinares-Lopez F., Pedregosa. F., and Vert, J.-P. Efficient and modular implicit differentiation. Neural Information Processing Systems, 2022.
Advances in
Bradley, R. A. and Terry, M. E. Rank analysis of incom-
plete block designs: I. the method of paired comparisons. Biometrika, 39(3/)):324-45 1952.
Chowdhery, A., Narang, S., Devlin, J., Bosma, M., Mishra, G., Roberts, A., Barham, P., Chung, H. W. Sutton, C., Gehrmann, S., et al. PaLM: Scaling language modeling with pathways. Journal of Machine Learning Research, 24(240):1-113, 2023.
Impact statement
This paper explores /-divergence generated loss functions for classification or language modelling. We do not foresee any specific ethical or societal implications arising directly from this work.
Christiano, P. F., Leike, J., Brown, T., Martic, M., Legg, S., and Amodei, D. Deep reinforcement learning from human preferences. In Advances in Neural Information Processing Systems, 2017.
Acknowledgments
We thank the anonymous reviewers for their comments that helped improve the paper.
Cichocki, A. and Amari, S.-i. Families of alpha- beta- and gamma-divergences: Flexible and robust measures of similarities. Entropy, 1236)11532-1568, 2010.
References
Csiszár, I. On information-tyye -tppe measure of difference of probability distributions and indirect observations. Studia Sci. Math. Hungar. 2:299-318 1967.
Agarwal, R., Vieillard, N., Zhou, Y., Stanczyk, P., Garea, S. R., Geist, M., and Bachem, O. On-policy distillation of language models: learning from self-generated mis- takes. In Proceedings of the International Conference Learning Representations (ICLR), 2024.
Dehghani, M., Djolonga, J., Mustafa, B., Padlewski, P., Heek, J., Gilmer, J., Steiner, A. P., Caron, M., Geirhos, R., Alabdulmohsin, I., Jenatton, R., Beyer, L.., Tschannen, M., Amab, A., Wang, X., Riquelme Ruiz, C.,, Minderer, M., Puigcerver, J., Evei, U., Kumar, M., Steenkiste, S. V., Elsayed, G. F., Mahendran, A.. Yu, F., Oliver, A., Huot, F., Bastings, J., Collier, M., Gritsenko, A. A., Birodkar, V., Vasconcelos, C. N., Tay, Y., Mensink, T., Kolesnikov, A., Pavetic, F., Tran, D., Kipf, T., Lucic, M., Zhai, X., Key- sers, D., Harmsen, J. J., and Houlsby, N. Scaling vision transformers to 22 billion parameters. In Proceedings of the International Conference on Machine Learning (ICML), pp. 7480-7512 2023.
Ali, S. M. and Silvey, D. A general class of coefficients of divergence of one distribution from another. Journal of the Royal Statistical Society: Series B (Methodological), 28( 28(1):13-112,, 1966.
Aminian, G., Bagheri, A., JafariNodch, M., Karimian, R., and Yassaee, M.-H. Robust semi-supervised learning via f-divergence and a-rényi divergence. In Proceedings of the IEEE International Symposium on Information Theory (ISIT), pp. 1842-1847. IEEE, 2024.
Belousov, B. Bregman divergence of alpha-divergent 2017. URL http: www.w/wwww......ooooo///..o....w be belousov. net
Duchi, J., Khosravi, K., and Ruan, F. Multiclass classifi- cation, information, divergence and surrogate risk. The Annals of Statistics, 46(6B):3246 - 3275, 2018.
Belousov, B. and Peters, J. f-divergence constrained policy improvement, arXiv preprint 1801.00000.000 2017.
Friedland, S., Gheorghiu, V., and Gour, G. Universal uncer- tainty relations. Physical review letters, 111(23), 2013,
Ben-Ban Ben-Bassat, M. f-entropies, probability of error, and feature selection. Information and Control, 39(3):227-242, 1978.
Garcia-Garcia D. and Williamson, R. C. Divergences and risks for multiclass experiments. In Proceedings of the Annual Conference on Learning Theory (COLT), pp. 28- 1. JMLR Workshop and Conference Proceedings, 2012.
Blondel, M. and Roulet, V. The elements differentiable programming. arXiv preprint arXiv:2403.1 140440.3.466,,,. 2024.

---

Loss Functions and Operators Generated by /Diveergences
Go, D., Korbak, T., Kruszewski, G., Rozen, J., Ryu, N., and Dymetman, M. Aligning language models with prefer- ences through f-divergence minimization. In Proceed- ings of the International Conference on Machine Learn- ing (ICML), 2023.
Novello, N. and Tonello, A. M. f-divergence based classifi- cation: beyond the use of cross-entropy. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
Nowozin, S., Cseke, B., and Tomioka, R. f-gan: Training generative neural samplers using variational divergence minimization. Advances in neural information processing systems, 2016.
He, K., Zhang, X., Ren, S., and Sun, J. Deep residual leamn ing for image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition (CVPR), pp. 770-778, 2016.
Peters, B., Niculae, V., and Martins, A. F. Sparse sequence- to-sequence models. In Proceedings of the Annual Meet- ing of the Association for Computational Linguisties (ACL), 2019.
Hinton, G., Vinyals, O., and Dean, J. Distilling the knowledge in neural network. arXiv preprint orXiv:1503.02550 2015.
Hoffmannn J., Borgeaud, S., Mensch, A., Buchatskaya, E., Cai, T., Rutherford, E., de Las Casas, D., Hendricks, L. A., Welbl, J., Clark, A., Hennigan, T., Noland, E., Millican, K., van den Driessche, G., Damoe, B., Guy, A., Osindero, S., Simonyan, K., Elsen, E., Vinyals, O., Rae, J.. and Sifre, L. An empirical analysis of compute- optimal large language model training. In Advances in Neural Information Processing Systems, 2022.
Radford, A., Wu, J., Child, R., Luan, D., Amodei, D., Sutskever, I,, al. Language models are unsupervised multitask learners. OpenAl blog, 1(8):9, 2019.
Rafailov, R., Sharma, A., Mitchell, E., Manning, D., Er- mon, S., and Finn, C. Direct preference optimization: Your language model is secretly reward model. In Ad- vances in Neural Information Processing Systems, 2024.
Krantz, S. G. and Parks, H. R. The implicit function theorem: history, theory, and applications. Springer Science & Business Media, 2002.
Raffel, C., Shazeer, N., Roberts, A., Lee, K., Narang, S., Matena, M., Zhou, Y., Li, W., and Liu, P. J. Exploring the limits of transfer learning with a unified text-to-text transformer. Journal of Machine Learning Research, 21 (1 (140):1-67, 2020.
Li, Y. and Turner, E. Rényi divergence variational infer- ence. Advances in neural information processing systems, 2016.
Rényi, On measures of entropy and information. In Pro- ceedings of the fourth Berkeley symposium on mathemati- cal statistics and probability, volume 1: contributions to the theory of statistics, volume 4, pp. 547-562. University of California Press, 1961.
Lin, C.-Y. Rouge: A package for automatic evaluation of summaries. In Text Summarization Branches Out, pp. 74-81,200 2004.
Liu, P. J., Novak, R., Lee, J., Wortsman, M., Xiao, L., Ev- erett, K., Alemi, A. A., Kurzeja, M., Marcenac, P., Gur, I., Korblith, S., Xu, K., Elsayed, G., Fischer, I., Pen- nington, J., Adlam, B., and Dickstein, J.-S. Nanodo: A minimal transformer decoder-only language model im- plementation in JAX., 2024. URL http://githii com/ com/goog google deepmind..d.eee dleepinnd/anndo.. nanodo.
Russakovsky, O., Deng, J., Su, H., Krause, J., Satheesh, S., Ma, S., Huang, Z., Karpathy, A., Khosla, A., Bernstein, M., Berg, A. C., and Fei-Fei, L. ImageNet large scale visual recognition challenge. International Journal of Computer Vision 115:211 1 1-252, 2015.
Sbert, M., Chen, M., Poch, J., Feixas, M., and Chen, Entropies from f-divergences. In Proceedings of the Inter- national Conference on Modeling Decisions for Artificial Intelligence (MDAI), pp. 47-60. Springer, 2024.
Martins, A. and Astudillo, R. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the International Conference on Machine Learning (ICML), 2016
Sharma, G. and Sazim, S. A geometric way to find the
measures of uncertainty from statistical divergences for discrete and finite probability distributions. arXiv preprint arXiv:2106. 0e21106.1880 14874, 2021.
Narayan, S., Cohen, B., and Lapata, M. Don't give me the details, just the summary! Topic-aware convolu- tional neural networks for extreme summarization. In Proceedings of the Conference Empirical Methods in Natural Language Processing (EMNLP), pp. 1797-1807. Association for Computational Linguistics, 2018.
Su, J., Ahmed, M., Lu, Y., Pan, S., Bo, W., and Liu, Y. Roformer: Enhanced transformer with rotary position embedding. Neurocomputing, 568:127063, 2024.
Nguyen, X., Wainwright, M. J., and Jordan, M. I. On surrogate loss functions and f-divergences. The Annals of Statistics, 37(2):876 904, 2009.
Terjék, D. MboreauYYssiae f-divergences. In Proceedings of the International Conference on Machine Learning (ICML), 2021.
10

---

Loss Functions and Operators Generated by f- -Divergenncss
Wang, C., Jiang, Y., Yang. C., Liu, H., and Chen, Y. Beyond reverse KI: Generalizing direct preference optimization with diverse divergence constraints. In Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Wortsman, M., Liu, J., Xiao, I.., Everett, K. E., Alemi, A. A., Adlam, B., Co-Reyes, J. D., Gur, I., Kumar, A., Novak, R., Pennington, J., Sohl-Dickstein, J., Xu, K., Lee, J., Gilmer, J., and Kornblith, S. Small-scale prox- ies for large-scale transformer training instabilities. Proceedings of the International Conference on Learning Representations (ICLR), 2024.
Zhu, D., Ying, Y., and Yang. T. Label distributionally robust losses for multi-clas classification: Consistency, robust- ness and adaptivity. Proceedings of the International Conference on Machine Learning (ICML), 2023.

---

Loss Functions and Operators Generated by /Diveergences
A. Experimental details and additional experiments
A.I. ImageNet classification
When using the KL divergence, our training yields 76.87% accuracy on ImageNet. This result matches previous benchmarks of ResNet50 trained on ImageNet, such as https: / github. com.//gosll//laaxttree/mmmmmm http:///mmmmeeemmelse/eee/ee examples/ im genet.
Accuracy
0.770
0.765
0.760
1.2
1.6
a
Figure 5. Validation accuracy ImageNet when using the a-divergence generated loss, with 21., Accuracy is maximized near
1.5.
A.2. Pretraining NanoDO models
The 1.2B NanoDO model's architecture and hyperparameters follow from the setup in Wortsman et al. (2024). The model is similar to GPT2 (Radford et al., 2019) but incorporates modern features like rotary positional embeddings (Su et al., 2024) and lk-layernorm (Dehghami al., 2023). To evaluate the trained model, we use the next-token prediction accuracy on the validation dataset. Our results (Table 3) show that a-divergence on par, and slightly more performant, than the classifical KL approach.
Table 3. Next-token prediction accuracy
Divergence
Accuracy (%)
Kullback-Leiblrr
48.66
Chi-square
46.75
a-divergence (a = 1.5)
48.70
A.3. Standard deviation of accuracy results over random seeds
Table 4. Mcan, standard deviation, min, max accuracies over random seeds for the ImageNet classification results.
Divergence
Mean
Standard Deviation
Min
Max
Kullback-Leibler
0.7684
0.0007
0.7676
0.7692
Chi-square
0.7604
0,0015
0.7587
0.7621
a-divergence (a = 1.5)
0.7758
0.0013
0.7743
0.7776
Aasse--Shanoon
0.7246
0.0014
0.723
0.7266
Squared Hellinger
0.7281
0.0008
0.7272
0.729
12

---

Loss Functions and Operators Generated by /DDivergences
Table Mean, standard deviation, min, Rouge-2 scores over 5 random seeds for the supervised fine-tuning (SFT) results.
Divergence
Mean
Standard Deviation
Min
Max
Kullback-Leinn
9.77
0.02
9.75
9.8
Chi- square
11.15
0.1
11.02
11.31
a-divergence (a 1.5)
14.27
0.04
14.2
14.32
Jensen-Shannon
7.95
0.08
7.87
8.07
Reverse Chi-squared
9.55
0.1
9.44
9.7
Table Mean, standard deviation, min, max Rouge-2 scores over 5 random seeds for the distillation results.
Divergence
Mean
Standard Deviation
Min
Max
Kullback-Leibler
16.64
0.05
16.57
16.71
Chi-square
14.17
0.09
14.01
14.26
a-divergence (a = 1.5)
17.43
0.13
17.19
17.6
Jensen-Shannon
16.51
0.06
16.43
16.6
Reverse Chi-squared
16.3
0.05
16.25
16.38
A.4. Computational cost
Bisection convergence. We experimentally validate the exponential convergence of our proposed Algorithm Our results are in Figure and confirm the theoretical convergence rate.
Mean absolute error
Observed Theoretical
10P
10-2
10-4
10-6
5
10
15
Number of iterations
Figure Error for computing root Proposition using our bisection based Algorithm function the number iterations. We use the output with 30 iterations proxy for the true root. We compare the measured error (dark purple) with the theoretical 2 error (light purple). This experiment run on TPU.
Overall cost. We propose experiment validating that our proposed losses generated by /-divergences and associated operators do not affect the runtime of standard learning pipelines. For this, we consider a Residual Network (He et al., 2016) ResNet18, with either the standard softargmax or the softargmax associated with the a = 1.5 divergence. For different batch sizes, we profile the time needed for running both the model and the f-softargmaxx Results in Figure 7 show that the runtimes are approximately equal.
13

---

Loss Functions and Operators Generated by b- f-Divergences
Ours
Standard
Time (sec.)
0.010
0.005
128
256
512
1024
Batch size
Figure 7. Profiled time for an input of shape (b, 224, 224 to be processed by ResNet18 followed /-softargmax corresponding the 1.5-divergence (dark purple) or the standard Kullback-Loii Leibler divergence (light purple, which case we use the standard JAX implementation) for different batches size b. This experiment is on TPU.
A.5. Impact of prior distribution sigmoids
1.0
0.75
0.8
0.5
0.6
q:
0.25
0.1
0.4
0.2
0.1
0.0
2
0 0
2
Figure 8. Illustration of + sigmoid, (1 21 91)) for a E (0.1, 0.5, 0.75, 0.9} and for f(u) = log the generating function the KL divergence. We see that sigmoid, (1 Qh 91)) at is equal to g.. Intuitively, if a model is uncertain and produces value of 0, then the model outputs qı. the prior probability of the positive class.
14

---

Loss Functions and Operators Generated by /Diieeggences
Reverse Kullback-Leibler
P1
P2
2.5
2.5
2.5
0.0
0.0
0.0
-2.5
2.5
-2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
61
01
01
Squared Hellinger P2
P1
P3
2.5
2.5
2.5
0.0
0.0
0.0
-2.5
2.5
-2.5
-2.5
0.O
-2.5
0.0
2.5
2.5
0
0
01
Reverse Chi-Square P2
P1
2.5
2.5
2.5
0.0
0.0
0.0
-2.5
-25
-2.5
-2.5
0.0
2.5
2.5
0.0
2.5
-2.5
0.0
2.5
0
01
01
Jeffrey P2
P1
2.5
2.5
2.5
6
0.0
0.0
0.0
-2.5
2.5
-2.5
-2.5
0.0
-2.5
0.0
2.5
-2.5
0.0
2.5
61
0
Consen-Shannor P2
P1
2.5
2.5
2.5
C 0.0
0.0
0.0
-2.5
-2.5
-2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
0
0
01
0.0
0.5 Value
1.0
Figure 9. Illustration of (p..P2.Pa) softargmax, (0,, Oz 0: q) when varying 0,, 02 E It for diverse f f-divergences and with
(1, 1)..
15

---

Loss Functions and Operators Generated by f- /Divergeencss
Kullback-Leibler, prior =0.333 0.33, 0.33)
P1
P2
33
2.5
2.5
2.5
0.0
0.0
0.0
-2.5
-2.5
-2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
-2.
0.0
2.5
0,
01
0
Kullback-Leibler, prior q =0.877 0.12, 0.02)
P1
P2
33
2.5
2.5
2.5
82
0.0
6'
0.0
6
0.0
-2.
-2.5
-2.5
-2.
0.0 01
2.5
-25
0.0
2.5
-2.5
0.0
2.5
01
01
Jensen-Shannon, prior q =00.33, 0.33, 0.33)
P1
P2
Р3
2.5
2.5
2.5
62
0.0
0.0
0.0
-2.5
-2.5
-2.5
-2.5
0.0 0,
2.5
-2.5
0.0
2.5
-2.5
0.0
2.5
0,
0
Jensen-Shannon, prior q =0.87, 0.12, 0.02)
P1
P2
33
2.5
2.5
2.5
82
0.0
6
0.0
6
0.0
-2.5
-2.5
-2.5
-2.5
0.0
2.5
-2.5
0,0
2.5
-2.5
0,0
2.5
0
01
0
0.0
0.5 Value
1.0
Figure 10. Illustration of (pi, P2.ps) = softargmax, (0,,
(O0, 02. 0: q) when varying 0,, O2 E for diverse / f-divergences and priors 4.
16

---

Loss Functions and Operators Generated by f- -Divergenncss
B. Examples of o /divergeeecss
KL
f(u)= u log u, dom(f) = R.
f((u) log
fv) = exp(v U 1), dom(f+) = IR
(v) exp(u - 1)
Dy(p,)) a) - KL q) = (p, loooog (p, log 4)
Generalized KL
f(u) u log (u - 1), dom(J) = R
f'()) log
(v) exp(u) - 1, dom(f.)) = IR
(v) exp(u)
Dy(p, - GKL(p,9) = (p,logr) (p, log a) - (p.1) 1) + (g. 1)
Reverse KL
f() log u, dom(s) R++
f'()) - 1/2
[.) 1 logg -v), dom(f.) = IR__ (c.f. Proposition 2)
[L))) -1/V
Dy(p,9) ạ) = KL(G.p)
Jeffreys
f(u) (u 1) log u dom(f) = R++
f'(u) = log +
f-(v) ([exp[I-o))
log W() 1, dom(f=) R (c.f. Proposition 3)
(v) ([exp(1-o)
Dy(p,9) + (logp log q, 4)
Remark: W is the Lambert function
Jensen- Shannon
f(u) u log u (u + 1) log u-1)) dom(f) R
('( log log [+1) log
2n
a+I
fv) = log(2 - exp(u)), df.( = log 2) (c.f. Proposition 4)
(v) = 2 2-exp(v) 2exp(-r)-1
exp(v)
Dy(p,2) = 2 JS(p, q) = KT(p,m) + KL.(g,m) = (p. log pt0 ) + <1. log ( (22)) where = 20 9).
Squared Hellinger
f(u) (Vi- - 1)2 u 2/ 1, dom(f) = R4
f'(u) 1 Vu
fv( 1-v dom(f=) (0,0, 1) (c.f. Proposition 5)
fv(r) [I-UJ=
Dy(p,q) =2
SH((p)) 2
Chi-squared divergence (a.k.a. Pearson divergence)
f(u) 1 (u² 1), dom(s) R
17

---

Loss Functions and Operators Generated by /Divergeencss
('(x)
fv( {(²² + +),, f-m
R (c.f. Proposition 6)
f((v) =
Dy(p, 9) = (p, = = Σラ j=
%
Remark: other possible generating functions are f(u) (u 1)2 or f(u) =2(2² - -u). We choose f(u)
[["² - 1) as it satisfies ('(0) = 0.
Reverse Chi-squared divergence (a.k.a. Neyman divergence)
f(u) { 1), dom(f) R.
('(x) = 2u²
fx(v) = -2e + 1/2 dom(f=) = R. (c.f. Proposition 7)
fl(o) --2V
Dy(p.9) = (g.pp)
a-divergences
(v" -w- (w-=1)-a(u-) + UR (u) ala-1)
a +
f(u) =
dom(f) = R,
uloo (u 1)
i
if #1
f'(u) loga(u)
log u
a
if (0,1) ifa=
fv)
exp(v)
(c.f. Proposition 8)
(I+(a1))n)+
i E too), where 24 = max(x; 0)
(-00,
i 10,
dom f.
R
if 21
(1 + (a 1)u)
if E (0,1) if =a=1
f((v)= = exp. (v)= =
expv
(1
1) if (1,+
a (a- 1)(q, 1)) Remark: By analogy with the KL divergence case, the a-divergence be derived by setting f'(u) = loga (u) and by integrating. The constant of integration is chosen so to satisfy f(1) 0.
Dy(p.q) = (p, (p/g)) (p. (g. 1)) ললー
All functions I above can easily be shown to be strictly convex and differentiable. Note that both properties (strict convexity and differentiability) are defined on the interior of the domain of definition of the function, and therefore non -inddfferentiabilityy on its border (typically the left) does not matter to assess these properties.
C. Proofs
C.1. Conjugates
In this section, we derive the conjugate of f.
f-(v)= = max f(u)
for the divergences listed in Appendix B.
Reverse KL divergence.
18

---

Loss Functions and Operators Generated by /Diivergences
Proposition The conjugate of f(u) -log(u) for E dom f (0, +00) fa(t) -1 log(-u) for dom (-00, 0).
Proof. For v > 0, the function u + wv - f(u) is unbounded above by taking u + to0. For 0, we have lim uw - f(]) +oo fuv - f(u)} -cc. Hence maximizer u of H wu f(u) exists and is characterized by ("("). Since ('(u) -1 uu, have
v = f'() = v = - 1/2 = u u= ---1 f.()).
We then get
f.(v) +log(-1/0) -1 log(-v).
Jeffreys divergence.
Proposition 3. The conjugate of f(u) (u log for dom f (0.+00) is fa(e) W(exp(1-i)) log (exx((--)) - for dom f- = R, where W is the Lambert W function.
Proof. For any v E R, we have function u + ww fu) has a maximizer u" characterized by v (l()).
f(4) = -im fww [(u)} = -00. Therefore, for any v E R, the
Using (u) = log u + 1 a. we get
1.
== (u) = log
- 1 1---- 1/u + log(1/u) = exp(1 v) = cxp(1/u)1/u.
We recall the definition of the Lambert W function.
For 2 0, ((2) is the inverse of the function g(w) = exp(w)) ie., W(2) g g2) and W-'(w) = g(w). We then obtain
u 1 W(exp(1 - u)) = f()).
Furthermore, we have
we u log +
and therefore, we get
f=(u) = ue u log u + log
u log u
W(exp(1- - v)) + log W(exp(1 - v)) 1.
Jensen-Shannon divergence.
Proposition 4. The conjugate of f(u) = u logu - (u + log (u+l) for E dom = (0, (0,+00) is f.(v) =
log u - = log((2 exp(v)), for dom f. = (-o0,
Proof. We have for u + +o0, wo - u ulog u + (u +u++l 1) log((u (u + 1)/2) ~ wv - u log + u log u/2 ~ wu ullog 2, where ~ denotes asymptotic equivalence. Hence for v > log 2, we have lim uv f(u) = too. For v < llo 2, we have lim {uv - f(u)} = lim f(u)} = -0. Hence, for v < log 2, a maximizer u² of H uv - f(u) exists and is characterized by v = f'(").
19

---

Loss Functions and Operators Generated by f- /Divergeencss
Using ("( = log
2u we get
u+1
2u
exp(e)
exp(0)
(u) = exp(t)
2 exp(-u) exp(0) f1)).
2 cxp(e)
Let us now compute [+( wu ulog (u 1) log
u+l
We observe that
u +
u
u + 11) :) =log u U.
u + 1
= log
2 eeexp(1)
2
exp(e)
2
Therefore
1u+1 =u(log v) log u ulog u uE + log u V.
(u + log
Therefore
f+(e) = log u u= 12g(22 exp(u)).
Squared Hellinger divergence.
Proposition 5. The conjugate of f(u) = -Vū 1ia--132m (-o0,1).
1)2 for dom = [0, +oo) is f=(u) = I-p for u E dom f+ =
Proof. We have f(u) u(v 1) + 2 - 1. Hence, for 1, f(u) is unbounded above. For v < 1, lim tuv f(u)} -00 and f(u) has then maximizer u² in [0, + characterized by f'(u") We have f(u) = (Vū - 1)²= u 1u-24 2/ + 1. Using
1 Vi
v f'() = 1
= Vu-
--
(1 8
we get
v
WV
(1 v)2
Vu
1- u
(Vu - 1)2
(1 v)2
so that
2
(*(0)
(1 2)
(1 0)
1-
Chi-squared divergence (Pearson divergence).
Proposition 6. The conjugate of f(u) { - 1) for E dom f = -Ris
fu) 1/222 + for E dom f. = R.
Proof. f is a strongly convex function on R. Hence the maximizer u" + wv f(u) exists for any v E and characterized by f'(u") Using f'(u) u, we get f'(u) = u. Therefore.
(v) = ue f(u) = 2² (02 - 1) = (v² + 1).
1
2
Reverse Chi-squared divergence (Neyman divergence).
20

---

Loss Functions and Operators Generated by /Diveergences
2v + 2 for Proposition 7. The conjugate of f(u) = 2 - 1) for a E dom f = (0, (0,+ is [+(1) = -V-220 V E dom f= = (-)00))
Proof. For v > O, 1o i uv - f(u)} = toc. For v < O, oo
fuw f(u)} = lim - -f(u)} = -c0
Hence the maximizer u* of u. + ut f(u) exists and characterized by f'(u") = v. Using v = ('() 2u2. we get
1
2²
20
-- 20
Therefore,
fL(U) uo 1/1 1
-
11
---1
1)
v 2v
1
2
2u
2
1
V
a-divergences.
Proposition 8. The conjugate of f(u) (w"=1)-a(u-1) ala-1)
+ 1R (u) is
(1+(a-Ipp
if E (0
f+(v)=
exp(v)
if 11
(14(a-130))-1 oF
ifa E (o0), where 2+ max(x, 0)
a
for
(-00, - if E (0,1)
E dom
ifa21
Proof. Case E (0, As u + t00, we have vu [v" [ ~ (u+ u.
ala-1)
Hence for u I-a" 1om vu (u" (1-=1)) t00.
a(a-l)
For v S - lim إااسسا } = lom (u-=1)-o(--1) = - -0. Hence the maximizer ut of
ااa-l)
a(a-1)
up - (u) exists and is characterized by ('(x)) =v
We have
== f'u) = a-1 u= (1+ - f((
Therefore, denoting (1 (a 1)v), we have, for S T
(w) = wv - f(u)
ルナ (g 1) a(2 a(2
a(a-1 1)
alla 1)u + 12 - (a 1)
ala -1 1)
z -1
(1 (a - 1)u) 1 (1-))
a
Case E (20). For any v E IR, lima that g' decreasing on (0, +00].
- 1) 1 30. Denote g(u) vu
(1-=1)-a(u-1) Note a(a-1)
a(a-1)
21

---

Loss Functions and Operators Generated by /-Divergences
u < - we have y0(0)
--v S 0, hence (') for any u 0, that y is decreasing on [0, +00 and
uPPuz09(4) = ( = (0) in this case.
u > - g0) > 0. lim g'(u) oo and maximizer u² of g can be found by the necessary condition ("(")) c. Following the computations done in the case a E (0, 1), for > I-a we get
((()) (1 + (a 1)0)
a
f.(e) (1+ (a - l)b)
And for any E IR, we get
f*(0) (1+ (a 1)u)
a
f.(v) = (1+ (a 1)
where 2 m 0}.
Case a 1 Follows similar computations as for the KL case.
C.2. f-softargmaxes
KL divergence. With f = exp(v 1). we obtain
r¹ 1 log
Ij exp(0,).
Plugging r* back in (13), we obtain
qi exp(6,)
P Σド= 96 exp(0,)
Other choices of f do not lead to closed-form expressions.
C.3. f-softplus and f-sigmoid
In this section, we derive closed-form expressions for some instances of f-offiplus and f-sigmoid. In this binary classification setting, we define 0 = (0,s). s). (go, q1) and p = (1 - ,,,,,")) m)) By Proposition 1, we seek for
argmin + (- 9n f2 7).
FER
Since f. differentiable and convex, such a scalar 7* is characterized by
- dom f.;
(16)
8- r* dom f.,
(17)
gof((--") + qI r²) =.
(18)
KL divergence.
Proposition 9. Let s(u) = u log u. Then,
a exp(s)
(s))) qu + q1 exp(s)'
Proof. Using (18) and (v) exp(v 1), we obtain
1 log(go a exp(s)).
22

---

Loss Functions and Operators Generated by f- -Divergenccss
so that
' exp(s) 4o 41 exp(s)
Reverse KL divergence.
Proposition 10. Let f(u) - log(u). Then,
softplus, , (5: T" qu log(r") - 91 so) -90 + Q1)
sigmoid, (s; (a:nidd(ss:))
91
T*
where
V(90 + q1 8)² 490s)
90 + 92 +
Proof. Using f1( = -1/u, the root condition (18) can be written as
fo 7) +
+
40 + 91
T-8
(T - 8) + 9T =1
T(r 8)
ar² + br + c 0,
for
(go + 91 +
8.
Let us define the discrimant
1/2 4ac
= (go + + 8) Aqos
(0) 90 + 8)2 + 4go9
> O.
Therefore, the root condition (18) reads
+¹ E (TI, T2
b + VA
-b- VA
2a
2a
Since dom fa = R_, we have
(-7* E dom fa) and (s (s-7* 7* E dom fa) = r² 2 max(0, s}.
We have
90 + Q1 + (go + q1 + 8)2 490s
T2
1
72
40 + Q
v
(go
+ 41 8)2 4918)
23

---

Loss Functions and Operators Generated by J-Divergences
Hence for 0, 72 < 0, and for 2 72 < 0. So the unique solution the set of conditions (16), (17) and (18) is
= + qn + + (qo + 41 + 8)2 4gos
Plugging this value in the formulas for the softplus and the sigmoid in Proposition gives the result.
Numerically stable implementation.
For very large positive negative may get numerical issues of the form as-bs. For numerically stable implementation, one could write
T² qo h(g(s))
r² - q1 + h(-((s))
where
g(s) = 90 +
|g| ( + g²
-1)
if -1 otherwise
h(g) 2
g + g²2 + 4god1
We can then further rewrite
|øl (v1+
49091 g²²
-1) [gl
4go41/9²
+
Agugt 9"
+
4go
g2 + 4go + [g]
Censen-Shannon divergence.
Proposition 11. Let f(u) = u log u - (u 1) log (= Then,
softplus, q) = llo(("")) - 4o log(2 1/xx) - 91 log(2 1/(x" 7)
41
iggmoidy(;;) q) = x- -
where
x² b + VA 2a y exp(-s)
4y
b = 2(11 + y4o0 + 41)
C = + + 91
A = 1 4ac.
Proof.
Using (1 = 2exp(-n)-1 the root equation (18)
1
go
91
2exp(r) - +
2 exp((
Using the change of variables = exp(7) and = exp( - s), we obtain
40 41
2r
2ny 1
9o (2ny 1) + 91 (2x 1) (2x 1) 1)
ax + be c 0,
24

---

Loss Functions and Operators Generated by f- -Divergenccss
where we defined
a 4y b == -2(1 + + 34o + 91)
:= + 40 + 91-
Let us define the discriminant
A = 1/2 4ac.
We have
A 4y (11+ [1 go)y" + 91)y 1/2) 4(1 20 + 9i)
44|4(1 20) + + - - 4(1 20 + 21)]
16y9041
0,
where in the second line we used that y > 0 and minz ozr + fBx = 2/ for any a >> 0, > 0. Therefore, we have
-b + VA
x² = exp((T")
2a
Using (12) with
f+(-T")- - logg(2 -
= - -log(2 1/x*)
fa - T*) = - log(2 - exp(s - -*)) -log 1/ 1/(x" y)),
we obtain
softplus, (s) log(x") 20 log(2 1/x*) qi log(2 1/(x" y))
Similarly, using (13), we obtain
sigmoid,(s)
91
2 2exp(r* 8) 2 x y
Numerically stable implementation.
Working with exponential requires careful handling of any formula like uy vy or (uy)/(oy) that can easily reduce numerically oo NaN or 00/00 NaN. We therefore derive the expressions of x² and x y terms of numerically stable operations.
We have, denoting
1 + go and 3 = + 41
x+ a + By
+ 7 VV(a + 11- - 4cy-1
4
y
1/2
(ay' + Byy 1/2
11/2 + v /aay//2++ + +By -
4
1/2
4(1 + 90 + 41)
(ay)
1/2 + By 112) 1 1
1
4
(ay' 12 By 1/202
exp(s/2 + h(s/2)),
where
4(1 40 + exp(2g(s))
h(s) g(s) + log 1+ 11
2log(2)
g(s) = log (ac - + Bc*)
logsumexp( + log(1 + go), s + log(1 + qq)).
25

---

Loss Functions and Operators Generated by /Divergenncss
To summarize, we have
x² = =x==(s//++ r'y = cxp((8/2++ h(s/2)
and therefore
softplus, (s) s/2+ 90 log [2 -xpp h(s/2)]] 41 log [2 exp(s/2- (asis//-hs///)))
sigmoid,
91
2 2exp(-s/2+/h(s// 1
Squared Hellinger divergence.
Using f.(v) = (m-n and using the change of variable x == + 1, the root equation (18)
becomes
g0
q1
q1
=
(1+ +
2
+
8)²
x²
[I 5)2
20 s)² + q1 = x²2xx s)²
20(x² 2 - 2sr + 5²) + q12" 2x² 2sx + 8²)
20(22 2sx + 32) + q1 x²_ -2xx
3 + 2xx²
axt bx + cx² + dx + +e= 0,
where
0 =
-2s
C = 2 90 q1
2sgo
= -os
This is quartic equation, which can be solved in closed form.
C.4. Proof of Proposition 1
First, the maximum defining the softmax and softargmax is well defined since it is strictly concave problem non-empty bounded set. Denoting (p) = qf (p/g)), such that (p) = f' (p/9)) and (0) (0), we can apply Lemma and get
&
max (p, 0) DE
,;1(p//a)) = inf T +
Σ Jn
(19)
TER
j=1
==1
with (0) lim 00 (z) E RRU --00.
Since (0, +00) S dom fl, and q f ((>=n ")) and f for Jmax E sr Oy are well defined.
We can then define
Tmin == Omax - ('(grnlan)
Tmax 0max fr (E -1)
where 0 = 0juexx Since > 0, jjmax
Ej Uj and since f' is increasing, we have Tiin Tmax-
We can then analyze the following function on [Tmin Tmax]]:
h(T) = 1=1
of (max{(; -T, f (0))).
26

---

Loss Functions and Operators Generated by f-Divergences
First, need ensure that can compute derivatives of this function in [Tminy Tmax]- From Lemma 2, have I" (max (03 (0))) (J, (0; - -) and dom(f + AR_ dom U (-00, I (0)]. Therefore, if f' > 00, the domain of (f + 'R Y. is unbounded below. Otherwise, if (0) -00, since im f S dom f" the domain of f", and so of (J + LR. Y unbounded below. Denoting a dom( ( + 1R Ya, since Omax Tmin = [' dom frr then have m
Tuin < a and therefore
Z min
max T < a
fj < a, for all j E (1....,kk)
dom N'.
We can then show that h' (Fmin) S and h (Tmax) 2 Indeed, we have
h (Taii))
=
Σ 4j 9. - Tmin: (((1))
Vimus S' (max((max Tmin (0)))
Tiaas 1' (max { (1 f' (())))
where in (i) used that q 0 and (max f'()))) 0 for any y E dfm(f + 4B. per Lemma 2, and in (ii),
used that q q 0. f is increasing and (p) = p for any p E dom
Similarly, have
('(Tmax)
1 Σj= qi (max((0, Tmax f((}))
2 Σソ= S'd (max(0mma Tmaxx ((())))
=1 Σイ= S. max (or(rrr() (E+))1) ,))())
(6))
where in (1) we used that Ej a,d s (E, ag) max; by a, 2 0 with here aj g0 > 0 and by f.
Timaxs ('()) and in (ii) we used the same reasoning as for h' (Tmin)-
Finally, we show that h is increasing on [Tmins Tmax]. For Imax. we have that
T T Tmax
0max T 2 Omax Tmax f (4imma)) >> ('(0).
Since (ma))) (f++ is increasing on dom f (-00, f(0] (see Proposition 2), T -ff--(mmas (6max T, f' (()))) is increasing [Tmin; mmax] and so h'(7) = Σ/= 93 (max/0, T, (0))) is increasing [Tmins Tmax (as sum of an increasing and non decreasing function).
Overall, h is well defined and strictly convex on [Tminn Tmax] such that h (T((Tin) < 0 and h hT(mmax) 2 0. Hence, we have
h(T) FCR
inf
min
M(T).
and the unique minimizer 7* can then be found by solving the first order optimality condition h'(7*) = 0 in [Tmins Tmax]] This gives the expression of 7* in the claim and plugging r" back into (19) gives the expression of the softmax. The expression of the softargmax follows from Lemma 1.
C.5. Lemmas
27

---

Loss Functions and Operators Generated by /DDivergences
Lemma Given i strictly convex differentiable univariate scalar functions Q2 such that (0, +00) - dom U2y, for any 0 E RR*, we have
sup (p, 0)
Σ (p(p:))
inf T
+
Σ (St
+ 18 (0, T).
DEA
FER
j=1
where
(Ωz + 1 (z) (max T (,
and (0) = 20 (x) R U { ). Given a minimizer T" of the right hind-side, the maximizer p" of the left hand side is given by
= (i3 L,, Y0, - -*)
Proof. We have
k
sup (p. 0) A
Σ (g(p)) sup inf (p. 0)
Σ
()(p), + (1 (1-(p,1) 1))T
PERt TER
f=1
inf 7 + max (p, 0 r1) - Σ (g(pgs
rER
pERt
i=1
h (22 + uR. ) (0, - -))
inf T +
TER
j=1
where the second equality stands from strong duality, using that the simplex is convex, that the 22 are convex functions, and that the maximization problem is strictly feasible since (0, +o0) S domm2
We can then apply Lemma 2, and get
& Σ (p(p,) inf T TER
& Σ
max (p,0) 0) DEΛ
(mxx - (2
==1
with (0) = lim 20 2g (x) E RRU(-o0)
Given minimizer 7" of the right hand-side, the maximizer pa of the left
hand side is given by strong duality as
Pj
argmax p5(0, -
P,20
whose full expression follows from Lemma 2.
Lemma Given a strictly convex differentiable univariate scalar function such that (0, +oo) C dom f. we have, for any y e F.
(f+ + 12, (v) sup (
z20
with f'(0) = lim,- (x) E RU (-0). For y E dom(s + 1R, )₂ = dom f' (- (')), we have
(f+m).((v)) = (max(y, (()))) 2 0. Finally, (f + KR,Y, Y. is increasing on (dom f + 1R, y \ ( f')
Proof. Fix some y R, denote (0) lim,- I' (x) E IR U ( (-oc) and
M(x) = ry f(x).
28

---

Loss Functions and Operators Generated by /Diivergences
y (0) (so provided that (') > -o). then (a) y (a)) 0 for all 0, since I' increasing. So h is decreasing (0, +00) and the maximum of on R,+ is reached 0, that is
sup xy f(x) - - #20
y f'(x) > 0 for all æ < 0 since f is increasing. Therefore h(x) < h(0) for all æ < 0. Finally, since (0) S O, the supremum of h on IR is necessarily greater or equal than 0, that is,
y 2 f'(), then =
su - f(x = -up zy f(z) = (v).
#20
=E3
Note that for (0) (provided that (0) -o0), we then get
(0) -10)
x20
and
(l) argmax 0.
zER
Combining the two cases above, we get
sup.y - f(x) = ('(() ify -f(O) ff
x20
f (0)))
For the derivative, first note that since ʃ + UR, is strictly convex, its convex conjugate is differentiable on its domain of definition. Then, given E dfm(f dR, Y. we have from the previous considerations,
((mxxxyy-(())) = (v)
if > f((0)
(f (f+
10
#20
Using that (0() = 0, we get
4f+ Y.(y) = (
The expressions above also show that dom( (s + 1R Y, dom U (-00,
In any case, for y dom((s + tR)
(f (f+1))(())) argmax f(x) 2 0,
#20
by definition of the maximization set.
Finally, since f is strictly convex and differentiable on (0, +o0), f invertible on (f) where f() 10m (x) and (c()) lim (x)x). Moreover, for any y E (J'(), we then have (ʃ 2 K(y)
fv = (f')-(()), (g). where (f')-1 is the inverse of f on (f'()), f(cz)). Since f is increasing (0, t00), (f") is
also increasing on (f'(),, (co)) and so are ft and (f + 4R, Y. Noting that (f'(0), (co)) = dom f. (-00, f( (dom / + K (=oc,"((0) (0)) concludes the claim.
29


Reviews:
[{"review_id": "1vZIEODlvN", "reviewer": "Reviewer_9x5b", "summary": "The paper investigates a general framework for generating loss functions using f-divergences, extending the well-known logistic loss (cross-entropy). It introduces a new set of operators, namely f-softmax and f-softargmax, and develops a novel bisection algorithm for computing them. The experimental results focus on evaluating the effectiveness of these loss functions in image classification (ImageNet) and language modeling, including pretraining, supervised fine-tuning (SFT), and distillation.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their feedback. \n\n> **The experimental study is rather limited**\n\nWe disagree that our experimental study is limited. We evaluate the proposed losses on tasks of different data modalities, including both vision (ImageNet) and text generation tasks, which cover different training strategies (from scratch, finetuning, and distillation).  All other reviewers appreciated the experimental efforts (\"experimental designs [...] make sense to me\", \"experiments are well designed\", \"Experiments appear to be soundly designed\").\n\n> **accuracy without variance**\n\nThank you for this suggestion. Please see the results in our response to reviewer EvEL, where we report the mean, standard deviation (std), minimum (min), and maximum (max) across 5 independent runs for each divergence in the ImageNet, SFT, and distillation experiments. To summarize, we found that the standard deviation is low.\n\n\n> **underlying reason why a particular case of f-divergence provides some gains**\n\n\nWe agree that such insights—knowing when and why a loss outperforms others on specific tasks—would be beneficial. However, this is very challenging. Indeed, even in the case of the standard cross-entropy loss, its induced loss landscape in model parameters and the ease or difficulty of optimizing over it are generally hard to analyze for practically relevant tasks. We believe that, just as the convergence rates of optimizers have not always dictated deep learning practices, theoretical properties of f-divergence losses may not be reflected in deep learning experiments. For this reason, we prefer to present a methodological approach that showcases the actual performance of different losses, upon which the community can build. Through these experiments, we observed that the alpha divergence with $\\alpha=1.5$ appears to provide a good trade-off between the standard KL divergence and a sparsemax loss.\n\n> **This paper builds on Fenchel–Young losses rather than fundamentally innovating**\n\nAt the beginning of Section 3, we clearly state “In this paper, we propose to study Fenchel–Young losses and associated operators when the regularizer is defined as $\\Omega_f(p; q) = D_f(p, q)$\". We believe this hasn’t been done before.\n\n> **The use of f-divergences in loss functions is already well studied**\n\nAs we cover in our ample related work section, there are indeed existing works using $f$-divergences to derive loss functions, such as Nguyen et al. (2009), as you mention. However, these works do not use the same mathematical formulation: they are not based on Fenchel-Young losses. We are the first to study $f$-divergences as the regularizer in Fenchel-Young losses. \n\n> **\"f-Divergence Based Classification: Beyond the Use of Cross-Entropy\" Novello, Tonello**\n\nWe will add this citation, thank you.\n\n> **How interesting are such loss functions in state-of-the-art systems in language processing?**\n\n\nOur pretraining experiments use  a model with 1.2 billion parameters. While state-of-the-art models often use a larger number of parameters, we argue that this is a size where models already start to be very useful. In fact, many organizations have a 1 billion-parameter model in their offering (e.g., Gemma, Mistral, etc). In addition, our training pipeline relies on state-of-the-art components and training recipes. Our pretraining experiments are implemented with nanodo [1], we use modern decoder-only transformers with rotary embeddings and QK layer normalization, and train them on the large-scale C4 corpus, following the empirical scaling laws found in prior work [2]. This approach allows us to explore the impact of our loss functions within a framework that reflects current practices. To summarize, we argue that our experiments are conducted in a real-world setting and are therefore informative.\n\n[1] Peter Liu, et al. NanoDO: A minimal Transformer decoder-only language model implementation in JAX. http://github.com/google-deepmind/nanodo, 2024.\n\n[2] Mitchell Wortsman, et al. \"Small-scale proxies for large-scale transformer training instabilities.\" ICLR, 2024.\n\n> **Can you report some gains on networks based on attention using different (more complex) loss functions?**\n\nWe are not sure if we understand the question correctly. While it would be possible in principle, our work does not explore using f-softargmax as attention layers (i.e. intermediate layers). Our work focuses on loss functions (i.e. output layers).\n\n> **Do the so-called new entropies satisfy the axiomatic formulation of entropy? What are their properties and their interest?**\n\nAn $f$-divergence gives rise to a well-defined negative $f$-entropy (Cichocki & Amari, 2010) if $f$ is strictly convex (Blondel et al, 2020, Section 4.1). These entropies have different sensitivity to changes in the probability of an event, which we visualize in Figure 1.", "claims_and_evidence": "The paper is very clear in its content and claims, all claims are clearly articulated. However, not all claims are justified and some are a bit overstated. \nGeneralizing or extending loss functions by replacing KL divergence with the class of f-divergence functions is a well-studied topic, with numerous contributions (and techniques) and it is fairly a standard procedure. There is no novelty in that aspect here. Note that the generalization of the logistic loss by replacing the KL divergence with more general f-divergences has been done at least in the bayesian setting (see work by Andrea Tonello and others). Works like Nguyen et al. (2009), Martins & Astudillo (2016), and Go et al. (2023) already explored similar directions. This paper builds on Fenchel–Young losses rather than fundamentally innovating on them, and the use of Fenchel–Young losses (Blondel et al., 2020) as a general framework was already well-established. The new \"entropies\" cannot be claimed entropies simply because they are generated in the same way as KL or Tsallis divergence. Do they satisfy the axioms of information measures?", "methods_and_evaluation_criteria": "The potential benefits or interest of the proposed loss functions are demonstrated on image classification (ImageNet), language model post-training, and distillation. The experimental study is rather limited, failing to test these ideas in other SOTA architectures, so how general the results and the claims could be?", "theoretical_claims": "Yes, the theoretical claims have been checked. The study rigorously defines a new class of operators that generalize softmax, this is correctly done, although there is no technical difficulty. The convexity and differentiability analysis are also correct, based on the framework of Fenchel–Young losses.", "experimental_designs_or_analyses": "The experiments are valid and correct but they are limited, not representative and the gains are not very pronounced or significant (nor are sufficiently supported by technical explanations and justifications why certain things work and others not). Stating the accuracy without the variance makes it hard to see whether the proposed loss function outperform others in ImageNet. Then, while α = 1.5 performs well, fine-tuning across different datasets and architectures might require additional adjustments (the parameter sensitivity needs to be further studied).", "relation_to_broader_scientific_literature": "The use of f-divergences in loss functions is already well studied and extensions beyond KL for cross-entropy have been done before.\nMany ideas here are reformulations or extensions rather than fundamentally new insights.\n\nThe theoretical part, in terms of defining f-divergences, the new entropies, etc. is well established in the information theoretic community and there are many, more general results, alternatives (Sharma-Mittal), etc.", "essential_references_not_discussed": "Although the literature on this topic is very large, there are no essential references missing, apart from examples where f-divergences have been used in loss functions (specifically in cross entropy), such as \"f-Divergence Based Classification: Beyond the Use of Cross-Entropy\" Novello, Tonello, and https://arxiv.org/pdf/2501.18537v1. A clearer comparison highlighting the differences between this work and prior approaches should be provided.", "other_strengths_and_weaknesses": "Strengths:\n- The authors explore the potential gains by extending KL divergence in cross-entropic loss functions.\n- There is an attempt to provide some analytical results (not always new or of technical depth though).\n- The algorithm seems to be relevant in practice as its implementation cost seems to be low. The main computational contribution is the bisection method for efficiently solving the f-softargmax transformation.\n\nWeaknesses:\n- Limited novelty, both in terms of f-divergences results (per se) or extending loss functions (including cross entropy) using f-divergences. There is a rich literature that has explored the benefit from replacing KL divergence by f-div. \n- The gains are not important and there is no theoretical/analytical or even experimental justification on why a particular value of a-divergence (as defined by Tsallis entropy) provides gains (but not some other widely used/well known cases of this rich family of divergences.\n\nOverall, a well-structured, incremental improvement on f-divergence-based losses, but not an important (or groundbreaking) contribution in theory (and practice).", "other_comments_or_suggestions": "Nothing major to comment.", "questions_for_authors": "1) What is the underlying (even fundamental) reason why a particular case of f-divergence provides some gains (not significant ones though)? Does it have to do with the properties it follows? (in terms of (a)symmetry, or satisfying the partition inequality, etc.).\n\n2) How interesting are such loss functions in state-of-the-art systems in language processing? Can you report some gains or interesting results on networks based on attention using different (more complex) loss functions?  \n\n3) Do the so-called new entropies satisfy the axiomatic formulation of entropy? What are their properties and their interest?", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "ZKGNm5jSYk", "reviewer": "Reviewer_EvEL", "summary": "The authors propose a framework including operators (f-softmax, f-softargmax, f-softplus and f-sigmoid) and loss functions that generated by f-divergences for multi-class classification. Mathematical derivations and efficient computation algorithm are provided. The practical performance are demonstrated on ImageNet classification and language model settings.\n\n## update after rebuttal\nThank the authors for their rebuttal and I'd like to keep my evaluation score as weak accept.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for their constructive feedback. We hope we have addressed your comments.\n\n> **it would be better to include performance variance**\n\nThank you for this suggestion. We repeated our experiments with multiple random seeds and reported the standard deviation across independent runs. Specifically, for each $f$-divergence generated loss, we used 5 random seeds, and applied the loss on ImageNet classification, finetuning, and distillation experiments. The results are below. Overall, the standard deviations of all losses are small, which we believe is because the training datasets are large enough to diminish the effect of different realization of random initial parameters.\n\n**ImageNet**:\n\n|divergence|accuracy mean|accuracy std|accuracy min|accuracy max|\n|---|---|---|---|---|\n|cs|0\\.7604|0\\.0015|0\\.7587|0\\.7621|\n|js|0\\.7246|0\\.0014|0\\.723|0\\.7266|\n|squared\\_hellinger|0\\.7281|0\\.0008|0\\.7272|0\\.729|\n|alpha divergence ($\\alpha=1.5$)|**0\\.7758**|0\\.0013|0\\.7743|0\\.7776|\n|kl|0\\.7684|0\\.0007|0\\.7676|0\\.7692|\n\n**Supervised finetuning experiment**:\n\n|divergence|ROUGE-2 mean|ROUGE-2 std|ROUGE-2 min| ROUGE-2 max|\n|---|---|---|---|---|\n|cs|11\\.15|0\\.1|11\\.02|11\\.31|\n|js|7\\.95|0\\.08|7\\.87|8\\.07|\n|rcs|9\\.55|0\\.1|9\\.44|9\\.7|\n|alpha divergence ($\\alpha=1.5$)|**14\\.27**|0\\.04|14\\.2|14\\.32|\n|kl|9\\.77|0\\.02|9\\.75|9\\.8|\n\n**Distillation experiment**:\n\n|divergence|ROUGE-2 mean| ROUGE-2 std| ROUGE-2 min| ROUGE-2 max|\n|---|---|---|---|---|\n|cs|14\\.17|0\\.09|14\\.01|14\\.26|\n|js|16\\.51|0\\.06|16\\.43|16\\.6|\n|rcs|16\\.3|0\\.05|16\\.25|16\\.38|\n|alpha divergence ($\\alpha=1.5$)|**17\\.43**|0\\.13|17\\.19|17\\.6|\n|kl|16\\.64|0\\.05|16\\.57|16\\.71|\n\n\n> **There is another previous work that shares a similar idea by Zhu, Dixian, Yiming Ying, and Tianbao Yang. \"Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity.\" International Conference on Machine Learning. PMLR, 2023.**\n\nWe will add this citation, thank you.\n\n> **There is no theoretical insight why certain variants (alpha-divergence with alpha = 1.5) perform better than others.**\n\nWe agree that such insights—knowing when and why a loss outperforms others on specific tasks—would be beneficial. However this is very challenging. Indeed, even in the case of the standard cross-entropy loss, its induced loss landscape in model parameters and the ease or difficulty of optimizing over it are generally hard to analyze for practically relevant tasks. We believe that, just as the rates of convergence of optimizers have not always dictated deep learning practices, theoretical properties of f-divergence losses may not be reflected in deep learning experiments. For this reason, we prefer to present a methodological approach that showcases the actual performance of different losses, upon which the community can build. Through these experiments, we observed that the alpha divergence with $\\alpha=1.5$ appears to provide a good trade-off between the standard KL divergence and a sparsemax loss.\n\n> **compare to other classification losses in the literature, such as SVM losses and other CE loss variants.** \n\nIn addition to our $f$\\-divergence generated losses, we tried the (multiclass) hinge loss. However, despite trying various learning rates and warm-up, we were unable to make the hinge loss work well on ImageNet. We suspect that the hinge loss, as a non-smooth loss, requires a completely different hyper-parameter tuning. Our observation here aligns with findings from the paper of Zhu et al., (2023) you mention, which demonstrates that applying multiclass hinge loss to Tiny ImageNet results in accuracy only slightly better than random (See their Table 7, where “CS” denotes the Crammer-Singer multiclass hinge loss, which we also used).\n\n[1] Zhu et al, 2023, ICML, Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity.", "claims_and_evidence": "The claims are clear and convincing to me.", "methods_and_evaluation_criteria": "The methods and evaluation criteria make sense to me.", "theoretical_claims": "I didn't check details for the proof but the theoretical claims in the main content make sense to me.", "experimental_designs_or_analyses": "The experimental designs and analyses generally make sense to me. \n\nHowever, it would be better to include performance variance or standard deviation for multiple replicates with different random seeds or data splits.", "relation_to_broader_scientific_literature": "The proposed method could be used for multi-class classification task.", "essential_references_not_discussed": "There is another previous work that shares partially similar idea that generate loss function with convex regularization, which can recover logistic and SVM losses with KL divergence. \n\nZhu, Dixian, Yiming Ying, and Tianbao Yang. \"Label distributionally robust losses for multi-class classification: Consistency, robustness and adaptivity.\" International Conference on Machine Learning. PMLR, 2023.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper writing and presentation is good. \n2. The motivation is clear and the flow of mathematical derivations are neat. \n3. The authors ensure the efficiency of the proposed method, which only takes negligible overhead in practice (Appendix A.3).\n\n\nWeaknesses:\n\n1. There is no theoretical insight why certain variant ($\\alpha$-divergence with $\\alpha=1.5$) performs better than others. \n2. The experiments don't include performance variance for multiple different replicates (as mentioned before).\n3. It could make the proposed method more convincing by compare other classification losses in the literature, such as SVM losses and other CE loss variants. The current compared baselines is a little limited.", "other_comments_or_suggestions": "Please see previous comments.", "questions_for_authors": "Please see previous comments.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "jUbniPnv0m", "reviewer": "Reviewer_aCJJ", "summary": "This paper proposes a generalization of entropy-based loss functions (such as logistic loss and softmax) by incorporating f-divergences. Specifically, the generalization is formulated using Fenchel-Young duality, where the standard Shannon entropy regularization is replaced with f-entropies. The authors demonstrate that several existing loss functions, including sparsemax and entmax, emerge as special cases of this framework under different choices of f-divergences. Furthermore, the paper provides detailed practical considerations on the computation and differentiation of the proposed loss functions, ensuring their feasibility for large-scale learning tasks. The empirical evaluation on ImageNet and language modeling datasets validates the effectiveness of this approach, with the α-divergence (α = 1.5) achieving the best performance.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback. \n\n> **clarify under what conditions non-uniform reference measures might be useful.**\n\nThank you for your suggestion. Non-uniform reference measures may be relevant in classification tasks with known imbalanced classes. We directly tried the relevance of the approach on pre-training in the Nanodo codebase and did not find any particular gains in that setting.\n\n> **The paper lacks a theoretical explanation for why some choices lead to better optimization or generalization.**\n\nWe agree that such insights—knowing when and why a loss outperforms others on specific tasks—would be beneficial. However this is very challenging. Indeed, even in the case of the standard cross-entropy loss, its induced loss landscape in model parameters and the ease or difficulty of optimizing over it are generally hard to analyze for practically relevant tasks. We believe that, just as the rates of convergence of optimizers have not always dictated deep learning practices, theoretical properties of f-divergence losses may not be reflected in deep learning experiments. For this reason, we prefer to present a methodological approach that showcases the actual performance of different losses, upon which the community can build. Through these experiments, we observed that the alpha divergence with $\\alpha=1.5$ appears to provide a good trade-off between the standard KL divergence and a sparsemax loss. \n\n\n> **including additional references in variational inference techniques using f-divergences.**\n\n\nThank you for your suggestions. We will include them.\n\n> **How does it perform compared to temperature-scaled softmax or label-smoothed softmax? Including these comparisons could provide insights**\n\nRegarding temperature, we believe temperature scaling is not useful at training time, as $\\beta$ can be absorbed into the logits $\\theta$. However, it could indeed be useful at inference time. We will add a remark to make this clarification.\n\nRegarding label-smoothing, we believe it would be interesting to try, though this technique can be applied to any loss and is therefore relatively orthogonal to our work.\n\nIn addition to $f$-divergence generated losses, we tried the (multiclass) hinge loss. However, despite trying various learning rates and warm-up, we were unable to make the hinge loss work well on ImageNet. We suspect that the hinge loss, as a non-smooth loss, requires a completely different hyper-parameter tuning. Our observation here aligns with findings from the paper Zhu et al., (2023) mentioned by Reviewer EvEL, which demonstrates that applying multiclass hinge loss to Tiny ImageNet results in accuracy only slightly better than random (See their Table 7, where “CS” denotes the Crammer-Singer multiclass hinge loss, which we also used).\n\n[1] Zhu et al, 2023, ICML, Label Distributionally Robust Losses for Multi-class Classification: Consistency, Robustness and Adaptivity.\n\n> **Typos**\n\nThanks for spotting them. They are now corrected.", "claims_and_evidence": "The abstract claims two primary contributions:\n\n1. Generalizing Shannon entropy to f-entropy produces loss functions that are advantageous.\n\nThis is well supported by theoretical analysis (Fenchel-Young framework) and empirical results (image classification and language modeling)\n\n2. The generalization allows for non-uniform reference measures, which could be useful.\n\nThis is not well addressed. The authors themselves note in line 343 (lhs) that using a non-uniform reference measure did not lead to performance improvements. Although I recognize the possibility to use non-uniform reference to incorporate prior knowledge, this paper did not include informtation how this incorporation could be beneficial.\n\nGiven this, I recommend that the authors reconsidr making Claim 2 in the abstract or clarify under what conditions non-uniform refernece measures might be useful.", "methods_and_evaluation_criteria": "The proposed methods and evaluation criteria are generally appropriate for the problem.\n1. Theoretical insight is strong: The framework unifies prior approaches like sparsemax and entmax under a broader family of entropy-based loss functions.\n2. Practical implementation is well considered: The paper addresses efficient computation of the loss and gradient, making the method feasible for large-scale applications.\n3. Empirical evaluation is sufficient: The experiments on image classification (ImageNet) and language modeling show consistent performance improvements, supporting the claims.\n\nHowever, there is a key issue that remains unaddressed:\n1. What differentiates members of the f-divergence family?\n- It is unclear why certain f-divergences (e.g., α-divergence with α = 1.5) improve performance, while others (e.g., Jensen-Shannon divergence) degrade it.\n- The paper lacks a theoretical explanation for why some choices lead to better optimization or generalization.\n- A deeper analysis of the effect of different f-divergences on training dynamics or model representations would strengthen the claims.", "theoretical_claims": "I reviewed the proofs and did not find any major issues.", "experimental_designs_or_analyses": "The experiments are well designed and support the primary claim that incorporating f-divergence could improve the training process and bring improvements in performance.", "relation_to_broader_scientific_literature": "The topic of this paper has the potential to impact the broader community of machine learning.\n\n- It introduces a new family of loss functions under a unified framework based on f-divergences, offering a fresh perspective on entropy-basesd regularization. Nevertheless, the authors should provide insights into selection of specific f-divergences.\n\n- The variational problem under f-divergences is also relevant to automatic implicit differentiation techniques, which are increasingly used in optimization and deep learning. This paper provides a concrete example.", "essential_references_not_discussed": "I suggest including addition refernces in variational inference techniqeus using f-divergences. f-divergence has been extensively studied in variational inference. For instance:\n\n- Nowozin et al., f-gan: Training generative neural samplers using variational divergence minimization. NeurIPS 2016.\n- Li & Turner. Rényi divergence variational inference. NeurIPS 2016.", "other_strengths_and_weaknesses": "Strengths:\n\n1. The paper is well-written and easy to follow.\n2. The appendix provides substantial details that assist understanding of the paper.\n\nWeaknesses:\n1. Limited comparison beyond f-divergence losses. The proposed loss function is only compared within the f-divergence family, including sparsemax and entmax. How does it perform compared to temperature-scaled softmax or label-smoothed softmax? Including these comparisons could provide insights into why certain f-divergences are more beneficial than others and whether the proposed approach offers advantages beyond the existing methods.", "other_comments_or_suggestions": "Typos:\n- Line 161 and throughout the paper: Jeffrey**s** (instead of Jeffrey) divergence is the correct way of reference.\n- Line 617: Propositon --> Proposition\n- Line 636: coresponding --> corresponding", "questions_for_authors": "NA", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "h4WvsWEXbb", "reviewer": "Reviewer_LzrN", "summary": "The paper proposes using Fenchel-Young losses derived from f-divergences to perform image classification and language model pretraining, fine-tuning, and distillation.   The authors present an efficient bisection method for solving for the f-softmax function involved in optimizing the proposed loss.   The paper analysis f-Fenchel-Young losses in the context of image classification and language model pretraining, fine-tuning, and distillation, along with ablations for softmax aggregation of f-trained logits at generation time.", "questions": "", "limitations": "", "rating": 3, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "We thank the reviewer for their constructive feedback and their interest in this work.\n\n> **m1) Why is the relation between (5) and (6) an upper bound? Is it tight e.g. for  f of Legendre type?**\n\nThere was a typo in these equations, the lower or equal sign should be replaced by a greater than or equal sign. This bound is detailed in Proposition 3 of (Blondel et al., 2020). Indeed it is tight if $\\Omega$ is of Legendre type, that is, if $f$ is of Legendre type on $(0, +\\infty)$. This is the case for the KL, reverse KL, Jensen to cite a few examples but not the case for the chi-squared or $\\alpha$ divergences, for example.\n\n> **m2) Do the authors have any comment on the role of  q=1  vs.  q=1/K? The latter seems more principled. Panel 3 in Fig 10 (q=1/K) resembles Panel 5 of Fig 9 (q=1)**\n\nThis is an excellent question. $f$-entropies are recovered with $q=1$, not $q=1/k$. Using $q=1$ or $q=1/k$ can lead to slightly different regularization function $\\Omega$, and losses, due to the fact that $f$ can be non-homogeneous (that is, $f(p/q)q \\neq f(p)$). Mathematically, the losses differ since\n\n$$\n\\mathrm{softmax}_f(\\theta; \\mathbf{1}/k) \n    = \\alpha \\sup\\_{p \\in k \\triangle^k} \\langle p, \\theta \\rangle \n    - D\\_f(p, \\mathbf{1})\n    \\neq \\alpha \\  \\mathrm{softmax}\\_f(\\theta; \\mathbf{1})\n$$\nwhere $k \\triangle^k = \\{ k p, p \\in \\triangle^k\\}$ is a scaled simplex. Numerically, we did not observe changes in training curves when using one or the other. \n\n> **1) What is the novelty compared to the original Fenchel-Young losses? Blondel et. al 2019 \"Learning with Fenchel-Young Losses\"**\n\nOur paper sets out to study Fenchel-Young losses when the regularization $\\Omega$ is set to a $f$-divergence, which to our knowledge hadn’t been studied before. In doing so, we draw an interesting parallel between entropies already used in Blondel et al (Shannon, Gini, Tsallis) and $f$-divergences (KL, chi-square, alpha divergences).\n\nProposition 1 in our paper can be thought of as a generalization of Proposition 9 in Blondel et al. When $q$ is non-uniform, their proposition does not apply, while our proposition does. When $q$ is uniform, our proposition tackles the case where $f$ is not differentiable on 0. In addition, we prove that there is a *unique* solution to the root problem on the considered interval, which Blondel et al did not prove. Our proof does not go through KKT conditions and rather relies on conjugate calculus. We also provide detailed computations in the appendix that take into account some potential numerical instabilities of naive implementations.\n\nOn the empirical side, we demonstrate the proposed losses on tasks of different data modalities, including both vision (ImageNet) and text generation tasks, which cover different training strategies (from scratch, finetuning, and distillation). In addition, we obtained a novel empirical insight: using the classical soft-argmax works well even if we trained with our f-divergence based losses. This suggests that the choice of the loss used at training time, not the choice of the $f$-softargmax used at inference time, impacts accuracy the most. We hope that our results give a good glimpse of the losses’ potential as well as their limitations.\n\n> **2) Do the authors have any comment on why modifying the FY loss improves performance in general? (It's ok not to, I often dread this question when deriving generalized losses!)**\n\nUnfortunately, we do not have a good theoretical answer to this question, see also the answer we provided to reviewers 9x5b, EvEL and aCJJ. Our goal here was first and foremost to provide experimental results on recent tasks (pretraining, fine-tuning, distillation of LLMs) with a methodological approach to build these losses. We hope that such experimental results may help the community understand the relevance of different losses. We thank the reviewer for the numerous avenues they already proposed.\n\n> **3) Is the f-soft-argmax invariant to addition by a constant?**\n\nYes, this is the case. This comes from the fact that $\\langle p, \\theta + c \\rangle - \\Omega(p) = \\langle p, \\theta \\rangle - \\Omega(p) + c$ when $p$ belongs to the simplex.\n\n> **4) Temperature scaling during training or inference might also be considered (?).**\n\nWe believe temperature scaling is not useful at training time, as $\\beta$ can be absorbed into the logits $\\theta$. However, it could indeed be useful at inference time. We will add a remark to make this clarification.", "claims_and_evidence": "The paper clearly steps through the derivation of the proposed losses using the analogy with the well-known softmax and soft-argmax functions.   I have minor questions regarding specific claims in *Other Comments* below.", "methods_and_evaluation_criteria": "The methods and evaluation criteria appear sound.\n\nIt appears that one challenge of comparing training with different $f$ is that the training losses are different and thus validation or test losses can not be easily compared.   Thus, the authors use accuracy for classification or next-token prediction, and downstream task (summarization) scores for finetuning and distillation.", "theoretical_claims": "I am familiar with Proposition 1 from existing work, and the convergence of the algorithm appears correct.", "experimental_designs_or_analyses": "Experiments appear to be soundly designed.   Language model distillation and fine-tuning are particularly active areas of research.", "relation_to_broader_scientific_literature": "The representation of the f-softmax in Proposition 1 moves far beyond Wang et. al 2024 Thm 1.   Even Eq 13 needs to be rearranged using $(f_*^\\prime)^{-1} = f^\\prime$ to recover their result.    Since this discussion may not be necessary, the citation could also be dropped.", "essential_references_not_discussed": "One might also wonder how to directly optimize the Fenchel-Young loss in Eq 11.   It seems that CvxPy Layers [1] might be used, although this optimization over the vocabulary (rather than bisection over a scalar) seems less efficient and convenient.\n\n[1] Agrawal et. al 2019, \"Differentiable Convex Optimization Layers\"", "other_strengths_and_weaknesses": "The paper is well-written and comprehensive.   \n\nIt would be useful to explicitly emphasize novelty (see below), for the review process at the very least.   \n\nIn any case, the applications to language models are creative and interesting.", "other_comments_or_suggestions": "*Notation for the Loss Function*\n\nI initially thought it might be more insightful to write the loss in Eq. 5 as\n$\\min \\limits_{\\boldsymbol{\\theta}} -\\langle \\boldsymbol{\\theta}, \\boldsymbol{y} \\rangle + \\text{softmax}_{\\Omega}(\\boldsymbol{\\theta})$ (i.e. a conjugate optimization), but I also see why the authors need to include this in Eq. 11.\n- The main concern is that $\\Omega(\\boldsymbol{y})$ is a constant in Eq. 5 and the reader has to parse that this term does not contribute to the optimization (amidst use of $\\boldsymbol{p}$ above and $\\boldsymbol{q}$ later).   Perhaps the authors could add a comment here.\n- However, the payoff is that we can optimize Eq. 11 with respect to q.   \n\n\n\n\n*Minor questions:*\n\nm1) Why is the relation between (5) and (6) and upper bound?   Is it tight e.g. for $f$ of Legendre type?\n\nm2) Do the authors have any comment on the role of $q=1$ vs. $q=1/K$?   The latter seems more principled.   Panel 3 in Fig 10 ($q = 1/K$) resembles Panel 5 of Fig 9 ($q=1$)", "questions_for_authors": "1) \nWhat is the novelty compared to the original Fenchel-Young losses?  Blondel et. al 2019 \"Learning with Fenchel-Young Losses\"\n- a similar bisection scheme appears in their Algorithm 1\n- Tsallis 1.5, sparsemax, etc. are discussed.   JSD and Hellinger appear to fall under the original framework, even if not stated explicitly or tested empirically\n- generalization to reference $q$ is a distinction from Blondel et. al 2019 but appears to be known (perhaps up to $0 \\in \\text{dom}(f^\\prime)$)\n- I invite the authors to spell these out for those of us reviewers and ACs less versed in this line of work.   \n*I will strongly argue for acceptance if this concern is met.*    I do value the empirical study, especially in language modeling settings.\n\n\nMy remaining questions arise mostly from a genuine interest in the work rather than a critical evaluation informing the review score.\n\n\n2) \nDo the authors have any comment on why modifying the FY loss improves performance in general? (It's ok not to, I often dread this question when deriving generalized losses!)\n- In particular, for SFT/Distillation, one of my hypothesis in reading the paper was that allowing for sparsity might be useful for next-token prediction where greedy/top-k/nucleus procedures perform well.   \n- However, the soft-max decoding from f-softmax logits appears to undermine this hypothesis\n- Do we expect any interesting interaction between sparse-f finetuning/distillation and decoding procedures beyond standard (temperature=1) decoding?\n\n\n3) \nIs the f-soft-argmax invariant to addition by a constant?   ($\\text{softargmax}_f(\\theta + c, q) = \\text{softargmax}_f(\\theta, q)$)\n- $\\text{softmax}_f(\\theta + c, q) = \\text{softmax}_f(\\theta, q) + c$ is stated in Terjek 2021, and I've proven it myself at some point.   This corresponds to shifting $\\tau^* \\rightarrow \\tau^*+c$ and appears to not modify the softargmax.   \n- This would be a useful property to specify in general.\n- I came to this by thinking whether one might also consider f-softmax aggregation from standard logits (the reverse setting of Lines 353-376R).   Although \"open-logits\" is an uncommon access model, one could recover softmax logits up to a constant from next-token probabilities and use this for f-softmax decoding (if the above holds).    \n\n4) \nTemperature scaling during training ($\\text{softmax}_{\\beta f}$) or inference (scaling $\\beta$ with given $\\theta$) might also be considered (?).   Some initial experiment or commentary could serve to highlight this as a direction for exploration for practictioners.", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Meta-review: This paper studies Fenchel-Young losses when the regularizer is defined as a $f$-divergence. The framework recovers previously studied losses (cross-entropy, sparsemax, $\alpha$-entmax) and leads to new ones, potentially using non-uniform reference measures. 

Reviewers highlighted the clarity of the paper and the solid experimental design as main strengths of this work. They pointed out as weaknesses the limited novelty -- many ideas here are reformulations or extensions rather than fundamentally new insights, and the best performing loss setting corresponds to a previously studied loss. Yet, the framework (combination of Fenchel-Young losses and $f$-divergences) appears never to have been studied before in its full generality, and the proposed Algorithm 1 is also new, which makes me lean towards acceptance.
Decision: Accept (poster)