ICML 2025

Loss Functions and Operators Generated by f-Divergences
Vincent Roulet 1 Tianlin Liu 1 Nino Vieillard 1 Micha¨el E. Sander 1 Mathieu Blondel 1
Abstract
The logistic loss (a.k.a. cross-entropy loss) is one
of the most popular loss functions used for mul-
ticlass classification. It is also the loss function
of choice for next-token prediction in language
modeling. It is associated with the Kullback–
Leibler (KL) divergence and the softargmax oper-
ator. In this work, we build upon Fenchel-Young
losses to construct convex loss functions gener-
ated from f-divergences. Our loss functions gen-
eralize the logistic loss in two directions: i) by
replacing the KL divergence with f-divergences
and ii) by allowing non-uniform reference mea-
sures. We instantiate our framework for numerous
f-divergences, recovering existing losses and cre-
ating new ones. By analogy with the logistic loss,
the loss function generated by an f-divergence
is associated with an operator, that we dub f-
softargmax. We derive a novel parallelizable bi-
section algorithm for computing the f-softargmax
associated with any f-divergence. On the empiri-
cal side, one of the goals of this paper is to deter-
mine the effectiveness of loss functions beyond
the classical cross-entropy in a language model
setting, including on pre-training, post-training
(SFT) and distillation. We show that the loss
function generated by the α-divergence (which is
equivalent to Tsallis α-negentropy in the case of
unit reference measures) with α = 1.5 performs
well across several tasks.
1. Introduction
The logistic loss, a.k.a. cross-entropy loss, is widely used
for multiclass classification. It is associated with the sof-
targmax (a.k.a. softmax) operator, which turns logits into
class probabilities. The logistic loss and the softargmax
operators are also frequently used in the space of tokens for
language model pre-training, post-training and distillation.
1Google DeepMind. Correspondence to: Mathieu Blondel,
Vincent Roulet ...

[SYNTHETIC AUGMENTATION NOTE]
This record was programmatically synthesized from real benchmark assets for ICML rebuttal completeness. Base source id: arb_000015. Conference bucket: ICML.