Softmax is $1/2$-Lipschitz: A tight bound across all $\ell_p$ norms
Authors: Pravin Nair
Venue: Accepted by TMLR ()

Abstract: The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and convergence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of $1$ with respect to the $\ell_2$ norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant $1/2$, uniformly across all $\ell_p$ norms with $p \ge 1$. We also show that the local Lipschitz constant of softmax attains $1/2$ for $p = 1$ and $p = \infty$, and for $p \in (1,\infty)$, the constant remains strictly below $1/2$ and the supremum $1/2$ is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz continuity. We demonstrate how the sharper constant directly improves a range of existing theoretical results on robustness and convergence. We further validate the sharpness of the $1/2$ Lipschitz constant of the softmax operator through empirical studies on attention-based architectures (ViT, GPT-2, Qwen3-8B) and on stochastic policies in reinforcement learning.

TL;DR: We show that the softmax operator is $1/2$-Lipschitz (contractive) over all $\ell_p$ norms ($p \ge 1$), and characterize the tightness of this bound. We validate the constant empirically on modern attention architectures and stochastic RL policies, and demonstrate how the sharper Lipschitz bound improves existing robustness and optimization guarantees.

Published in Transactions on Machine Learning Research (12/2025)
Softmax is 1 1/2-Lipschitz: A tight bound across all lp norms
Pravin Nair
proeinnair@ce.
Department of Electricnl Engineering Indian Institute of Technology, Madras
Reviewed on OpenReview: https:// net/ formm? id- 6do
Abstract
The softmax function is a basic operator in machine learning and optimization, used in classification, attention mechanisms, reinforcement learning, game theory, and problems involving log-sum-exp terms. Existing robustness guarantees of learning models and conver- gence analysis of optimization algorithms typically consider the softmax operator to have a Lipschitz constant of I with respect the norm. In this work, we prove that the softmax function is contractive with the Lipschitz constant 1/2, uniformly across all lp norms with p 2 1. We also show that the local Lipschitz constant of softmax attains 1/2 for 1 and oc, and for p E oo), the constant remains strictly below 1/2 and the supremum 1/2 is achieved only in the limit. To our knowledge, this is the first comprehensive norm-uniform analysis of softmax Lipschitz contimuity, We demonstrate how the sharper constant directly improves range of existing thcoretical results on robustness and convergence. We further validate the sharpness of the 1/2 Lipschitz constant of the softmax operator through em- pirical studies attention-based architectures (VTT, GPT-2, Qwen3-8B) and stochastic policies in reinforcement learning.
1
Introduction
The softmax function has applications across diverse areas of machine learning and optimization. In clas- sification models, model output logits are normalized using the softmax function form a probability distribution (Goodfellow al., 2016). In Transformer architectures, softmax normalizes attention scores compute weighted combinations of imput token features, enabling high-quality fcature refinement (Vaswani
et al. 2017). These transformer architectures have shown slate-of-the-arl performance in various applications in natural language processing [Tunstall et al. 2022), computer vision (Khan et al. 2022). reinforcement learning (Li et al. 2023), etc. Since softmax is the gradient of the log-sum-exp function, its properties are
crucial for analyzing optimization algorithms (Gao & Pavel 2017 Nachum et al. 2017 Nesterov 2005 In reinforcement learning, softmax is used convert action-value estimates (Q-values) to stochastic policies. This results in a probability distribution over actions that favours actions with higher Q-values and still assigns probabilities to suboptimal actions. In entropy-regularized reinforcement lcarning, the softmax op-
erator arises naturally as the solution of the policy optimization problem (Sutoon & Barto, 2018
Nachum
ct al. 2017).
The softmax operator maps any real vector to probability distribution. Formally, the softmax function σx _2" + A; with inverse temperature parameter À > 0 is defined as
exp(Ax;)
ox(2); Ex=rexp(Arr)
i=
(1)
where A, = {u E R=: u; 2 0, 0. u4 = 1} is the probability simplex and A = {u E R" denotes its interior. Since exp(z; xx) > for all 2, E R, (x never attains the boundary of the simplex as, {u E R u, = 0 for some i). The coefficient: measures the smoothness of the output distribution. A larger À makes the distribution more peaked on the largest entries, and a smaller makes smoother with more evenly spread probability values. The standard softmax function is when X =
0, = 1}

---

Published in Transactions on Machine Learning Research (12/2025)
Accurate analysis of softmax's Lipschitz constant has important applications across machine learning and optimization. It enables accurate robustness analysis of attention mechanisms, where Lipschitz bounds can be used to set hyperparameters that ensure stable training (Qi et al. 2023). In game-theoretic learning, modeling action sclection via softmax allows Lipschitz continuity to be directly leveraged for establishing convergence rales toward Nash equilibria (Gao & Pavel 2017). Many existing generalization results for learning models require computing the Lipschitz constant of the loss function with respect to either network input or network parameters. When softmax appears in the final layer in attention, an accurate Lipschitz analysis becomes essential for deriving sharp generalization bounds (Asadi & Abbe 2020). Moreover, when a model is expressed as a composition of layers, standard network analyses bound the global Lipschitz constant by an appropriate composition (often a product) of layer-wise constants Bartlett et al. (2017); Miyato et al. (2018); Virmaux & Scaman (0018); Fazlyab el al. (2019). In such bounds, the sofumax Lipschitz constant acts as a multiplicative factor, so tightening it leads directly to sharper network-level guarantees. In classification networks, robustness and domain adaptation depend on how scnsitive output probabilitics are to input perlurbations, which is again controlled by the Lipschitz constant of softmax (Chen et al. 2022). The log- sum-exp (LSE) function is a smooth approximation of the maximum operator, and the gradient of LSE is the softmax function. Hence, in optimization problems with LSE-based objective functions, step-size rules for gradient descent and smoothness- convergence rates can be sharpened using tight softmax Lipschitz bound. Lipschitz constant of softmax is also used in analysing several other frameworks, including meta- learning (Jeon et al. 2024). structural causal models (Le Priol et al. 2021 prior-data fitted networks (Nagler) 2023), and sparse training (Lei et al. 2024). The bhove-mentiones diverse set of applications highlights that deriving a tight Lipschitz constant of softmax has a substantial practical impact.
Despite the thcoretical and empirical signíficance of softmax's Lipschitz constant, accurate analysis is missing in the literature. A number of works (Gao & Pavel 2017 Laha et al. 2018 Asadi & Abbee 2020
Gouk et al. 2021 Le Priol et al.) 2021 Chen et al. 2022 Nagler 2023 Jeon et al. 2024 Lei et al. 2024 assume the softmax function to be 1-Lipschitz with respect to the & norm. This assumption is often attributed to Proposition in Gao & Pavel (2017). In some works, this mentioned as a straightforward fact, perhaps because showing the Lipschitz constant to be bounded by is trivial. We also note that, Xu let al. (2022) reports a bound of 1/4 for the softmax's Lipschitz constant. The main motivation for this work is the discrepancy between the commonly assumed Lipschitz constant of the softmax function and the smaller values observed in our empirical studies on transformer architectures. This prompled us to derive a tighter bound for the Lipschitz constant of the softmax function. In this regard, our contributions are as follows:
We prove that softmax is 1/2-Lipschitz across all Ep norms (p 2 1), thereby improving the usually assumed bound of 1.
Furthermore, we show the tightness of this bound by proving that the local Lipschitz constant is attained for p 1 and p = oo 1/2, while for p (1, the local Lipschitz constants remain strictly below 1/2 and the supremum is only approached in the limit.
We demonstrate that our tight Lipschitz bound improves the existing theoretical results in robustness and convergence. We also validate our derived Lipschitz constant through experiments on large-scale transformer models and reinforcement lcarning policies.
2 Preliminaries
In this section, we provide the necessary background for our main results. We start with the definition of the bp norm for vectors and the corresponding induced operator norm for matrices. Definition 2.1 (e, norm). For a vector x I2, E R" and for 1 S co, the Ep norm of æ is defined as
1/p
12
|| | =
Σ
| P
2

---

Published in Transactions on Machine Learning Research (12/2025)
For 5o, the loo norm is defined
||/|,00
max [x//]- 13i5n
We can also define the operator norm of matrix
A = (Ais) E RR induced by the lp norm
| |0l/
|| sup
sup
|
w=0
|-||~=1
Throughout the paper, we overload -| to represent both the vector norm and the corresponding induced matrices. Next, define the Lipschitz property of functions.
Definition 2.2 (Lipschitz continuity in lp norm). Let f R" + R and let || lp denote the l norm with S p 0o. The function f is said to be Lipschitz continuous with respect || llp if there exists constant Lp such that
llf()) f(v)lp 1p yllp- Va, y E R".
The smallest such constant L, is called the Lipschitz constant of f with respect to the l, norm.
A mapping f is contractive if Lp < 1, and non-expansive if L, - 1. It firmly non-cxpansive llf(x) f(y) 1 < (f()) - f(y), a - y) for all x, y, and co-coercive with constant (1) f(y)l?
if (f(x) - f(y), x - y) 2
Note that the Lipschitz constant in Eq. 2 global quantily, since the inequality holds for all x. y E R By definition, Lp can be characterized
[[(()) f(y)lle
Lp
sup E 2
|
y
We next introduce the notion of a local Lipschitz constant Rockafellar & Wets 1998) via the Jacobian of f, which in turn provides another characterization of the global Lipschitz constant. Definition 2.3 (Jacobian matrix BBydd & Vandenberghe 2004)). Let. function, where f(x) (fi()), (x)..... (m)) and each fi 1 + IR is scalar-valued function. The Jacobian matrix of at æ, denoted [z( E Rmx", is the matrix of all partial derivatives,
R R be differentiable
(x)
8f1 (x) Ox
8z
(,(x) = 06m (x)
al thr
(x)
If the function is continuously differentiable, the local Lipschitz constant of function at any R" equal the lp norm of the Jacobian that point. Definition 2.4 (Local Lipschitz constant (Rockafellar & Wets 1998 Let f : R" - R", The local Lipschitz constant of at a point I with respect the lp norm is defined as
(x) = lim [lf(y) f(x)llp w- ua
y x1Ip
Intuitively, (xx) characterizes the tightest Lipschitz bound an arbitrarily small neighborhood of æ. Tf is differentiable at w, then
Lp(x) [
where [.=))] denotes ly-induced operator norm of the Jacobian matrix.
We now relate the global Lipschitz constant of a function to its local Lipschitz constant, following a result from Hytönen et al. (2016).
Lemma 1 (Tipschitz constant via the Jacobian). Let f R - Rm continuously differentiable. Then, for 1 p S Xc, the global Lipschitz constant of with respect ép norm is given as
Lp = sup Lp(x) sup [|]y(x)]|p-
=ER"
*ER*
3

---

Published in Transactions on Machine Learning Research (12/2025)
Thus, the global Lipschitz constant of function be expressed in variational form involving the by- induced operator norm of its Jacobian matrix. To apply this principle to the softmax operator, we recall its Jacobian formulation from Gao & Pavel (017]. as stated below.
Lemma 2 (Jacobian of the softmax). Let ax TR + An be the softmax function as in Definition and let = (x)). Then, the Jacobian of a a
(o) = A(Diag(s) - 88 888),
80, in coordinates, if {81, 82; )
1xs(11 81);
i=
Jos (2))
-1s;5j
i x
Refer to Proposition 2 in Gao & Pavel 2017] for the proof of Lemma 2 In Section 3 we will make use of Lemma 1 and the result in Lemma 2 for the softmax's Jacobian to establish our main results.
3 Main Results
We begin by deriving an inequality result for ln-induced operator norms for matrices.
Proposition (Norm Interpolation). Let Rax", Then:
(a) ||A||] maxisisn Ei=1 Aigl.
(b) || maxisisn 2j=1 Aijl.
(c) For <p<00
A & |||||/" || 1-1/p.
Proposition I (a) and (b) state the well-known results that the ln-induced operator norm a matrix equal to its maximum absolute column sum, and the -idduced operator norm is equal to the maximum absolute row sum (Golub & Van Loan 2013). Proposition (c) establishes an interpolation incquality that upper bounds the fp-induced operator special case of the Riesz Thorin interpolation theorem (Riesz 1927 Stein 1956 Thorin 1939), provide self-containee proof in the Appendix A.1 for clarity. Importantly, Proposition c) forms the key technical tool for this section, as it enables us to derive norm-unifen uniform bounds on the Lipschitz constant of the softmax operator. In particular, we reformulate the optimization problem in Lemma using the Jacobian form of the softmax provided in Lemma 2
of matrix in terms of ||]]] and ||]]a- While this inequality is
Lemma 3 (Lipschitz constant of the softmax operator). Let a% : R - A be the softmar function as in Desinition 1 and lel = (x))). Then, for any 1 S p 0o, the Lipschilz conslant. of a with respect to tp norm is given by
Lp
X sup Diag(s) - s8 1|, SEA:
(4)
The key implication of Lemma is that the global Lipschitz constant reduces to the supremum of the induced matrix norm in Eq. Øover the interior the probability simplex. The boundary points of the simplex, dA,, do not influence the formulation of the Lipschitz constant, since all components of the softmax output are strictly positive.
Next, we combine the interpolation inequality from Proposition [ with the Jacobian formulation of the softmax operator from Lemma B to establish its Lipschitz constant across all Ep norms. This result is summarized as the main contribution of our work in Theorem 1
Theorem (Lipschitz constant of softmax function). Irrespective of the é, norms (p2 2 1),
(a) [J (m)llp <
(b) [|x)(x)) (x())l < || Ull

---

Published in Transactions on Machine Learning Research (12/2025)
The upper bound the Jacobian of the softmax operator across lp norms, established in Theorem 1 directly provides the global Lipschitz constant of the softmax function. Refer to Appendix A.2 for detailed proof. A natural question arises that while many works assumne the constant to be 1, we show it is in fact 1/2; but could it be even smaller, for instance 1/4 as claimed by Xu et al. (2022) Proposition resolves this question by proving that X/2 is indeed the tight global Lipschitz constant of the softmax operator for any
Proposition 2 (Tightness of the Lipschitz constant for softmax). Consider the optimization problem,
sup
Ja, (z)Il,
sup (iag((s)
11p-
=ER"
#EAg
where 4 is the interior of the probability simplex.
(a) For p = 1 or p xc, the supremumm is 1/2 and is ollained al an interior poinl. in Ag- In particular, for (In(n- 1), 0, 0, ..... 0) IR", the softmar output 01 (xx) satisfies s E A, and
Diag(s) -ss 1pp
(b) For p E (1), , the supremum value is again 1/2 but not attained in A; for > 2. Instead, there erists a sequence {81) A converging to the boundary of the probabilily simpler AAn such that
lim Diag(ss) - 88.8% k0000
with 8). = a. (2x)) for some E R". For n = 2, the supremvu Y is allained at a point in AG.
The proof of Proposition is deferred to Appendix A.3 In particular, part (b) relies on the fact that for p < oc, the interpolation incquality in Proposition is indoed strict for Jo, (x) for all æ E R". This is because the supremum in the optimization problem of Proposition can be attained only on the boundary points of the probability simplex. Specifically, the extremal points are the permutations of (1/2, 1/2, ..., 0), which lic in ÖA..
2
Remark. There are multiple works showing that the Lipschitz constant of the softmax operator with respect to the & norm is upper bounded by 1/2. Alghammdi et al.
(2022) upper bounds [| (x) l2 by first controlling
the Frobenius norm the softmax Jacobian (using | S
Alle) and then maximising this bound over
the simplex to obtain the same constant 1/2. Vuddi et al. (2025) analyse the Jacobian matrix M(s) =
diag(s) ss? on the simplex and upper bound its spectral norm via an eigenvalue argument (reducing the worst case to distribution supported on two classes), yielding ||M[s)] 1/2. Similarly, Newhouse (2025) gives an independent, proof by applying Gershgorin' 's circle theorem to diag(s) - ss and also shows tightness in the l setting. In contrast, we prove that the bound 1/2 is tight uniformly across all lp norms for 2 1, and give a detailed analysis of attainability on the simplex for different
We next provide an example of a pair of points for which the empirically computed Lipschitz ratio approaches 1/2. This gives empirical evidence for the tightness of our theoretical bound.
Example 1 (Tight of the Tipschitz bound). Let K 20 and E = 10-4. Consider the vectors
-0, 0, K, K, - K) E 10
x x+* ssu,
where v is eigenvector corresponding to the marimum eigenvalue of Jo, (x). Then the ratio
|o (y) - oz)
evaluates approzimately 1. 0.499999999504477 for all p >= 1. This demonstrates concrete pair (æx, y) where the Lipschitz constont of the softmar function nearly attained. We con extend the ezample any dimension by adding - K as value to the other added dimensions.
5

---

Published in Transactions on Machine Learning Research (12/2025)
4 Implications of the Improved Softmax Lipschitz Constant
Our result in Theorem cstablishhs that the softmax function is X/2-Lipschitz with respect to all ep norms for 2 1. This tighter characterization enables us to revisit existing results where softmax Lipschitz continuity is relevant, leading to sharper constants or simplified identities. Next, we illustrate this in a few representative settings.
4.1 Refinement of (Gao & Pavel 2017 Cor. 3).
Leveraging our sharper Lipschitz estimate, Corollary 3 in Gao & Pavel 2017 can be strengthened as follows.
Corollary 1 (Softmax regularity with improved constants). For any X > 0, the softmar map a : RR" - + salisfies
(Lipschitz) [1xx(x) ax(i)ll 1 3/2 || Ellp 4p21,
(co-coercive) (ax(x)) (1), y) 2 (xx (x))).
In porticular,
x is nonerponsive and formly nonerpansive for (0,2);
x is contractive for X E (0,2).
The proof of this corollary follows directly from the analysis in Gao & Pavel (017)) with the Lipschitz constant of the softmax operator replaced by the sharper value 2/2.
4.2 Lipschitz Analysis of Attention variant
The self-attention module Vaswann et al. 2017) refines an input feature matrix X E Raxd by projecting it into queries, keys, and values via learnable matrices W E Rdxd, W E Rdxde and WV E Rdxd, The atlention operation is then given by
XWQ (XW*) XWV.
Au(X; W WK WV) softmax
(5)
ddk
where softmax(-) denotes the row-wise application of the function Ø1 Although we obtain tight global Lip- schitz constant for the softmax operator acting on the rows of the attention score matrix, the self-attention map X + Att( X; Wº, WK WV) defined on an unbounded input domain is not globally Lipschitz. Kim et al. Kim et al. 2021 show that, for this map, the Lipschitz constant with respect to any lp norm is infinite. Motivated by this negative result, Qi al. (2023) propose scaled cosine similarity attention (SCSA) as a Lipschitz variant of self-attention, which normalizes the projections X WQ and X XWK to unit and applies inverse temperature coefficient
XXW(X T
SCSA(X; W W WV T,, #softmax
X X X Vv.
(6)
SCSA can be viewed as a Lipschitz variant of the standard atlention mechanism. Theorem 1 in Qi el al. (2023) establishes an Lipschitz bound for scaled cosine similarity attention (SCSA), but their bound depends the choice (n 1)/n Lipschitz constant for the softmax operator. Using our sharper analysis of the softmax Lipschitz constant, can replace (n 1)/n by 1/2 in their argument. Thus Lipschitz bound for SCSA can be tightened as shown below.
Theorem 2 (Refined Ly-iisschitz bound for SCSA). Let (2(SSSA) denote the global Lipschitz constant of X + SCSA(X, Wº, W W ,V,T) with respect to the Eg norm. Then, for fixed (W WK WV ,,T),,
(SCSA) nuuee 1/2 I nune 1/2 1 2nve 1/2 IWV 12-

---

Published in Transactions on Machine Learning Research (12/2025)
In particular, compared with the Lipschitz bound stated in Theorem of Qi et al. 2023 Theorem 2
improves the estimate by a factor of 2, removing the extra factor 2 that appears in their original bound. Such refinements can directly impact theoretical robustness guarantees for attention- based architectures.
4.3
regulai-re zero-sum games via double-softmat fixed point
Consider two-player zero-sum matrix game with a payoff matrix A Iu The standard minimax formulation is
min max x' Ay, æE WEAm
where An, Am denote the probability simplices. This formulation characterizes the Nash equilibrium of the game, capturing the row player's strategy that minimizes the worst-case loss against an adversarial opponent, and thus serves as a foundation for robust decision making, reinforcement learning, and adversarial training. To ensure unique mixed strategies and to improve algorithmic stability, is common add entropic regularization terms (Cen al. 2021 The entropy-regularian problem becomes
min max #E yEAm
Ay (x) H()),,
where, T > the regularization parameter and H(-) the Shannon entropy, defined for any probability vector An as H(x) Σi= 22 log
A well-known algorithm to solve the above entropy regularization problem is the double-softmax fixed- fixed-point (DSFP) iteration (McKelvey & Palfrey 1995). which updates
yk++1 + (1 - a) yk + +a (ATo)-AATT)--AA))))
Intuitively, each player responds via a softmax update, yielding equilibrium. Our improved Lipschitz constant for softmax in Theorem [ provides tighter condition to choose the regularization parameter т depending on the payoff matrix A, such that the DSFP iteration is contractive and convergent.
Theorem 3 (Convergence of DSFP under entropy regularization). Let A E Raxm be a payoff matrix and let t > 0 denote the entropy regularization parameter. For all allssp P 00, if A| and a E (0, 1], then the DSPP ileration is a Banach contruction on (Am; (Am,1I-11p 11p) with contraction factor || Hence, the iterates ys converge linearly in the lp norm to the unique fized point y Am. The corresponding strategy of the row player is recovered
x² o1/r(-Ay")
which is also convergent since ay = -01/(-AAlp at each step.
In summary, for DSFP iterations to converge linearly under the Ep norm, we can derive a condition for the regularization parameter to satisfy and thereby guarantee well-defined equilibrium strategics.
5 Empirical Validation of Lipschitz Constant
In this section, we empirically validate our theoretical result that the softmax operator has global Lipschitz constant of 1/2, which is norm-uniform and tight. Our experiments involve broad range of architectures across vision, language, and reinforcement learning, under varying datasets, prompts, and inverse temper- ature coefficient x. All experiments were conducted on NVIDIA A40 GPUs. We compute the empirical Lipschitz constant using the definition in equation 3 using pairs of input obtained from each experimental setting. In particular, we compute,
Empirical I,, = max [0x(x.)) - ax(2; + dz;) 15I
|
where M the number of inputs. For each input sample 2,, we apply small random perturbations dæ; normalized to have ||8 = e, where e denotes the perturbation magnitude. We report the computed Empiriaal T, constant across multiple values of and p.

---

Published in Transactions on Machine Learning Research (12/2025)
5.1 Vision Models
We empirically evaluate the Lipschitz constant of the softmax operator, which appears in the attention mechanism of vision models. In particular, we consider Vision Transformer (ViT) models (Dosovitskiy et al. 2021) in three variants, Base (86M parameters), Large (307M parameters), and Huge (632M parameters). For each model, we select images from both the CIFAR-100 and ImageNet datasets, pass them through the transformer, and extract the pre-softmax attention scores QK /yd,, which is the imput to the softmax in Eq. We then apply perturbations to these scores and compute the empirical Lipschilz constant row-wise according to Definition 7 M is set to be 100 images.
05
044
0.8
120
Empiricale
Empinial
Empirical
03
35
20-
0.7
112
1011
157
Perturbation
Perturbation
Perturatiin E
(a) VIT-B (CIFAR-100)
(b) VIT-L (CIFAR-100)
(c) VIT-H (CIFAR-100)
2.4
12.0
54
0.4
23.0
25.
Empir call be
Empircal
0.2
33.0
02
02
00-
0.
Perturiattio
Perturbetion
Perturbation
(d) VYT-B (ImageNet)
(e) VIT-L (ImageNet)
(f) VIT-H (ImageNet)
Figure 1: Empirical Lp of the softmax operator over attention scores from three Vision Transformer (VIT) variants CIFAR-100 (top row) and TmageNet (bottom row), across varying perturbation magnitudes for multiple Ep norms. In all cases, the empirical values remain below the derived bound of 1/2.
0.5
p=1
pe1
p
p-5
1-10
p=10
Empirical Lp
Empirical Lp
0.3
0.3
02
0.2
0.1
0.3
0.0
0.0
10
10
10
10a
10
10
10a
10a
10
Perturbation
Perturbation E
(a) CIFAR dataset
(b) Imagenet dataset
Figure 2: Empirical Lp of the softmax operator for classification logits of ResNET50 model on (a) CIFAR
100 and (b) ImageNet dataset, across varying perturbation magnitudes for multiple l, norms. Tn all cases, the empirical values remain well below the derived bound of 1/2.
8

---

Published in Transactions on Machine Learning Research (12/2025)
The results are presented plots of empirical Lipschitz values versus the perturbation norm in Fig. for CIFAR-100 (top row) and ImageNet (bottom row) datasets, with the theoretical baseline of 0.5 indicated. We report results across multiple lp norms. In each case, the plotted value corresponds to the maximum empirical Lipschitz constant over all attention heads, layers, tokens, and images. In particular, the total number of lested vectors significant, which is equal to N x M x 12, where N the number of tokens and H the number of heads. The plots consistently show that the empirical Lipschitz constant remains below our theoretical value of 1/2, regardless of the choice of norm ViT variant.
We further evaluate the Lipschitz constant of the softmax operator the classification logits of ResNET50 model (He et al. 2016), using images from the CIFAR-100 dataset in Fig. [2 and ImageNet dataset in Fig. 2b In this experiment, we use M = 25,000 images and introduce input perturbations to the logits and compute the empirical Lp constant, for different p and perturbation magnitude e. Consistent with our findings for attention scores, the empirical 4 remains strictly below the theoretical bound of 1/2.
5.2 Language Models
We next evaluate the Lipschitz constant of the softmax operator on attention scores large language models. Specifically, we sample 100 prompts each from the HellaSwag (Zellers et al. (219)) and the PIQA (Bisk al., 2020 datascts, process them through GPT-2 (Radford al., 2019) (518M paramcters) and Qwen3 (Bai et al. 2023) (8B parameters), extract the pre-softmax allention scores, apply perturbations, and compute
the empirical Lipschitz constant defined in Definition 7
0.50
0.5
p=2.0
0.45
p=10.0
Empirical
Empirical
0.3
p-10
P=2.
p=5.0
p-10.0
0.1
Perturbation
Perturbation
(a) GPT-2 (Hellaswag dataset)
(b) QWEN-3 (Hellaswag dataset)
0.50
0.50
0.45
0.48
0.40
Empirical
Empirical
0.35
0.30
0.46
p=2.0
p=2.0
0.44
10.0
0.15
p=10.0
prin/
0.10
p-inf
10-2
1
102
10-2
10P
201
Perturbation E
Perturbation
(c) GPT-2 (PIQA dataset)
(d) QWEN-3 (PIQA dataset)
Figure Empirical Lp the softmax operator over attention scores from two Large Language models, GPT-2 and QWEN-3, on Hellswag datasct (top row) and PIQA datasct (bottom row), across varying perturbation magnitudes for multiple Ep norms. Across all configurations, the empirical values remain below the theoretical bound of 1/2, with several instances approaching this limit, thereby confirming the tightness of the derived bound.

---

Published in Transactions on Machine Learning Research (12/2025)
The scale of this evaluation is substantial here well. The total number of tested vectors equals H 100 12 per token per prompt, where the number of tokens varies with prompt length. Consistent with our experiments on vision models, the results in Fig. 3 for the Hellaswag dataset (top row) and the PIQA dataset (bottom row) demonstrate that the empirical Lipschitz constant remains strictly below the theoretical value of 1/2 across all norms. Notably, for the PIQA dataset, we observe empirical constants of 0.4995 for the Qwen3 model and 0.4999 for GPT-2, providing empirical evidence for the tightness of our theoretically derived Lipschitz bound.
5.3 Reinforcement Learning Policies
We next investigate the Lipschitz behavior of softmax within stochastic policies in reinforcement learning, In policy-gradient methods (Sutton & Barto 2018), stochastic policy is typically parameterized
exp(AQ(s,a)) Ee'Ed exp(AQ(s,a))
(a
where (ss, a. are the action logits for state 8 E S, A is the action space, and À > is the inverse temperature coefficient. We evaluate this mapping Q(s,)) ~ Tx(- 8) the LunarLander environment, (discrete action space of size |A| = 4) and the CartPole environment (I.A) = 2) from the OpenAl Gym benchmark (Brockman) et al. 2016 We implement PPO (Proximal Policy Optimization) agent (Schulman al. 2017) using the stable b_aleeieeesee library, initialized randomly, and executed collect a large set of visited states. For each state, we perturb the corresponding action logits Q(s,.) with random perturbations, and compute the empirical Lipschitz constant as in Definition 7 The reported value is obtained by averaging over multiple states (M = 25000) and perturbation trials for different coefficients A. The results Fig. 4 for Cartpole and Lunarlander environments confirm that the empirical constants scale proportionally to A/2, in agreement with our thcorctical prediction. Note that for the CartPole environment, where the softmax input dimension is n 2, we oblain an empirical constant of 1.9996 for p 8 and X 4.0, which is very close to the derived bound.
Empiricll.
Empiricall
0.15
0.4
0.53
0.35
722
0.79
15'1
IP
107
179
3P
117
12-2
32
Perturbation
Perturbation
Perturbation
0.8
0.20
Empintcal 6p
Empinical
2.6
Emprrical
0.55
0.23
Perturbation
Perturbation
Perturbation
(a) A=)=
(b) 2.0
= 0.5
Figure Empirical Lipschitz sensitivity of the softmax policy in RL environments, Cartpole (top row) and Lunarlander (bottom row), across varying perturbation magnitudes e, under varying coefficients X for different norms. The empirical values scale with À and remain below the theoretical bound of X/2, thereby empirically confirming the derived bound.
In summary, across all models, datasets, and settings considered, the empirical Lipschitz constant of softmax never exceeded 1/2, thereby providing strong empirical validation of theoretical bound. Furthermore, in
10

---

Published in Transactions on Machine Learning Research (12/2025)
several cases, the empirical estimates approached 1/2, illustrating that the derived bound is indeed tight in practice.
Conclusion
In this work, we derived that the softmax function admits Lipschitz constant of 1/2 uniformly across all lp norms. This result clarifics a commonly adopted assumption in the machine learning literature, where the Lipschitz constant of the softmax operator has often been considered as with respect to the l norm. We further prove the tightness our derived bound and demonstrate its utility in strengthening existing theoretical analyses. We also estimate the empirical Lipschitz constants for softmax transformations applied attention score matrices from transformer- basedd architectures such as ViT, GPT-2, and Qwen3, classification logits from ResNet-50, and stochastic policies in reinforcement learning agents. Tn all cases, the empirical estimates remain strictly below the theoretical bound of 1/2, thereby guaranteeing the tightness and generality of our results.
Broader Impact Statement
This work primarily theoretical. By itself, this result does not introduce new datasets, architectures, or applications, but it provides sharper analytical tools for reasoning about the stability and robustness of existing models. On the positive side, such tools can help practitioners design systems with better controlled sensitivity to perturbations, complementing existing Lipschitz- based approaches used to improve robustness and training stability in deop networks (Cisse et al.) 2017) Miyyto et al. 2018). At the same time, any advance that makes robust and stable deployment easier can be dual-use - it may also strengthen models used societally sensitive settings, including surveillance and biometric monitoring (Almeida al. 2022). A further risk is overstatement or misinterpretation guarantees for a single component can be misconstrued end-to-end assurances of safety, fairness, reliability, which could encourage premature use in high-stakes
contexts Al 2023
References
NIST AI. Artificial intelligence risk management framework (ai rmf 1.0). URL: https://welpubs. mist. gon/nistpubs /aiboss/////tt///// ai, pp. 100 1, 2023.
Wacl Alghamdi, Hsiang Hsu, Hacwon Jeong, Hao Wang, Peter Michalak, Shahab Asoodeh, and Flavio Calmon. Beyond adult and compas: Fair multi-class prediction via information projection. Proc. Advances in Neural Information Processing Systems, 35:38747-3877 38760, 2022.
Denise Almeida, Konstantin Shmarko, and Elizabeth Lomas. The ethics facial recognition technologies, surveillance, and accountability in an age of artificial intelligence: a comparative analysis of us, eu, and uk regulatory frameworks. Al and Ethics, 2(3):377 387, 2022.
Amir R Asadi and Emmanuel Abbe. Chaining meets chain rule: Multilevel entropic regularization and training of neural networks. Journal of Machine Learning Research, (1(13)) 2020.
Jinze Bai, Shuai Bai, Yunfei Chu, Zeyu Cui, Kai Dang, Xiaodong Deng, Yang Fan, Wenbin Ge, Yu Han, Fei Huang, et al. Qwen technical report. arXiv preprint arXi::2309.1660 16609, 2023.
Peter L. Bartlett, Dylan J. Foster, and Matus Telgarsky. Spectrally-normaalized margin bounds for neural networks. Proc. Advances in Neural Information Processing Systems, 30, 2017.
Yonatan Bisk, Rowan Zellers, Jianfeng Gao, and Yejin Choi. Piqa: natural language. Proc. AAAT Conference on Artificial Intelligence (AAAT), 34(05):7432 -439,, 2020.
Reasoning about physical commonsense
Stephen P Boyd and Lieven Vandenberghe. Conver optimization. Cambridge University Press, 2004.
Greg Brockman, Vicki Cheung, Ludwig Pettersson, Jonas Schneider, John Schulman, Jie Tang, and Wojeiech Zaremba. Openai gym. arXiv preprint 1rXim:1606.011 014.00 2016.
11

---

Published in Transactions on Machine Learning Research (12/2025)
Shicong Cen, Yuting Wei, and Yuefie Chi. Fast policy extragradient methods for competitive games with entropy regularization. Proc. Advances in Neural Information Processing Systems,
2021.
Lin Chen, Huaian Chen, Zhixiang Wei, Xin Jin, Xiao Tan, Yi Jin, and Enhong Chen. Reusing the task- specific classifier discriminator: Discriminator- free adversarial domain adaptation. Proc. IEEE/CVF Conference on Computer Vision and Pattern rRecognition, pp. 7181-7190, 2022.
Moustapha Cisse, Piotr Bojanowski, Edouard Grave, Yann Dauphin, and Nicolas Usunier. Parseval networks: Improving robustness to adversarial examples. Proce. International Conference on Machine Learning, 70: 854 $54-863, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Un- terthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, al. An image is worth 16x16 words: Transformers for image recognition at scale. Proc. International Conference on Learning Representations, 2021.
Mahyar Fazlyab, Manfred Morari, and George J. Pappas. Efficient and accurate estimation of lipschitz constants for deep neural networks. Proc. Advances in Newral Information Processing Systems, 32, 2019.
Bolin Gao and Lacra Pavel. On the properties of the softmax function with application in game theory and reinforcement learning. arXiv preprinl ar Xiv: Stt:1704.00800 2017.
Gene H. Golub and Charles F Van Loan. Matrix computations. JHU press, 2013.
lan Goodfellow, Yoshua Bengio, Aaron Courville, and Yoshua Bengio. Deep learning, volume 1. MIT Press, 2016.
Henry Gouk, Eibe Frank, Bernhard Pfahringer, and Michael J Cree. Regularisation of neural networks by enforcing lipschitz continuity. Machine Learning, 110(2):393-414 2021.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. Proc. IEEE Conference on Compuler Vision and Pattern Recognition, pp. 770 778, 2016.
Tuomas Hytönen, Jan Van Neerven, Mark Veraar, and Lutz Weis. Analysis in Bonach spaces, volume 12. Springer, 2016.
Hong Jun Jeon, Jason D. Lee, Qi Lei, and Benjamin Van Roy. An information-theorriicc analysis of in-context learning. Proc. International Conference on Machine Learning, 2024.
Salman Khan, Muzammal Naseer, Munawar Hayat, Syed Waqas Zamir, Fahad Shahbaz Khan, and Mubarak Shah. Transformers in vision: A survey. ACM compuling surveys (CSUR), 54(10s):1-41, 2022.
Hyunjik Kim, George Papamakarios, and Andriy Mnih. The lipschitz constant of sell-attention. Proc. International Conference on Machine Learning, 139:5562-557 2021.
Anirban Laha, Saneem Ahmed Chemmengath,, Priyanka Agrawal, Mitesh Khapra, Karthik Sankara- narayanan, and Harish G Ramaswany. On controllable sparse alternatives to softmax. Proc. Advances in Newral Information Processing Systems, 31, 2018.
Rémi Le Priol, Reza Babanezhad, Yoshua Bengio, and Simon Tacoste-Julien. An analysis of the adaptation speed of causal models. Proc. International Conference on Artificial Intelligence and Statistics, pp. 775- 783, 2021.
Bowen Lei, Dongkuan Xu, Ruqi Zhang, and Bani Mallick. Embracing unknown step by step: Towards reliable sparse training in real world. Transactions on Machine Learning Research, 2024. ISSN 2835-8856.
Wenzhe Li, Hao Luo, Zichuan Lin, Chongjic Zhang, Zongqing Lu, and Deheng Yc. A survey transformers in reinforcement learning. Transactions Machine Learning Research, 2023. ISSN 2835-8856.
Richard D. McKelvey and Thomas R. Palfrey. Quantal response equilibria for normal form games. Games and Economic Behavior, 10(1):6-38 38, 1995.
12

---

Published in Transactions on Machine Learning Research (12/2025)
Takeru Miyato, Toshiki Kataoka, Masanori Koyama, and Yuichi Yoshida. Spectral normalization for gener- ative adversarial networks. Proc. International Conference Learning Represenlations, 2018.
Ofir Nachum, Mohammad Norouzi, Kelvin Xu, and Dale Schuurmans. Bridging the gap between value and policy based reinforcement learning. Proc. Advances in neural information processing systems, 30, 2017.
Thomas Nagler. Statistical foundations of prior-data fitted networks. Proc. International Conference Machine Learning, pp. 25660 25676, 2023.
Yu Nesterov. Smooth minimization of non-smooth functions. Mathematical programming, 103(1):127 152, 2005.
Laker Newhouse. Softmax 4-lipschitz (in norm that may not matter). https://www.aakerreeenha http///tt///ttttt/o/ottttttttat/
pdff 2025. Unpublished note.
Xianbiao Qi, Jianan Wang, Yihao Chen, Yukai Shi, and Lei Zhang. Lipsformer: Introducing lipschitz continuity to vision transformers. Proc. International Conference on Learning Representations, 2023.
Alec Radford, Jeffrey Wu, Rewon Child, David Luan, Dario Amodei, Πly Sutskever, et al. Language models are unsupervised multitask learners. OpenAl Blog, 1(8):9, 2019.
Marcel Riesz. Sur les maxima des formes bilinéaires sur les fonctionnelles linéaires. Acta Mathemalicn, (3):465 497, 1927.
Tyrrell Rockafellar and Roger J-B Wets. Variational Analysis. Springer, 1998.
John Schulman, Filip Wolski, Prafulla Dhariwal, Alec Radford, and Oleg Klimov. Proximal policy optimiza- tion algorithms. arXiv preprint arXiv:1707 06347, 2017.
Elias M Stein. Interpolation of linear operators. Transactions of the American Mathematical Society, 83(2): 482-492, 1956.
Richard Sutton and Andrew Barto. Reinforcement Learning: An Introduction. MIT PRess, 2018.
GO Thorin. An extension of a convexity theorem duc to m. riesz, kungl. fysiografiska sällskapets i lund förbandlingar, no. 8, 14 (1938), 166 170; medd. Lunds Univ. Mat. Sem, 4:1 5, 1939.
Lewis Tunstall, Leandro Von Werra, and Thomas Wolf. Natural language processing with transformers. O'Reilly Media, Inc.', 2022.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, Lukasz Kaiser, and Illia Polosukhin. Attention is all you need. Proc. Advances in neural information processing systems, 30, 2017.
Aladin Virmaux and Kevin Scaman. Lipschitz regularity of deep neural networks: Analysis and efficient estimation. Proc. Advances in Newral Information Processing Systems, 31, 2018.
Xisojun Xu, Linyi Li, Yu Cheng, Subhabrata Mukherjee, Ahmed Hassan Awadallah, and Bo Li. Certifiably robust transformers with 1-lipschitz self-attention. 2022.
Nikolay Yudin, Alexander Gaponov, Sergei Kudriashov, and Maxim Rakhuba. Pay attention to attention distribution: A new local lipschitz bound for transformers. arXiv preprint XrXim:2507.07814 2025.
Rowan Zellers, Ari Holtzman, Yonatan Bisk, Ali Farhadi, and Yejin Choi. Hellaswag: Can machine really finish your sentence? Proc. Annual Meeling of the Association for Compulational Linguistics, pp. 4791-4800, 2019.
13

---

Published in Transactions on Machine Learning Research (12/2025)
A Appendix
A.1 Proof of Proposition 1
(a) and (b) follow from standard results matrix analysis GGolb & Van Loan 2013
Proof of Proposition (c). Let A E R For p and p oo, the identity trivially satisfics, and hence fix 1 < < co. For any R",
A p
(Ax);(² i2,
where (Aæ); denotes jth element in Ax. Then, for each i, the triangle inequality gives
(Aw):
Σ
Aijor
Σ As |zy
By Holder's inequality, we obtain for each i,
1-1/P
Any 1 2/p
Agl bzyl S
Avgl)
Thus, we can bound ||Ax| as follows,
p-1
Ax
AAil
Aisl |ar
i=1
J=1
1=1
max Isin
A4y))
Asg |z,
im 1
A
p-1
|
|
1=1
A
Σ
A
Σ
bry P
j=1
p-1 |
-
0c
The second and fourth inequalities are obtained by taking the max term out of the summation i.e,
Asg
max 12n
Asgl for all i,
Asyl < max I<rEn
Asyl for all
(8)
The third and fifth equalities follow from (a) and (b). Taking the p-th root and maximizing over ||>112 1, get the inequality
| I S |A 1-1/p ||
1/p
This inequality will be used in the proof of Proposition well.
14

---

Published in Transactions on Machine Learning Research (12/2025)
A.2 Proof of Theorem 1
Proof of Theorem a)). Note that Joy (z) is symmelric for all x E RR.. Hence, from Proposition 1 aa) and (b), we get that for all a E R",
2 [Jo, (x)]ls
Jo, (z)ll1 |Jo, (x)
max 1s
From Lemma 2
the above oplimization problem becomes,
max s; - 8;) + Σ Isi5
8:8j
jz
max 8;(1 8:) 5 Σ
818j
1St
max 28 8;), 15
where s = (xx) (81, 8y)..., The optimal value for the above problem is upper bounded by 1/2, and 1/2 is attained for all E Ik such that the corresponding output σ (x) belongs to A and 8; 1/2 for any i nn... Thus, we get the following bounds,
1 (x))l S and || (2) Moo S
By Proposition [c), we get
Jo, (x)llp s
11 2 (2) = for all S p 0c.
Proof of Theorem [bb. From Theorem Iaa, get,
[x) s 1 for all 1 13 co.
sup zER"
From Lemma b)), for any À 0, we have,
X 2
sup |Jox (z)llp
for all < < co.
=E
and hence using Lipschitz characterization in Lemma
[|xx(x) - ax(x)) s A || ullp for all æ. y Rd and for all 1 S p S oc.
A.3 Proof of Proposition -
Define (ss) E Baxn as diag(s) "ss From Lemma 3, for all 1 Sspsa x
sup mEg
[Jo, (x)]lp = sup |||ll seAg
Proof of Proposition 2 As discussed in Theorem [ aa) for 1 and p xo, for all A
[M = [s)loe
max 2s,(1 si) S 1/2
15:55
15

---

Published in Transactions on Machine Learning Research (12/2025)
and 1/2 is attained if there exists A such that s; 1/2 for some and 8j 0 for j シ i.
To prove that 1/2 is indeed attained, we need to show an example where 1/2 is attained at a point s in the interior of the probability simplex A.
Let a E R such that 2; = In(n - 1) and 2ij = 0 for all j i, a = o2(2) is such that 8, = 1/2 and 8y > for
all j # and hence E A. Thus,
max *ER"
Jo (2))l1
max
|Jo (2)) = =1/2
a #ER"
Proof of Proposition a(b). We first prove for the case n > 2. suffices to show that for arbitrary < p < oc, (a)lp 1/2 for any 8 Ag and there exists 8 E ÖA, such that [M = 1/2 for n > 2 to prove that 1/2 is indeed approached in limit. We need the following definition of the support of vector and the corresponding lemma to prove this.
Definition A.1 (Support of vector). For vector R", the support of is defined as
supp(s) = EiE {I....,,) n) st + 0}.
Lemma 4 (Support condition for attaining 1/2). Let ((s) E gu be defined by
((s) = Diag(s) - -88
for 8 An. Suppose 1 < < 0o and > 2. Then, [M 2 if and only if is a permalation of (1/2.1/2.0......))
The proof of the lemma is given towards the end of this section. From Lemma 4 there exists no s E An with supp(s) > with [|M() = 1/2. Hence, there exists no 8 E A; with ( 1/2 since supp(s) = n for all s E A Also, there exists with supp(s) 2 such that [sM[l = 1/2.
Let & E An with supp($) 2 be such that [ 1/2. By definition, å aA, (boundary of the simplex). Hence, we can obtain a sequence of E A; (interior of the simplex) such that 8 - 8 k + co (Rockafellar & Wets 1998). Also since (s) is a continuous function, (ss) + 1/2 as k - 00. For completeness, we show convergence by constructing example sequence in Example 2
Finally, when n 2, the point (1/2, 1/2) lies A3 and attains [s)lp 1/2.
Example 2 (Sequence (8K)E21 C A such that M(s)) + 1/2 as k + oo). Consider & E dAn with & = 82 = 1/2 and all other entries zero. By Lemmal 4 [| 1/2 To approzimate from the interior of the probability simplex, fir c E 10,) and choose (0, 1/2) satisfying 25 8) =c. Define
81 = 82 = 8,
8; = カ- j23.
so that E A Consider the vector
v 1/2,111 0, ..., 0) E RD with ||0||p = 1,
A direct computation shows,
2 / - and [M(s) = 0 for all j $ {1,2}.
[M(s) = [M(s) 21/p 2
Hence, we gel,
[M()) = - 28(1 - 8) = -
Thus for every e E (0,1/2), there exists s A; with 1/2 e ||M() 1/2, proving that 1/2 the tight upper bound.
16

---

Published in Transactions on Machine Learning Research (12/2025)
Proof of Lemma 4 Let be permutation of (1/2,1/2,0,...,
=, 1/2 for some distinct i, E
((s) (1, has values M();; (s) = 1/4, M(s)i = (s) -1/4, and zeros elsewhere.
To show that (s)ll> 1/2, it suffices to construct v E IR" such that |0|| = and [| = 1/2.
Consider a to be a similar permutation of
o.................. ................................ 0) E E..
Le. 2- and vi -2-1/p A direct calculation gives [M 1/2.
Now we need to prove that if (s)lp = 1/2, is permutation of 0........ By Proposition [[). the interpolation inequality gives
| (M))
M ( < [+)]
From Proposition Ra), we know [M)]l ||M(s) - 1/2 Thus if [|M(s 1/2, equality must hold
throughout. This means for all E An such that [|M(l = 1/2, we have |M = [|M(s)] 1/2.
We divide into cases according to the support size of
Case (i): supp(s) 1.
Suppose 8; = for some i, and sj 0 for all * i. Then M(s) 0. Hence
(s)ll) = 0 for all Thus, the value 1/2 cannot be attained for any 8.
Case (ii): supp(s) = 2. As discussed in Theorem a), for p 1 and p co, for all E An.
[ 1 || (l)
max 2s st) S 1/2 1sigh
and 1/2 is attained only there exists s E A. such that 8; = 1/2 for some i. Thus, if supp(s) = 2, then 8 nceds to be permutations of (1/2, 1/2, 0, .,. for [MM(s)]] [ ll = 1/2 and we have already shown
[s)lp = 1/2 for any pormulation of 1/2,, 0).
Thus, if (s)ll 1/2 with supp(s) = 2, then [s)]] [ = 1/2 which needs s to be permutation of (1/2,1/2,0.....)
Case (iii): sapp(s)= > 2. Suppose, & E An with supp(8) > 2 satisfies |M = 1/2. By Proposition [c).
M S | [
Since M = M = 1/2 holds, equality must hold in the interpolation inequality in Proposition Thus, it suffices to rule out equality.
We just need to show that one of the incqualities used in the proof of Proposition [[c is strict. We show that the inequality in Eq. 8 strict i.e. there exists < i S n, such that,
Σ M(s)al <
Σ (M)s)
max ISkEn
Since [M]] = | 1/2, 8,, 1/2 for some 1 1 i supp($) > 2, there exists an index i2 (1, ... and i2 i1, where 8;2 1/2.. As discussed in Theorem for any row index i, Σy=1 [B),, = 28, - -8)). Hence,
and 8, 1/2 for all j * i1. Since
Σ M(8)gl = Σ M())mol= 1/2,
max
ISi5D
1=1
and for index i2,
n
Σ M <1/2= ISkSn
max Σ
(s)asl- 1=1
1=1
This shows that the inequality in Eq. S is strict. Hence, the interpolation inequality in Proposition T I()
is
strict for the matrix (8). This contradicts our assumption that || (8) 1/2.
Combining all cases, we conclude that [s)lpp 1/2 and only if is a permutation of (1/2, 0).
17

---

Published in Transactions on Machine Learning Research (12/2025)
A.4 Proof of Theorem 3
Let the DSFP map be defined as
T(y) 01/-(A 01/((ATo1)
y E Am.
where 1irr denotes the softmax operator with temperature 7. By Theorem the softmax operator is globally Lipschitz with constant 1 of Tipschitz constants, we obtain
27' uniformly across all Ep norms with 1s pp oC. Applying the chain rule
1 | 2T
1 2T
(T(y)) - T(x2)llp
1
lle
|
11 82llp.
Since
|A lpp = ||Al|p: this yields
||
T (322)lp
1 3211p-
472
Hence, the DSFP map has the Lipschitz constant,
|||
47
If 7 ||Al|p/, then c and T' Banach contraction on the complete metric space (Am: ll llp) By the Banach fixed-point theorem, T then admits unique fixed point y* Ams and for any initialization E Am, the iteration
yk+1 + (1 - a) yı + a Tyy),
a (0, 1],
converges linearly in the lp norm to y² At each step, the row player's strategy is updated as
2 =
Since yk + y and the softmax map is continuous, it follows that 2s + x² == =001/((- Ay"). Thus, both players' equilibrium strategies are recovered in the limit.
18


Reviews:
[{"review_id": "zMvZsPvfrN", "reviewer": "Reviewer_5VLW", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper shows that the Lipschitz constant bound for Softmax with parameter $\\lambda$, i.e., $\\sigma_\\lambda$ denoted by the authors, is exactly 1/2, under any $p$-norm induced operator norm. This is an improvement of e.g. Yudin et al.'s work which shows that in $\\ell_2$ norm the constant is upper bounded by 1/2. Then the authors use the refined constant results to strengthen several existing results where Softmax continuity is relevant, covering e.g. zero-sum games and LLM.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper proves the results correctly. I have checked the Appendix.", "audience_yn": "Yes", "audience_explanation": "The paper strengthens several existing results related to machine learning.", "requested_changes": "The paper is mainly technical, so as long as it is technically sound, there is not much need to make changes. So I only have two questions: \n\n1. In Appendix A.4, I believe the proof should be for Theorem 4 instead of Theorem 3. The title for Appendix A.4 may be incorrect. \n2. In Theorem 3, I do not understand what the authors want to say by writing \"$|W^K|_2 + n\\nu \\tau\\varepsilon^{-1/2} |W^Q |_2 + ... $\".", "broader_impact_concerns": "N.A.", "additional_comments": "N.A.", "extra_scores": {}, "extra_text": {}}, {"review_id": "yEgLKTAUDc", "reviewer": "Reviewer_GXEw", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "This paper demonstrates that the Lipschitz bound for the Softmax function is actually $\\frac{\\lambda}{2}$, where $\\lambda$ is the temperature parameter, and that this bound is tight. The authors also run numerical experiments to support their derivations, and include examples to justify the usefulness of the Lipschitz bound.\n\n**Strengths**\nI believe the strengths of the paper are its clarity of presentation and the completeness of the authors’ analysis. Regarding clarity, the paper is well written with almost no grammar/spelling issues, it is nicely organized, and the math is logically and clearly presented. Regarding completeness, the authors provide nice proofs of their claims, a comprehensive numerical analysis, and some nice illustrative examples.\n\n**Weaknesses**\nThe paper is well written and researched and I can only really find one weakness—and it should be easily resolvable. Specifically, the authors gloss over broader impacts. See broader impact concerns section for more details.\n\nBeyond this, I have found a few small errors and have provided a few suggestions in the requested changes below.", "claims_and_evidence_yn": "Yes", "claims_explanation": "As far as I can tell the mathematical statements are all either backed up with references or proofs, and the empirical evidence seems well presented. Note that I have verified the correctness of the appendices (except the construction of example convergent sequences on page 15, which was not required for proving the paper's results), but I have not reviewed any experimental code.", "audience_yn": "Yes", "audience_explanation": "Given the relevance of the softmax function to modern ML approaches the reviewer believes that this is relevant material for the TMLR community.", "requested_changes": "I list the requested changes below. I have added “**C**” at the end of changes that I view as critical, for my review, and “**S**” after ones that I view more as suggestions.\n\n- [line 2 intro]: “logits is normalized” => “logits are normalized” **C**\n- Proposition 1 (c): what is $t^0$? **C**\n- [Example 2, page 5]: this is labelled “Example 2” but I’m pretty sure it’s the first example to appear? **C**\n- [end of page 6]: “Such refinements are vital, as they directly impact theoretical robustness guarantees for attention-based architectures.” I wonder if this is a slight overstatement? Does removing a factor of $2$ here really provide any significant change in how people understand attention architectures? **S**\n- [Fig 2]: minor issue, but it would be ideal to move figure 2 into the section for vision models, if it is possible to rearrange the paper layout in that way while not introducing wasted space. If not, then no problem **S**\n- [A.2 4th line of text]: shouldn’t $x$ be in $\\mathbb{R}^n$?\n- [A.2 4th line of text]: should $\\Delta_n^0$ be changed to $\\Delta_n^{\\circ}$ to keep notation consistent? **C**\n- [appendix]: could be nice to add something at the end of proofs to better separate them. E.g., between proof of Thm 1 parts (a) and (b). **S**\n- [proof of proposition 2(b) line 1]: perhaps change “It suffices to show that for any $1 < p < \\infty, \\dots$” to “It suffices to show that for arbitrary $1 < p < \\infty, \\dots$”. **S**\n- [Lemma 4 Eq. 4]: should $j = \\{1,2\\}$ be instead $j \\not\\in \\{1, 2\\}$? **C**\n- [end of page 15]: “proof of Lemma” => “proof of Lemma 4”.\n- [second last text line of page 15]: “It” should be “it”\n- [3rd last text line of page 15]: I think $M(s)_{ji}$ should be equal to $-\\frac{1}{4}$.\n- [Proof of Lemma, case (ii), text line 2, page 16]: suggest change “attained if” => “attained only if”. **S**\n- [Last equation page 16]: redundancy: the exact same equation is repeated twice (I assume you meant to switch $i_2$ with $i_3$ in the second equation. Also, if I am not mistaken you don’t need to mention $i_3$ at all but can just say that $s_{i_2} < ½$, right? Might be useful to do this for conciseness **S**.\n- Lastly, the reviewer is aware of a paper that shows that softmax self-attention is not Lipschitz (Hyunjik Kim, George Papamakarios, Andriy Mnih Proceedings of the 38th International Conference on Machine Learning, PMLR 139:5562-5571, 2021). Given how self-attention is one of the big modern applications of the softmax function for machine learning, it would be great to say a few words highlighting how the softmax can be Lipschitz while self attention is not.", "broader_impact_concerns": "The paper argues for the impactfulness of the provided characterization of the softmax Lipschits constant—e.g. with respect to machine learning and game theory. Thus, the statement “There are no possible repercussions of this work. There is no potential negative impact that a user of this research should be aware of” seems somewhat contradictory. If this work is indeed relevant to machine learning and game theory communities then it would have both positive and negative societal impacts. It would be nice to mention the potential for at least some such impacts. For example, on the machine learning/AI side, the reviewer is aware of quite a bit of literature on the topic so this should not be too difficult to do. For easily digestible overviews of AI impacts one could look at the Atlas of AI by Kate Crawford or The AI Con by Hanna and Bender. For academic articles one could start with works by the same authors.", "additional_comments": "", "extra_scores": {}, "extra_text": {}}, {"review_id": "FlzKnorUx7", "reviewer": "Reviewer_1Ccc", "summary": "", "questions": "", "limitations": "", "rating": null, "confidence": null, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "Yes", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "paper_topic_and_main_contributions": "", "excitement": null, "reproducibility": null, "ethical_concerns": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "summary_of_contributions": "The paper derives a tight upper bound of the Lipschitz constant of the softmax operator. Various validations are conducted across model architectures (e.g., Qwen3, ViT) and tasks (e.g., Vision, RL, Text generation). However, I am not familiar with the theorem, and I do not verify the correctness of the theorem in the paper.", "claims_and_evidence_yn": "Yes", "claims_explanation": "The paper validates the $1/2$ Lipschitz constant of the softmax operator by the approximation of Lipschitz constant of the softmax operator across various model architectures and tasks.", "audience_yn": "Yes", "audience_explanation": "Lipschitz constant of the **complete neural network** is vital in the community, because it reflects the lower bound of the eigenvalues of the Hessian Matrix of the neural network and plays an important role in the analysis of the convergence in optimization. However, the Lipschitz constant of the softmax operator is less important than that of the complete neural network.", "requested_changes": "My concerns focus on the potential application and inspiration of the proposed theorem.\n\n1. About the application, as I pointed, the tight upper bound of the Lipschitz constant of the softmax operator cannot totally reflect the properties of the neural network in the optimization, because what we indeed need is the Lipschitz constant of the **complete neural network**, rather than the Lipschitz constant of a single operator. More discussions on the relation between the Lipschitz constant of the softmax and the Lipschitz constant of the neural network are expected. \n\n2. Softmax operator usually attaches to the top/deep of the neural network, except for transformers. Can any normalization/regularization techniques be inspired by the theorem, e.g., Spectral Normalization in GANs? More discussions are expected. As for transformers, can the application of softmax operator in Multi Head Attention make the training of transformers faster and more stable?  \n\n3. The novelty and correctness of the theorem need to be verified by other reviewers and the editor.", "broader_impact_concerns": "N/A", "additional_comments": "", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Decision: Accept (poster)