What is Sparsemax?. A useful variation of softmax | by Michael Larionov, PhD | Towards Data Science

In machine learning, there are several very useful functions, for example, sigmoid, relu, softmax. The latter is widely used in multi-class classification problems as an output layer of the Neural networks:
This function has a useful property: the sum of its elements is one, which makes it very useful to model probabilities. It is also differentiable everywhere and the derivative is never zero, which make it useful in the backpropagation algorithms. in contrast, the derivative of the argmax function, that softmax is called to replace, is always zero. Another useful property is that this function preserves the support of the posterior probability distribution because the output probabilities are never zero, however small values they may be.
But sometimes you want to have a sparse output, only preserving several values with non-zero probability. For this scenario, André F. T. Martins and Ramón F. Astudillo proposed a new function called sparsemax in their paper From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification.
From Softmax to Sparsemax:

A Sparse Model of Attention and Multi-Label Classification

Andre F. T. Martins  ́

†] ANDRE.MARTINS@UNBABEL.COM

Ramon F. Astudillo  ́

† RAMON@UNBABEL.COM

†Unbabel Lda, Rua Visconde de Santarem, 67-B, 1000-286 Lisboa, Portugal  ́

]

Instituto de Telecomunicac ̧oes (IT), Instituto Superior T  ̃ ecnico, Av. Rovisco Pais, 1, 1049-001 Lisboa, Portugal  ́



Instituto de Engenharia de Sistemas e Computadores (INESC-ID), Rua Alves Redol, 9, 1000-029 Lisboa, Portugal

Abstract

We propose sparsemax, a new activation func-
tion similar to the traditional softmax, but able

to output sparse probabilities. After deriving

its properties, we show how its Jacobian can be

efficiently computed, enabling its use in a net-
work trained with backpropagation. Then, we

propose a new smooth and convex loss function

which is the sparsemax analogue of the logis-
tic loss. We reveal an unexpected connection

between this new loss and the Huber classifi-
cation loss. We obtain promising empirical re-
sults in multi-label classification problems and in

attention-based neural networks for natural lan-
guage inference. For the latter, we achieve a sim-
ilar performance as the traditional softmax, but

with a selective, more compact, attention focus.

1. Introduction

The softmax transformation is a key component of several

statistical learning models, encompassing multinomial lo-
gistic regression (McCullagh & Nelder, 1989), action se-
lection in reinforcement learning (Sutton & Barto, 1998),

and neural networks for multi-class classification (Bridle,

1990; Goodfellow et al., 2016). Recently, it has also been

used to design attention mechanisms in neural networks,

with important achievements in machine translation (Bah-
danau et al., 2015), image caption generation (Xu et al.,

2015), speech recognition (Chorowski et al., 2015), natural

language understanding (Hermann et al., 2015; Sukhbaatar

et al., 2015; Rocktaschel et al.  ̈ , 2015), and computation

learning (Graves et al., 2014; Grefenstette et al., 2015).

There are a number of reasons why the softmax transfor-
Proceedings of the 33 rd International Conference on Machine

Learning, New York, NY, USA, 2016. JMLR: W&CP volume

48. Copyright 2016 by the author(s).

mation is so appealing: it is simple to evaluate and differ-
entiate, and it can be turned into a (concave) log-likelihood

function by taking the logarithm of its output. Alternatives

proposed in the literature, such as the Bradley-Terry model

(Bradley & Terry, 1952; Zadrozny, 2001), the multinomial

probit (Albert & Chib, 1993), the spherical softmax (Ol-
livier, 2013; de Brebisson & Vincent  ́ , 2015), or softmax

approximations (Bouchard, 2007), while advantageous in

certain scenarios, lack some of these convenient properties.

In this paper, we propose the sparsemax transformation.

Sparsemax has the distinctive feature that it can return

sparse posterior distributions, assigning zero probability to

some output variables. This property makes it appealing

for filtering large output spaces, predicting multiple labels,

or as a component to identify which of a group of variables

are relevant for a decision, making the model more inter-
pretable. Crucially, this is done while preserving most of

the attractive properties of softmax: we show that sparse-
max is also simple to evaluate, it is even cheaper to differ-
entiate, and that it can be turned into a convex loss function.

To sum up, our contributions are as follows:

• We formalize the new sparsemax transformation, derive

its properties, and show how it can be efficiently com-
puted (§2.1–2.3). We show that in the binary case sparse-
max reduces to a hard sigmoid (§2.4).

• We derive the Jacobian of sparsemax, comparing it to the

softmax case, and show that it can lead to faster gradient

backpropagation (§2.5).

• We propose the sparsemax loss, a new loss function that

is the sparsemax analogue of logistic regression (§3). We

show that it is convex, everywhere differentiable, and

can be regarded as a multi-class generalization of the Hu-
ber classification loss (Huber, 1964; Zhang, 2004).

• We apply the sparsemax loss to train multi-label linear

classifiers (which predict a set of labels instead of a sin-
gle label) on benchmark datasets (§4.1–4.2).

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

• Finally, we devise a neural selective attention mecha-
nism using the sparsemax transformation, applying to a

natural language inference problem (§4.3).

2. The Sparsemax Transformation

2.1. Definition

Let ∆K−1

:= {p ∈ R

K | 1

>p = 1, p ≥ 0} be the

(K − 1)-dimensional simplex. We are interested in maps

from R

K to ∆K−1

. Such maps are useful to convert a vec-
tor of real weights (e.g., label scores) to a probability distri-
bution (e.g. posterior probabilities of labels). The classical

example is the softmax, defined componentwise as:

softmaxi(z) = exp(zi)

P

j

exp(zj )

. (1)

A limitation of the softmax transformation is that the re-
sulting probability distribution always has full support, i.e.,

softmaxi(z) 6= 0 for every z and i. This is a disadvan-
tage in applications where a sparse probability distribution

is desired, in which case it is common to define a threshold

below which small probability values are truncated to zero.

In this paper, we propose as an alternative the following

transformation, which we call sparsemax:

sparsemax(z) := argmin

p∈∆K−1

kp − zk

2

. (2)

In words, sparsemax returns the Euclidean projection of the

input vector z onto the probability simplex. This projection

is likely to hit the boundary of the simplex, in which case

sparsemax(z) becomes sparse. We will see that sparsemax

retains most of the important properties of softmax, having

in addition the ability of producing sparse distributions.

2.2. Closed-Form Solution

Many algorithms have been proposed to project onto the

simplex (Michelot, 1986; Pardalos & Kovoor, 1990; Duchi

et al., 2008). We start by recalling the result that such pro-
jections correspond to a soft-thresholding operation. Be-
low, we denote [K] := {1, . . . , K} and [t]+ := max{0, t}.

Proposition 1 The solution of Eq. 2 is of the form:

sparsemaxi

(z) = [zi − τ (z)]+, (3)

where τ : R

K → R is the (unique) function that satis-
fies P

j

[zj − τ (z)]+ = 1 for every z. Furthermore, τ

can be expressed as follows. Let z(1) ≥ z(2) ≥ . . . ≥

z(K) be the sorted coordinates of z, and define k(z) :=

max n

k ∈ [K] | 1 + kz(k) >

P

j≤k

z(j)

o

. Then,

τ (z) =

P

j≤k(z)

z(j)



− 1

k(z)

=

P

j∈S(z)

zj



− 1

|S(z)|

, (4)

Algorithm 1 Sparsemax Evaluation

Input: z

Sort z as z(1) ≥ . . . ≥ z(K)

Find k(z) := max n

k ∈ [K] | 1 + kz(k) >

P

j≤k

z(j)

o

Define τ (z) = (

P

j≤k(z)

z(j))−1

k(z)

Output: p s.t. pi = [zi − τ (z)]+.

where S(z) := {j ∈ [K] | sparsemaxj

(z) > 0} is the

support of sparsemax(z).

Proof: See App. A.1 in the supplemental material.

In essence, Prop. 1 states that all we need for evaluating

the sparsemax is to compute the threshold τ (z); all coordi-
nates above this threshold (the ones in the set S(z)) will be

shifted by this amount, and the others will be truncated to

zero. We call τ in Eq. 4 the threshold function: this piece-
wise linear function will play an important role in the se-
quel. Alg. 1 illustrates a na ̈ıve O(K log K) algorithm that

uses Prop. 1 for evaluating the sparsemax. More elaborate

O(K) algorithms exist based on partitioning and median-
finding (Blum et al., 1973; Pardalos & Kovoor, 1990).1

2.3. Basic Properties

We now highlight some properties that are common to soft-
max and sparsemax. Let z(1) := maxk zk, and denote by

A(z) := {k ∈ [K] | zk = z(1)} the set of maximal compo-
nents of z. We define the indicator vector 1A(z)

, whose kth

component is 1 if k ∈ A(z), and 0 otherwise. We further

denote by γ(z) := z(1) − maxk /∈A(z) zk the gap between

the maximal components of z and the second largest. We

let 0 and 1 be vectors of zeros and ones, respectively.

Proposition 2 The following properties hold for ρ ∈

{softmax,sparsemax}.

1. ρ(0) = 1/K and lim→0+ ρ(

−1z) = 1A(z)/|A(z)|

(uniform distribution, and distribution peaked on the

maximal components of z, respectively). For sparsemax,

the last equality holds for any  ≤ γ(z) · |A(z)|.

2. ρ(z) = ρ(z + c1), for any c ∈ R (i.e., ρ is invariant to

adding the same constant to each coordinate).

3. ρ(Pz) = Pρ(z) for any permutation matrix P (i.e., ρ

commutes with permutations).

4. If zi ≤ zj , then 0 ≤ ρj (z) − ρi(z) ≤ η(zj − zi), where

η =

1

2

for softmax, and η = 1 for sparsemax.

Proof: See App. A.2 in the supplemental material.

1

In practice, there are observed linear time algorithms which

are often a better choice. See Condat (2014, Table 1), who provide

an empirical comparison among several of these algorithms.

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

− 3 − 2 − 1 0 1 2 3

t

0.0

0.2

0.4

0.6

0.8

1.0 softmax1 ([t,0])

sparsemax1 ([t,0])

Figure 1. Comparison of softmax and sparsemax in 2D (left) and 3D (two righmost plots).

Interpreting  as a “temperature parameter,” the first clause

of Prop. 2 shows that the sparsemax has the same “zero-
temperature limit” behaviour as the softmax, but without

the need of making the temperature arbitrarily small. We

will use this fact in the experiments in §4.2 to control the

degree of sparsity achieved with the sparsemax activation.

Prop. 2 is reassuring, since it shows that the sparsemax

transformation, despite being defined very differently from

the softmax, has a similar behaviour and preserves the same

invariances. Note that some of these properties are not sat-
isfied by other proposed replacements of the softmax: for

example, the spherical softmax (Ollivier, 2013), defined as

ρi(z) := z

2

i

/

P

j

z

2

j

, does not satisfy properties 2 and 4.

2.4. Two and Three-Dimensional Cases

For the two-class case, it is well known that the softmax

activation becomes the logistic (sigmoid) function. More

precisely, if z = (t, 0), then softmax1(z) = σ(t) :=

(1 + exp(−t))−1

. We next show that the analogous in

sparsemax is the “hard” version of the sigmoid. In fact,

using Prop. 1, Eq. 4, we have that, for z = (t, 0),

τ (z) =







t − 1, if t > 1

(t − 1)/2, if −1 ≤ t ≤ 1

−1, if t < −1,

(5)

and therefore

sparsemax1

(z) =







1, if t > 1

(t + 1)/2, if −1 ≤ t ≤ 1

0, if t < −1.

(6)

Fig. 1 provides an illustration for the two and three-
dimensional cases. For the latter, we parameterize z =

(t1, t2, 0) and plot softmax1(z) and sparsemax1

(z) as a

function of t1 and t2. We can see that sparsemax is piece-
wise linear, but asymptotically similar to the softmax.

2.5. Jacobian of Sparsemax

The Jacobian matrix of a transformation ρ, Jρ(z) :=

[∂ρi(z)/∂zj ]i,j , is of key importance to train models with

gradient-based optimization. We next derive the Jacobian

of the sparsemax activation, but before doing so, let us re-
call how the Jacobian of the softmax looks like. We have

∂softmaxi(z)

∂zj

= (δij e

zi P

k

e

zk − e

zi e

zj )/(

P

k

e

zk )

2

= softmaxi(z)(δij − softmaxj (z)), (7)

where δij is the Kronecker delta, which evaluates to 1 if

i = j and 0 otherwise. Letting p = softmax(z), the full

Jacobian can be written in matrix notation as

Jsoftmax(z) = Diag(p) − pp>, (8)

where Diag(p) is a matrix with p in the main diagonal.

Let us now turn to the sparsemax case. The sparsemax is

differentiable everywhere except at splitting points z where

the support set S(z) changes, i.e., where S(z) 6= S(z+d)

for some d and infinitesimal .

2 From Eq. 3, we have that:

∂sparsemaxi

(z)

∂zj

=

(

δij −

∂τ(z)

∂zj

, if zi > τ (z),

0, if zi ≤ τ (z).

(9)

From Eq. 4, the gradient of the threshold function τ is:

∂τ (z)

∂zj

=

 1

|S(z)|

if j ∈ S(z),

0, if j /∈ S(z).

(10)

Note that j ∈ S(z) ⇔ zj > τ (z). Therefore we obtain:

∂sparsemaxi

(z)

∂zj

=



δij −

1

|S(z)|

, if i, j ∈ S(z),

0, otherwise. (11)

Let s be an indicator vector whose ith entry is 1 if i ∈ S(z),

and 0 otherwise. We can write the Jacobian matrix as

Jsparsemax(z) = Diag(s) − ss>/|S(z)|. (12)

It is instructive to compare Eqs. 8 and 12. We may re-
gard the Jacobian of sparsemax as the Laplacian of a graph

whose elements of S(z) are fully connected. To compute

2

For those points, we can take an arbitrary matrix in the set of

generalized Clarke’s Jacobians (Clarke, 1983), the convex hull of

all points of the form limt→∞ Jsparsemax(zt), where zt → z.

From Softmax to Sparsemax: A Sparse Model of Attention and Multi-Label Classification

it, we only need S(z), which can be obtained in O(K) time

with the same algorithm that evaluates the sparsemax.

Often, e.g., in the gradient backpropagation algorithm, it is

not necessary to compute the full Jacobian matrix, but only

the product between the Jacobian and a given vector v. In

the softmax case, from Eq. 8, we have:

Jsoftmax(z)·v = p

(v−v ̄1), with v ̄ := P

j

pjvj , (13)

where

denotes the Hadamard product; this requires a

linear-time computation. For the sparsemax case, we have:

Jsparsemax(z)· v = s

(v − vˆ1), with vˆ :=

P

j∈S(z)

vj

|S(z)|

.

(14)

Interestingly, if sparsemax(z) has already been evaluated

(i.e., in the forward step), then so has S(z), hence the

nonzeros of Jsparsemax(z) · v can be computed in only

O(|S(z)|) time, which can be sublinear. This can be an im-
portant advantage of sparsemax over softmax if K is large.

Computational efficiency. In sum, the computational

needs of softmax and sparsemax are compared as follows:

• At training time, sparsemax can backpropagate gradients

faster due to the sparsity (as stated above).

• At inference time, the softmax forward pass is faster, but

asymptotically both are linear time. In our experiments

(§4), even with the O(K log K) Algorithm 1, similar

runtimes were achieved with softmax and sparsemax.

3. A Loss Function for Sparsemax

Now that we have defined and analyzed the sparsemax

transformation, we will use it to design a new loss function

that resembles the logistic loss, but can yield sparse poste-
rior distributions. Later (in §4.1–4.2), we apply this loss to

label proportion estimation and multi-label classification.

3.1. Logistic Loss

Let D := {(xi

, yi)}

N

i=1 be a dataset, where each xi ∈ R

D

is an input vector and yi ∈ {1, . . . , K} is a target label. We

consider empirical risk minimization problems of the form

minimize

λ

2

kWk

2

F +

1

N

X

N

i=1

L(Wxi + b; yi),

w.r.t. W ∈ R

K×D, b ∈ R

K, (15)

where L is a loss function, W is a matrix of weights, and

b is a bias vector. The loss function associated with the

softmax is the logistic loss (or negative log-likelihood):

Lsoftmax(z; k) = − log softmaxk(z)

= −zk + logX

j

exp(zj ), (16)

where z = Wxi + b, and k = yi

is the “gold” label. The

gradient of this loss is, invoking Eq. 7,

∇zLsoftmax(z; k) = −δk + softmax(z), (17)

where δk denotes the delta distribution on k, [δk]j = 1 if

j = k, and 0 otherwise. This is a well-known result; when

plugged into a gradient-based optimizer, it leads to updates

that move probability mass from the distribution predicted

by the current model (i.e., softmaxk(z)) to the gold label

(via δk). Can we have something similar for sparsemax?

3.2. Sparsemax Loss

A nice aspect of the log-likelihood (Eq. 16) is that adding

up loss terms for several examples, assumed i.i.d, we obtain

the log-probability of the full training data. Unfortunately,

this idea cannot be carried out to sparsemax: now, some

labels may have exactly probability zero, so any model that

assigns zero probability to a gold label would zero out the

probability of the entire training sample. This is of course

highly undesirable. One possible workaround is to define

L



sparsemax(z; k) = − log  + sparsemaxk

(z)

1 + K

, (18)

where  is a small constant, and +sparsemaxk(z)

1+K is a “per-
turbed” sparsemax. However, this loss is non-convex, un-
like the one in Eq. 16.

Another possibility, which we explore here, is to construct

an alternative loss function whose gradient resembles the

one in Eq. 17. Note that the gradient is particularly im-
portant, since it is directly involved in the model updates

for typical optimization algorithms. Formally, we want

Lsparsemax to be a differentiable function such that

∇zLsparsemax(z; k) = −δk + sparsemax(z). (19)

We show below that this property is fulfilled by the follow-
ing function, henceforth called the sparsemax loss:

Lsparsemax(z; k) = −zk+

1

2

X

j∈S(z)

(z

2

j −τ

2

(z))+1

2

, (20)

where τ

2

is the square of the threshold function in Eq. 4.

This loss, which has never been considered in the literature

to the best of our knowledge, has a number of interesting

properties, stated in the next proposition.

Proposition 3 The following holds:

1. Lsparsemax is differentiable everywhere, and its gradient

is given by the expression in Eq. 19.

2. Lsparsemax is convex.

3. Lsparsemax(z + c1; k) = Lsparsemax(z; k), ∀c ∈ R.

The idea is to set the probabilities of the smallest values of z to zero and keep only probabilities of the highest values of z, but still keep the function differentiable to ensure successful application of backpropagation algorithm. This function is defined as:
Here τ(z) is called threshold function and it defines the support function S(z) that contains all non-zero indices of p. A python implementation of sparsemax function is below:
Running it and softmax on the same values we can indeed see that it does set some of the probabilities to zero, where softmax keeps them non-zero:
np.around(sparsemax([0.1,  1.1, 0.2, 0.3]), decimals=3)array([0. , 0.9, 0. , 0.1])np.around(softmax([0.1,  1.1, 0.2, 0.3]), decimals=3)array([0.165, 0.45 , 0.183, 0.202])
It is also interesting to see how the two functions are different in the two-dimensional case. In this case softmax becomes a sigmoid function, and sparsemax can be represented in this form:
The picture below illustrates how they are different:
Note, that the sparsemax function is not differentiable everywhere (but neither is relu), but where it is is an easy computation:
Here |S(z)| is the number of elements in the support S(z).
The obvious problem with sparsemax is vanishing gradient. You can see that the derivative turns to zero once z becomes large. The authors acknowledged the problem and even proposed a new loss function in place of cross-entropy loss. However I believe the main advantage of sparsemax is not in the output layer, but in the middle of the neural network, for example, in the attention mechanism.
You can find the code for this article in my github repository.