Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models
Authors: Phil Chen
Masha Itkina
Ransalu Senanayake
Mykel Kochenderfer
Venue: NeurIPS 2021 Poster (2021)
Keywords: Deep Learning or Neural Networks
Sparsity and Feature Selection
Variational Inference
(Application) Natural Language and Text Processing

Abstract: Many applications of generative models rely on the marginalization of their high-dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computationally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-likelihood is undefined for sparse probability distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. In this work, we present ev-softmax, a sparse normalization function that preserves the multimodality of probability distributions. We derive its properties, including its gradient in closed-form, and introduce a continuous family of approximations to ev-softmax that have full support and can be trained with probabilistic loss functions such as negative log-likelihood and Kullback-Leibler divergence. We evaluate our method on a variety of generative models, including variational autoencoders and auto-regressive architectures. Our method outperforms existing dense and sparse normalization techniques in distributional accuracy. We demonstrate that ev-softmax successfully reduces the dimensionality of probability distributions while maintaining multimodality.

Evidential Softmax for Sparse Multimodal Distributions in Deep Generative Models
Phil Chen, Masha Itkina, Ransalu Senanayake, Mykel J. Kochenderfer
Stanford University (philhe, mitkina, ransalu, mykel)@stanford. edu
Abstract
Many applications of generative models rely on the marginalization of their high- dimensional output probability distributions. Normalization functions that yield sparse probability distributions can make exact marginalization more computation- ally tractable. However, sparse normalization functions usually require alternative loss functions for training since the log-iikelhood is undefined for sparse probabil- ity distributions. Furthermore, many sparse normalization functions often collapse the multimodality of distributions. normalization function that preserves the multimodality of probability distributions. We derive its properties, including its gradient in closed-form, and introduce a continuous family of approximations to ev-sofimax that have full support and can trained with probabilistic loss functions such as negative log-likelihone and Kullback-Leible divergence. We evaluate our method on variety of generative models, including variational autoencoders and auto-regressive architectures. Our method outperforms existing dense and sparse normalization techniques in dis- tributional accuracy. We demonstrate that ev-softmax successfully reduces the dimensionality of probability distributions while maintaining multimodality.
this work, we present ev-softmax, a sparse
Introduction
Learning deep generative models over discrete probability spaces has enabled state-of-the-art perfor- mance across tasks in computer vision (T]. natural language processing (2)(44, and robotics ( The probability distributions of deep generative models often obtained from neural network using the softmax transformation. Since the softmax function has full support, the logarithm of the output probabilities is well-defined, which is important for common probabilistic loss functions such as the negative log likelihood, cross entropy, and Kullback-Leebllr (KI.) divergence.
Several applications highlight the importance of achieving sparsity in high dimensional dense distributions to make downstream tasks computationally feasible and perhaps even interpretable (8]- [1]. In discrete variational autoencoders (VAEs), for instance, calculating the posterior probabilities exactly requires marginalization over all possible latent variable assignments, which is intractable for large latent spaces [9. [10]. Stochastic approaches to circumvent exact marginalization include the score function estimator 12). [13), which suffers from high variance, and continuous relaxations of the discrete latent variable, such as Gumbel-Softmat (14), [15], which introduce bias.
Alternatively, high-dimensional latent spaces can tractably marginalized using normalization functions that produce sparse probability distributions. Several sparse alternatives to softmax have been proposed, including sparsemax [8], a-entmax (16), and sparsehourglass [17]. However, such approaches often collapse the multimodality of distributions, resulting in unimodal probability distributions [10] (see Fig. []. When marginalizing over discrete latent spaces, maintaining this multi- modality is crucial for applications such image generation, where the latent space is multimodal
in nature [ and machine translation, where multiple words can be valid translations and require
35th Conference on Neural Information Processing Systems (NcurIPS 2021).

---

(1, 0, 0)
Input
Softmax
Sparsemax
Ev-Softmax (Ours)
(0, 1, 0)
(0, 0, 1)
Figure 1:
We evaluate the softmax, sparsemax, and ev-softmax (ours) functions on an example input point v IR' and plot the resulting values on the plane + U2 + U3 = 1. We plot the points = (0.4, 1.4, -0.8), softmax(v) (0.25, 0.67, 0.07). sparsemax(v) = (0, 1. 0). and er-softmax(v) (0.27, 0.73, 0). Sparsemax yields unimodal distribution, while ev-softmax yields himodal distribution which is close that of softmax.
context distinguish between the optimal word choice 9 Furthermore, these approaches either require the construction of alternative loss functions to the negative log- log-likelihood, which is undefined
for cero-valued output probabilities (8). [16] 18] or they are applied as a post-hoc transformation at test time [10]. For the latter case, the latent distributions obtained through these existing sparse normalization functions and their resulting posteriors are not trained directly.
The post-hoc sparsification procedure introduced by Itkina er al. [10 provides a method for obtaining sparse yet multimodal distributions at test time. We generalize this method to sparse normalization function termed evidential softmax (ev-softmax) which can be applied during both training and test time. We define continuous approximations of this function that have full support and are thus compatible with typical loss terms, such as negative log- log-likelihood and KL divergence. We use this full-support approximation of ev-softmax directly optimize discrete generative models for objective functions with probabilistic interpretations. This approach is in contrast with post-hoc sparsification methods that optimize a proxy model (10) and methods that use non- non-probabilistic objective functions, such as sparsemax [8] We evaluate these discrete generative models at test time with the ev-softmax function obtain sparse yet multimodal latent distributions.
Contributions: This paper proposes a strategy for training neural networks with sparse probability distributions that is compatible with negative ligeelkeo and KL-divergence computations. We generalize the evidential sparsification procedure developed by Itkina et al. (10) to the ev-softmax function. We prove properties of ev-sofimax and its continuous approximation, which has full support. Our approach outperforms existing dense and sparse normalization functions in distributional accuracy across image generation and machine translation tasks.
Background
Notation: We denote scalars, vectors, matrices, and sets as a, a, A, and A, respectively. The vectorized Kronecker delta ò; is defined to be 1 at the ith index and 0 at all the other indices. The
K K-dimensional simplex written as A == {E E IR K+1 1"e g 1, 2 0}. The KL divergence of probability distribution from distribution is denoted as (L q).
2.1 Discrete Variational Autoencoders
To model a data generating distribution, we consider data x E Rdd categorical input variable y E y. categorical latent variable z Z, and parameters f E R². In the VAE model, is sampled from prior po (z), and we generate x conditioned on z as po (x z). The conditional variational autoencoder (CVAB) [19] extends this framework by allowing for the generative process be controlled by input y. The latent variable z is thus conditioned on the input y in the prior po y).
The VAE estimates parameters & and an auxiliary distribution (z x) to calculate the evidence
lower bound (ELBO) of the log-likelihode for the observed data [20]:
log pe (x) > (xix)) log Pe (x 2) - (L (z x) || pz))))
The CVAE estimates parameters á of an auxiliary distribution (z x,y) to obtain the ELBO [19
log (x yy) 2 (x)x,) log (x z) KL(go y) pe y)).

---

The subsequent discussion focuses on VAEs, and analogous expressions for CVAEs may be obtained by replacing Po (x)).. qs x). and (z(z) with (x y). 9s(z x,y), and ps y), respectively.
Maximizing the VAE ELBO using gradient-based methods involves computing the term (ol log px z) in Eq. [ which requires marginalization over the auxiliary distribu- tion qa (z) x). Many Monte Carlo approaches estimate this quantity without explicit marginalization over all possible latent classes z E 2. The score function estimator (SFE) is an unbiased estimator which uses the identity (xxE log (x 2) (xxx z) V log go(2 x) to estimate the gradient through samples drawn from go (z x). However, the SFE is prone to high variance, and in practice, sampling-based approaches generally require variance reduction methods such as baselines, control variates, and Rao-Blackwellle Bleekwellllitniioonn techniques (21), Alternatively, a low-variance gradient estimate can be obtained by the repcrameterization trick, which separates the parameters of the distribution from the source of stochasticity [20]. The Gumbel--ofttax and Straight-Throun
reparameterizations rely on a continuous relaxation of the discrete latent space [4). (15). In contrast to these stochastic approaches, our method, like sparsemax (8), a-entmax [16), and sparsehourglass
[77. focuses on obtaining sparse auxiliary distributions gs (z x) to enable exact marginalization over all latent classes E 2 for which the auxiliary distribution has nonzero probability mass.
2.2 Normalization Functions
common normalization function that maps vectors V E K to probability distributions in A is the softmax function:
SOFTMAX(V)2 x e
(3)
One limitation of this function is the resulting probability distribution always has full support, i.e. SOFTMAX(Y); * 0 for all and k. Thus, if the distributions 9s in Eq. (I) and Eq. (2) are outputs of the sofimax function, then calculating the expectations over go exactly requires marginalizing over all one-hot vectors z E RK which can be computationally intractable when is large (9). (10).
Recent interest in sparse alternatives has led to the development of many new sparse normalization functions (8) [TO]
16 18). The sparsemax function projects inputs onto the simplex achieve a sparse distribution 8). The a-entmax family of functions generalizes the softmax and sparsemax
functions through learnable parameter a which tunes the level of sparsity [16). Sparsehourglass is another generalization of sparsemax with controllable level of sparsity (17). Despite the controls, we find that these methods achieve sparsity at the expense of collapsing valid modes.
2.3 Post-Hoc Evidential Sparsification
Of particular interest is a sparse normalization function that maintains multimodality in probability distributions. Balancing sparsity and multimodality has been shown be beneficial for tasks such as image generation [10] or machine translation (16). Itkina et al. (10) propose a post-hoc sparsification method for discrete latent spaces in CVAEs which preserves multimodality in distributions. The method calculates evidential weights as affine transformations of an input feature vector and interprets
these weights as evidence either for a singleton latent class { } or its complement { [z depending on the sign of the weights. This interpretation, with origins in evidential theory (22). [23] allows for
the distinction between conflicting evidence and lack of evidence, which is normally not possible with ordinary multinomial logistic regression. Conflicting evidence can be interpreted as multimodality
whereas lack of evidence can correspond to sparsity [10].
For given dataset and model, the resulting post-hoc sparsification is as follows. A CVAE is trained on a dataset using the softmax normalization and the ELBO in Eq. D At test time, for given query y from the dataset, the learned weights of the last layer of the encoder 3 E [J+1))x and the corresponding output of the last hidden layer Φ(y) R the evidential weights are:
K
wi Buk K
Bue
(4)
K
E=1
for each A E {1,.... K). Evidence in support of latent class 24 then corresponds to w max(O, wx) and evidence against it w max(0, wb)). The resulting sparse dis- tribution is:
p(zx y) x 1m # 0)
(5)
3

---

where the evidential belief mass my is:
mg e e
-1+
II tfk
(6)
for each & {1...., KK... K}. Further information can be found in Appendix A.
3 The Evidential Softmax Transformation
Although the evidential sparsification procedure is successful at providing sparse yet multimodal normalizations of scores, can only be performed post-hoc, without the possibility of being used at training time. Furthermore, it requires O(JK²) computations to calculate all the weights for each input, with hidden features in the last layer and K latent classes.
We propose a transformation, which we call evidential softmax (ev-softmax), on inputs v E R K We claim that this transformation is equivalent that of the post-hoc evidential sparsification procedure with the advantages of having a simple closed form and requiring only O(K) computations.
Let T # Em= be the arithmetic mean of the elements of v. We define our transformation as,
EVSOFTMAX(Y): & [1 2 JJe*.
Note that the post-hoc sparsification procedure in Section 2.3 takes input vectors Φ E IR' and the weights 3 E IR (J of the neural network's s last layer whereas the ev-softmax function takes as input = 3 Φ E R We formalize the equivalence of ev-softmax with the post-hoc sparsification
procedure follows:
Theorem 3.1. For any input vector &o E R', weight matrix B E R define the score vector = 3 do and variables Wk. w my, and p(zz) as in Section 2.3 for all l E (K....,K). the probability distribution of is non-atomic, then EVSOFTMAX())
p for each k.
The proof of this theorem uses the following two lemmas:
Lemma 3.2. Let v = K E U; Then U2 - - = wh for all k ( [l,...,K},
Proof. The result follows from algebraic manipulation of Eq. 4.. Full details are in Appendix B. +
Lemma 3.3. For my as defined in Eq. 66 L(m, 0} 1 Luv > f} for all k {L...., K}.
Proof. Observe that the condition mm = 0} = 1 1 (m, + 0} equivalent to the conjunction of the following two conditions: 1) u2 < 0 and 2) there exists some l o k such that wy 2 0. From Lemma 3.2 these statements are equivalent to U/ S о and there exists some f * 1 such that T or there must exist some & { k such that > B. In both cases, there exists o such that 02 2 U. Thus, we see that U2 S T implies there exists some l + A such that 2 T. We conclude that [mm 0} [u < F and the result follows.
U. Note that if Uą < E, then either
We briefly sketch the proof of Theorem 3.1 and refer the reader to Appendix B for the full derivation. Lemma 3.2 implies that e* x ckk In addition, combining Lemma 3.3 with the non-atomicity of the distribution of v yields that {mm * 0} -1 (wx 2 ea (Lemma B.3). Eq. 7 follows from these two results.
3.1 Properties
Table gives list of softmax alternatives, desirable properties of a normalization function, and which properties are satisfied by each function as inspired by the analysis of Laha et al. (17). Refer to Appendix for proofs these properties. These results show that ev-sofimax satisfies the same properties as the sefimax transformation. While idempotence can be useful, these normalizations are generally only applied once, and in this regime the idempotence does not provide any further guarantees. Likewise, scale invariance is less applicable in practice because the scale of the inputs to

---

Table 1: Summary of properties satisfied by normalization functions. Here. v denotes that the property satisfied. The * denotes that the property is satisfied conditional on either the imput or some parameter that is independent of the input. Other properties satisfied by all considered normalization functions include invariance with respect permutation of the coordinates and existence of the Jacobian. These properties proven in Appendix C.
Property
Softmax
Sparsemax
8
Sparschourglass
Ey-Softmax (Ours)
Idempotence
Monotonic
Translation Invariance
Scale Invariance
Full Domain
Lipschitz
1
1
K4
normalization functions is often of the learned features of data, and the scale generally provides measure of confidence in the predictions.
The Jacobians of softmax and ev-softmax have analogous forms (see Appendix C):
aSoFTMAX(v): SOFTMAX(V); - SOFTMAX(V),)
(8)
Ou,
@EVSOFTMAX(V): EVSOFTMAX(Y)(6G EVSOFTMAX(Y))).
For the Jacobian of the proposed ev-softmax function, there are two cases to consider. First, when either or both of EVSOFTMAX(Y); and EVSOFTMAX(V), are exactly O, we have @EVSOrTMAX(V): Intuitively, this occurs because classes that have a probability of 0 do not contribute to the gradient and also remain under small local changes in the input. Second, when both EVSOFTMAX (v), and EYSOFTMAX(V); are nonzero, the partial derivative is equivalent to that softmax over the classes that have nonzero probabilities. Hence, optimizing models using the ev-softmax function should yield similar behaviors to those using the softmax function.
do;
One notable connection between ev-softmax and sparsemax is that the supports of both functions can be expressed in terms of mean conditions their inputs. For an input v RR the ith class given zero probability by ev-softmax if 0; < K Σk= Ug. Analogously, denoting the size of the support of the ith class given zero probability by sparsemax U; S - Ei= (())) where U(ee) is the kth largest element in sparsification, whereas ev-softmax considers the arithmetic mean before sparsification.
sparsemax on
(8). Thus, sparsemax considers the arithmetic mean after
Figure presents a visualization of an input point in IR3 and the results upon applying softmax, sparsemax, and ev-softmax to this point. Informally, the interior of the simplex is the of trimodal distributions, the edges constitute the set of bimodal distributions, and the vertices are the unimodal distributions. Thus, this input provides example of when sparsemax yields unimodal distribution whereas ev-softmax does not, Note that I is the fowest-dimensional space for which the ev-softmax function is nontrivial. In I², ev-softmax is equivalent the argmax function, which piece-wise constant and has zero gradients everywhere. However, the focus of ev-softmax is on balancing multimodality with sparsity for spaces with higher dimension than IR.
3.2 Training-time Modification
Since ev-sofmmax outputs a sparse probability distribution, the negative log-likellnn likellkkooood and KL diver- gence cannot be directly computed. To address this limitation, modify ev-sofimax during training time as follows:
EvSOFTMAX (V); x (1 2 t) + *
(10)
for some small In practice,
to 10-6
The KI. divergence with the above training modification takes the same form as the e-KI. used for overcoming sparsity in language modeling 24] (25). As approaches 0, the gradient of the negative
5

---

Even
Odd
1.0
Probability
0.6
0.4
0.0
Sohmax
8
Spunsemax
Fntmax-1155 1.5
Post-hoc Evidential
By Softmax (Ous)
Z
Z
Using the proposed normalization function ev-sofimax at training time yields a more accurate distribution on the MNIST dataset than sofimax, sparsemax. enimax- 1.5, and post-hoc evidential baselines. The horizontal axis shows the decoded image for each latent class and the vertical axis shows the probability mass. Note that the latent space for the post-hoc evidential method is the same that of softmax, since the method applied only at test time to the latent space distribution. Unlike ev-softmax, the sparse distributions produced by sparsemax and entmax-1. 1.5 collapse the latent space.
Figure 2:
log likelihood loss term approaches a form that mirrors that of the softmax function:
V. log (V)) 6; SOFTMAX(V)
(11)
lim V. log [EVSOFTMAXmin. (v); 5; EVSOFTMAX(Y), c-0
(12)
for any finite-dimensional, real-valued vector A derivation of Eq. 12 is given in Appendix D.
While the Gumbel-Softmax relaxation also enables gradient computations through discrete spaces, Eq. (2 establishes the convergence and closed- form solution of the gradient as the relaxation goes O, which applies uniquely the ev-softmax function. Note that the limit of the log-likelihood of distribution generated by the training-time time modification (i.e. lim,- log iggEESS EVEOOFMOAA.... (v)))) is undefined, but Eq. 12) shows that the limit of the gradiens of the ligeliidl is defined.
4 Experiments
We demonstrate the applicability of proposed strategies the tasks of image generation and machine translation! We compare our method to the softmax function as well as sparse normalization functions sparsemax and enfmax-1.5, [16]. We also compare with the post-hoc evidential sparsifi- cation procedure 10 Sparsehourglass is not used as it is similar to a-enimax. Experiments indicate that ev-softmax achieves higher distributional accuracy than softmax, sparsemax, entmax-1.5, and post-hoc evidential sparsification by better balancing the objectives of sparsity and multimodality.
4.1 Conditional Variational Autoencoder (CVAE)
To gain intuition for the ev-sofimax function, we consider the multimodal task of generating digit images for even and odd queries (y E (even, odd}) on MNIST. We use a CVAE with |Z| = 10 latent classes. Since the conditional prior distribution given y (even, odd] is uniform over 5 digits, this task benefits from both sparsity and multimodality of a normalization function.
We follow the CVAE architecture and training procedure of Itkina et al. [10) for this task. For each of the four normalization functions we consider (ev-sofmmax, sparsemax, entmax-1. 1.5, and sofimax), we train encoder, which consists of two multi-layer perceptrons (MLPs), and decoder. The first MIP of the encoder receives an input query y and outputs categorical prior distribution (z y) the latent variable generated by the normalization function. The second MLP takes as input a feature vector x in addition to the query y, and outputs the probability distribution for the
Model:
"ooee for the experiments is available

---

Even
Odd
Wasserstein distance
0.15
Softmax
Sparsemax
0.1
Entmax-1
Post-hoc Evidential
Ey-Softmax (Ours)
3,000
6,000 0
3,000 6,000
Iteration
Iteration
Figure 3: Wasserstein distance between the predicted categorical prior and the ground truth prior distributions.
Our method outperforms softmax, sparsemax, and entmax- 1.5 over random seeds. Lower is better.
posterior gu (z y), generated using the same normalization function. The decoder consists of single MLP that takes as imput the latent variable z and generates an output image x. Each model was trained for 20 epochs to minimize the ELBO of Eq. (2). Further details can be found in Appendix E.
Results and discussion: Fig- 2 we show the decoded images of the latent classes trained using each of the normalization functions, well as the corresponding prior distributions over these latent classes for even and odd inputs. Using ev-softmax at training time yields most sensible images for both even and odd inputs. The CVAE trained with ev-softmax learns a distribution with five nonzero probability modes for both even and odd inputs. All but one the digits are represented by the latent classes of the ev-seftmax distribution, and they are generated by the even and odd priors almost perfectly. contrast, the sefimax distribution results in nonzero probability mass assigned to all latent classes for both inputs, producing distributional accuracy that is qualitatively worse than that of ev-softmax. While the post-hoc evidential distribution also has multiple nonzero probability modes for both even and odd imputs, the distribution is less accurate, demonstrating the advantage of softmax in training using the sparse latent distribution. The decoded images of enimax- 1.5 show that enimax-1. 1.5 reduces the posterior latent space to fewer than classes, decreasing the representational power of the generative model. Over 10 random seeds and hyperparameter grid-search, sparsemax single mode.
collapsed both the prior and posterior distributions
To quantify the distributional accuracy each CVAE model, we compute the Wasserstein distance (Appendix E) between the model's 's predicted categorical prior p(z y) and the ground truth prior, shown in Fig. 3 We see that ev-softmax converges the fastest and obtains the lowest distance from the ground truth distribution. Sparsemax and entmar-1.5, converge toward the true prior as they collapse the prior and posterior distributions. The model trained with softmax converges more slowly and to higher distance from the ground truth distribution because lacks the sparsity that ev-softmax offers, resulting in worse distributional accuracy. The post-hoc evidential procedure yields the second lowest Wasserstein distance, but it converges more slowly than ev-softmax.
the other hand, do not allow the model to
4.2 Semi-supervised Variational Autoencoder
To evaluate our method a semi- cemi-supervised setting, we consider the semi-supervised VAE of Kingma al. [26). We evaluate the classification performance of the model on the MNIST dataset, using 10% of the labeled data and treating the remaining data as unlabeled. In this regime, we expect that sparsity remains key ingredient since intuitively associate each digit with one latent class, while multimodality reflects uncertainty in the generative model.
We follow the architecture and training procedure of Correia et al. (9. We model the joint probability of an MNIST image E Rd,, a continuous latent variable E R and a discrete latent variable {ō1.... δ10) (x) (x z, hb)(z)((h) We assume h has an nedimenssinal
Model:
standard Gaussian prior and z a uniform prior.
We compare ev-softmax to other sparse normalization functions, including sparsemax and enmmax-13 1155. We also train a model using sofimax and apply post-hoc evidential sparsification test time. Since we follow the procedure of Correia et al. (9) we additionally report the results they obtained for stochastic methods. Unbiased gradient estimators that they report include SFE with a moving average baseline; SFE with a self-critic baseline (SFE+) 27 neural variational inference and learning (NVIL) with
7

---

learned baseline [28]; and Sum-and-sampp Sato-Blackwellicee version of SFE (2I). They also evaluate biased gradient estimators including Gumbel-Softmax and straight-through Gumbel-Softman Softmax (ST Gumbel) [4], 15 Each model was trained for 200 epochs.
Results and discussion: shows that using ev-sofimax yields the high- classification accuracy of models using either Monte Carlo methods or marginalization. Compared sparse- max and entmax- 1.5, ev-softmax main- tains higher multimodality in the output distribution, reducing the latent space by 84% on average. In particular, both sparsemax and enfmax-1. 1.5 reduce the latent space by almost 90%, but at the cost of decreased accuracy. On the other hand, post-hoc evidential sparsification preserves more latent classes than softmax but with a higher variance, and it also achieves a lower accuracy. This suggests that in this supervipe set- ting, training with sparsity and multi- modality in the output distribution are both important for distributional accu- racy, which benefits overall performance.
Table The results of the semi-supervised VAE on MNIST show that ev- ev-softmax yields the best cemi-supervise performance. We report the mean and standard error of the accuracy and number of decoder calls for 10 random seeds.
Method
Accuracy (%)
Decoder calls
Monte Carlo SFE
94.75 + .002
1
SFE+
96.53 ,001
2
NVIL
96.01 + .002
1
Sum&Sample
96.73 + .001
2
Gumbel
95.46 + .001
1
ST Gumbel
86.35 + .006
1
Marginalizaaion Softmax
96.93 + .001
10
Sparsemax
96.87 = .001
1.01 0.01
Entmax-1.5
97.20 + .001
1.01 = 0.01
Post-hoc evidential
96.90 + .001
1.72
1 0.05
Ev-softmax
97.30 + .001
1.64 + 0.01
4.3 Vector-Quantized Variational Autoencoder (VQ-VAE)
We demonstrate the performance of our approach on much larger latent space by training a Vector Quantized- Variational Autoencoder (VQ-VAE) model on tinyImageNet [29] for im- age generation. The VQ- VAE learns discrete latent variables embedded in continuous vector spaces (T]. We use the tinylmageNet dataset due to com- putational constraints.
Table We sample images from each of the latent classes and compute downstream classification performance (top-5 accu- racy and top- 10 accuracy) and average number of nonzero latent classes. Our method yields images with the highest top-5 and top- 10 accuracy scores. K indicates the average number of nonzero latent classes.
Method
Acc-5
Acc-10
K
Softmax
38.4 + 0.2
48.8 + 0.3
512
Sparsemax
40.0 0.2
52.0 0.2
46 + 0.3
Model: We train the VQ-VAE ar- chitecture in two stages. First the VQ-VAE model is trained assuming uniform prior over the discrete latent space. For each normalization method, we train an autoregressive prior over the latent space from which we sample. We consider a latent space of 16 x 16 discrete latent variables with K = 512 classes each. We use the same PixelCNN (30) as Itkina et al. [IO] for the autoregressive prior. For each normalization function, we train a PixelCNN prior using that function to generate the output probability over the latent space. To evaluate the accuracy of our VQ- VQ-VAE models, we train a Wide Residual Network classifier (3T) on tinylmageNet and calculate its classification accuracy on the images generated by the VQ- VAE models. Further experimental details are given in Appendix E.
Entmax- 1.5
38.4
0.1
90 0.3
Post-hoc evidential
38.2 0.2
48.4 0.3
102 = 0.2
Ev-softmax
43.6 + 0.3
55.6
0.2
77 + 0.2
Table shows that ev-sofimax outperforms other sparse methods in accuracy and is able generate images with the most resemblance the ground truth. Our method reduces the number of latent classes by 85% on average, more than the 83% of entmax-1.8 1.5 but less than the 91% of sparsemax. With the exception of post-hoe evidential sparsification, the sparse methods all outperform softmax, suggesting that sparsity improves the model's 's performance on the task of image generation by discarding irrelevant latent classes. The underperformance of the post-hoc evidential model is expected, as post-hoc evidential sparsification is applied only at test time to model trained with the softmux function, which bounds its performance. Since the VQ-VAE samples directly from the output distribution of the PixeICNN prior, the probabilistic nature of this downstream task
Results and discussion:
8

---

Table 4: BLEU, ROUGE, and METEOR scores, number nonzero elements in the attention on EN- corpus of IWSLT 2014. We report the mean and standard error for the BLEU score and number of nonzero elements over 5 random seeds. Our method outperforms all baselines in all metrics and maintains higher number of nonzero attended elements than other sparse normalization functions. Higher is better for all metrics. The asterisk denotes that ev-softmax outperformed the method with significance level of 0.05, and denotes a significance of 0.01.
Metric
Softmax
Sparsemax
Entmax-1.5
Post-hoc Evidential
Ev-sofumax
BLEU
29.2 + 0.06
29.0 + 0.05
29.2 0.07
29.2 = 0.05
29.4 0.05
ROUGE-1
59.31
58.47**
58.94*
59.09*
59.32
ROUGE-2
35.62
34.76
35.20*
35.42*
35.74
ROUGE-L 56.09
55.39**
55.75*
55.93*
56.18
METEOR
57.02*
56.33**
5.83**
56.84*
57.20
# attended
39.5 + 11.5
2.3 + 0.54
4.1 + 1.3
3.8 + 0.93
8.2 + 1.3
suggests that optimizing the prior for log-likelihood allows the model to learn prior distribution that achieves better performance on the downstream task.
4.4 Machine Translation
We evaluate our method in the setting of machine translation. Attention- based translation models gen- erally employ dense attention weights over the entire source sequence while traditional statistical machine translation systems usually based lattice of sparse phrase hypotheses [9]. Thus, sparse attention offers the possibility of more interpretable attention distributions which avoid wasting probability mass on irrelevant source words. Furthermore, language modeling often requires the context of multiple words correctly identify dependencies within an input se- quence (2). Hence, we investigate the impact of the multimodality our proposed normalization func- tion on the quality of translations.
does
that
make
you
curious
Softmax
Cenniiion ia Bot
Sparsemax
Entmax-1
Ev-softmax
Figure 4: Attention weights produced by each nor- malization method for the first word of the trans- lated sentence. Sparsemax and entmax- both tend over two words while ev-sofimax and softmax attend over the entire source. The resulting pre- dictions are macht (ev-softmax), Ist (sparsemax), and Machen (entmax-1.3 1.5, softmax). The ground truth translation macht sie das neugierig?. Al- hough the attention distributions ev-sofimax and sofimax are similar in this example, er-softmax still significantly sparsifies attention away from superfiuous inputs.
Model: We train transformer models for each of the sparse normalization functions using the OpenNMT Neural Machine Translation Toolkit (32]. Each nor- malization function is used in the self-attention layers. We use byte pair encoding (BPE) to encode rare and unknown words and ensure an open vocabulary (33). Due to computational constraints, we finetune trans- former models on the English-German translation dataset from IWSLT 2014 (34] These models were pretrained on EN-DE corpus from WMT 2016 (35). We finetune for 40000 iterations. Hyperparameters and further experimental details are given in Appendix E.
Results and discussion: As shown Table 4 our method obtains the highest BLEU (36), ME- TEOR (37). and ROUGE (38) scores of the considered normalization functions while reducing the average number of words attended 8.2. In comparison with the other sparse normalization functions, ours maintains the highest number of attended source words. This supports the notion that attending over subset of the source words benefits performance, but the performance of sparsemax and entmax-1.5 1.5 could be hurt by attending over too few words. For short sentences, ev-softmax maintains nonzero attention over the entire sentence, as Fig. F demonsttatts.. This allows for the model to identify relationships between words in the entire source sequence, such as tense and conjugation, enabling more accurate translation.
Related Work
Softmax Alternatives: There considerable interest alternatives to the softmax function that pro- vide greater representational capacity [39]. (40) and improved discrimination of features [1 Sparse-

---

max 19, its generalization a-entmax [16]. and its maximum a posteriori adaptation SparseMAP 42 introduce sparse normalization functions and corresponding loss functions. While the a-entmax family of functions contains parameter to control the level of sparsity, the functions are all sparse for a with no known continuous approximation with full support. Therefore, these methods are all trained with alternative loss functions that do not directly represent the likelihood of the data. Another controllable sparse alternative to softmax is sparsehourglass [17]. We baseline against only sparsemax and entmax-1.5 as more research has been conducted on these two methods and entmax-1.5 is semantically similar to sparsehourglass. In contrast, we develop a normalization function which has continuous family of approximations that can be used during training time with the negative log-likelibe likelihoood and KL divergence loss terms. This flexibility enables our method to substitute the softmax transformation in any context and model.
Discrete Variational Autoencoders:
Recent advancements in VAEs with discrete latent spaces have introduced strategies for learning discrete latent spaces to tackle wide-ranging set of tasks including
image generation [T) 43), video prediction [44), and speech processing [45] For discrete VAEs,
the gradient of the ELBO usually cannot be calculated directly (14) While the SFE 12 [3
enables Monte marlo-based approaches to estimate the stochastic gradient of the ELBO, it suffers from high variance and is consequently slow to converge. Continuous relaxation via the Gumbel- Softmax distribution allows for an efficient and convenient gradient calculation through discrete latent spaces [14), [15]. but the gradient estimation is biased and requires sampling during training. Our work, in contrast, enables direct marginalization over a categorical space without sampling. Similar to our work, Correia et al. [9] apply sparsemax to marginalize discrete VAEs efficiently. While also generate a sparse distribution over the latent space as in Correia et al. [9]. we better balance sparsity with the multimodality of the distribution. Prior work evidential sparsification focused on a post-hoc method for discrete CVAES [10]. We extend this method to more classes of generative models and develop modification apply the method at training time.
machine translation, sparse attention reflects the intuition that only a few source words are expected to be relevant for each translated word. The global attention model considers all words on the source side for each target word, which is potentially prohibitively expensive for translating longer sequences [46). The softmax transformation has been replaced with sparse normalization functions, such as the sparsemax operation (TT) and its generalization, a-entmax [47). substitute sofimax in the attention layers of machine translation models. Prior work on sparse normalization functions for attention layers has focused on global attention layers in recurrent neural network models. In contrast, study the effect of incorporating these sparse normalization functions in transformers, where they are used in self-attention layers.
Sparse Attention:
Like our work, both sparsemax and o-entmax
6 Conclusion
We present ev-softmax, a novel method to train deep generative models with sparse probability distributions. We show that our method satisfies many of the same theoretical properties as softmax and other sparse alternatives to softmax. By balancing the goals of multimodality and sparsity, our method yields better distributional accuracy and classification performance than baseline methods. Future work can involve training large computer vision and machine translation models from scratch with our method and examining its efficacy at a larger scale on more complex downstream tasks.
7 Broader Impact
Our work focuses method to train generative models that output sparse probability distributions. We apply our method in the domains of image generation and machine translation. Our work has the potential to improve the explainability and introspection of generative models. Furthermore, our work aims to reduce the computational cost of training generative models, which are used in unsupervised and semi-supervised approaches to learning. Although our empirical validation shows that our method can improve the accuracy of generated distributions, the use of sparse probability distributions can also amplify any inherent bias present in the data modeling assumptions. One limitation of our work is that we do not provide explicit theoretical guarantees on the conditions for yielding an output probability of zero for a class. We hope that our work motivates further research into improving the accuracy of generative modeling and reducing bias of both models and data.
10

---

References
[1] A. van den Oord, Vinyals, and Kavukeuoglu. "Neural discrete representation learning," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 6306-6315.
[2] A. Vaswani, N. Shazeer, Parmar, Uszkoreit, L. Jones, N. Gomez, Kaiser, Polosukhin, "Attention is all you need," in Advances in Neural Information Processing Systems (NeurIPS), 2017.
[3] Radford, K. Narasimhan, T. Salimans, and Sutskever, "Improving language understanding by generative pre-training." 2018.
[4] Z. Yang. W. Chen, F. Wang, and B. Xu, "Improving neural machine translation with conditional sequence generative adversarial nets," arXiv, 2017.
[5] T. Salzmann, B. Ivanovic, P. Chakravarty, and M. Pavone, "Trajectron++:: Dynamically-feann feasible trajectory forecasting with heterogeneous data." in European Conference on Computer Vision (ECCV), 2020.
[6] J. Co-Reyes, Y. Liu, A. Gupta, B. Eysenbach, P. Abbeel, and S. Levine, "Self-consistent trajectory au- toencoder: Hierarchical reinforcement learning with trajectory emheddings." in International Conference Machine Learning (ICML), PMLR, 2018. pp. 1009-1018.
[7] V. Fortuin, M. Hüser, F. Locatello, H. Strathmann, and G. Rätsch, "SOM-VAE: Interpretable discrete representation learning on time series," in International Conference on Learning Representations (ICLR), 2019.
[8] A. Martins and Astudillo, "From softmax to sparsemax: A sparse model of attention and multi-label classification," in International Conference on Machine Learning (ICML), 2016, pp. 1614-1623.
[9] Correia, V. Niculae, Aziz, and A. Martins, "Efficient marginalization of discrete and structured latent variables via sparsity," Advances in Neural Information Processing Systems (NeurIPS), vol. 33, 2020.
[10] M. Itkina, B. Ivanovic, R. Senanayake, M. J. Kochenderfer, and M. Pavone, "Evidential sparsification of multimodal latent spaces in conditional variational autoencoders." Advances in Neural Information Processing Syxtems (NeurIPS), vol. 33, 2020.
[11] C. Malaviya, P. Ferreira, and A. F. Martins, "Sparse and constrained attention for neural machine translation," in Annual Meeting of the Association for Computational Linguistics (ACI.), 2018, pp. 370- 376.
[12] R. Williams, "Simple statistical gradient-following following algorithms connectionis learning." Machine Learning, vol. 8, no. 3-4, pp. 229-256, 1992.
[13] P. W. Glynn, "Likelihood ratio gradient estimation for stochastic systems." Communications of the ACM, vol. 33, no. 10, pp. 75 84, 1990.
[14] E. Jang. S. Gu, and B. Poole. "Categorical reparameterization with Gumbel-Softmax." in International Conference on Learning Reprexentations (ICLR), 2017.
[15] C. J. Maddison, A. Mnih, and Y. W. Tch, "The Concrete distribution: continuous relaxation of discrete
random variables," " in International Conference on Learning Representations (ICLR), 2017.
[16] B. Peters, V. Niculae, and A. F. Martins, "Sparse
models," in Anmual Meeting of
the Association for Computational Linguistics (ACL). 2019, pp. 1504-1519.
[17] A. Laha, S. A. Chemmengath, P. Agrawal, M. Khapra, K. Sankaranarayanan, and H. G. Ramaswamy, "On controllable sparse alternatives softmax," in Advances in Neural Information Processing Systems (NeurIPS), 2018. pp. 6422 6432.
[18] V. Niculac, A. Martins, M. Blondel, and C. Cardie, "SparseMAP: Differentiable sparse structured inference," " in International Conference Machine Learning (ICML), PMLR, 2018, pp. 3799-3808. [19] K. Sohn, H. Lee, and X. Yan, "Leaming structured output representation using deep conditional generative models," in Advances in Neural Information Processing Systems (NeurIPS), 2015, pp. 3483-349)
[20] D. P. Kingma and M. Welling, "Auto-encoding variational Bayes," in International Conference on Learning Representations (ICLR), 2014.
[21] R. Liu, J. Regier, N. Tripuraneni, M. Jordan, and J. Mcauliffe, "Rao-llackwellizeed stochastic gradients for discrete distributions," in International Conference on Machine Learning (ICMI), PMLR, 2019, pp. 4023-4031.
[22] A. P. Dempster, "A generalization of Bayesian inference," Classic works of the Dempster-Shate Theory of Belief Functions, pp. 73-104, 2008
[23] T. Denoeux. "Logistic regression, neural networks and Dempster-Shafer theory: A new perspective." Knowledge-Bor Based Systems, vol. 176, pp. 54-67, 2019.
[24] B. Bigi, "Using Kullhack-l.ehller distance for text categorization," in European Conference on Informa- tion Retrieval (ECIR), Springer, 2003, pp. 305-319.
[25] J. Hullman, N. Diakopoulos, and F. Adar, "Contextifier: Automatic generation of annotated stock
visualizations," in SIGCHI Conference on Human Factors in Computing Systems, ACM, 2013, pp. 2707- 2716.
11

---

[26] D. P. Kingma. D. J. Rezende. S. Mohamed, and M. Welling. "Semi-supervised learning with deep generative models." in Advances in Neural Information Processing Systems (NeurlPS), 2014, pp. 3581- 3589.
[27] S. J. Rennic, F. Marcheret, Y. Mrouch, J. Ross, and V. Goel, "Self-eritical sequence training for image captioning," " in IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017, pp. 7008- 7024.
[28] Mnih and K. Gregor, "Neural variational inference and learning in helief networks," in International Conference on Machine Learning (ICML), PMIR, 2014, pp. 1791-1799.
[29] Y. Le and X. Yang. "Tiny ImageNet visual recognition challenge," CS23IN: Convolutional Neural Networks for Visual Recognition (Stanford University). 2015.
[30] A. van den Oord and B. Schrauwen, "Factoring variations in natural images with deep Gaussian mixture models," in Advances in Neural Information Processing Systems (NeurlPS), 2014.
[31] S. Zagoruyko and N. Komodakis, "Wide residual networks," in Proceedings of the British Machine Vision Conference (BMVC), 2016, pp. 87.1
[32] G. Klein, F. Hernandez. V. Nguyen, and J. Senellart, "The OpenNMT neural machine translation toolkit: 2020 edition," in Conference of the Association for Machine Translation in the Americas (AMTA), 2020, pp. 102-109.
[33] R. Sennrich, B. Haddow, and A. Birch, "Neural machine translation of rare words with subword units," in Annual Meeting of the Association for Computational Linguistics (ACL). 2016, pp. 1715-1725.
[34] M. Cettolo, M. Federico, L. Bentivogli, N. Jan, S. Sebastian, S. Katsuitho, Y. Koichiro, and F. Christian, "Overview the IWSLT 2017 evaluation campaign." International Workshop on Spoken Language Translation (IWSLT), 2017, pp. 2-14.
[35] O. Bojar, R. Chatterjee, C. Federmann, Y. Graham, B. Haddow, M. Huck, A. J. Yopes, P. Kochn, Logacheva, C. Monz, et al., "Findings of the 2016 Conference on Machine Translation," in Conference Machine Translation (WMT), 2016, 6, pp. 131-198.
[36] K. Papineni, S. Roukos, T. Ward, and W.-J. Zhu, "BLEU: A method for automatic evaluation of machine translation," in Annual Meeting of the Association for Computational Linguistics (ACL), 2002, pp. 31 1-
318.
[37]
M. Denkowski and A. Lavie, "METEOR 1.3: Automatic metric for reliable optimization and evaluation of machine translation systems." Sixth Workshop on Statistical Machine Translation, Association for Computational Linguistics, 201 1, pp. 85-91.
[38] C.-Y. Lin, "ROUGE: A package automatic evaluation summaries," in Text Summarization Branches Out, Association for Computational Linguistics, 2004, pp. 74-81.
[39] S. Kanai, Yamanaka, Y. Fujiwara, and Adachi, "Sigsoftmax: Reanalysis of the softmax bottleneck," in Advances in Neural Information Processing Systems (NeurIPS), vol. 2018, 2018, pp. 286-296.
[40] R. Memisevic, C. Zach, M. Pollefeys, and G. E. Hinton, "Gated softmax classification," in Advances in Neural Information Processing Systems (NeurIPS), vol. 23, 2010, pp. 603-1611..
[41] W. Liu, Y. Wea, Z. Yu, and M. Yang, "Large-margin softmax loss for convolutional neural networks.," in International Conference Muchine Learning (ICML), PMLR, vol. 2, 2016, 7.
[42] V. Niculae and M. Blondel, "A regularized framework for sparse and structured neural attention," in Advances in Neural Information Processing Systems (NeurIPS), 2017, pp. 3340-3350.
[43] A. Razavi, A. van den Oord, and O. Vinyals, "Generating diverse in Advances in Neural Information Processing Systems (NeurlPS), 2019, pp. 14837-14847
images with VQ-VAE-2,"
[44] J. Walker, A. Razavi, and A. d. Oord, "Predicting video with VQVAE," arXiv, 2021.
[45] C. Gârbacca, A. van den Oord, Y. Li, F. S. Lim, A. Lucbs, O. Vinyals, and T. C. Walters, "Low bit-rate speech coding with vq-vac and wavenet decoder," in IEEE International Conference Acoustics, Speech, and Signal Processing (ICASSP), 2019, pp. 735-739.
[46] M.-T. Luong, H. Pham, and C. D. Manning, "Effective approaches iitention-based neural machine translation," in Conference on Empirical Methods Natural Language Processing (EMNLP), 2015, pp. 1412-1421.
[47] B. Peters, V. Niculae, and A. F. Martins, "Sparse suneencetoosequeeee models," in Annual Meeting of the Association for Computational Linguistics (ACL). 2019, pp. 1519.
[48] D. Kingma and J. Ba, "Adam: A method for stochastic optimization,"" in International Conference on Learning Representations (ICLR), 2014.
12


Reviews:
[{"review_id": "ZmGpTi5h7sZ", "reviewer": "Reviewer_G99K", "summary": "The paper introduces a normalizing function called evidential softmax (ev-softmax). Based on principles of evidential theory, ev-softmax is able to assign zero probability to classes that lack evidence of the data. It can be seen as a generalization of the work done by Itkina et al. (2020), the crucial difference being that ev-softmax can be applied at training time. The experiment section shows improved performance compared to other sparse methods, on a variety of tasks.", "questions": "", "limitations": "", "rating": 8, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "_Originality_: The submission introduces a novel and differentiable normalizing function that is a generalization of the method described in Itkina et al. (2020). It is clear how this work differs from that contribution, and related work is adequately cited, as far as I know.\n\n_Quality_: The methods described in this submission are well supported by evidential theory and the authors do a good job in their explanation. Furthermore, they use their method in a variety of tasks, in which they empirically show that their method has superior performance when compared to sparsemax, 1.5-entmax, and softmax.\n\n_Clarity_: This submission is very well written and well detailed. Results could be easily reproduced since the authors provided the code.\n\n_Significance_: The method proposed in this submission can be an asset to researchers and practitioners in the future as a promising sparse alternative to softmax, where the reason for this sparsity is theoretically grounded in evidential theory. The improved results of ev-softmax on four different tasks also show that the method has practical applications.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I believe the authors have adequately addressed the societal impact of their work.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}, {"review_id": "P_TeBISjqrW", "reviewer": "Reviewer_R6Na", "summary": "This paper shows a sparse normalization function that preserves the multimodality of probability distributions to  train deep generative models.\n\nThe authors apply their methods to different VAE architectures and demonstrate the effectiveness of their methods.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The overall framework of the paper is good. However, the motivation of the paper is not very clear. Compared to the previous works about VAE, this paper is not very innovative.", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors do not fully demonstrate the limitations of their work. In addition, they do not clearly give the experimental results to show the benefits of considering the multimodality and sparsity of the data distribution.", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "1.5"}, "extra_text": {}}, {"review_id": "N-5hmcaKfwp", "reviewer": "Reviewer_kvJZ", "summary": "The paper proposes a new sum-to-one activation function, namely evidential softmax, which is specialized for the sparse multimodal activation.\nWith the suggested transformation, the authors resolve the existing post-hoc process of evidential sparsification.\nThe authors provide various experiments to support their proposed sparse activation function.\n\n", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "\nOriginality:\nRemoving the post-hoc process and finding the equivalent form seems quite novel work.\nHowever, I'm a bit confused of the messeage of the paper.\nThe motivation in the introduction and the final purpose that can be seen in the experimental result section of the paper seems quite distinguished.\n\nQuality:\nThe paper sound okay technically, however, I'm not convinced with the experimental section and the related questions are separately listed below.\n\nClarity:\nI cannot say that the paper is well-written due to the overall configuration.\nI wonder why the discrete VAE came up in the first in the background.\nIt would be better if Section 5 come before Section 4.\n\nSignificance:\nThe paper is explained through theoretical backups.\nHowever, the overall performance is too marginal and Table 2 seems to be taken from [9] but there are no comment on this.\nAlso, there is no confidence interval in VQ-VAE experiment.\nAs a minor comment, it would be more convinced if the authors applied various \\alpha values for the entmax.\n\nQuestions:\n- In Table 1, compared to the other sum-to-one activation functions, EvSoftmax has less nice properties. How does the authors think that which property unsatisfaction gives better result?\n- How does the result change in Fig 1 if the ratio over the labels is unbalanced? (including the extremely unbalanced case)\n- What does decoder calls imply? And is it good or bad if the number is small?\n- SS-VAE experiment result shows little difference across various models. What becomes when using extremely small amount of labeld data? (for example, 1%) Also, how does the performance gap change when using more complex datasets such as OMNIGLOT or CIFAR?\n- How is the top-1 accuracy result in the VQ-VAE experiment?\n- How does overall performance change as \\epsilon differs?\n- I can't find any comparison on the post-hoc process [10] in the experiment section. How differ the performance are?\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "The authors pointed out the limitation of their work, and the work seems that it does not have any potential negative societal impact.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "8"}, "extra_text": {}}, {"review_id": "2v7Z8Tw9SHa", "reviewer": "Reviewer_fAmj", "summary": "This paper proposes a sparse normalization function called evidential softmax (ev-softmax). Specifically, it extends the post-hoc sparsification approach of Itkina et al, to propose a normalization function that can be applied during both training and testing. Notably, the authors claim that ev-softmax preserves the multimodality of the distribution, while providing sparse output distributions. The authors argue that sparsity is likely to help with interpretability and also improve task performance, while multimodality is useful when the output space is multimodal. They also show experiments showing that previous sparse normalization functions collapse multimodality of distributions. \n\nThe formulation of the ev-softmax function is accompanied with relevant proofs showing that a continuous family of approximations exist for ev-softmax, which make it possible to train with NLL and KL divergence based losses. Experiments on image generation using a generative model (VAE) and machine translation (using an autoregressive model) show some improvements over previous sparse normalization techniques on distributional accuracy and classification performance.\n", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper provides a clear description of the formulation of the ev-softmax operation and a comprehensive list of properties exhibited by this function. The argument for building sparse yet multimodal distributions is also convincing. However, I'm not sure I completely understand the advantage of applying evidential softmax at training time. It would be great if the authors could clarify. Further, I believe the authors could have still compared with only applying post-hoc sparsification using the method of Itkina et al just to provide a data point for comparison. I believe this could be discussed more in the paper to emphasize the novelty of the work.\n\nOne other comment I had about the description is that currently, it might not be giving enough intuition to the readers about why ev-softmax gives sparse & multimodal distributions. This might be because that this work is an extension of the work of Itkina et al and some details are redundant, but I still think it could be worth illustrating in more depth how this function behaves and why it fits the desired criteria.\n\nOn the experimental front, I think the paper could be improved. The improvements on some of the tasks (in tables 2, 4) seem very small and significance tests are missing. I also wonder if the authors could have used considered different values of alpha for the alpha-entmax baseline. Following are my questions/concerns about the experiments:\n- Re: the MNIST experiments in sec 4.1, both sparsemax and entmax-1.5 seem to have degenerate performance in this experiment but they perform reasonably well in the other image generation experiments. Do you know why this is the case? \n- Did you consider varying the value of alpha in entmax? If yes, does that help improve the multimodality of the distributions?\n- How well does the wide residual network used for evaluating the models in the exp in 4.3 perform? Instead of using a model for evaluation, did you consider performing human evaluation on a smaller sample?\n- The improvements on the machine translation experiments appear to be very small. Have you performed significance testing (perhaps using a paired bootstrap test)? Computing multiple automatic metrics such as ROUGE, METEOR and performing human evaluation would be useful too.\n- How can one guarantee that it is multimodality that is helping performance? I understand the # attended words is higher / latent space reduction is (sometimes) higher with ev-softmax. However, there is lack of evidence showing that multimodality helps performance.\n- How expensive is the ev-softmax in terms of time and space, relative to other sparse normalization alternatives?\n- One of the motivations posed in the intro is multimodality, especially in the context of machine translation where multiple reference words are likely. However, in the experiments, this effect hasn't really been validated. The function has instead been applied to the attention distribution.\n\n \nOther comments:\n- In Sec 2.1, could you expand upon your earlier claim that the Gumbel-Softmax estimator introduces bias?\n- Some citations / comparisons to other relevant work such as Zhao et al (Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection) are missing.\n- Line 226: appears to be a typo.\n\nOverall, I think the paper is written well and has good motivation. The experiments have some weaknesses highlighted above and performance improvements seem small. Also, the description of the intuition behind the approach (why it induces multimodality + sparsity) would be useful.\n", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "I would appreciate if the authors could expand upon the discussion in section 7 about the possibility of bias amplification through sparsification. I also wonder if there is relevant work that can be cited to add on to this discussion.\n", "review_text": "", "extra_scores": {"needs_ethics_review": "No", "ethics_review_area": "", "time_spent_reviewing": "4"}, "extra_text": {}}]

Author rebuttal:

Meta-review: The paper introduces a normalizing function called evidential softmax (ev-softmax). Based on principles of evidential theory, ev-softmax is able to assigns zero probability to some classes. It extends Itkina et al. (2020) by recovering their transformation and allowing its usage at training time. The experiment section shows (relatively small) performance gains compared to other sparse methods on several simple tasks and on a machine translation task.

Most reviewers agree that this is a good paper that proposes an interesting contribution, closes the loop with Itkina et al. (2020),  and establishes an interesting link between evidential theory and sparse transformations. The main weaknesses which have been pointed out are  lack of details about some of the experiments (clarified in the rebuttal), lack of camparison with alpha-entmax for other values of alpha (not crucial but a nice-to-have), and lack of discussion of how ev_softmax relates to other sparse transformations (which the authors elaborate on in the rebuttal, but could be further expanded). This changes seem doable at camera ready time, therefore I recommend acceptance. 

I urge the authors to follow the recommendations of the reviewers. Reporting other values of alpha should be simple given that the code for alpha-entmax is publicly available. More importantly, the relation between ev_softmax and entmax/sparsemax should be clarified:
- The claim that ev_softmax(x)_j = 0 => sparsemax(x)_j = 0 is not correct - the supports of ev_softmax and sparsemax are generally not a subset of each other (check comment with counterexamples)
- Note that the support of sparsemax can also be expressed as a mean condition on the largest scores — see Alg 1 in https://arxiv.org/pdf/1602.02068.pdf. The zeros of sparsemax are the entries where x_j <= mean(top_k(x)) - 1/k, where k is the size of the support. I think this deserves discussion.
- When x in R^2, sparsemax is a hard sigmoid (Fig 1 in the ref above) but ev_softmax seems to be a 0/1 loss, piecewise constant, with zero gradients everywhere. This means you can’t backpropagate through ev_softmax for this 2-dimensional case. I think this an important limitation that should be discussed (sparsemax and entmax don’t have this limitation).
Decision: Accept (Poster)