Venue: NeurIPS 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: summary: The paper proposes a new sum-to-one activation function, namely evidential softmax, which is specialized for the sparse multimodal activation.
With the suggested transformation, the authors resolve the existing post-hoc process of evidential sparsification.
The authors provide various experiments to support their proposed sparse activation function.


main_review: 
Originality:
Removing the post-hoc process and finding the equivalent form seems quite novel work.
However, I'm a bit confused of the messeage of the paper.
The motivation in the introduction and the final purpose that can be seen in the experimental result section of the paper seems quite distinguished.

Quality:
The paper sound okay technically, however, I'm not convinced with the experimental section and the related questions are separately listed below.

Clarity:
I cannot say that the paper is well-written due to the overall configuration.
I wonder why the discrete VAE came up in the first in the background.
It would be better if Section 5 come before Section 4.

Significance:
The paper is explained through theoretical backups.
However, the overall performance is too marginal and Table 2 seems to be taken from [9] but there are no comment on this.
Also, there is no confidence interval in VQ-VAE experiment.
As a minor comment, it would be more convinced if the authors applied various \alpha values for the entmax.

Questions:
- In Table 1, compared to the other sum-to-one activation functions, EvSoftmax has less nice properties. How does the authors think that which property unsatisfaction gives better result?
- How does the result change in Fig 1 if the ratio over the labels is unbalanced? (including the extremely unbalanced case)
- What does decoder calls imply? And is it good or bad if the number is small?
- SS-VAE experiment result shows little difference across various models. What becomes when using extremely small amount of labeld data? (for example, 1%) Also, how does the performance gap change when using more complex datasets such as OMNIGLOT or CIFAR?
- How is the top-1 accuracy result in the VQ-VAE experiment?
- How does overall performance change as \epsilon differs?
- I can't find any comparison on the post-hoc process [10] in the experiment section. How differ the performance are?

limitations_and_societal_impact: The authors pointed out the limitation of their work, and the work seems that it does not have any potential negative societal impact.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper proposes a sparse normalization function called evidential softmax (ev-softmax). Specifically, it extends the post-hoc sparsification approach of Itkina et al, to propose a normalization function that can be applied during both training and testing. Notably, the authors claim that ev-softmax preserves the multimodality of the distribution, while providing sparse output distributions. The authors argue that sparsity is likely to help with interpretability and also improve task performance, while multimodality is useful when the output space is multimodal. They also show experiments showing that previous sparse normalization functions collapse multimodality of distributions. 

The formulation of the ev-softmax function is accompanied with relevant proofs showing that a continuous family of approximations exist for ev-softmax, which make it possible to train with NLL and KL divergence based losses. Experiments on image generation using a generative model (VAE) and machine translation (using an autoregressive model) show some improvements over previous sparse normalization techniques on distributional accuracy and classification performance.

main_review: The paper provides a clear description of the formulation of the ev-softmax operation and a comprehensive list of properties exhibited by this function. The argument for building sparse yet multimodal distributions is also convincing. However, I'm not sure I completely understand the advantage of applying evidential softmax at training time. It would be great if the authors could clarify. Further, I believe the authors could have still compared with only applying post-hoc sparsification using the method of Itkina et al just to provide a data point for comparison. I believe this could be discussed more in the paper to emphasize the novelty of the work.

One other comment I had about the description is that currently, it might not be giving enough intuition to the readers about why ev-softmax gives sparse & multimodal distributions. This might be because that this work is an extension of the work of Itkina et al and some details are redundant, but I still think it could be worth illustrating in more depth how this function behaves and why it fits the desired criteria.

On the experimental front, I think the paper could be improved. The improvements on some of the tasks (in tables 2, 4) seem very small and significance tests are missing. I also wonder if the authors could have used considered different values of alpha for the alpha-entmax baseline. Following are my questions/concerns about the experiments:
- Re: the MNIST experiments in sec 4.1, both sparsemax and entmax-1.5 seem to have degenerate performance in this experiment but they perform reasonably well in the other image generation experiments. Do you know why this is the case? 
- Did you consider varying the value of alpha in entmax? If yes, does that help improve the multimodality of the distributions?
- How well does the wide residual network used for evaluating the models in the exp in 4.3 perform? Instead of using a model for evaluation, did you consider performing human evaluation on a smaller sample?
- The improvements on the machine translation experiments appear to be very small. Have you performed significance testing (perhaps using a paired bootstrap test)? Computing multiple automatic metrics such as ROUGE, METEOR and performing human evaluation would be useful too.
- How can one guarantee that it is multimodality that is helping performance? I understand the # attended words is higher / latent space reduction is (sometimes) higher with ev-softmax. However, there is lack of evidence showing that multimodality helps performance.
- How expensive is the ev-softmax in terms of time and space, relative to other sparse normalization alternatives?
- One of the motivations posed in the intro is multimodality, especially in the context of machine translation where multiple reference words are likely. However, in the experiments, this effect hasn't really been validated. The function has instead been applied to the attention distribution.

 
Other comments:
- In Sec 2.1, could you expand upon your earlier claim that the Gumbel-Softmax estimator introduces bias?
- Some citations / comparisons to other relevant work such as Zhao et al (Explicit Sparse Transformer: Concentrated Attention Through Explicit Selection) are missing.
- Line 226: appears to be a typo.

Overall, I think the paper is written well and has good motivation. The experiments have some weaknesses highlighted above and performance improvements seem small. Also, the description of the intuition behind the approach (why it induces multimodality + sparsity) would be useful.

limitations_and_societal_impact: I would appreciate if the authors could expand upon the discussion in section 7 about the possibility of bias amplification through sparsification. I also wonder if there is relevant work that can be cited to add on to this discussion.
review_title: null

final_score: rating: 8: Top 50% of accepted NeurIPS papers, clear accept
review_content: summary: The paper introduces a normalizing function called evidential softmax (ev-softmax). Based on principles of evidential theory, ev-softmax is able to assign zero probability to classes that lack evidence of the data. It can be seen as a generalization of the work done by Itkina et al. (2020), the crucial difference being that ev-softmax can be applied at training time. The experiment section shows improved performance compared to other sparse methods, on a variety of tasks.
main_review: _Originality_: The submission introduces a novel and differentiable normalizing function that is a generalization of the method described in Itkina et al. (2020). It is clear how this work differs from that contribution, and related work is adequately cited, as far as I know.

_Quality_: The methods described in this submission are well supported by evidential theory and the authors do a good job in their explanation. Furthermore, they use their method in a variety of tasks, in which they empirically show that their method has superior performance when compared to sparsemax, 1.5-entmax, and softmax.

_Clarity_: This submission is very well written and well detailed. Results could be easily reproduced since the authors provided the code.

_Significance_: The method proposed in this submission can be an asset to researchers and practitioners in the future as a promising sparse alternative to softmax, where the reason for this sparsity is theoretically grounded in evidential theory. The improved results of ev-softmax on four different tasks also show that the method has practical applications.
limitations_and_societal_impact: I believe the authors have adequately addressed the societal impact of their work.
review_title: null

final_score: rating: 6: Marginally above the acceptance threshold
review_content: summary: This paper shows a sparse normalization function that preserves the multimodality of probability distributions to  train deep generative models.

The authors apply their methods to different VAE architectures and demonstrate the effectiveness of their methods.

main_review: The overall framework of the paper is good. However, the motivation of the paper is not very clear. Compared to the previous works about VAE, this paper is not very innovative.
limitations_and_societal_impact: The authors do not fully demonstrate the limitations of their work. In addition, they do not clearly give the experimental results to show the benefits of considering the multimodality and sparsity of the data distribution.
review_title: null

Meta-review: The paper introduces a normalizing function called evidential softmax (ev-softmax). Based on principles of evidential theory, ev-softmax is able to assigns zero probability to some classes. It extends Itkina et al. (2020) by recovering their transformation and allowing its usage at training time. The experiment section shows (relatively small) performance gains compared to other sparse methods on several simple tasks and on a machine translation task.

Most reviewers agree that this is a good paper that proposes an interesting contribution, closes the loop with Itkina et al. (2020),  and establishes an interesting link between evidential theory and sparse transformations. The main weaknesses which have been pointed out are  lack of details about some of the experiments (clarified in the rebuttal), lack of camparison with alpha-entmax for other values of alpha (not crucial but a nice-to-have), and lack of discussion of how ev_softmax relates to other sparse transformations (which the authors elaborate on in the rebuttal, but could be further expanded). This changes seem doable at camera ready time, therefore I recommend acceptance. 

I urge the authors to follow the recommendations of the reviewers. Reporting other values of alpha should be simple given that the code for alpha-entmax is publicly available. More importantly, the relation between ev_softmax and entmax/sparsemax should be clarified:
- The claim that ev_softmax(x)_j = 0 => sparsemax(x)_j = 0 is not correct - the supports of ev_softmax and sparsemax are generally not a subset of each other (check comment with counterexamples)
- Note that the support of sparsemax can also be expressed as a mean condition on the largest scores — see Alg 1 in https://arxiv.org/pdf/1602.02068.pdf. The zeros of sparsemax are the entries where x_j <= mean(top_k(x)) - 1/k, where k is the size of the support. I think this deserves discussion.
- When x in R^2, sparsemax is a hard sigmoid (Fig 1 in the ref above) but ev_softmax seems to be a 0/1 loss, piecewise constant, with zero gradients everywhere. This means you can’t backpropagate through ev_softmax for this 2-dimensional case. I think this an important limitation that should be discussed (sparsemax and entmax don’t have this limitation).
Decision: Accept (Poster)