SlowGuess/NAACL_2022_OCR huggingface dataset:
Table 4: Results of Human Evaluation across 270 random examples (with repetitions) from WMT'13 En  $\rightarrow$  Ru test split. Scores are on a 4-point scale.
The results of human evaluation are shown in Table 4. Cohen's  $\kappa = 0.56$ , indicating moderate agreement between annotators. As we can see, all three models give approximately the same translation quality, and all three are significantly inferior to the reference translation. This is generally consistent with the results of 1.5-ReLU and 1.5-entmax in Table 1, but at the same time casts doubt on the softmax lag behind 1.5-ReLU and 1.5-entmax as the BLEU metric suggests.
In Appendix D we give a few examples where 1.5-ReLU translates better than 1.5-entmax and vice versa.
Sparse seq2seq models. Our proposed  $\alpha$ -ReLU transformation is based on the  $\alpha$ -entmax transformation of Peters et al. (2019), which in turn is a generalization of the sparsemax transformation (Martins and Astudillo, 2016). In our work, we study sparseness at the output of a neural network. Nevertheless, there are a number of works aimed at sparsification within a neural network. For example, Malaviya et al. (2018); Peters et al. (2019); Correia et al. (2019) show that sparsemax and  $\alpha$ -entmax can replace softmax in the attention mechanism with some success. A recent work of Zhang et al. (2021) attempted to replace softmax with a component-wise ReLU in the attention mechanism. Unfortunately, in its pure form, this replacement leads to the inability of the model to learn at all, since its loss function does not decrease during optimization. The authors solve this problem by adding a normalizing layer on top of the attention layer.
These and other works (Zhang et al., 2019) state that sparsity in the weights of attention produces more interpretable patterns. However, Meister et al. (2021) questioned this claim and were unable to find clear evidence to support it. Therefore, in this
work, we focused on the application of  $\alpha$ -ReLU to the output of the transformer model, and not to the mechanism of attention, but at the same time we do not deny the possibility of studying the latter.
Self-normalization. Self-normalizing training aims to bypass the need of normalization during inference time. This is done by tweaking the learning mechanism so that the sum of all predictions sums (approximately) to a constant value. Theoretical work on why this works is poorly understood (Andreas et al., 2015) but early work in neural machine translation has shown its empirical value. Vaswani et al. (2013) achieves that by using noise-contrastive estimation (the neural model is used to re-rank the output of a hierarchical phrase-based machine translation system). Noise-contrastive estimation is also the standard training mechanism for word2vec (more popular than the alternative hierarchical softmax), which also eschews any expensive normalization. Differently, Devlin et al. (2014) changes the training loss to include a factor that encourages the normalizing factor to be 1. At inference time, this is just assumed and decoding time is reported to achieve a  $15\mathrm{x}$  speed-up.
We believe that the main limitations of our work are as follows:
-  $\alpha$ -ReLU's output is still not a probability distribution, as required by the classical formulation of a probabilistic classification model.
-  $\tau$  evaluation requires either running the data through an untrained model with  $\alpha$ -entmax at the output, or deriving a formula similar to (4) for each individual architecture.
- Our approach only works for the case when  $\alpha$ -ReLU is used at the output of the model, but it is not clear how to use it as an alternative to softmax/ $\alpha$ -entmax in the attention layer.
The last mentioned limitation leads to the potential risk of inability to learn if  $\alpha$ -ReLU is misused in the intermediate layers of the neural network such as attention layers. The experiments of Zhang et al. (2021) using vanilla ReLU (2-ReLU with  $\tau = 0$  in our notation) instead of softmax to produce attention weights lead to a divergence of the loss function of the Transformer model. This translates into a waste of energy, especially when training
large models on large datasets. Therefore, we believe that in the future, a preliminary mathematical analysis and/or experiments with small models on small datasets should be carried out as to why the unnormalized distribution of attention weights leads to the inability of the model to learn.
It seems that the sparsity of the output is natural for (sub)word prediction models. Nevertheless, sparsity does not have to come with slowdown of computations, as our work shows. The proposed transformation,  $\alpha$ -ReLU, gives a sparse output, shows competitive performance, and is as fast as softmax. The reduced dependency on the vocabulary size seems particularly important in translation, where neural models are moving more and more towards multi-lingual ones, which in general have a much higher vocabulary size in order to accommodate enough tokens for all languages.
A natural extension of this work will be the evaluation of  $\alpha$ -ReLU in the problem of open-ended text generation, as well as a replacement for softmax in the attention layers of Transformer models.
Our standalone implementation of  $\alpha$ -ReLU in PyTorch is available at https://github.com/MaxatTezekbayev/alpha-relu.
The authors would like to thank Christopher Dance, André Martins, Bill Byrne, Vlad Niculae and anonymous reviewers for their feedback. The work of Maxat Tezekbayev and Zhenisbek Assylbekov is supported by the Nazarbayev University Collaborative Research Program 091019CRP2109.
Jacob Andreas, Maxim Rabinovich, Michael I Jordan, and Dan Klein. 2015. On the accuracy of self-normalized log-linear models. In Proceedings of the 28th International Conference on Neural Information Processing Systems-Volume 1, pages 1783-1791.
Barry C. Arnold, N. Balakrishnan, and H. N. Nagaraja. 2008. A First Course in Order Statistics (Classics in Applied Mathematics). Society for Industrial and Applied Mathematics, USA.
Jimmy Lei Ba, Jamie Ryan Kiros, and Geoffrey E Hinton. 2016. Layer normalization. arXiv preprint arXiv:1607.06450.
Alexandre Berard, Ioan Calapodescu, Marc Dymetman, Claude Roux, Jean-Luc Meunier, and Vassilina Nikoulina. 2019. Machine translation of restaurant reviews: New corpus for domain adaptation and robustness. In Proceedings of the 3rd Workshop on Neural Generation and Translation, pages 168-176, Hong Kong. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Chris Callison-Burch, Christian Federmann, Barry Haddow, Philipp Koehn, Christof Monz, Matt Post, Radu Soricut, and Lucia Specia. 2013. Findings of the 2013 Workshop on Statistical Machine Translation. In Proceedings of the Eighth Workshop on Statistical Machine Translation, pages 1-44, Sofia, Bulgaria. Association for Computational Linguistics.
Ondrej Bojar, Christian Buck, Christian Federmann, Barry Haddow, Philipp Koehn, Johannes Leveling, Christof Monz, Pavel Pecina, Matt Post, Herve Saint-Amand, Radu Soricut, Lucia Specia, and Ales Tamchyna. 2014. Findings of the 2014 workshop on statistical machine translation. In Proceedings of the Ninth Workshop on Statistical Machine Translation, WMT@ACL 2014, June 26-27, 2014, Baltimore, Maryland, USA, pages 12-58. The Association for Computer Linguistics.
Mauro Cettolo, Jan Niehues, Sebastian Stüker, Luisa Bentivogli, and Marcello Federico. Report on the 11th iwslt evaluation campaign, iwslt 2014.
Gonçalo M. Correia, Vlad Niculae, and André F. T. Martins. 2019. Adaptively sparse transformers. In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 2174-2184. Association for Computational Linguistics.
Jacob Devlin, Rabih Zbib, Zhongqiang Huang, Thomas Lamar, Richard Schwartz, and John Makhoul. 2014. Fast and robust neural network joint models for statistical machine translation. In Proceedings of the 52nd Annual Meeting of the Association for Computational Linguistics (Volume 1: Long Papers), pages 1370-1380, Baltimore, Maryland. Association for Computational Linguistics.
Angela Fan, Mike Lewis, and Yann N. Dauphin. 2018. Hierarchical neural story generation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 1: Long Papers, pages 889-898. Association for Computational Linguistics.
Ari Holtzman, Jan Buys, Li Du, Maxwell Forbes, and Yejin Choi. 2020. The curious case of neural text degeneration. In 8th International Conference on Learning Representations, ICLR 2020, Addis Ababa, Ethiopia, April 26-30, 2020. OpenReview.net.
Arthur Jacot, Clément Hongler, and Franck Gabriel. 2018. Neural tangent kernel: Convergence and generalization in neural networks. In Advances in Neural Information Processing Systems 31: Annual Conference on Neural Information Processing Systems 2018, NeurIPS 2018, December 3-8, 2018, Montreal, Canada, pages 8580-8589.
Guillaume Klein, Yoon Kim, Yuntian Deng, Jean Senellart, and Alexander Rush. 2017. OpenNMT: Open-source toolkit for neural machine translation. In Proceedings of ACL 2017, System Demonstrations, pages 67-72, Vancouver, Canada. Association for Computational Linguistics.
Chaitanya Malaviya, Pedro Ferreira, and Andre F. T. Martins. 2018. Sparse and constrained attention for neural machine translation. In Proceedings of the 56th Annual Meeting of the Association for Computational Linguistics, ACL 2018, Melbourne, Australia, July 15-20, 2018, Volume 2: Short Papers, pages 370-376. Association for Computational Linguistics.
Andre F. T. Martins and Ramón Fernandez Astudillo. 2016. From softmax to sparsemax: A sparse model of attention and multi-label classification. In Proceedings of the 33nd International Conference on Machine Learning, ICML 2016, New York City, NY, USA, June 19-24, 2016, volume 48 of JMLR Workshop and Conference Proceedings, pages 1614-1623. JMLR.org.
Clara Meister, Stefan Lazov, Isabelle Augenstein, and Ryan Cotterell. 2021. Is sparse attention more interpretable? In Proceedings of the 59th Annual Meeting of the Association for Computational Linguistics and the 11th International Joint Conference on Natural Language Processing, ACL/IJCNLP 2021, (Volume 2: Short Papers), Virtual Event, August 1-6, 2021, pages 122-129. Association for Computational Linguistics.
Kishore Papineni, Salim Roukos, Todd Ward, and Wei-Jing Zhu. 2002. Bleu: a method for automatic evaluation of machine translation. In Proceedings of the 40th Annual Meeting of the Association for Computational Linguistics, July 6-12, 2002, Philadelphia, PA, USA, pages 311-318. ACL.
Ben Peters and André F. T. Martins. 2021. Smoothing and shrinking the sparse seq2seq search space. In Proceedings of the 2021 Conference of the North American Chapter of the Association for Computational Linguistics: Human Language Technologies, NAACL-HLT 2021, Online, June 6-11, 2021, pages 2642-2654. Association for Computational Linguistics.
Ben Peters, Vlad Niculae, and André F. T. Martins. 2019. Sparse sequence-to-sequence models. In Proceedings of the 57th Conference of the Association for Computational Linguistics, ACL 2019, Florence, Italy, July 28- August 2, 2019, Volume 1: Long Papers, pages 1504-1519. Association for Computational Linguistics.
Matt Post. 2018. A call for clarity in reporting bleu scores. arXiv preprint arXiv:1804.08771.
Rico Sennrich, Barry Haddow, and Alexandra Birch. 2016. Neural machine translation of rare words with subword units. In Proceedings of the 54th Annual Meeting of the Association for Computational Linguistics, ACL 2016, August 7-12, 2016, Berlin, Germany, Volume 1: Long Papers. The Association for Computer Linguistics.
Felix Stahlberg and Bill Byrne. 2019. On NMT search errors and model errors: Cat got your tongue? In Proceedings of the 2019 Conference on Empirical Methods in Natural Language Processing and the 9th International Joint Conference on Natural Language Processing, EMNLP-IJCNLP 2019, Hong Kong, China, November 3-7, 2019, pages 3354-3360. Association for Computational Linguistics.
Ilya Sutskever, Oriol Vinyals, and Quoc V. Le. 2014. Sequence to sequence learning with neural networks. In Advances in Neural Information Processing Systems 27: Annual Conference on Neural Information Processing Systems 2014, December 8-13 2014, Montreal, Quebec, Canada, pages 3104-3112.
Constantino Tsallis. 1988. Possible generalization of boltzmann-gibbs statistics. Journal of statistical physics, 52(1):479-487.
Ashish Vaswani, Noam Shazeer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N. Gomez, Lukasz Kaiser, and Illia Polosukhin. 2017. Attention is all you need. In Advances in Neural Information Processing Systems 30: Annual Conference on Neural Information Processing Systems 2017, December 4-9, 2017, Long Beach, CA, USA, pages 5998-6008.
Ashish Vaswani, Yinggong Zhao, Victoria Fossum, and David Chiang. 2013. Decoding with large-scale neural language models improves translation. In Proceedings of the 2013 Conference on Empirical Methods in Natural Language Processing, pages 1387-1392, Seattle, Washington, USA. Association for Computational Linguistics.
Biao Zhang, Ivan Titov, and Rico Sennrich. 2021. Sparse attention with linear units. In Proceedings of the 2021 Conference on Empirical Methods in Natural Language Processing, EMNLP 2021, Virtual Event / Punta Cana, Dominican Republic, 7-11 November, 2021, pages 6507-6520. Association for Computational Linguistics.
Jiajun Zhang, Yang Zhao, Haoran Li, and Chengqing Zong. 2019. Attention with sparsity regularization for neural machine translation and summarization. IEEE ACM Trans. Audio Speech Lang. Process., 27(3):507-518.
IWSLT'14 De→En
- Architecture: Transformer, embedding size 512, 6 layers, 8 heads, hidden size 1024, shared vocabulary.
- Batch size: 4096 tokens (with gradient accumulation for 8 steps).
- Optimizer:  $ADAM$ ,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.998$ , noam decay, learning rate 2.0, 4000 warmup steps.
- Dropout: 0.3
- No label smoothing.
WMT'14 En→De
- Architecture: Transformer, embedding size 512, 6 layers, 8 heads, hidden size 2048, shared vocabulary of 40K tokens, shared embeddings and decoder embeddings.
- Batch size: 4096 tokens (with gradient accumulation for 4 steps).
- Optimizer: ADAM,  $\beta_{1} = 0.9$ ,  $\beta_{2} = 0.998$ , noam decay, learning rate 2.0, 8000 warmup steps, average decay 0.0005.
- Dropout: 0.1.
- Attention dropout: 0.1.
- No label smoothing.
WMT'13 En→Ru Same as in WMT'14 En→De, except that Dropout is 0.3.
A.1 GPU Power Consumption
Dataset	IWSLT'14 En→De	WMT'14 De→En	WMT'13 En→Ru
GPU(s)	1 × RTX 2080 Ti	2 × RTX 3090	4 × Tesla V100 SXM2
Power consumption, W	250	2×320	4×300
Training time, hours
softmax	15.41	30.06	28.43
sparsemax	24.43	73.33	58.82
1.5-entmax	26.11	79.89	61.20
1.5-ReLU	16.50	31.44	24.21
TOTAL hours	82.44	214.72	172.67
TOTAL kW-hours	20.61	137.42	207.20
GRAND TOTAL kW-hours	365.23
Table 5: Power consumed by GPUs for training.
We do not report  $CO_{2}$  consumption, as experiments were run in different countries, making aggregate statistics difficult to compute. The largest experiment (on WMT'13), were run in France, which benefits from a very low  $CO_{2}$  emission intensity in its electrical mix.
Notation. We let  $\mathbb{R}$  denote the real numbers. Bold-faced lowercase letters (x) denote vectors in Euclidean space, bold-faced uppercase letters (A) denote matrices, plain-faced lowercase letters  $(x)$  denote scalars,  $\| \cdot \|$  denotes the Euclidean norm:  $\| \mathbf{x} \| \coloneqq \sqrt{\mathbf{x}^\top \mathbf{x}}$ . The gradient of  $f: \mathbb{R}^d \to \mathbb{R}$  is denoted by  $\nabla f$ . The Jacobian of  $\mathbf{z} \mapsto g(\mathbf{z})$  is denoted by  $\mathbf{J}_g(\mathbf{z})$ . Also, we denote  $\mathrm{ReLU}(x) := [x]_+ := \max\{x, 0\}$ ,  $[d] := \{1, \ldots, d\}$ ,  $\Delta^{d-1} := \{\mathbf{p} \in \mathbb{R}^d \mid \sum_i p_i = 1, p_i \geq 0\}$ ,  $\mathbf{e}_y := (0, \ldots, 0, 1, 0, \ldots, 0)$  where  $1$  is at  $y^{\text{th}}$  position.
B.1 Proof of Lemma 1.
First, let us calculate the Jacobian of the mapping  $\mathbf{z} \mapsto \alpha\text{-ReLU}(\mathbf{z})$ . Recall that
$$
\alpha \text {- R e L U} _ {i} (\mathbf {z}) := [ (\alpha - 1) z _ {i} - \tau ] _ {+} ^ {\frac {1}{\alpha - 1}}.
$$
Therefore, the partial derivatives are given by
$$
\begin{array}{l} \frac {\partial [ \alpha \mathrm {- R e L U} _ {i} (\mathbf {z}) ]}{\partial z _ {i}} = \frac {1}{\alpha - 1} \cdot [ (\alpha - 1) z _ {i} - \tau ] _ {+} ^ {\frac {1}{\alpha - 1} - 1} \cdot (\alpha - 1) = [ (\alpha - 1) z _ {i} - \tau ] _ {+} ^ {\frac {2 - \alpha}{\alpha - 1}} \\ = [ \alpha - \operatorname {R e L U} _ {i} (\mathbf {z}) ] ^ {2 - \alpha}, \\ \end{array}
$$
$$
\frac {\partial [ \alpha \mathrm {- R e L U} _ {i} (\mathbf {z}) ]}{\partial z _ {j}} = 0. \qquad i \neq j
$$
Thus, the Jacobian can be written concisely as
$$
\mathbf {J} _ {\alpha - \operatorname {R e L U}} (\mathbf {z}) = \operatorname {d i a g} \left\{\left[ \alpha - \operatorname {R e L U} (\mathbf {z}) \right] ^ {2 - \alpha} \right\}, \tag {6}
$$
where raising to power is done component-wise (i.e.  $\mathbf{x}^{\beta} = [x_{1}^{\beta},\ldots ,x_{d}^{\beta}]$ ), and  $\mathrm{diag}[\mathbf{x}]$  is a diagonal matrix with  $\mathbf{x}$  on its diagonal.
Recall the definition of the Tsallis  $\alpha$ -entropy:
$$
\mathrm {H} _ {\alpha} [ \mathbf {p} ] := \frac {1}{\alpha (\alpha - 1)} \left(1 - \sum_ {j} p _ {j} ^ {\alpha}\right).
$$
Its gradient w.r.t.  $\mathbf{p}$  is
$$
\nabla_ {\mathbf {p}} \mathrm {H} _ {\alpha} [ \mathbf {p} ] = - \frac {1}{\alpha - 1} \mathbf {p} ^ {\alpha - 1},
$$
Combining this with (6), and using the chain rule, we have
$$
\begin{array}{l} \nabla_ {\mathbf {z}} \operatorname {H} _ {\alpha} [ \alpha - \operatorname {R e L U} (\mathbf {z}) ] = \left[ \mathbf {J} _ {\alpha - \operatorname {R e L U}} (\mathbf {z}) \right] ^ {\top} \cdot \left(- \frac {1}{\alpha - 1} [ \alpha - \operatorname {R e L U} (\mathbf {z}) ] ^ {\alpha - 1}\right) \\ = \left[ \operatorname {d i a g} \left\{\left[ \alpha - \operatorname {R e L U} (\mathbf {z}) \right] ^ {2 - \alpha} \right\} \right] ^ {\top} \cdot \left(- \frac {1}{\alpha - 1} [ \alpha - \operatorname {R e L U} (\mathbf {z}) ] ^ {\alpha - 1}\right) \\ = - \frac {1}{\alpha - 1} [ \alpha - \operatorname {R e L U} (\mathbf {z}) ] ^ {2 - \alpha} \odot [ \alpha - \operatorname {R e L U} (\mathbf {z}) ] ^ {\alpha - 1} \\ = - \frac {1}{\alpha - 1} \alpha - \operatorname {R e L U} (\mathbf {z}), \tag {7} \\ \end{array}
$$
where  $\odot$  is the Hadamard product (element-wise multiplication), and we used  $\mathrm{diag}[\mathbf{x}] \cdot \mathbf{y} = \mathbf{x} \odot \mathbf{y}$ . Taking into account (7), the gradient of the  $\alpha$ -ReLU loss (3) w.r.t.  $\mathbf{z}$  is
$$
\begin{array}{l} \nabla_ {\mathbf {z}} \ell (\mathbf {z}, y) = \nabla_ {\mathbf {z}} \left[ (\alpha \text {- R e L U} (\mathbf {z}) - \mathbf {e} _ {y}) ^ {\top} \left(\mathbf {z} - \frac {\tau}{\alpha - 1} \mathbf {1}\right) \right] + \nabla_ {\mathbf {z}} \mathrm {H} _ {\alpha} [ \alpha \text {- R e L U} (\mathbf {z}) ] \\ = (\alpha \mathrm {- R e L U} ({\bf z}) - {\bf e} _ {y}) + {\bf J} _ {\alpha \mathrm {- R e L U} ({\bf z})} ^ {\top} \left({\bf z} - \frac {\tau}{\alpha - 1} {\bf 1}\right) - \frac {1}{\alpha - 1} \alpha \mathrm {- R e L U} ({\bf z}) \\ = (\alpha \mathrm {- R e L U} ({\bf z}) - {\bf e} _ {y}) + \frac {1}{\alpha - 1} \left[ \mathrm {d i a g} \{[ \alpha \mathrm {- R e L U} ({\bf z}) ] ^ {2 - \alpha} \} \right] ^ {\top} [ (\alpha - 1) {\bf z} - \tau {\bf 1} ] - \frac {1}{\alpha - 1} \alpha \mathrm {- R e L U} ({\bf z}) \\ = (\alpha \text {- R e L U} (\mathbf {z}) - \mathbf {e} _ {y}) + \frac {1}{\alpha - 1} \underbrace {\left[ (\alpha - 1) \mathbf {z} - \tau \mathbf {1} \right] _ {+} ^ {\frac {2 - \alpha}{\alpha - 1}} \odot \left[ (\alpha - 1) \mathbf {z} - \tau \mathbf {1} \right]} _ {\alpha \text {- R e L U} (\mathbf {z})} - \frac {1}{\alpha - 1} \alpha \text {- R e L U} (\mathbf {z}) \\ = \alpha - \operatorname {R e L U} (\mathbf {z}) - \mathbf {e} _ {y}, \\ \end{array}
$$
where in the fourth line we used  $[\mathbf{x}]_{+}^{\beta}\odot \mathbf{x} = [\mathbf{x}]_{+}^{\beta}\odot [\mathbf{x}]_{+} = [\mathbf{x}]_{+}^{\beta +1}$ . This concludes the proof.
B.2 Approximation of  $\tau$  for 1.5-entmax
We derive the formula (4) in two steps: first in Lemma 2, we approximate  $\tau(\mathbf{z})$  of 1.5-entmax when  $\mathbf{z}$  is an arbitrary random sample from the normal distribution with zero mean and variance  $\sigma^2$ ; next in Lemma 3, we compute  $\sigma^2$  for the case when  $\mathbf{z}$  is the pre-softmax vector of logits in the Transformer model.
Lemma 2. Let  $z_{1},\ldots ,z_{d}$  be independent and identically distributed random variables from the normal distribution  $\mathcal{N}(0,\sigma^2)$ . Then the thresholding function of 1.5-entmax(z) can be approximated as
$$
\tau (\mathbf {z}) \approx \frac {\sigma}{2} \Phi^ {- 1} (1 - p ^ {*}),
$$
where  $\Phi^{-1}(\cdot)$  is a probit function, and  $p^*$  is the solution of
$$
\Phi^ {- 1} (1 - p) = m (p) - \sqrt {\frac {4}{\sigma^ {2}} \cdot \frac {\epsilon}{p} - s (p)}
$$
$$
\begin{array}{l} m (p) := \frac {1}{p - \epsilon} \left[ \phi \left(\Phi^ {- 1} (x) \right] _ {x = \epsilon} ^ {x = p} \right. (8) \\ s (p) := \frac {1}{p - \epsilon} \left[ - \phi \left(\Phi^ {- 1} (x)\right) \cdot \Phi^ {- 1} (x) + x \right] _ {x = \epsilon} ^ {x = p} - [ m (p) ] ^ {2} (9) \\ \phi (t) := \frac {1}{\sqrt {2 \pi}} e ^ {- \frac {t ^ {2}}{2}} \\ \epsilon := \frac {1}{d} \\ \end{array}
$$
Proof. Let  $z_{(1)} \geq \ldots \geq z_{(d)}$  be a sorting of  $z_1, \ldots, z_d$  in descending order. Peters et al. (2019) showed that
$$
\tau (\mathbf {z}) = \frac {M (k)}{2} - \sqrt {\frac {1}{k} - \frac {S (k)}{4}}, \tag {10}
$$
where  $k \in [d]$  is any index that satisfies
$$
\frac {z _ {(k)}}{2} \geq \frac {M (k)}{2} - \sqrt {\frac {1}{k} - \frac {S (k)}{4}} \geq \frac {z _ {(k + 1)}}{2} \Leftrightarrow z _ {(k)} \geq M (k) - \sqrt {\frac {4}{k} - S (k)} \geq z _ {(k + 1)} \tag {11}
$$
$$
M (k) := \frac {1}{k} \sum_ {i = 1} ^ {k} z _ {(i)}, \qquad S (k) := \frac {1}{k} \sum_ {i = 1} ^ {k} z _ {(i)} ^ {2} - [ M (k) ] ^ {2}.
$$
Approximating  $z_{(i)}$  by its asymptotic mean  $\sigma \Phi^{-1}\left(1 - \frac{i}{d}\right)$  (Arnold et al., 2008), and denoting  $\epsilon := \frac{1}{d}$ ,  $p := \frac{k}{d}$ , we have
$$
\begin{array}{l} M (k) \approx \frac {1}{k} \sum_ {i = 1} ^ {k} \sigma \Phi^ {- 1} \left(1 - \frac {i}{d}\right) \approx \frac {\sigma}{p - \epsilon} \int_ {\epsilon} ^ {p} \Phi^ {- 1} (1 - x) d x = \frac {\sigma}{p - \epsilon} \int_ {\epsilon} ^ {p} - \Phi^ {- 1} (x) d x \\ = \frac {\sigma}{p - \epsilon} \left[ \phi (\Phi^ {- 1} (x)) \right] _ {x = \epsilon} ^ {x = p} = \sigma m (p), \\ \end{array}
$$
where we approximated the average of finitely many numbers  $\{\Phi^{-1}(1 - i / d)\}_{i = 1}^{k}$  by the mean value of the function  $\Phi^{-1}(1 - x)$ , and then we used the fact that  $-\phi (\Phi^{-1}(x))$  is an antiderivative for the probit function  $\Phi^{-1}(x)$ ; and  $m(p)$  is defined by (8).
Similarly, for the second empirical moment, we have
$$
\begin{array}{l} \frac {1}{k} \sum_ {i = 1} ^ {k} z _ {i} ^ {2} \approx \frac {1}{k} \sum_ {i = 1} ^ {k} \left[ \sigma \Phi^ {- 1} \left(1 - \frac {i}{d}\right) \right] ^ {2} \approx \frac {\sigma^ {2}}{p - \epsilon} \int_ {\epsilon} ^ {p} [ \Phi^ {- 1} (1 - x) ] ^ {2} d x = \frac {\sigma^ {2}}{p - \epsilon} \int_ {\epsilon} ^ {p} [ \Phi^ {- 1} (x) ] ^ {2} d x \\ = \frac {\sigma^ {2}}{p - \epsilon} \left[ - \phi (\Phi^ {- 1} (x)) \cdot \Phi^ {- 1} (x) + x \right] _ {x = \epsilon} ^ {x = p}, \\ \end{array}
$$
$$
S (k) \approx \frac {\sigma^ {2}}{p - \epsilon} \left[ - \phi (\Phi^ {- 1} (x)) \cdot \Phi^ {- 1} (x) + x \right] _ {x = \epsilon} ^ {x = p} - [ m (p) ] ^ {2} = \sigma^ {2} s (p),
$$
where  $s(p)$  is defined by (9). Hence, finding  $k \in [d]$  that satisfies (11) is (approximately) equivalent to finding  $p \in (0, 1)$  that satisfies