Title: Tight Asymptotics of Extreme Order Statistics
Venue: NeurIPS 2025
Accepted: yes
Keywords: Order statistics

Asymptotic analysis
Abstract: A classic statistical problem is to study the asymptotic behavior of the order statistics of a large number of independent samples taken from a distribution with finite expectation. This behavior has implications for several core problems in machine learning and economics—including robust learning under adversarial noise, best-arm identification in bandit algorithms, revenue estimation in second-
price auctions, and the analysis of tail-sensitive statistics used in out-of-distribution detection.

The research question we tackle in this paper is: How large can the expectation of the $\ell$-th maximum of the $n$ samples be? For $\ell=1$, i.e., the maximum, this expectation is known to grow as $o(n)$, which can be shown to be tight. We show that there is a sharp contrast when considering any fixed $\ell>1$. Surprisingly, in this case, the largest possible growth rate for all fixed $\ell>1$ is $O(\frac{n}{\log(n)\log\log(n)})$ and $\Omega(\frac{n}{\log(n)(\log\log(n))^{1.01}})$. Our result is actually finer than the latter and provides a sharp characterization of the largest achievable growth rate for the expectation of the $\ell$-th maximum of $n$ i.i.d. samples. 

Beyond the theoretical analysis, we support our findings with extensive simulations. These empirical results highlight a notable phenomenon: although the multiplicative gap between the maximum and the second maximum grows quickly with 
$n$, the ratio remains approximately constant in 99\% of trials. This suggests that while worst-case growth is sharp and meaningful, typical-case behavior may be significantly more stable.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the behavior of the expected $\\ell$-th largest order statistics from $n$ i.i.d. samples (i.e., $(n-\\ell+1)$-th order statistic),  focusing on the case $\\ell \\ge 2$.\nThe authors show that, for all fixed $\\ell \\ge 2$, the growth rate of this expected value should be $O(\\frac{n}{\\log(n)\\log\\log(n)})$.\nFurthermore, they establish a characterization: for a distribution with finite mean, the condition $M_\\ell(n) = \\Omega(g(n))$ holds if and only if the monotonically increasing sublinear function $g(n)$ satisfies $g(n)$ satisfies$\\sum_n g(n)/n^2 < \\infty$, where $M_\\ell(n)$ denotes the expected $\\ell$-th largest order statistics.\n\nSTRENGTHS AND WEAKNESSES:\n### Strength\n\n* New theoretical insight for all fixed $\\ell\\geq 2$.\n\nThe paper illustrates how the expected $\\ell$-th largest order statistic, $M_\\ell(n)$ behaves fundamentally differently from the maximum order statistic. For distribution with the finite expectation, it was shown that $M_1=\\Omega(g(n))$ holds for any sublinear $g(n)$ [8], but the authors show that $M_{\\ell \\geq 2} = \\Omega(g(n))$ can hold iff $\\sum_n g(n)/n^2 <\\infty$. \n\n* Validation of theoretical results by numerical experiments: Simulations cover both finite-variance and infinite-variance cases and track the empirical rate against the theoretical upper– and lower–bound results, giving concrete support to the asymptotic claims.\n\n### Weakness\n\n1. The paper establishes the theoretical behavior of $M_\\ell$ that is distinct to $M_1$, but it largely leaves readers on their own to understand what these results mean for concrete examples or for practical tasks. (although there are general applications in Section 1)\nTherefore, it would be beneficial to have some discussions on the obtained results.\n\nFor example, while the authors explained that it is impossible to derive a specific sublinear function $g(n)$ satisfying $M_\\ell^F(n)=o(g(n))$ for given distribution $F$, one may wonder what $M_\\ell^F(n)$ looks like for familiar distributions that could be written in the closed formulation beyond $\\delta_k$ expression.\nIn this context it would be useful to illustrate the results with Fréchet-type heavy-tailed distributions, those whose tail quantile function is regularly varying with index $a>0$, i.e. $x_k = a k^a o(k^{\\epsilon})$ for any $\\epsilon>0$  (see Charras-Garrido & Lezaud, 2013 for more details).\nA classical example is the Pareto distribution with shape parameter $b>0$, for which $a=1/b$.\nFor such tails one can compute explicitly the sublinear order of $M_\\ell^F(n)$ making the sub-linear growth far more transparent than the abstract $\\delta_k$ expression.\nIncluding such examples would let readers grasp the sub-linear growth rate of $M_\\ell^F$ at a glance.\n\n2. The theorem is phrased “for every integer $\\ell\\geq 1$\", but the proof of Lemma 4 (Line 688) assumes $i$ is small enough relative to $n$ to ensure\n$$ \n\\left(1-\\frac{i^2}{n} \\right)\\left(1-\\frac{1}{n} \\right) e^{i/k} \\geq (1-1/e), \\quad k \\geq n+1.\n $$\nFor example, $i=7, n=122, k=123$ violates the inequality, so we need $n\\geq 123$ for fixed $i=7$.\nFor this reason, I think the statement of Theorem should be restated with an explicit range such as $n \\geq n_0(\\ell)$ for fixed $\\ell$ or for integer $\\ell$ small enough relative to $n$.\n\n---\nMyriam Charras-Garrido and Pascal Lezaud. Extreme value analysis: an introduction. Journal de la\nSoci´et´e Franc¸aise de Statistique, 154(2):66–97, 2013.\n\nQUESTIONS:\n1. If there are any misunderstandings in my assessment of the Strengths and Weaknesses, I would greatly appreciate it if you could point them out. It would be helpful to receive further explanation or clarification where appropriate.\n2. Could you elaborate on the distinctions among the three regimes of interest:\n(i) the central/quantile case (e.g., the median or fixed-rank quantiles),\n(ii) the extreme case (as in this paper, dealing with the largest or near-largest order statistics), and\n(iii) the intermediate case?\nFor example, the different motivations for studying each regime and how results from each are applied or interpreted in practice.\n\n3. I would also appreciate further insight into the contribution of this paper, especially in terms of why the results are theoretically or practically significant. For example, how such tight asymptotics for extreme order statistics (rather than just sample mean) particularly beneficial in best-arm identification examples? \n\n--- \n### Typos\n1. Subsection title 1.2. : overivew -> overview\n2. Citation [14]: Daniel Levy and et al. -> need to be revised.\n3. In the proof of Proposition 6, missing spaces in Lines 708 and 709: escalatordistribution -> escalator distribution.\n\nLIMITATIONS:\nyes", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 2, "CLARITY": 2, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-06-14"}

comments: SUMMARY:
This paper studies the behavior of order statistics, and specifically, the expectation of order statistics beyond the maximum. Previous work shows that for all constant distributions the expected maximum of n samples is $o(n)$, and this is tight. This paper studies the growth rate for the \ell-th order statistic. The picture is rather subtle: there exist infinitely many distributions and infinitely many n such that the \ell-th order statistic is an arbitrary sublinear function. However, there are limits to what can be done for a fixed distribution and all n. Specifically, $\Omega(n/(log(n) loglog(n) )$ is not possible for all n, but $\Omega(n/(log(n) loglog^{1+eps}(n) )$ is, for all $\eps > 0$.

STRENGTHS AND WEAKNESSES:
This is a nice, well-written paper that studies a fundamental problem in probability. The paper completely answers the question it sets out to solve: get bounds on the growth rate of the expected value of the \ell-th order statistic. The only weakness is that the scope might be a bit narrow. I.e., this problem is of interest to a small sub-community of NeurIPS, e.g., statisticians, or researchers in auction theory (the expected k-th order statistic is the revenue of a k-position auction). Still, I think that asking that a NeurIPS paper is of interest to everyone who attends NeurIPS is impossible, so I am in support of accepting this work.


======

Here are some typos (not sure where else I should point them out):
Typo in line 54: “ℓ-the” should be “ℓ-th”
Typo in line 55: “Our necessary condition for to have”
Typo in line 156: “escalatorprobability”

QUESTIONS:
N/A

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 3
IMPACT: 4
ORIGINALITY: 4
DATE: 2025-06-18

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors consider n iid random variables drawn from a distribution F and are concerned with the problem of the asymptotic (largest possible) grow of extreme order statistics (in the following, I denote with $M_1$ with the max, and $M_n$ with the minimum).\nFor the maximum it is well-known that $M_1 \\in o(n)$ and for any sub-linear function $g(n)$ there exists a distribution $F$ such that $M_1(n) \\ge g(n)$.\nThe authors thus focus on the case $l > 1$. Specifically, they show that for all $l$, the largest possible grow is $O(n/(log(n) log(logn))) and there exists a distribution such that the growth is $n/(log(n) log(logn)))^(1.01)$.\nThe authors also run experiments which shows how the although the multiplicative gap between the maximum and the second maximum grows quickly with n, the ratio remains approximately constant in 99% of trials.\n\nSTRENGTHS AND WEAKNESSES:\n# Strengths \n- This paper fills a well-motivated theoretical problem (see the applications subsection in Section 1)\n- The resulting characterization of the extreme order statistics is tight and interesting\n- The main text is overall well-written and clear\n\n# Weaknesses\n- Not much. Apart from the question below, and some (probably) **errors** that I have found in a proof (which I believe can be fixed without impacting the results). Below, I provide some detailed comments on the proofs together with the details on those errors. **Please, fix them (or correct me) during the rebuttal. For the moment, my score is Borderline Accept, but it is a placeholder. (I will increase it / decrease it based on the output of the rebuttal).**\n\n# Clarity of the appendix\nOverall, given the few pages in the appendix, I suggest the authors to provide more explicit steps while deriving their results. While proof-reading the results, I needed to spend some time verifying some (actually many) algebraic steps. Even a small tag near an inequality to justify a step would help with the workload on the review process a lot. I would also encourage the authors to provide more intuition on how the results were derived (i.e., more details on the main intuition behind the proofs). \n\n# Comments on the proofs\n### On Lemma 3 \n- Not really clear to me the comment on line 680\n  \n### On Lemma 4\n- In the proof of the upper bound, it seems that it is missing an $e^{n/k}$. Indeed, you are only reporting the upper bound on the Poisson. So maybe the result should involve an $e^2$? \n- For the LB, it is not clear how you use \"for $i$ small enough relative to $n$\" (line 688). It seems to me that it might be in the last step to assume that everything is positive, so that you can further proceed by lower bounding the binomial. Also, that last step is not clear to me. Please justify.\n- Related to the above. Given the statement of the Lemma, it is not clear that $n$ should be large enough wrt $l$.  \n- It might be nice to comment a bit in the paper on why a non-trivial UB is needed in Lemma 4, while a trivial UB can be used in Lemma 3\n\n### On Theorem 5\n- In the proof of Theorem 5, it seems that the authors are applying Proposition 2 for $F^{-}$. The result, however, is stated and proved only for $F^+$.  Although the upper bound holds, it is not clear how the authors are proceeding for the LB. (p.s., I can see that $F^+$ and $F^-$ are naturally close to each other in shape, and so that similar arguments could hold, in this sense I believe that the result is probably correct, but can you provide a formal argument for this?)\n- In Eq. 7, the upper bound does not seem to be needed in the rest of the proof. Right?\n\n### On Lemma 6 \n- Is there an extra n/x in the equation below line 740?\n- The equality after the \"Since...\" does not hold (line 742). There is an $i$ missing at the numerator so that we have n chooses i * i/n = (n - 1) chooses (i-1).\n- I feel that this should not be a problem for the statement, but can you please provide a corrected proof of the statement? \n\n# Other Comments\n- Typos: below line 156 \"escalatorprobability\"\n- Line 210, reference missing\n- Lines 708, 709 \"escalatordistribution\"\n- Suggestion: In the introduction, stress slightly more that $l$ fixed implies that we have no guarantees, e.g., for the min, as $l$ needs to be constant and hence cannot grow with $n$\n- Suggestion: A table somewhere would probably help the reader to maintain a clear big picture, i.e., the existing results + the new results with reference to the various theorems/proposition and their implications\n- Sentence at lines 220-221 not clear to me\n- Appendix A.9 is never mentioned in the main text\n- Proposition 5 states that for any sublinear $g(n)$, there exists a distribution $F$ such that $M^F_l(n) \\ge g(n)$ for infinitely many $n$'s. Could you comment more on why this implies the comment on lines 206-208? i.e., the fact that for a fixed $F$ one cannot find a specific sublinear function such that $M^F_l(n) \\in o(n)$ (e.g., maybe specifying with words that $g$ cannot depend on $F$ would help the reader)\n- Also, for the sake of clarity, it might be useful for the reader to write somewhere the implications of liminf and limsup and how they relate to the asymptotic notation\n\nQUESTIONS:\nCan you comment more on the experiments of Figure 3 and Corollary 1? It seems that the growth of M_2 is approximately $(n/log(n))^{1/2}$ (which contradicts the theory). Is this because the simulated values of n are too small?\n\nLIMITATIONS:\nTheoretical work. I do not foresee a direct path to potential negative societal aspects.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-25"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper answers a basic question in the asymptotic behaviour of the expectation of the second, and more generally $\\ell$’th maximum of i.i.d. random variables.\n\nWhile the case of the max ($\\ell=1$), it is known that the growth rate can be up to $o(n)$ - in the sense that there is a linear upper bound known, but there exists a finite expectation  distribution $F$ such that $M_{1}(n)=\\Omega(g(n))$ - the growth rate for $\\ell>1$ is tightly characterised as both $O(\\frac{n}{\\log(n)\\log\\log(n)})$ and $\\Omega(\\frac{n}{\\log(n)(\\log\\log(n))^{1.01}})$. \n\nThe authors also introduce a reduction to what is referred to as an escalator distribution; a convenient measure discretisation of a a cumulative distribution $F$, the order statistics of which both upper and lower bound the corresponding order statistics of F. \n\nTheory is supplemented with extensive experiments  and characterising behaviour of empirical vs. worst-case separations.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n* The paper offers a surprising perspective on the behaviour of the $\\ell$’th order statistic for $\\ell > 1 $ with extremely general and tight results (up to a minuscule $\\log\\log(n)^{0.01}$ factor), applying to all distributions with finite expectation.\n* The techniques, notably the escalator distribution reduction, are elegant and provide a clean path to bounding asymptotics of order statistics rigorously.\n* The applicability of the results is broad, including value-at-risk estimation, option pricing, auction theory, load balancing in compute, and - as noted in Section 1.1 - several high-impact areas in machine learning theory (e.g., adversarial risk, best-arm bandits, OOD detection).\n* The paper zeroes in on concrete cases (finite variance, higher moments), illustrating how growth rates degrade systematically under additional distributional constraints.\n* Results fill a natural and under-explored gap in the literature between known results for the maximum and the poorly understood behaviour of lower extreme order statistics.\n\nWeaknesses:\n- Presentation is somewhat notation-heavy, and highly technical, which may limit accessibility for non-theoretical or applied ML audiences. Considering the theoretical interest of results, this is a rather minor weakness.\n* The paper could do more to explicitly connect theoretical findings to practical thresholds, and provide concrete examples like those elaborated in Section 1.1.\n\nQUESTIONS:\nQuestions:\n* The escalator distribution reduction is elegant. Is there a possibility of extending this technique to dependent sample scenarios, or is independence fundamentally required for the escalator arguments to hold?\n* Given the broad applicability, could you clarify practical guidelines for using your results in robust risk or OOD detection pipelines? Specifically, how might practitioners identify whether their system is in a “regime” where your bounds meaningfully constrain the top $\\ell$ scaling - is it as simple as characterising the maximal degree of finite moments?\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 4, "ORIGINALITY": 4, "DATE": "2025-07-01"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies the worst-case asymptotics of extreme order statistics, e.g. $M_{\\ell}(n)$  the $\\ell$-th greatest value among $n$ i.i.d. positive random variables, for a fixed $\\ell$ as $n$ goes to infinity. For $M_1(n)$, is has been previously shown that $M_1(n)=o(n)$, and $\\liminf M_1(n)/g(n)>0$ for any $g(n)$ sublinear in $n$ tightly characterizing the worst-case asymptotics. Contrasting with this previous result, this papers shows that while $\\limsup M_{\\ell}(n)/g(n) >0$ remains true for all sublinear functions $g(n)$, it does not necessarily hold anymore for $\\liminf$. In particular, it is shown that $M_{\\ell}$ consistently dominates $g(n)$ if and only if $\\sum_n g(n)/n^2<\\infty$, providing a complete picture on the case $\\ell>1$. Additional results are provided when additional moments are finite.\n\nSTRENGTHS AND WEAKNESSES:\n## Strengths \n\n- It provides a comprehensive theoretical analysis of the extreme order statistics, and highlights a very non-intuitive subtle difference in asymptotic behavior between $M_1$ and $M_{\\ell}$ for $\\ell>1$. \n- The paper is clearly written and the related works are precisely presented.\n\n## Weaknesses \n\n- Given that this paper is based on a quite technical aspect of limit behavior of order statistics, it would be better to be as unambiguous as possible on the meaning of the $\\Omega(g(n))$ and other Landau notations. In some fields, these notations sometimes implicitly assume that the ratio $f(n)/g(n)$ converges, which is clearly not the case here as $\\limsup$ and $\\liminf$ are distinct. \n- The paper is lacking a bit in terms of intuition in why there is such a difference between $M_1$ and $M_2$, and why there is a difference in terms of $\\limsup$ and $\\liminf$. Why does this phenomena not appear at $\\ell=4$ for instance?\n- The paper mentions potential applications, and while I really appreciate the result, I still struggle to see how this would concretely impact existing bounds or results in other topics. It would be great to highlight on a small example possible consequences, for instance to detail how the the main result concretely affects BAI (being one of the example suggested).\n- This paper would benefit from additional visual aids, such as plotting the escalator transformations and the worst case distributions. It might be a good use of the additional space to add these plots.\n\nQUESTIONS:\n- By the fact that the threshold $g(n)$ does not depend on $\\ell$, does this suggest that all of the $M_{\\ell}$ for $\\ell>1$ have all the exact same worst-case asymptotic behavior? Why is that the case?\n- If we have that $\\sum_n g(n)/n^{\\ell} <\\infty$, can we say anything more about $M_{\\ell}$? \n- What can we say about the asymptotics of order statistics gaps $M_{\\ell}-M_{\\ell+k}$?\n- line 679, why can we upper bound $1/(k-1)^i$ by $1/(n-1)^i$ given that $k\\leq n$? I must have missed something obvious.\n\nThere are some small typos to correct:\n- line 55 Conditions for to have\n- line 221 There exists infinite ... sublinear functions. (confusing sentence)\n\nLIMITATIONS:\nYes.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}