Title: Sampled Estimators For Softmax Must Be Biased
Venue: NeurIPS 2025
Accepted: yes
Keywords: Softmax

Negative Sampling

Extreme Classification
Abstract: Models requiring probabilistic outputs are ubiquitous and used in fields such as natural language processing, contrastive learning, and recommendation systems. The standard method of designing such a model is to output unconstrained logits, which are normalized into probabilities with the softmax function. The normalization involves computing a summation across all classes, which becomes prohibitively expensive for problems with a large number of classes. An important strategy to reduce the cost is to sum over a sampled subset of classes in the softmax function, known as the sampled softmax. It was known that the sampled softmax is biased; the expectation taken over the sampled classes is not equal to the softmax function. Many works focused on reducing the bias by using a better way of sampling the subset. However, while sampled softmax is biased, it is unclear whether an unbiased function different from sampled softmax exists. In this paper, we show that all functions that only access a sampled subset of classes must be biased.
With this result, we prevent efforts in finding unbiased loss functions and validate that past efforts devoted to reducing bias are the best we can do.


Peer reviews:
comments: SUMMARY:
This paper proves that an unbiased estimator based on sampled indices for full softmax must be biased for general cases.

STRENGTHS AND WEAKNESSES:
**Strength**
- The results in the manuscript are mathematically sound with rigorous and detailed proof presented. The assumptions are mild, practical and reasonable.
- The conclusion is of interest to machine learning systems involving millions of possible classes to predict.

**Weaknesses**
- The results simply prevent investigations for finding unbiased estimators but do not provide any other deeper insights such as how to reduce the bias with the help of conclusions in the paper.

QUESTIONS:
- Line 122. Positive classes are not defined. Moreover, what if the positive classes of other instances are identical to the current instance? Are they still so-called negative?
- Assumption 2.3 excludes adaptive sampling methods. The authors argue that this requires computing the full softmax. But is it possible to select sampling distribution after obtaining logits with a sampling approach rather than computing the full softmax?
- Is it possible to quantify the bias?
- How do existing techniques align with the paper's conclusion? Are there inspirations for new bias-reduction methods if we still adopt the biased estimator?
- The sampled estimator is biased for general cases and the paper proves it by crafting counter-examples. What are the cases / conditions under which the estimator is unbiased?

LIMITATIONS:
Yes.
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-27

comments: SUMMARY:
The paper provides a proof that no unbiased sampled estimator of the softmax function exists. The authors argue that this long suspected but never formally proven property could guide research into future softmax estimators.

The authors also give a thorough review of several related works which attempted to mitigate or avoid this bias.

The paper’s proof leverages a contradiction, showing that an estimator cannot exist which is unbiased in two cases: one where all logits are constant and one where all logits take on one of two values.

STRENGTHS AND WEAKNESSES:
The provided proof is clear, formal, and relatively easy to follow. While the simplicity of the proof does not disqualify it from publication, it represents the sole output of the paper, meaning the bar for novelty and impact should be set high.

Unfortunately, the applications of the proved result appear relatively limited. No scenarios come to mind in which one would expect a softmax estimator to be unbiased across the entire space of logit vectors.

Though this analysis certainly helps to guide the conversation, it seems reasonable to expect further analysis from a publication of NeurIPS’ caliber (e.g., bounds on the bias based on k or distributional assumptions under which an unbiased estimator is possible).

QUESTIONS:
Q1 What sorts of distributional assumptions on the logits would be required to construct an unbiased estimator for k << n?

Q2 Given a fixed k << n, can one get close to an unbiased estimator?

Q3 Does this property have implications for adaptive softmax estimators or softmax gradient estimators?

LIMITATIONS:
yes
TITLE: Official Review
CLARITY: 4
IMPACT: 2
ORIGINALITY: 2
DATE: 2025-06-30

comments: SUMMARY:
The authors consider the problem of approximating the softmax using sampling of its arguments. They prove theoretically that any sampling-based estimator that samples a fixed number of logits (and that is not allowed to depend on the logits) must necessarily be biased.

STRENGTHS AND WEAKNESSES:
Strengths:
- The negative result provided by the paper is interesting, and supports / encourages the development of efficient biased estimators

Weaknesses: 
- While the construction provided rules out unbiased estimates for the softmax, I don’t think it technically rules out unbiased estimates of, e.g., the log softmax (which is typically used in loss computation), which would be sufficient to obtain unbiased stochastic gradient estimates.  It would be great if the authors could comment on this.
- Moreover, the theoretical analysis rules out *exactly* unbiased estimators, leaving open important questions of approximability (see below).

QUESTIONS:
- Does the analysis shed any light on the difficulty of *approximating* the softmax?  This would be useful for estimators that navigate a bias-variance tradeoff.
- The analysis also doesn’t seem to rule out estimators that use a stochastic number of samples (e.g., based on a stopping rule), correct?  If it does, it would be great to point it out.

LIMITATIONS:
Not explicitly discussed; some of the points mentioned above could be mentioned.
TITLE: Official Review
CLARITY: 3
IMPACT: 2
ORIGINALITY: 3
DATE: 2025-06-30

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nIn this work, the authors address a problem related to the use of the softmax function over a large number of classes. To handle the computational cost of the full softmax, practitioners often use a sampled softmax, which is known to be a biased estimator. This paper proves that any estimator that relies on a sampled subset of classes must be biased, under the assumption that the sampling distribution is independent of the logit values. The main contribution is a formal impossibility proof that confirms a widely held belief in the field, suggesting that research should continue to focus on bias reduction rather than elimination.\n\nSTRENGTHS AND WEAKNESSES:\n#### Strengths\n\n1. The paper provides a solid and rigorous proof for a widely-held, but previously unproven, belief in the machine learning community. Formalizing this impossibility result is a valuable contribution that helps solidify the theoretical foundations of large-scale classification.\n\n2. The paper is well-written, and the proof is presented clearly. The logical flow is easy to follow, and the authors do a good job of setting up the problem and their assumptions. The technical quality of the mathematical argument is high.\n\n#### Weaknesses\n\n1. While the result is a useful formalization, it proves something that was already strongly suspected and implicitly assumed by most researchers in the area. Therefore, the work confirms the existing research direction (bias reduction) rather than opening up new ones. Its overall impact might be limited.\n\n2. The work is entirely theoretical. It does not offer new insights into the nature or magnitude of the bias for different estimators, nor does it provide any guidance for designing better-biased estimators. The lack of any empirical validation or practical takeaways makes the contribution feel incremental.\n\nQUESTIONS:\n1. The proof relies on constructing specific configurations of the logit values (e.g., all equal, or split into two groups). This is a valid and powerful proof technique. Have the authors considered alternative proof strategies, perhaps from information theory or functional analysis, that might offer a different perspective on why this unbiased estimation is impossible?\n\n2. The problem definition and proof are based on sampling k indices from n without replacement. Does the impossibility result also hold for sampling with replacement? It seems intuitively that it should, as this would be a less informative sample.\n\nLIMITATIONS:\nYes, the authors have adequately addressed the limitations. They are very clear that their work is a theoretical impossibility proof and does not provide a new algorithm.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 3, "IMPACT": 2, "ORIGINALITY": 2, "DATE": "2025-07-03"}