A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding
Authors: Alexander Koujianos Goldberg
Giulia Fanti
Nihar B Shah
Venue: NeurIPS 2025 spotlight ()
Keywords: peer review
randomized top-k selection
robust optimization
ranking
game theory
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: Many decision-making processes involve evaluating and selecting items, including scientific peer review, job hiring, school admissions, and investment decisions. These domains feature error-prone evaluations and uncertainty about outcomes, which undermine deterministic selection rules. Consequently, randomized selection mechanisms are gaining traction. However, current randomized approaches are ad hoc and, as we prove, inappropriate for their purported objectives. We propose a principled framework for randomized decision-making based on interval estimates of item quality. We introduce MERIT (Maximin Efficient Randomized Interval Top-$k$), which maximizes the worst-case expected number of top candidates selected under uncertainty represented by overlapping intervals. MERIT provides optimal resource allocation under an interpretable robustness notion. We develop a polynomial-time, practically efficient algorithm and prove our approach satisfies desirable axiomatic properties not guaranteed by existing methods. Experiments on synthetic peer review data from grant funding and conferences demonstrate that MERIT matches existing algorithms' expected utility under fully probabilistic models while outperforming them under our worst-case formulation.

A Principled Approach to Randomized Selection under Uncertainty: Applications to Peer Review and Grant Funding
Alexander Goldberg
Giulia Fanti
Nihar B. Shah
Carnegie Mellon University fakgoldbe, gfanti, nihars minaaraaaaddreee Saadrrs).adddwe.cccmm edu
Abstract
Many decision-making processes involve evaluating and selecting items, including scientific peer review, job hiring, school admissions, and investment decisions. These domains feature error-prone evaluations and uncertainty about outcomes, which undermine deterministic selection rules. Consequently, randomized selection mechanisms are gaining traction. However, current randomized approaches are ad hoc and, prove, inappropriate for their purported objectives. We propose principled framework for randomized decision-making based on interval estimates of item quality. We introduce MERIT (Maximin Efficient Randomized Interval Top-k), which maximizes the worst-case expected number of top candidates se- lected under uncertainty represented by overlapping intervals. MERIT provides optimal resource allocation under an interpretable robustness notion. We develop a polynomial-tine time, practically efficient algorithm and prove our approach satisfies desirable axiomatic properties not guaranteed by existing methods. Experiments on synthetic peer review data from grant funding and conferences demonstrate that MERIT matches existing algorithms' expected utility under fully probabilistic models while outperforming them under our worst-case formulation.
Introduction
In many applications like scientific funding, job hiring, school admissions, and startup investment, decision makers evaluate and select items based imperfect assessments. Recently, there has been growing interest in introducing randomization into selection processes to address uncertainty in evaluations, reduce reviewer burden, encourage high-risk proposals, and combat reviewer partiality. In fact, many funding agencies have already adopted partial lotteries to allocate grant money, starting with the New Zealand Health Research Council in 2013 (3T). followed by the Swiss NSF in 2019 (1], and recently expanding to numerous agencies worldwide 114
50 58 Proposals for randomization have also emerged in college admissions [I 25 job screening 40 and startup investment (35].
In current deployments, decision makers collect peer-review assessments and run a lottery where selection probabilities derive from review scores. However, current procedures are ad hoc. We initiate a principled approach to randomizing decisions from evaluations, focusing on the question: Given imperfect evaluations of candidate quality, what is a suitable probability distribution over applicants for random selection?
We take the perspective of a funder selecting the highest quality grant proposals. A key motivation for randomization is uncertainty about relative quality. Many existing deployments describe peer review lotteries as random "tie-breaking" between proposals of equal quality [36] 53). Similar to
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

previous work [2I]. we assume the funder estimates numeric quality intervals for each proposal. These intervals may capture uncertainty due evaluation errors, miscalibration, subjectivity, or aleatoric uncertainty about future success. Crucially, we capture settings with "Knightian uncertainty,"
where a funder cannot assign a probability measure over possible outcomes [26-= a model widely
applied in policy-making [52 45 [7, financial investment [13 33 37 and R&D investment [3]
This assumption is particularly apt for peer review, where probabilistic models of reviewer errors have performed poorly in real deployments 51 Section "Miscalibration" J. decision makers lack ground truth data to evaluate model appropriateness, and future success is inherently difficult to predict [49 12 60 22). Without reliable probabilistic model, we assume decision makers describe uncertainty through quality intervals rather than point estimates. The funder draws conclusions only about relative ordering: overlapping intervals indicate insufficient evidence to conclude one proposal is better, while non-overlapping intervals indicate dominance. We develop theory and a practical algorithm for making funding decisions from interval quality estimates, with the following key contributions:
1. Modeling uncertainty as Knightian uncertainty intervals: Prior work assumes known probabilistic relationships between quality and scores 21 We show that in such fully Bayesian settings, deterministic selection always maximizes expected utility, making randomization unnecessary. Our model captures the motivation for randomizing using "Knightian" uncertainty intervals.
2. A principled approach to randomization: We formalize two key principles: ante optimal- ity maximizing worst-case utility over all rankings consistent with intervals-ant and ex post valid- respeccnn strict dominance relationships. Prior heuristic rules fail to satisfy both principles.
3. Efficient algorithm: We develop a polynomial time algorithm solving the maximin optimization problem, despite related graph problems being NP-hard [15] 42 61 2]. Our Maximin Efficient Randomized Interval Top-k (MERIT) algorithm runs in under 5 minutes on 10,000+ candidates on a standard laptop. Implementation available at alggttuu..... . com/ akgoldberg/Lottery
4. Axiomatic comparison: We initiate an axiomatic approach to comparing randomized mechanisms, identifying desirable properties of "monotonicity in budget", "stability", and "reversal symmetry." We prove MERIT prevents "maximal instability" and respects "reversal symmetry" while existing mechanisms do not.
5. Empirical comparison: We evaluate MERIT against existing methods using synthetic data based on real peer review data from conferences (NeurIPS 2024, ICLR 2025) and grant agencies (Swiss NSF 2020). MERIT performs comparably to existing methods in expected utility under linear reviewer error model used by the Swiss NSF (21) and many other prior works [16] 5 43 44). However, under our worst-case objective, algorithm significantly outperforms deterministic selection and the Swiss NSF's randomized approach.
For clarity of presentation, all formal proofs are deferred to Appendix A
2
Background and Approach
We begin by describing current deployments of randomized decisions in scientific funding and then motivate our approach.
2.1 Existing Deployments
In recent years, there have been many deployments of "peer review lotteries" in scientific funding decisions. Most deployments use an approach of siltoomiee-abvvethreetttee
Under this approach, the funder chooses minimum acceptable quality threshold and samples uniformly at random among all proposals that are above this threshold. This approach has been adopted by numerous funding agencies 14 55 58 55 31 Council, the British Academy (55) and as means of allocating oral presentations at the USENIX Security Conference [37]. However, as we describe in Section 3 randomize-above thrreshldd may violate desired principle of "ex post validity", which says that one proposal clearly dominates another, the stronger proposal should be funded if the weaker
proposal is funded.
Taking a different approach, the Swiss National Science Foundation (NSF) method that explicitly accounts for uncertainty about the quality of each proposal. They assume that
21 56
pioneered
2

---

each proposal has a latent true quality and review scores are generated based on these quality scores and reviewer-specific noise parameters. They assume priors on the model parameters and develop methods to obtain point estimates and confidence intervals for the true quality of each proposal. As described in Algorithm X the Swiss NSF then samples & proposals by setting "provisional funding line" as the k-th highest point estimate. All proposals with intervals strictly above the funding line are selected and all proposals strictly below the funding line are rejected. The remaining budget is allocated uniformly at random among proposals with intervals that overlap the funding line. We provide pseudocode for the Swiss NSF method and wandomize-above-trrshhee in Appendix E
The intuition provided by the Swiss NSF for their approach is that confidence intervals capture the funder's uncertainty estimating the proposal quality and randomizing decisions accounts for this uncertainty, thereby leading to better decisions. In contrast, as we prove in Theorem A. when the funder assumes that review data is generated from a fully specified Bayesian model, there exists deterministic selection of proposals that maximizes the funder's s expected utility for any utility function. Informally, if the funder knows the model that generates their data, then they do not need to randomize in order to maximize their expected utility. We present the formal proposition and proof in Appendix A. This result suggests one drawback of the Swiss NSF's method-in their model setting there may be a utility cost to randomization compared to choosing deterministically in an optimal manner. A second drawback is that the Swiss NSF's algorithm for selecting proposals from intervals violates natural axioms for selection rule, like monotonicity in the budget i and stability, show in Section 5
2.2 Our Approach
In our work, we propose model that captures the motivation for randomizing due to uncertainty about the relative quality of the proposals. We show that if the funder cares about their worst case utility they must randomize decisions in order to robustly optimize their utility.
Specifically, we consider a funder who estimates intervals for each proposal based on data. These intervals need not come from any one particular model, but they should capture the funder's inherent uncertainty about the relative quality of proposals.
Our interpretation of the quality intervals stems from the intuitive argument that if the intervals for two proposals overlap then the funder does not have enough evidence to distinguish between them. On the other hand, if the interval of proposal A dominates the interval of proposal B, then the funder has sufficient evidence to believe that A is better than B. Hence, the intervals define a partial ordering of proposals that represents set of conclusions by the funder regarding the relative quality of different proposals. This ordering is the canonical "interval order" for a set of intervals. It captures the spirit of how the Swiss NSF interprets confidence intervals in their setup iitcheyard00222ethiiningg
In practice, such intervals can arise from a variety of sources, including missing-date imputation, model ensembling, expert imput, multi-criteria aggregation, or robustness model misspecification for confidence intervals. We provide detailed explanations of such intervals in Appendix
3 Problem Formulation
Our method applies settings like admissions, scientific peer review, job screening, and financial investment, where decision makers estimate quality intervals and select top candidates based on these intervals. For concreteness, throughout our exposition, will describe a funder choosing proposals.
Consider a funder who receives n proposals. From these, the funder wishes to select the highest quality proposals. Note that n could be as large as thousands of proposals and & a fixed fraction of the total and can also be in the hundreds or thousands. For each proposal E [m: [ the funder estimates an interval [f,, u.] C R representing range of quality scores that the proposal could possibly take. A higher score indicates higher quality. The funder wishes to design a randomized selection mechanism to choose h proposals given the intervals. In order to design such mechanism, we adopt two primary principles which we term as ex ante optimality and ex post validity, described below.
'ee use the standard notation [x] to denote set (l...., xe for any positive integer
3

---

Ex ante optimality The funder's utility is the expected number of the true top-k proposals that they select, ranked by their quality. Formally, let [n|] + [n] denote a ranking of the proposals where for each proposal i E [n]] the rank of the proposal is denoted by o E [n]. If the funder samples proposals with marginal probabilities p E [0, 1)ah and the true ranking of proposals a, their expected utility is D P; { (i) & ke. Clearly, if the funder knew the true ranking a, then they would optimize utility by choosing deterministically. i.c., by setting pt = 1 for with o S k, and 0 otherwise.
However, recall that the funder uncertain about relative qualities of the proposals, as captured by overlaps in intervals. Any ordering that is consistent with overlapping intervals could be the true ranking. Hence, the intervals define a set of feasible rankings:
Σn = (a permutation of [n] Vi,j E [m], 4 4>43 = <o())
In other words, proposal has quality strictly above proposal j. then is ranked higher than j in all E En, As an example of an extreme case, if all n intervals overlap each other, then Σ,, consists of all possible permutations of the proposals.
For ex ante optimality, the funder optimizes their worst case utility over feasible rankings Σ,;
Σ p. (a(1)) k}
(I)
The funder maximizes their worst-case expected utility by choosing the optimal marginal probabilities p solving the maximin optimization problem:
max min pE(o,a]"; dEE,
(a)(
$ k).
[|el] mk
Finally, the funder randomly chooses n proposals with marginal probabilities corresponding to p.
The ex ante optimization problem has game theoretic interpretation that motivates the need for randomization. Our model corresponds to zero-sum Stackelberg game where the funder is the "leader" who selects l proposals. The funder faces an adversarial "follower" who chooses a ranking of proposals. The leader's utility is the number of top l proposals selected based on the adversary's 's ranking, while the adversary's 's utility the negation of the leader's. The Strong Stackelberg Equilib- rium (SSE) is exactly the solution to Objective 2 It is well known that in SSE, the leader may need to commit a randomized (or mixed) strategy.
post validity The ex post validity criterion requires that for any pair of proposals a and b, if D's quality interval lies strictly below a's interval and if b is selected, then must also be selected.
Formally, selection rule that takes as input set of quality intervals and outputs set of selected proposals S satisfies ex post validity, if for all pairs of intervals a, b E with fg > ub, and all outputs S selected with non-zero probability,
S a S.
The ex post validity criteria ensures that the actual selected set of proposals is legitimate to stakehold- In particular, if the funder rejects proposal that dominates accepted proposal, that would be unacceptable to the funder and to applicants.
While the post condition seems natural, the simple randomize-above-theer violate it: Suppose proposals a and & both lie above the threshold, a dominates Because a and b are entered into uniform lottery, may be rejected random, while b is accepted at random.
mechanism can

---

Efficient Algorithm
The ante optimization problem in
(
equivalent to solving the following linear program (LP):
max PER EEE
V
(3)
subject to v S
p: P:l(o(i) kj. Yo En.
Ps k and 0 S p: S 1, Vi
[n]
This LP has exponentially many constraints (.)) The minimum weight k-ideal problem on arbitrary partial orders is NP-hard [[]], and several related interval order problems are also NP-hard [2] 42) 61]. Despite this, we show the problem is solvable polynomial time using the ellipsoid method with a separation oracle (Section We develop practical cutting plane algorithm (Section (42)) and show how to ensure ex post validity efficiently (Section 4.3) We call this end-to-end algorithm Maximin Efficient Randomized Interval Top-k (MERIT).
4.1 Polynomial Time Algorithm
We now develop polynomial-time algorithm to solve linear program BB) Our approach solves the problem using polynomial time "separation oracle" [18] with the ellipsoid algorithm. A separation oracle checks whether proposed solution satisfies all the constraints. If the solution is feasible, the oracle confirms it. If not, it identifies (at least one) specific constraint that the solution violates. The separation oracle may be used solve the LP without enumerating all (exponentially many) constraints by starting with limited set of constraints and iteratively shrinking the possible feasible region of the LP through calls to the separation oracle. Our main result proves that this method yields a polynomial time algorithm:
Theorem 4.1 (Polynomial time solution). The linear program 3 can be solved within accuracy of the optimal solution in polynomial time with respect to and log(1/e) /c) using the ellipsoid algorithm with Algorithm Tas as separation oracle.
Algorithm Polynomial--nn time Separation Oracle
Input: Candidate solution: (p, v) E [0, 1]" x R with |p||a = k. number selected kt set of intervals {li, ui Ve(N) sorted in decreasing order of lower bound &
Output: A set of violated constraints (0 if (p, v) is feasible)
1: Z 2-+0 0
2: for i
1 to k do
3:
S. + [j E (i, n] intervals j and overlap)
4:
if |
(k: (i 1)) then
5:
Obtain S, by sorting S. by p and keeping only the k (i - smallest values
6: if > Σ + EJEs, Pj
- Z f S E then
7:
8: return Z
By convention, we take the empty summfrom j 1 to 0 to be 0.
The primary technical difficulty is the design of an efficient separation oracle. We present our separation oracle in Algorithm Given candidate solution (p, v), the separation oracle checks whether achieves worst-case objective value of at least v. If the worst-case objective value under p is greater than v, then the solution feasible, if not the oracle returns set of violated constraints. At a high level, the oracle works by constructing worst-case possible sets of top-k proposals. For each of the l ++ intervals with the largest lower bounds, the algorithm constructs the worst-case set of top-k proposals that includes intervals 1 to (i 1) and excludes interval in the top-k. Excluding an interval with large lower bound constrains the set of intervals that must be in the top-k; since all intervals strictly below the interval with the large lower bound must be excluded from the top-k.
5

---

In considering all such sets of intervals, the algorithm enumerates possible worst-case permutations with respect to p time O(rk). If the separation oracle finds permutation that gives objective values smaller than it returns this permutation, which corresponds to violated constraint in the I.P. If it does not find any such permutation, then (p, e) is feasible. This separation oracle used as a sub-routine in the ellipsoid algorithm to compute the optimal solution in polynomial time in Theorem
Lemma 4.2 (Polynomial-time separation oracle). For any candidate solution (p. v) to the linear program (3) Algorithm cearithmmreeurnnn returns 0 only if the candidate solution is feasible and returns non-empty set of violated constraints otherwise. Moreover, Algorithm 7 runs time O(n max k, logn)).
4.2 Practical Algorithm
The cutting plane algorithm is described in full Algorithm 2 The algorithm starts by solving a relaxation of the LP without any of the worst-case value constraints find initial (potentially infeasible) candidate solution (p, v). Then, the algorithm repeatedly calls the separation oracle to check the feasibility of the current candidate solution. If the candidate solution is feasible, it is optimal solution to the LP, since it is optimal for a relaxation of the full LP. If the candidate solution infeasible, the algorithm adds the constraints returned by the separation oracle and re-solves the LP.
The cutting plane algorithm converges to a feasible optimal solution quickly in practice because is initialized with useful set of constraints the feasible region of the problem. These constraints prune the problem and impose monotonicity and symmetry constraints on the marginal probabilities p, based the number of intervals above and below each proposal, which we define below.
Definition 4.3 (Number above (A) and below (B)). For each proposal à: A(1) = fr L, ui} and B(i) = (r: & >> V-
Definition 4.4 (Monotonically ordered subset), Subset M C [n]] is monotonically ordered if Vi [[M] - 1J, A(M[) & A(M(i+ + 1]) and B(M(i)) 2 B(M + +)).
Full analysis is in Appendix B
Algorithm 2 Cutting Plane Algorithm
Input: Number of proposals kt intervals 1 = [li 4s iijjj. max iterations T
Output: Ex ante optimal vector p
# Prune Intervals
1: For intervals strictly below 2 others, set p: = 0 and remove.
2: For intervals strictly above > k others, pı and remove.
3: Let a a= accepted intervals. Update <- k a.
# Initialize Linear Program
4: Compute A(i) = # proposals strictly above i, B(i) = proposals strictly below i.
5: Using A, B, partition intervals into w monotone subsets Mi...., M, (Alg. 5
6: Solve LP to obtain initial p,
min U.p
s.t.
P [0, 1] Vi, us E)
PM 2 PM Vi [| 1], M e (M....., M
# Add Cuts
7:
for T iterations do
8: C' <- SeparationOracle((p v), I)
9: if C' = ( then return p
D Feasible
else Add constraints from C' I.P and resolve for new (p, v)
D Infeasible
11: return Failure
4.3 Enforcing Fx Post Validity
A solution to the ante optimality I.P 3 returned by the Cutting Plane Algorithm (Algorithm) 2 or the Ellipsoid Algorithm, not guaranteed to output vector marginal probabilities, such that sampling proposals with these marginals always guarantees post validity. However, we prove that

---

we can post-process any solution to the ex ante optimization problem, and then sample with marginal probabilities p guarantee ex ante optimality and ex post validity simultaneously. This stands in contrast to the commonly used haandomize-ahve--hreete not guarantee ex post validity as described in Section [
approach to randomization, which does
Theorem 4.5 Post-ppocesssng for ex post validity). Given any ex ante optimal p. Algorithm enables the funder to sample l proposals while satisfying both ante and ex post conditions and computable in time (n
Theorem 4.5 applies the post-processing algorithm given Algorithm t a solution from the Cutting
Plane Algorithm. For any b [n] with t. > ub, Algorithm terminates with Pa 1 or pa 0. Moreover, Algorithm 3 never decreases the objective value of p. Hence, applying post-proee pooteeroessiing to an ex ante optimal solution is without loss of optimality and ensures that any sampling method that selects proposals with marginal probabilities satisfies ex post validity.
We then implement the sampling step using systematic sampling (described in Appendix Cy.
[22 which has runtime (
Algorithm 3 Post-Prie Post-Processing of for Ex Post Validity
Input: Vector of marginal probabilities sequence of intervals {le, u/]eel Output: Vector of marginal probabilities p
1: Order the intervals by increasing u.
2: for E [m] do
3: if Pb then
4: continue
5:
for from n to (b + 1) do
6:
if la > up and Pa < then
7:
d f min Pb, 1 - Pa}
8:
Pb + Pb d
9:
Pa t Pa +
4.4 Full Algorithm
The complete MERIT algorithm solves the ex ante optimization with post-processing for ex post validity (Algorithm 33 followed by sampling. The algorithm is provably polynomial time using the ellipsoid method (Theorem 4.1). In practice, we use the cutting plane algorithm (Algorithm 2 which efficient albeit with non-polynomial theoretical convergence.
Algorithm MERIT Algorithm
Input: Number of proposals to select kt set of intervals / = (li, un (e(M))
Output: Selection of k proposals
1: Compute an ex ante optimal vector of marginal probabilities p using Algorithm g
2: Apply ex post validity post-processing p (Algorithm 3
3: Sample k proposals from [n]] with marginal probabilities of inclusion given by p (Algorithm) 6
Axiomatic Comparison
In applications like scientific funding or college admissions, there is no agreed upon ground-truth measurement selection quality. Hence, it is unclear how to empirically measure whether one algorithm performs better than another. Therefore, inspired by social choice theory [10], we initiate an axiomatic comparison of MERIT with alternative methods from Section 2.1 deterministic top-k selection, randomize above threshold, and the Swiss NSF method. We analyze the behavior of MERIT and alternatives with respect to natural axioms.
5.1 Defining Axioms
We propose three natural desiderata for algorithms selecting proposals from quality assessments. We begin by defining generic "randomized selection rule."
7

---

Definition 5.1 (Selection rule). A selection rule receives as imput n quality estimates I {(l,, e, u4) Kefn where e, u, are lower and upper limits on item i's quality and e; E [L,, Us] is point estimate. Given budget denote the marginal selection probabilities.
(L.....n), n., outputs a subset of [n] of size k. Let k) [0, 1
This captures methods that do not use intervals (deterministic selection), use only intervals (MERIT), and use both intervals and point estimates (Swiss NSF). Next, we define three axioms.
First, "monotonicity in budget" requires that increasing should not decrease any proposal's selection probability:
Definition 5.2 (Monotonicity in budget), A selection rule respects monotonicity in budget if for any input I and all budgets E in= - 1], p(I, + 1), 2 p(I, k): vi E [n].
Second, selection rule should be "stable" "stable""channing one interval should not drastically change algorithm behavior. We define an undesirable form of instability. algorithm exhibits "maximal instability" if changing single interval by an arbitrarily small amount can switch behavior between deterministic selection (minimum entropy) and uniform random sampling (maximum entropy):
Definition 5.3 (Maximum instability). selection rule is maximally unstable if there exist inputs I and J differing by arbitrarily small 0 in one proposal's ' quality estimate and budget & e (2...., 2} such that p(I, 1 (uniform random sampling on 7) whereas p(J, k) E {0, 1) (deterministic on J).
selection algorithm should avoid maximum instability. We restrict budget to f2...., 2} since stability with respect to changing a single proposal is not meaningful when choosing only one proposal to accept or reject.
Finally, inspired by "reversal symmetry" from social choice theory 46). when selecting of 2 proposals, if the quality scale is reversed (all intervals flipped), the selection rule should flip the selection probabilities:
Definition 5.4 (Reversal symmetry). For input {lli, ei, u4) ue) where 4, es, uy E [0, 1jvi, let [R] {11 u, 1 e,, ())xe(n) be the reversed input. A selection rule selecting 1 of n = proposals respects reversal symmetry for any flipped inputs I and [(R), p(I, = (P1.P2) and p (R) 1) = (12,
5.2 Theoretical Analysis
Deterministic top-k selection meets these criteria but does not account for uncertainty (violating our ex ante requirement). We characterize randomized mechanisms in Theorem 5.5
Theorem 5.5 (Axiomatic analysis). Existing randomized algorithms have the following properties:
(a) Swiss NSF and never maximally unstable.
threshldd both exhibit maximun instability, while MERIT is
(b) Swiss NSF, cattommiiz--bbov--eeeeehoe and MERIT all violate monotonicity in budget.
(c) is not possible to simultaneously satisfy ex ante optimality and monotonicity in budget.
(d) Swiss NSF and randomize-abovy threshold violate reversal symmetry, while MERIT satisfies reversal symmetry.
While all randomized selection rules considered vioalte monotonicity, funder could enforce monotonicity in budget for MERIT by solving sequence of optimization problems from 1 to k, but this may come at loss of ex ante optimality, as we describe in Appendix D
We formally prove Theorem 5.5 in Appendix 5.5
6 Experimental Comparison of Methods
We evaluate MERIT using real peer review data from the Swiss NSF grant reviews [1], NeurIPS 2024 conference papers, and ICLR 2025 submissions. We compare performance under both (1) expected utility in a probabilistic model of reviewer behavior and (2) our worst-case utility objective. We provide additional ablations and qualitative insights into different lottery outcomes in Appendix G

---

Deterministic (mean)
Deterministic (model)
Swiss NSF
MERIT
0.95
0.8
Worst-case Utility
+0.1
Expected Utility
0.9
0.6
+0.193
0.4
+0.136
0.85
0.2
0.8
Swiss NSF
NeurlPS LOO
NeurlPS Gauss
NeurIPS Subj
ICLR LOO
ICLR Gauss
ICLR Subj
Swiss NSF
Conference
(a) Linear miscalibration model.
(b) Worst-case over interval ordering model (our model).
Figure 1: Proportion of top-k proposals selected by different methods with quality data generated under Swiss NSF model of linear miscalibration and under our model of worst-case over feasible rankings. MERIT matches performance of algorithms designed for the Swiss NSF's linear model, with expected utility averaged over 50 samples of synthetic data and error bars showing 95% CI for the sample mean. The gap between MERIT and other methods in the worst-case over intervals defined by our model can be substantial, as shown by the gaps in NeurIPS Gaussian, NeurIPS Subjectivity, and ICLR Subjectivity.
Expected Utility under Linear Miscalibration We evaluate expected utility under the Swiss NSF's widely-adopted linear miscalibration model [21] [6] [5, where reviewer scores are linear combinations of true quality, reviewer bias, and Gaussian noise. We simulate two settings: (1) Swiss NSF grants (350 proposals, 10 reviewers, 80 proposals/reviewer) and (2) CS conference review (1000 papers, 1000 reviewers, 5 papers/reviewer), selecting the top one-third in both cases. We generate 50 synthetic datasets with parameters dg = 2, d₁ = 1, o. 0.5 matching prior work (54 and estimate 50% confidence intervals following Swiss NSF methodology.
Worst-Case Utility Our worst-case objective I maximizes the minimum expected fraction of true top-k proposals over all rankings consistent with score intervals. We evaluate on three real datasets: Swiss NSF 2020 grants (n 353), NeurIPS 2024 accepted papers (n 4035), and ICLR 2025 submissions (n = 11520). For Swiss NSF, we use their linear model intervals. For NeurIPS and ICLR, generate intervals using three methods: (1) leave-one-out (LOO) ranges, (2) Bayesian credibility intervals under a Gaussian model, and (3) bebbeettivity-ajjustt intervals based on reviewer emphasis different criteria 28 38
Full details are in Appendix F
Results Figure [ shows our main results. Under the Swiss NSF's Bayesian model, MERIT performs comparably to Swiss NSF's randomized method and deterministic model-based selection in expectation, while deterministic mean-based selection performs poorly. In the worst-case setting, MERIT substantially outperforms all baselines across all datasets, achieving up to 0.19 higher utility than Swiss NSF. The optimality gap is particularly large when intervals are wide (e.g.. NeurIPS LOO, NeurIPS Gaussian), where deterministic selection and even Swiss NSF's randomized approach perform poorly. These results suggest MERIT maintains expected utility under probabilistic models while providing superior worst-case robustness. We provide additional ablations and experiments in Appendix G with similar results.
Computational Efficiency All experiments run on standard 2019 MacBook Pro. Using Gurobi 12.0.1 20 to solve LPs, MERIT completes under five minutes even for 11,000+ proposals. Detailed runtime analysis is in Appendix H

---

7 Related Work
Designing Better Evaluation Processes Many works diagnose flaws in peer review- such as miscalibration and arbitrariness in opinions of reviewers and propose improvements [51] Many models assume linear miscalibration [16 [
44 48 12I. including those used in the Swiss NSF (21), but these often perform poorly in practice [27]. likely due to the complexity of real- world miscalibration MT. A recent approach [59] addresses arbitrary miscalibration and shows that randomized estimators can improve ranking accuracy. Related work in hiring proposes algorithmic solutions prejudice and uncertainty about unknown options [24 30]. Both approaches make stronger assumptions on known model of errors in hiring process than our work.
43
Selection Under Uncertainty and Robust Optimization Our method falls under the umbrella of distributionally robust optimization (DRO) [41]. Our main contribution is a formulation specific to scenarios where decision makers want to select l high quality items under ambiguity sets defined by intervals. Though similar in spirit to robust portfolio optimization [62], our assumptions differ as investors typically have probabilistic models and seek risk-averse diversification. Recent work on conformal prediction for selection [22] focuses on bounding false positives above a fixed threshold, whereas we select the top-k items without assuming a fixed quality threshold or reliable probabilities.
Randomized Social Choice Our work connects to the literature on randomized social choice, which examines how incorporating randomness into voting rules can enhance desired properties [T0 9.
Unlike our setting, most social choice models assume no underlying ground-truth quality. A related line of work in social choice theory "distortion" interprets voters' rankings noisy reflections of latent utilities and seeks aggregation rules that perform well under this uncertainty [4]. Our approach is conceptually similar: we consider rankings consistent with intervals of quality and design a randomized mechanism that maximizes the worst-case expected utility of the selected top-k items.
8 Discussion
We introduce MERIT, a computationally efficient lottery for top-k selection under uncertainty. By relying solely on intervals rather than fully specified generative models, MERIT respects real-world funder constraints while providing a principled robust optimization solution. Our case studies demonstrate scalability to tens of thousands of candidates. An additional benefit of MERIT is that it can handle additional constraints may prefer to implement a uniform lottery, where all candidates subject to randomization are selected with equal probabilities. The optimization problem used in MERIT can be modified to constrain the lottery to uniform sampling, thereby implementing the ex ante optimal uniform lottery that respects ex post validity constraint (discussed in Appendix I).
the form of the lottery. For example, in some cases, funders
Limitations. MERIT exploits only interval ordering, appropriate when credible probabilistic models are unavailable but potentially sub-optimal with well-calibrated predictive models. We assume fixed proposal costs; variable budgets require accounting for both quality and budget allocation. Additional pairwise quality information (e.g., from common reviewers) is not currently exploited but could improve performance. Finally, MERIT may be less interpretable than threshold- -based methods like Swiss NSF
Future work. Several extensions merit investigation. First, variable-cost candidates: allowing partial funding or incorporating cost-quality trade-offs in budget-constrained settings, with attention to incentive compatibility. Second, additional ordering information: developing efficient heuristics for general partial orders beyond interval orders, perhaps via additional monotonicity constraints. Third, richer utility functions: extending beyond 0-1 utility to positional scoring rules (e.g., Borda count) while maintaining tractability. Fourth, cost-quality trade-offs: analyzing efficient trade-offs between reviewer resources and decision quality to guide when and how to randomize. Finally, equilibrium effects: understanding how randomization alters applicant and reviewer incentives and strategic behavior. Progress on these questions can inform both theoretical understanding and practical implementation of randomized selection mechanisms.
Progress on these questions can inform both theoretical understanding and practical implementation of randomized selection mechanisms.
10

---

Acknowledgments
This work supported in parts by NSF 1942124 and ONR N000142212181, well the Gates Foundation. The views and opinions expressed in this study are those of the authors and do not necessarily reflect the views positions of the sponsors.
References
[1] David Adam. Science funders gamble on grant lotteries. Nature, 575(7784):574-555
2019. doi: 10.1038/641586-01-003333 URL https:///wwit http:////wwwwaauure.oommm www.:eatarreccoom cot. wwt.oa/aaeee///////oommcmm
[2] Ranendu Adhikary, Kaustav Bose, Satwik Mukherjee, and Bodhayan Roy. Complexity of maximum cut interval graphs. Discrete & Computational Geometry, 70(2):30--322, 2023.
[3] Sara Amoroso, Pietro Moncada-Pateend-Caase Paternò-Castello, and Antonio Vezzani. R&d profitability: the role of risk and knightian uncertainty. Small Business Economics, 48:331-343, 2017.
[4] Elliot Anshelevich, Aris Filos-Ratsikas, Nisarg Shah, and Alexandros A Voudouris. Distortion social choice problems: The first 15 years and beyond. arXiv preprint arXiv:2103.00911, 2021.
[5] Yukino Baba and Hisashi Kashima. Statistical quality estimation for general crowdsourcing tasks. In ACM SIGKDD International Conference on Knowledge Discovery and Data Mining, 2013.
[6] Rina Foygel Barber, Emmanuel Candes, Aaditya Ramdas, and Ryan Tibshirani. Predictive inference with the jackknife++. The Annals of Statistics, ::(1))486-53 507,, 2021.
[7] Yakov Ben-Haim and Maria Demertzis. Decision making in times of knightian uncertainty: An info-gap perspective. Economics, 10(1):20160001, 2016.
[8] Joël Berger, Margit Osterloh, and Katja Rost. Focal random selection closes the gender gap in competitiveness. Science Advances, 6(47)xeabb2142 2020. doi: 10.1 126/sciad.abbb2142. URL
org httg:/oi//10 wwt.:/1111/ss.....oo http://ccciaaaabb2222222222
[9] Florian Brandl, Felix Brandt, and Hans Georg Seedig. Consistent probabilistic social choice. Econometrica, (445):1339-168 2016.
[10] Felix Brandt, Vincent Conitzer, Ulle Endriss, Jérôme Lang, and Ariel D Procaccia. Handbook of Computational Social Choice. Cambridge University Press, 2016.
[11] Lyle Brenner, Dale Griffin, and Derek J Koehler. Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment. Organizational Behavior and Human Decision Processes, 97(1):64-81, 2005.
[12) Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurlPS experiment. Xrxiv Preprint (txxx:2009.097744 2021.
[13] Larry G Epstein and Tan Wang- Intertemporal asset pricing under knightian uncertainty. In Uncertainty in Economic Theory, pages 429-471. Routledge, 2004.
[14] European Research Council. Peer review not perfect but still the best sys- tem that we have. htttps://rc.. europa. eu/ news events/magazing article/rr peer- Beer-revise sver-review-nee not perfect-stilll system we- wavee November 2023. Accessed: 2025-05-07.
[15] Ulrich Faigle and Walter Kem. Computational complexity of some maximum average weight problems with precedence constraints. Operations Research, 02(4):688-693 1994.
[16] Peter A. Flach, Sebastian Spiegler, Bruno Golénia, Simon Price, John Guiver, Ralf Herbrich, Thore Graepel, and Mohammed Zaki. Novel tools to streamline the conference review process: Experiences from SIGKDD' 09. SIGKDD Explorations Newsletter, 11(2):63-67, 2010. ISSN 1931-0145. doi: 10.1 145/1 009400.1809404441110009900010119 1809413. URL http::////cee acm. org/10.../0. 1145/
1809400. 1809413

---

[17]
Martin Charles Golumbic. Algorithmic Graph Theory and Perfect Graphs. Academic Press, New York, NY, USA, 1980.
[18] Martin Grötschel, László Lovász, and Alexander Schrijver. The ellipsoid method and its consequences in combinatorial optimization. Combinatorica, 1:169-197, 1981.
[19] Martin Grötschel, László Lovász, and Alexander Schrijver. Geometric Algorithms and Combi- natorial Optimization. Springer- Verlag, Berlin, 1988.
[20] Gurobi Optimization, LLC. Gurobi Optimizer Reference Manual, 2025. URL httpp://wwr
gurobi com
[21] Rachel Heyard, Manuela Ott, Georgia Salanti, and Matthias Egger. Rethinking the funding line at the swiss national science foundation: Bayesian ranking and lottery. Statistics and Public Policy, 9(1):1 11:1100-211, 121, 2022.
[22] Ying Jin and Emmanuel Candes. Selection by prediction with conformal p-values. Journal of Machine Learning Research, 24(244): 2023. URL http::/jjlz: www.llr.rggpppers////2
02-1176.htm
[23] L. G. Khachiyan. A polynomial algorithm in linear programming. Doklady Akademii Nauk SSSR, 244(5): 1979.
[24] Jon Kleinberg and Manish Raghavan. Selection problems in the presence of implicit bias. arXiv Preprint arXiv:1801.03533 2018.
[25] Jonathan Klick. More of declaration than a constitution. The Daily Penn- sylvanian. February 2024.
URL htttps:/www. wedddmmic-idoooogiii Guest Column.
com attaaaarticc//e/// /22/c////202//
penn- www.woowwwddissee
[26] Frank Hyneman Knight. Risk, Uncertainty and Profit, volume 31. Houghton Mifflin, 1921.
[27] John Langford. ICML acceptance statistics, 2012. http:///hunc..ret//p25111 (nline:
accessed 14-May-2018).
[28] Carole Lee. Commensuration bias in peer review. Philosophy of Science, 12(2)127221288) 2015.
[29] Jinsook Lee, Emma Harvey, Joyce Zhou, Nikhil Garg, Thorsten Joachims, and René F Kizilcec. Algorithms for college admissions decision support: Impacts of policy change and inherent variability. arXiv preprint arXiv:2407.11199 2024.
[30] Danielle Li, Lindsey R Raymond, and Peter Bergman. Hiring as exploration. Technical report, National Bureau of Economic Research, 2020.
[31] Mengyao Liu, Vernon Choy, Philip Clarke, Adrian Barnett, Tony Blakely, and Lucy Pomeroy. The acceptability of using a lottery to allocate research funding: survey applicants. Research Integrity and Peer Review, 5:1-7, 2020.
[32] William G Madow. On the theory of systematic sampling, II. The Annals of Matheratical Statistics, 20(3):333-354, 1949.
[33] Andrey Malenko and Anton Tsoy. Asymmetric information and security design under knightian uncertainty. Available at SSRN, 3100285, 2020.
[34] Charles Manski. Nonparametric bounds on treatment effects. The American Economic Review, (0(2):319-323) 1990.
[35] David McKenzie. Identifying and spurring high-growth entropreneurship: Experimental ev- idence from a business plan competition. American Economic Review, 107(8):278-2307) 2017.
[36] Nature Editorial. The case for lotteries as a tiebreaker of quality in research funding. Nature :/tw:://wwwww
609(7928):633, September 2022. doi: 10. 1al.133//41188-02021 586-0022-02959--2 nature. com/ httmm//rr..eeeeed1116----22----2
Accessed 2025-05-14.
12

---

[37] Kiyohiko G Nishimura and Hiroyuki Ozaki. Ireeeessible investment and knightian uncertainty. Journal of Economic Theory, 136(1):668 694, 2007.
[38] Ritesh Noothigattu, Nihar Shah, and Ariel Procaccia. Loss functions, axioms, and peer review. Journal of Artificial Intelligence Research, 70: 10:1481-1515, 2021.
[39] Jakub Nowotarski and Rafal Weron. Computing electricity spot price prediction intervals using quantile regression and forecast averaging. Computational Statistics, 30:791-8033 2015.
[40] Margit Osterloh and Bruno S Frey. Dealing with randomness. Management Revue, 30(4): 331-345, 2019.
[41] Hamed Rahimian and Sanjay Mehrotra. Frameworks and results in distributionally robust optimization. Open Journal Mathematical Optimization, 3:1 85, July 2022. ISSN 2777-5860. doi: 10.5802/ojmo.15 15. URL http://dx.. doi. www.orgg//00 wttg///..002///// 5002550020/// .15
[42] Ramamurthy Ravi, Ajit Agrawal, and Philip Klein. Ordering problems approximated: Single- processor scheduling and interval graph completion. In Automata, Languages and Programming: 18th International Colloquium Madrid, Spain, July 8-12, 1991 Proceedings 18, pages 751-762. Springer, 1991.
[43] Magnus Roos, Jörg Rothe, and Bjöm Scheuermann. How to calibrate the scores of biased reviewers by quadratic programming. In AAAI Conference on Artificial Intelligence, 2011.
[44] Magnus Roos, Jörg Rothe, Joachim Rudolph, Björn Scheuermann, and Dietrich Stoyan. A statistical approach to calibrating the scores of biased reviewers: The linear vs. the nonlinear model. In Multidisciplinary Workshop on Advances in Preference Handling, 2012.
[45] Doriana Ruffnn. Some implications of knightian uncertainty for finance and regulation. FEDS Notes, April 2014. doi: 10. 10110016/2380-77722001.. URL
htttps:/wwit www federal gov /ttpe//wwwweeteeeeeeeee/eeeeee/d of Come-implicas www.:///wwwaa...tttataaaoaoooaa. html
[46] Donald G. Saari. Geometry of Voting, volume of Studies in Economic Theory. Sprinerr Berlin Heidelberg, 1 edition, 2012. ISBN 978-3-642-48644499 doi: 10.1007/978.3-622-486444
[47] Martin Saveski, Steven Jecmen, Nihar Shah, and Johan Ugander. Counterfactual evaluation of peer-review assignment policies. Advances in Neural Information Processing Systems, 36: 58765-58788 2023.
[48] Hermann Schichl and Meinolf Sellmann. Consensual affine transformations for partial valuation aggregation. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 33, pages 2612-2619, 2019.
[49] Sara Schroter, Wim EJ Weber, Elizabeth Loder, Jack Wilkinson, and Jamie Kirkham. Evalua- tion of editors' abilities to predict the citation potential of research manuscripts submitted the bmj: a cohort study. Bmj, 379, 2022.
[50] Science Foundation Ireland. Innovate for ireland programme. https://www.sff ie/ http:///uuuuuinngccaaaaaaonnntee 07.
August 2024. Accessed: 2025-05-
[51] Nihar B Shah. An overview of challenges, experiments, and computational solutions in peer review. http://bitt.ly/Peerreeieees Communications of the ACM), June 2022.
(Abridged version published in the
[52] Cass Sunstein.
Knightian uncertainty. Available at SSRN 4662711, 2023.
[53] Swiss
National 2021.
Science Foundation.
Drawing lots as a bre-breaker,
March
URL https://wwe enf httpchehh/yy.f222200088//////e/ Accessed 2025-05-14.
news- 210331- - drawing lots- -ass-as-a-- tie- -ieakeereere
[54] Sijun Tan, Jibang Wu, Xiaohui Bei, and Haifeng Xu. Least square calibration for peer reviews. Advances in Neural Information Processing Systems, 34:27069-27088 2021.
13

---

[55] The British Academy. Partial randomisation trial extended after diversity of appli- cants and award holders increases. httpp://www. thep://wwwwwteerrrtssscccceeeyr ac a...a.cukk//. uk/news/
Pertial-randonee
April 2025.
Accessed: 2025-05-07.
[56] Giulia Trentacosti. Using lotteries to allocate research funding: Perspectives from switzerland, September 2021. hRL rug. http://wwww/..bboor///aaaae/////
[57] USENIX Securriy 2025 Program Committee, USENIX security 2025 conference format. https thub. org.a//ewec.....i......aoa..ooan
using- Iorttrttteer-ssee to allocate- -
lang=en
University of Groningen Library Open Science Blog.
April 2024. Version 1.1.
[58]
Volkswagen Foundation.
Partially randomized procedure
lottery and peer re-
view.
https: //w::://wwww http://wwwvolllssaggssttttuuuu de wtt.///ee/euddnnneeeeeeeeeeeeeee
oet.epalll-..oooooooooooooooooee and ane-rer---dpee------eeeere review
May 2025. Ac-
cessed: 2025-05-07.
[59] Jingyan Wang and Nihar B Shah. Your is my 1. your 3 is my 9; Handling arbitrary miscalibra- tions in ratings. arXiv Preprint arXiv:1806.0m 05085, 2018.
[60] Aidan S Weitzner, Matthew Davis, Andrew Han, Olivia O Liu, Anuj B Patel, Brian D Sites, and Steven P Cohen. How predictive is peer review for gauging impact? the association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal. Regional Anesthesia & Pain Medicine, 2024.
[61] Gerhard Woeginger. On the approximability of average completion time scheduling under precedence constraints. Discrete Applied Mathematics, 131(1):337-252 2003.
[62] Panos Xidonas, Ralph Steuer, and Christis Hassapis. Robust portfolio optimization: A catego- rized bibliographic review. Annals of Operations Research, 292(1):533-552, 2020.
A Proofs
A.1
Optimality of deterministic selection in fully Bayesian setting (Section 2.1
Here we provide the formal statement and proof of our informal claim that a funder can maximize their utility via a deterministic selection they assume fully Bayesian model of their data.
We prove a general statement for a funder who estimates a total ranking of proposals and receives a pre-specified utility for any possible estimate of the ranking of proposals and true qualities of the proposals. The top-k selection problem is a case of this estimation problem where the funder's 's utility specifies the utility of choosing proposals based on the estimated ranking of proposals.
Proposition A.1 (Optimality of deterministic selection in the fully Bayesian setting). Consider funder who estimates a ranking of proposals from review data in the following fully Bayesian setting. The funder observes review data y for set of proposals of true quality 0 X. The review data and true quality are generated jointll from a known model. Letting 11 denote the set of all permutations of proposals, the funder estimates a ranking of proposals and gains utility u(ā, where u II X + R.. The funder aims to choose (potentially randomized) estimator of the true ranking f + A(II) that maximizes their expected utility Ey.0, x [u(#,0) where ä f(y). In this setting. there always exists a deterministic f that maximizes the funder's 's expected utility.
Proof. Expand the expected utility by conditioning the observed data y as Ey.0, [u(ft, 0)) E, [Ba My, [uf, 0)]] For a given y. the funder maximizes the inner expectation by choosing any f E arg m Eayy ((('', f)[y), where the maximum over II always exists since the set of permutations finite. Hence, the deterministic estimator that takes (ly E arg max Eey [u( (0)y)] for all y E Y maximizes the funder's expected utility.
14

---

A.2 Polynomial time algorithm for ex ante optimization (Theorem 4.1
In order to prove the main theorem, we first prove Lemma 4.2 which establishes that the separation is correct and efficient.
oracle (Algorithm]
I
Proof of Lemma
4.2
Let o' be the smallest is found by the separation oracle. We begin by showing
that:
Σ p(lfo(1)
min Fin
For E [n]] let
K= [T C
[T],
3o En s.t. o(j) S kk S
be the set of all feasible selections of top-k intervals given Ln. Note that we can rewrite the objective
Σ ( Sk)
min Σ Tek
min OEE. 1=
Pj. JET
Now, we divide K into subsets, for i [k: + 1). define
K = (T C K: and [i 10-1] CT}
so that K() contains all sets in K that include intervals to (i 1) and exclude interval i and K = U K()) Let the intervals be sorted decreasing order of lower bound &. Then, for any i [n]. is not strictly above any of the intervals from 1 to (i 1). Additionally, letting
S. = {j E (i, n] intervals j and i overlap)
any interval in (i. n] \. S: must be strictly below interval Hence, any set of top-k intervals T ()))) contains intervals 1 to (i 1) and the remaining & - (i 1) intervals come from S, Additionally, for all j E St, U 2 l, while E S & so all intervals in S, overlap each other and any selection of these S; in the top is feasible. Hence:
k(t) {[i IJUA: A S, ||| (I- 1)}.
Finally, for fixed the minimum objective value over K(O) is to select the (A: - (i 1) proposals with the smallest values of p. Algorithm exxactl enumerates these worst-case constraints for each ( and hence returns the minimum over K.
Now, the separation oracle adds constraints to Z when v Ej=ips + Ejes, Pj- Hence the separation oracle returns 0 only if S Σ Py Ejes; Pj for every i, which precisely the condition for feasibility. Further, as argued above, the separation oracle considers feasible top-k; so any constraints added by the separation oracle represent valid constraints for the IP.
The algorithm runs in time Onn max (k. log n)), because finding S, requires most linear over (n) proposals for each of (k) proposals and initial sorting takes time O(n log n).. We can sort intervals p once, ensure that each linear scan for S, returns the intervals with smallest p with no additional sorting per iteration.
Now, we prove the main theorem by showing that the ellipsoid algorithm converges in polynomial time using this separation oracle as a sub-routine:
Proof of Theorem 4.1 We may relax the equality constraint of LP (3) Σ k to an inequality Ei Ps < k without loss of optimality, since increasing any P. (up to cannot decrease the worst-case value of v. This relaxation ensures that the feasible region lies within full-dimensional affine subspace of I and contains a nontrivial interior. In particular, the point p; = à = lies strictly inside the box constraints and satisfies all inequalities strictly, implying the feasible region is dill-oimensional.
Then, because our separation oracle runs in time polynomial in the classical ellipsoid algorithm 23 using the separation oracle to make cuts, solves the optimization problem to within accuracy in time (nly(n, (og(l/)),,,ooooooooo log( (R/r))) where R is the radius of Euclidean ball that contains the feasible
15

---

region and r is the radius of a Euclidean ball entirely contained in the feasible region. Clearly, R is upper bounded by poly(n). since each p; bounded in [0, 1] and v is bounded in [0. k). To establish lower bound on we invoke Theorem 6.2.2 in Grötschel, Lovász, and Schrijver [9, which states that if polyhedron == x E TR : Ar S b is full-dimensional, and the matrix and vector b consist of integers of maximum bit length U, then P contains a ball of radius at least 2 2-puly(n,U), Noting that all constraints in our model have integral coefficients of bit length at most log(k) < n. we have that r 2 2 poly(n), Hence, (or( is poly(n) and so the runtime of the algorithm is polynomial in and (1/el/
A.3
Ex post validity (Theorem 4.5
Proof of Theorem 4.3 We will prove that for any a, b E [n]] such that &, > Up and for any input Sagorithm@terminne
vector of marginal probabilities p. the post-processing and never decreases the objective value of p. Then, if any ex ante optimal p is given as input to the algorithm, after post-processing is still ex ante optimal. Moreover, any sampling method that respects the post-provs post-processed marginal probabilities will satisfy ex post validity, since if a dominates b, either is always sampled (Pa- = 1) or b never sampled (p)- = 0).
with pa.
ps
Let a, be any intervals with lg > ug. Let D = {d E [n|] La > [n be the set of all intervals strictly above interval a. Note that (1) all d D also strictly above b and that (2) b precedes when ordered by u and a precedes all E D. Hence, Algorithm 3 will process 6, then a, then d E D. After processing b = b, either ps, 0 or pa 1 for all d D. Because ps cannot increase in any subsequent iterations, if ph = 0, it will remain O until the algorithm terminates. If If = for all d E D, then because D. Pa 1. Furthermore, since pa 1 for all d E D. Pa. will not decrease in any subsequent iterations. Hence, the algorithm ends with = 0 or Pa = 1.
Further, for any E En, if (b) < k, then (a) < (b) < k. Hence, moving probability mass from ps to Pa cannot decrease min Σ (a)) & k).
A.4 Axiomatic comparison (Theorem
30
10
Figure 2: Example that violates maximum instability for Swiss NSF and randomize-above-thnss. with & = 3. Slightly decreasing the upper bound and point estimate of proposal 3, changes the algorithm's behavior from selecting the top 3 deterministically (left) to sampling among all 10 proposals uniformly at random (right).
(a) Maximum instability We will show that both Swiss NSF and randomize above threshold (with data-dependent threshold) are maximally unstable using example. Let intervals 1 to k be [0, 2] with point estimates of and let intervals l: + 1 to n be (0, 1 1- e] for some e (0, 1). Then, the Swiss NSF algorithm selects the first l proposals deterministically. Now, shift interval & to be [0, with point estimate of - e. Since all the intervals contain the k-the point estimate the Swiss NSF algorithm selects uniformly at random. Note that can be taken to be arbitrarily small, so the Swiss NSF is maximally unstable for any > O. If the randomize above threshold method chooses the threshold in a data-dependent manner, then the same example leads to maximal instability if the threshold is taken to be the k-th highest point estimate.
2c]
Now, to show that MERIT is not maximally unstable, we first observe that MERIT selects among all proposals uniformly at random only if all intervals intersect each other. Let < < n 1. Assume for the sake of contradiction that MERIT samples uniformly at random from all intervals and that there are two intervals and j with strictly above j. Since the algorithm ex post valid, if p; > then p. 1, but then p: PA 1, which is possible only if n k. yielding contradiction.
16

---

Now, to show that MERIT is nor maximally unstable, we establish that for any set of intervals where all intervals except for one overlap, the algorithm never chooses deterministically. Since all intervals but one overlap, the non-overlapping interval can either have the largest lower bound or the smallest upper bound.
First, consider the case where interval j has the largest lower bound. Then, we can partition the intervals into sets {j}, X, the set of intervals that intersect j. and / the set of intervals strictly below j. By symmetry and monotonic chain constraints (detailed in Appendix [B). the algorithm will
output marginal probabilities with at most 3 different values p)+ Px Ps, such that Pj 2 P= 2 pb- Since [X + [B] > k, the only input for which the algorithm could output lo 1 's is when [X] = k 1 and [B] n k > 1. In this case. j is always in the top k. so p,
and j is pruned from the problem. Then, our algorithm chooses k - 1 out of n - 1 intervals that all intersect, so the optimal solution is sampling& k uniformly at random from the remaining X U B intervals. Hence, our algorithm will not choose deterministically.
Now, consider the case where interval j has the smallest upper bound. Then, we can partition the intervals into 3 sets {j}. X, the set of intervals that intersect j. and A, the set of intervals strictly above j. By symmetry and monotonic chain constraints, our algorithm will output marginal probabilities
with at most 3 different values Pz.Pe.P2 Pa such that Pa 2 Px 2 Pj- Since A + X 2 k + 1, the only and X n
possible input for which the algorithm could output t 1' s is if |A| In this case, is never in the top k so the feasible top k could be any subset of size k from A U X. Therefore, selecting proposals uniformly at random from A U X is optimal, so MERIT will not choose deterministically.
> 0.
(b) (c) Monotonicity in budget We show that Swiss NSF, randomize above threshold, and MERIT are not monotonic in budget k, via the example shown in Figure 3
2
Figure 3: Example that violates monotonicity with respect to k for Swiss NSF and our MERIT
algorithm. When t 1, P2 = 1/2 for both algorithms. However, when I: 2, P2 = 1/3 for both algorithms.
In this example, taking k = 1, both the Swiss NSF method and MERIT randomize between proposals 1 and 2, so proposal is selected with probability 1/2. However, if the number of proposals selected is increased to k = 2, then proposal 1 is always selected and both algorithms sample uniformly at random from intervals 2 to 4, meaning that proposal 2 has selection probability of only 1/3. Hence, even though more proposals are being selected, proposal 2 is worse-off. The same example shows that randomize-abuu tbove-threshhold violates monotonicity, taking the threshold to be the point estimate or the lower bound of the k-th highest proposal.
This example also proves that it is not possible to simultaneously satisfy ex ante optimality and monotonicity in budget. For l: = 1, the unique ex ante optimal is (1/2, 1/2, 0, 0, 0) since feasible top 1 are {1} and {2). For k
the unique ex ante optimal p is (1, 1/3, 1/3, 1/3) since the feasible top 2 are {1, }} {1, 3]. {1, 43. Since these are unique ex ante optimal solutions, there is no sequence
of solutions for i equal to and 2 that satisfy monotonicity in budget and ex ante optimality simultaneously.
(d) Reversal symmetry To prove that Swiss NSF and (data-dependent) aandomize-boov--treehhe violate reversal symmetry, consider intervals (0, 1) and (0.1, 0.2) with point estimates 0.5 and 0.15.
17

---

Then, the Swiss NSF selection rule or wwwpiiiiee-bvvveteeeeeeeeeeeeeee point estimate, will accept interval and reject interval 2. However, if intervals are flipped so interval 1 stays the same, but proposal 2 has interval (0.8, 0.9) with point estimate of 0.85, then Swiss NSF samples uniformly at random between the two proposals. Now, when n = 2 and k = 1, MERIT only samples between the two proposals if their intervals overlap. If the intervals are horizontally flipped, then they still overlap and MERIT samples uniformly at random respecting reversal symmetry. If interval 1 lies strictly above interval 2, then after flipping interval 2 lies strictly above interval 1, so the marginal probabilities of selection are (1, 0) and (0, 1) respectively, respecting reversal symmetry.
with threshold set as the highest
B Full Cutting Plane Algorithm
As described in Section 4.2, we find that the cutting plane algorithm converges to a feasible optimal solution much faster when it initialized with additional useful constraints. These constraints prune decision variables and impose monotonicity and symmetry constraints on marginal probabilities based on the number of intervals above and below each proposal.
The cutting plane algorithm is described in full in Algorithm 2 The algorithm prunes intervals and initializing the LP with monotonicity constraints. The remainder of this section proves that these additions are without loss of optimality and ensure faster convergence.
We can think of A, B as defining a partial order: i j = A(i) S A(j) and B(i) > B Then, a monotonically ordered subset of intervals is a totally ordered subset or a "chain" and the minimum number of monotonically ordered subsets of intervals that covers set of intervals is the "width" of the partially ordered set of intervals. Note that this partial order defined by A and B is not the canonical interval order that we use to define the set of feasible permutations Σ,, The (A, B) partial order never was width larger than the interval order and often has much smaller width. At the extreme, if all intervals overlap, then the width of the interval partial order is n, while the width of the (A, B) partial order is 1, since all proposals have equal A and B.
As we now prove, the cutting plane algorithm converges in 0(k*+1) iterations, where denotes the number of monotonically ordered subsets partitioning the intervals (or the width of the (A, B) partial order. In theory, can grow linearly with k so this algorithm does not have the theoretical polynomial-time guarantee of the ellipsoid algorithm. In practice, w is often small for many sets of intervals.
Proposition B.1 (Cutting plane algorithm convergence). Letting w denote the number of monotoni- cally ordered subsets (chains) partitioning the intervals per Definition 4.4 Algorithm 2 converges to an optimal solution in (k iterations. The algorithm solves an LP with at most O(n + k constraints.
Proof. We first prove three lemmas that show that we can impose initial pruning, symmetry, and monotonicity constraints on p without loss of optimality.
Lemma B.2 (Pruning of optimal p). There exists an optimal p in which p. = 1, Vi E [n]] with B(i) 2 n k and pj- = 0, Vj E [r] with A() 2 k.
Proof. If B(1) 2 n - k. then is always included in the top & in any permutation of intervals. P/ < 1, then there must be some other interval with Py > 0. but setting p; = 1, Ps = 0 will not decrease the objective value, since is always in the top k. Similarly, if A(i) 2 k. then i is never included in the top k, so taking p; = 0 by shifting probability mass from i to any other proposal cannot hurt the objective value.
Lemma B.3 (Symmetry of optimal p). There exists an optimal p in which pt = Pý for all i, E [n] such that A(i) = A(j) and B(i) = B(j).
Proof. If A(i) A(j) and B(i) B(j), then and j have the same sets of intervals that are strietly above and strictly below each interval. Hence, for any permutation a E En, the permutation a' with i and j exchanged is also in Σ,,, so the objective value is maximized p; Pý-
Lemma B.4 (Ordering of optimal p by A and B). Let MI,... Ma, be a partitioning of [n] such
that each Mj is monotonically ordered. Then, there exists an optimal p for Objective (2) such that within each M, PMb] 2 PM for all j E [|M]] 1].
18

---

Proof. Let i,j E [ni] be any pair of proposals with A(1) - AG) and B(i) 2 BG). Let be a feasible solution to the optimization problem with Pj > Pc. We will show that exchanging the values of pj and ps can never decrease the objective value. Define equivalent to with and j exchanged:
Pj i
9r
Pt
r=j
Pr
otherwise.
Let o)) = P:l((i) S k). We want to show that minger, o(p.o) S minge,, v(g. rr). Consider any a E Σn. If (ii) < o(j), then u(p, Ø) I v(q.o). (f) > aj). define permutation to be equivalent to but with and exchanged:
[jj i
T(r)
(i)) r=j
( otherwise.
Note that v(p, o) 2 o)g,o), but T(g.T)= = v(p,o) 2 v(p, We now proposition that T E En- Let r [n] be any proposal such that o(r) (o(i), Then, r cannot be strictly above j. but since A(j) > A(i), r cannot be strictly above Similarly, r cannot be strictly below but since Bii) 2 B()., r cannot be strictly below j. Hence, must overlap both i and j. Therefore, exchanging and j does not violate any constraints, so r E Σn. Hence, we conclude that minger. v(p, σ)) I min F(g,o), Now, applying this exchange to every sequence of intervals within each M, yields the desired result.
Further, using symmetry and pruning together reduces the number of decision variables to Okk) from (nn).
Lemma B.5 (Pruning and symmetry give O(k) decision variables). After pruning per Proposition B.2 and grouping symmetric intervals per Proposition B.3 the optimization problem has < 22 decision variables.
Proof. After pruning all proposals with A(i) 2 k or B(i) n- k, there are at most k 1 proposals with B(i > O. The remaining intervals have B(i) = and can have possible values of A(i). they can be grouped into at most A - 1 groups with equivalent pú. Hence, the intervals can be grouped into at most 2k- - 2 decision variables.
Now, to complete the proof of correctness, observe that by Lemmas B.2 and B.4 the initial pruning and monotonicity constraints are without loss of optimality. We can additionally impose equality constraints p using symmetry (Lemma B.3 reduce the dimension of p Ok)). although this is not shown in Algorithm [2 simpler presentation. Because the linear program solved by the cutting plane algorithm a relaxation of the linear program (3), solution to the problem upper bounds the objective value of (3). By the correctness of the separation oracle (Theorem (.2). if the cutting plane algorithm converges
feasible (p, v), this therefore an optimal solution the full L.P.
To prove that the algorithm is guaranteed to converge within 6a+11 iterations observe that there are at most O(k") possible total orders of marginal probabilities (after applying symmetry and pruning) consistent with the partial order given by monotonicity constraints Рылía] Z PMr[++]] Vi E [IM]- VM E {M...... My. Note that for any total order of the proposals, there are k++1 possible constraints that the separation oracle can return, because the possible constraints are determined by the order of the Pt. Hence, there are at most 0(k*+1) cuts that could be added to the linear program and so the cutting plane algorithm must converge in ((S iterations. The initial LP only contains (nn) constraints, so the LP never has more than O + k k0+1y constraints.
B.1 Finding Minimal Set of Chains for (A, B) Partial Order
In order to partition intervals using A and B as per Proposition B.4 we need to compute such a partition, known as a chain covering of the set. We would like the partition into as few sets as possible in order to add as many constraints as possible to the problem and reduce runtime of our optimization algorithm. There are practical general methods solve this chain cover problem for any partial order in time O by computing the maximum matching of an appropriately constructed bipartite
19

---

graph. In our case, where the partial order has specific structure, we can solve the minimal chain cover problem even more efficiently in time On O' log The algorithm described below equivalent to an algorithm given in 17 [Chapter 7, Algorithm 7.1] for minimal coloring of permutation graph. For completeness, we reproduce the algorithm and proof of optimality our problem setting below.
Algorithm 5 Greedy minimal chain cover for the product order IR?
Require: A set of points {(a,, b,) 1i=1 with partial order i j > aj 2 aj and b; 2 bj. Ensure: A set C = {Ci...., C. of non- mon-increasing chains that covers [m].
Sort the indices in decreasing order of a so that as, 2 atz 2 ... 2 asa breaking ties by decreasing b
2: C C
3: for f- to n. do
4:
Let C. be the chain with the smallest buil(C) such that (mil(() 2 b..
5:
if C' exists then
6:
Append i to end of C
7:
else
8:
Create new chain Ceew f {it} and append it to C
9:
return C
Proposition B.6 (Correctness of minimal chain cover algorithm). Algorithm 5 returns a minimal size chain cover C = {Ci...., G.} of the partially ordered set ([n), =) defined by i ž j + a; 2 aj and b, by. It runs in time On I). n).
Proof. First, proposition that each C, is chain. Indices are processed in non-increasing a-order, so within any chain the a-coordinates never increase. A point is appended to C, only if the current tail q of C, satisfies b, 2 by: hence (a,, by)) ž (ap.
Now, to show that this set of chains is minimal we will invoke Dilworth's Theorem, which states that the minimal number of chains that cover a poset (its width) is equivalent the length of the longest antichain (sequence of incomparable elements.) We argue that the set of final tails of each chain forms an antichain. Let T {t_..... t. } be the final tails ordered that b, < b,, < < bi.- For any chains i, with < j we have at, 2 at,, otherwise ij would have been added to chain i. But, be; < b so 1 and fy are incomporable. Thús the elements of T are pairwise incomparable and 7 is an antichain of size v. Now, let A C [ri] be any antichain. Map each A cannot lie the same chain, hence |A|| S v. Therefore, the length of the longest antichain is v, so the width of the partially ordered set is v and the algorithm has returned a minimal number of chains.
A to the chain that contains
it. Two distinct elements
The algorithm can be implemented in time (n log n), because the set of chains is always sorted in increasing order of the b value of their tails so we use binary search to find the chain in which to insert each i (step 4.)
C Systematic Sampling
The method known as "Systematic Sampling", works by first computing cumulative probabilities
S S-= = Py. Then, it selects random starting point uniformly from the interval (0.1) and picks exactly one item from each of the l: intervals obtained by adding integers = 0. k to the starting point u. Each item selected if at least one of these evenly spaced points falls within its corresponding cumulative interval 1, Thus, the algorithm guarantees selecting exactly distinct items without replacement, each with the correct marginal probability pt. Additionally, we can initially shuffle the items uniformly at random, so that the algorithm replicates the expected behavior of uniform random sampling for items with equal values of p. The algorithm requires two passes over all the proposals and hence runs in time O(n).
20

---

Algorithm 6 Systematic Sampling
Require: Integer k, probability vector p E [0, 1] with Ep: = k
Ensure: A subset of k items sampled without replacement from [x] where item E [xi] included with marginal probability Pt-
1: Compute cumulative sums: So +- O, Ej= for i to
2: Sample Uniform(0,1)
3: for each m from 0 to k - do
4:
Include item in the sample where (u + m) (S
D Enforcing Monotonicity in Budget
In this section, we show that it is possible to solve MERIT sequentially and simultaneously satisfy ex post validity and monotonicity in budget k. This may come at the cost of ante optimality.
Algorithm 7 Solving Sequence of Optimization Problems for Monotonicity
Require: Sequence of intervals {[ei, ue]he[m) number selected k.
1: Let p()) (0......))
2: for E [:] do
3: Compute ante optimal p for selecting top-i proposals with the additional constraint
Pj 2 Pj Vj E [n] to obtain optimal p())
(8)
4: Post-process ( with Algorithm
5: return ((4)
Proposition D.1 (Monotonicity in budget k). Algorithm
guarantees that pl0) pl6-1) for all
i [k].
Proof. Let p(t-1) be the output i-1
i- and (()) be the output for iteration before post-processing
and p(+)) be the final output after post-processing. If an interval b is strictly below set of intervals
A, either pb (6-1)
pb (i-1) 0 and p6- (i-1) Va A. If pb (i-1) O. clearly Pb > p/: (i-1)
(6)
If Pb (4-1)
0 and pa
(i-1) O Va E A, then Pa
(()) = for all E A and Pb 2 Pb
((6)
(1-1) so the
post-processing will not decrease pb all and monotony will be satisfied.
E Existing Randomized Selection Algorithms
In this section, provide precise pseudocode for existing algorithms against which compare our MERIT approach in Section Note that for Randomize Above Threshold, may be chosen independently of the data (e.g., as absolute quality cutoff) based the data (e.g., as the k-th largest lower bound).
Algorithm 8 Swiss NSF Selection Algorithm [21]
Require: Set proposals with point estimates and intervals: number accepted
Rank proposals by decreasing point estimate and let e(1) be the point estimate of the k-th ranked proposal.
2: Let A be the of proposals with lower bound strictly above e(R)- R be the set of proposals with upper bound strictly below E(R), and P the set of proposals with intervals that contain C(R)-
3: Accept A, reject R, and accept (k: [Al) proposals chosen uniformly at random from P.
21

---

Algorithm 9 Randomize Above Threshold
Require: Set of proposals with point estimates and intervals; number accepted k.
1: Choose a threshold T (potentially based on the data.)
2: Reject all intervals strictly below T and select uniformly at random among the remaining.
F Details of Interval Generation
In this section, we provide additional details on how intervals may be generated.
F1 Interval Generation Methods
Below, we describe five potential methods of producing "Knightian" uncertainty intervals, where funder does not draw probabilistic conclusions about the intervals.
1. basedd intervals: in many grant funding panels most proposals have been reviewed by large fraction of the reviewers. For example, in the Swiss NSF process [21] over 91% pairs of reviewer-proposal pairs received scores. If every reviewer scored every proposal, then the funder need not worry about reviewer miscalibration--th tendency for different reviewers interpret the review scale differently [59]. Hence, the funder may first impute values for missing scores and then aggregate across reviewers. The funder can make minimal assumptions by adopting Manski bounds [34, (4).. Manski bounds generate intervals by imputing missing scores with a range of possible values from the minimum to maximum score. The aggregate scores are then given as an interval over the range of possible imputed values. Hence, the intervals represent the plausible set of values for each proposal's aggregate review score, without probabilistic interpretation.
Intervals from model ensembling: in many cases, funder may have a number of plausible models of their data, each of which produces point estimates. Hence, the funder can estimate all plausible models of the data and ensemble into intervals by taking quantiles of the point estimates. This type of ensembling has been applied to time-series forecasting problems where it is known as Quantile Prediction Averaging [39].
3. Intervals based on expert input: frequently cited motivation for randomization is concerns about prejudice, for example, against highly original ideas or junior researchers. The funder may not be able to reliably estimate such sources of error from their observational, potentially sparse data, but can rely on prior controlled experiments that establish the rough magnitude of prejudices in the review process.
4. Multi-criteria aggregation: Reviewers are often given a number of criteria on which proposals are rated, for example "intellectual merit" and "broader impact." The funder can consider multiple valid ways to aggregate criteria and generate intervals over the set of possible aggregations.
5. Robustness to mis-speciiicctton of model used to generate intervals: the funder may use a probabilistic model to estimate intervals and then draw conclusions about the ordering of the intervals, but may not trust other information about the distribution. This is common assumption in the literature on distributionally robust optimization (see [41 for survey), where the funder is said to be optimizing over "support-only" ambiguity set.
F2 Implementation of Interval Generation in Experiments
Our worst-case objective ( assumes that any ordering consistent with quality intervals could be the true ranking of proposals. The utility is then defined as the worst-case expected fraction of top-k proposals chosen over all possible orderings consistent with the intervals. We simulate generating such quality intervals in three scientific peer review scenarios and measure the performance of different selection methods with respect to our worst-case objective. First, we replicate the scientific grant funding process of the Swiss NSF. We use publicly released review data from their 2020 grant review process consisting of = 353 proposals [21]. We generate intervals using their linear model of reviewer miscalibration. Additionally, we generate imputation-based based intervals for the Swiss NSF data using Manski bounds to impute missing reviewer-proposal scores and aggregating scores across reviewers using the median. Second, we use paper review data from the NeurlPS 2024 conference.
22

---

available on OpenReview, which includes n 4035 accepted papers. We simulate a process of allocating long talks (orals and spotlights) among accepted papers, inspired by the USENIX Security 2025 [57 Third, we use paper review data from the ICLR 2025 conference, available on OpenReview, which includes all = 11520 papers submitted to the conference. We simulate allocating paper acceptances at this conference. For NeurlPS and ICLR, there is no standard method to generate intervals. We therefore implement three different methods for generating intervals on NeurIPS and ICLR data:
(1) Leave-one-oul intervals (1.00): taking inspiration from "jacknife" or leave-one-out intervals [6] we compute the range of possible mean review scores, leaving out one reviewer at a time for cach paper.
(2) Gaussian error model credibility intervals: similar to the Swiss NSF's intervals, we assume Gaussian model generates review scores based on underlying true quality scores and infer credibility intervals for the true quality. Specifically, for paper i, we assume paper has true quality 0, ~ (0, 2), precision T; Gamma(L, (1, 1), and review scores the paper are drawn i..d. from [(0, Ti). The parameters of the priors are chosen to closely match those of the Swiss NSF model. We infer 50% credibility intervals for 0 given the observed review scores using MCMC.
(3) Subjectivity intervals: NeurlPS 2024 and ICLR 2025 both asked reviewers to provide numerical scores of papers' soundness, presentation, and contribution in addition overall scores. Previous works have observed that different reviewers may have different subjective views of which criteria matter to paper's quality introducing arbitrariness into overall review scores (28). One proposed approach to mitigate this subjectivity is to learn mapping from sub-criteria to an overall score based on peer review data (8 and then use this mapping to adjust the review score. We generate intervals by applying this method to adjust scores and taking the interval to be all values in between original scores and subjectivity adjusted scores.
We note that there are no confidence intervals for estimates generated in the worst-case setting as these are generated for a single dataset with single set of intervals, where there is no randomness in the data generation process.
G Additional Experiments
In this section, we provide additional experimental results comparing the performance of randomized selection rules.
G.1 Ablations
We additionally conduct ablations on parameters of both models that provide insight into the relative performance of different methods under varying model settings. First, in Figure 4 we show utility of different methods in the Swiss NSF's linear miscalibration model varying the degree of miscalibration. As the magnitude of miscalibration increases, deterministic selection using mean score degrades greatly in performance it does not account for eтror due to miscalibration at all. Meanwhile, the other methods perform similarly and maintain fairly high expected utility, even with a large degree of miscalibration. We observe similar results in additional ablations of the miscalibration model and when selecting the top one-tenth of proposals instead of top one-third, as shown in Appendix G
In Figure 5 we test Manski bounds on the Swiss NSF dataset of grant reviews, where we impute missing values with the full range scores. We artificially drop reviews to increase the sparsity of review scores, leading to worse utility. With 40% sparsity, deterministic achieves near zero utility as almost all intervals overlap. At all sparsity levels, MERIT outperforms both Swiss NSF and Deterministic selection. In fact, at sparsity of 05 to 0.25, the optimality gap between Swiss NSF and MERIT is the same as that of deterministic selection and MERIT
G.2 Qualitative Comparison of Outcomes
In order to give insight into how differing approaches may lead to different types of lotteries in actual peer review settings, we present qualitative differences between MERIT and the Swiss NSF approach.
In Table we provide a high-level comparison of the marginal probabilities of the sampling proposals given under MERIT and the Swiss NSF algorithm. One simple comparison point is on the number of
23

---

0.8
0.-
Determinetto (mean socre)
Swiss NSF
Mt-II
0.8-
Expectod
0.6
(menn
6.7
Detarministic (modal score)
Swiss NSF
Wordt Case UtIe
0.6
0.4
da (Linaar (rrtion))
0.3
(a) Swiss NSF simulated data.
Expected Uniley
0.9
0.1
0.8-
0.15
0.20
0.25
0.30
0.35
0.40
Fradton Proppss Pairs Missing (Sparsity)
6.7
Figure 5: Worst-case utility over Manski bound intervals as a function the fraction of reviewer- proposal pairs missing, with random dropping of review scores to increase sparsity for the Swiss NSF panel review dataset. Error bars show boot- strapped 95% CIs for the sample mean over 50 trials of randomly dropping review scores.
Ss (Linear Misca lbratin)
(b) Conference simulated data.
Figure Ablation study comparing methods under the Swiss NSF's model of linear miscal- ibration with varying levels of miscalibration. Error bars show bootstrapped 95% CIs for the sample mean over 50 samples of randomly gen- erated data from the model.
proposals that entered into lottery (% Random) under each algorithm. We find that MERIT tends to randomize over an equal or greater number of candidates than the Swiss NSF algorithm-on Swiss NSF data, NeurIPS LOO, NuerIPS Gaussian, and ICLR Subjectivity the two algorithms randomize similar number of candidates, while NeurlPS Subjectivity, ICLR LOO, and ICLR Gaussian, MERIT randomizes over more candidates than Swiss NSF. The biggest difference between the two approaches with respect the outcome, is that the Swiss NSF rule assigns the same probability to every proposal that enters its lottery tier, whereas MERIT allows a range of probabilities: the broadest span is NeurIPS Gaussian where marginal probabilities range from 0.16 to 0.83. This reflects the additional flexibility of MERIT' lottery. Finally, the relative weight MERIT puts in the "certain accept" group compared Swiss NSF (p 1) is not uniform-it is higher than Swiss NSF for NeurlPS Subjectivity, essentially tied on the original Swiss NSF set, and lower for the two ICLR variants, the split between guaranteed and lottery funding depends the structure of the intervals in each study rather than following a single trend.
MERIT
Swiss NSF Algorithm
Dataset
% Accept
% Random
Range of p
% Accept
% Random
Value of
Swiss NSF
28.3
3.4
(0.5, 0.5)
28.0
3.7
0.54
Swiss NSF (Manski Bounds)
5.4
26.9
(0.75, 0.94)
5,4
26.9
0.92
NeurlPS LOO
3.4
16.5
(0.36, 0.94)
3.4
17.9
0.34
NeurIPS Gaussian
2.2
25.7
(0.16, 0.83)
4.0
27.5
0.20
NeurIPS Subjectivity
4.5
18.7
(0.14, 0.45)
1.6
10.4
0.78
ICLR LOO
11.1
32.4
(0.51, 0.88)
22.0
22.8
0.44
ICLR Gaussian
9.5
34.4
(0.45, 0.87)
21.2
27.7
0,4
ICLR Subjectivity
17.6
24.3
(0.50, 0.88)
16.2
25.2
0.63
Table 1 Comparison of marginal probabilities of acceptance by MERIT and Swiss NSF on each dataset. "Accept" guaranteed to be selected (p 1), while "Random" entered into lottery
(0 < p < 1).
24

---

H
Additional Analysis of Computational Runtime Case Studies
In Figure we show the runtime in seconds of the MERIT algorithm (including all pre- precesssssnng and post-processing steps) on each dataset as a function of the acceptance rate. For all methods, the algorithm runs in under five minutes. Runtime increases with acceptance rate, which is expected because the number of constraints grows with the number of selections A:. We find that the cutting plane algorithm converges in under 30 iterations for all datasets, meaning that it solves under 30 linear programs. Furthermore, the largest linear program solved has 25,000 constraints when choosing k = 5760 of the n = 11520 ICLR papers, suggesting that the cutting plane algorithm scales well in and k.
Siisa HST (a Dourdo
KOUF
No LOO
K2LR doasciinn
Neattre Oukcutar
12UK Suijacii
202
202
153
153
Accuplance Vaalo (811)
Figure 6: Runtime of MERIT on standard personal laptop (in seconds) as a function of acceptance rate using review data from the Swiss NSF (n = 353), NeurlPS 2024 accepted papers (n = 4035) and ICLR 2025 papers (n 11520).
In Figure 7 present the number of cuts and the number of iterations it takes for the cutting plane algorithm to converge. We note that the size of the LP solved and convergence rate could potentially be optimized further by strategically pruning cuts from the linear program at each iteration, but even without additional optimizations the algorithm yields practical performance.
hor Manak (oundo)
leurte Sutadiiyyy
Secpeciviyyy
Neath's LOG
CURUUC
7200
Narbaar of Incstere
Nunburet 0/5
1200
1000
Acceptance
Acceptence Etale (bm))
Figure 7: Convergence of cutting plane algorithm as a function of acceptance rate using review data from the Swiss NSF (n = 353), NeurIPS 2024 accepted papers (n 4035) and ICLR 2025 papers (n = 11520). The number of cuts corresponds to the size of the largest linear program solved in a single iteration of the cutting plane algorithm and the number of iterations corresponds to the total number of LPs solved before convergence.
I Ex-Ante Optimal Uniform Random Lottery
An additional benefit of MERIT is that it can be easily adapted to handle constraints on the form of the lottery. In some contexts, funders or decision-makers may prefer to implement a uniform lottery, where all candidates subject to randomization are selected with equal probability. This form of randomization may viewed as simpler, more transparent, and more acceptable to participants than one in which probabilities differ across candidates.
25

---

To accommodate such requirements, the optimization problem underlying MERIT can be modified constrain all randomized candidates to share common probability of selection. This yields the ex anle optimal uniform lottery that still satisfies the ex post validity constraints. The resulting optimization problem is given below:
max RER,
V
subject
p:l1o(11 S k), Yo E En.
(4)
Σ ps k and 0 P. 1, [n] 1=1
Pr E (0,1,c), ViE
(5)
P: 1 or Pj Vi.j E [n] 4 2 u0
(6)
The optimization problem is equivalent to the original optimization problem 2 with the addition of constraint (5), which forces the lottery be uniform, and constraint (6). which ensures ex post
validity. These additional constraints turn the lincar program into mixed integer program (MIP). Notably, this MIP has polynomial in n constraints, excepting the worst-case ordering constraints present in the original optimization problem. Hence, this be solved using Algorithm (the
MERIT cutting plane algorithm) initialized with the additional integer constraints.
Empirical Performance. Figure compares the uniform variant of MERIT (MERIT Uniform) against the base MERIT and the Swiss NSF mechanism across datasets. The results show that the uniform constraint has minimal effect on expected utility, while preserving improving worst-case robustness under our model.
Deterministic (mean)
Deterministic (model)
Swiss NSF
MERIT Uniform
MERIT
0.95
Worst- cast Utility
0.8
+0.101
Expected Uulity
0.9
0.6
+0.1
0.4
+0.0K5
0.85
0.2
0.8
Swiss NSF
NeurlPS LOO
NeurlPS Gauss
NewrIPS Subj
ICLR
ICLR Gauss
ICLR
Swiss NSF
Conferenee
(a) Linear miscalibration model.
(b) Worst-case interval ordering model (our model).
Figure 8: Proportion of top-k proposals selected by different methods with quality data generated under the Swiss NSF model of linear miscalibration and under our model of worst-case over feasible rankings. MERIT Uniform matches performance of algorithms designed for the Swiss NSF's linear model and recovers much of the gap between MERIT and other methods in the worst-case over intervals defined by our model
Table 2 compares the qualitative outcomes under different algorithms. Notably, MERIT Uniform achieves nearly identical marginal acceptance probabilities and competitive runtime performance, suggesting that funders can adopt this more interpretable form of randomization without substantial efficiency loss.
26

---

MERIT
Swiss NSF
MERIT Uniform
Dataset
% Ace
Rand
p
Ace
% Rand
p
Acc
% Rand
P
Swiss NSF
28.3
3.4
0.5-0.5
28.0
3.7
0.54
28.3
3.4
0.5
NeurIPS LOO
3.4
16.5
0.96-0.94
3,4
17.9
0.34
3.8
16.1
0.36
NeurIPS Gaussian
2.2
25.7
0.16-0.88 0.83
4.0
27.5
0.20
4.9
19.3
0.24
NeurIPS Subjectivity
4.5
18.7
0.14 0.45
1.6
10.4
0.78
3.6
16.1
0.37
ICLR LOO
11.1
32.4
0.51-0.88
22.0
22.8
0.44
11.1
25,2
0.83
ICLR Gaussian
9.5
34.4
0.87
21.2
27.7
0.40
17.9
25.6
0.56
ICLR Subjectivity
17.6
24.3
0.49-0.88
16.2
25.2
0.63
23.0
18.4
0.49
Table 2: Comparison of marginal probabilities acceptance by MERIT, Swiss NSF, and MERIT Uniform each dataset. "Acc" guaranteed be selected (p 1), while "Rand" entered into lottery (0 < p < 1). MERIT shows the range of probabilities, while Swiss NSF and MERIT Uniform assign single uniform probabilities.
Finally, Table compares the runtime of the original and uniform version of MERITThe runtime of MERIT Uniform can significantly slower than MERIT due to solving an integer program, but still runs in reasonable time, rumning in 40 mins instead of 4 mins in the slowest case of ICLR Gaussian, which has over 10,000 candidates from which to choose. Additionally, we performed no additional optimizations of solver parameters in evaluations of MERIT Uniform, there may be further speed ups that are possible.
Table 3: Runtime Comparison for MERIT and MERIT Uniform
Dataset
Runtime (seconds)
MERIT
MERIT Uniform
Swiss NSF
0.046
0.043
Swiss NSF (Manski Bounds)
0.029
0.036
NeurIPS LOO
0.941
1.301
NeurIPS Gaussian
5.618
43.278
NeurIPS Subjectivity
1.883
11.319
ICLR LOO
43.561
194.366
ICLR Gaussian
242.222
2,346.693
ICLR Subjectivity
21.060
22.225
27

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: We detail our contributions in the abstract and introduction, which describe each main section of our paper (novel problem definition, efficient algorithm, axiomatic comparison, and demonstration of practical applicability via case studies.)
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification:
Provided in the Discussion section.
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). ). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
28

---

Answer: [Yes]
Justification: We include all propositions and proofs are in the supplemental material, with proof intuition in the main text.
Guidelines:
The answer NA means that the paper does not include theoretical results. All the theorems, formulas, and proofs in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We explain our setup in Section and provide additional details on implemen- tation and experiments in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
29

---

Question: Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Answer: [Yes]
Justification: We include zip file of our code the technical appendix. The data all based on existing publicly available data, and we provide code to replicate our interval generation methods from this data.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (https://nips.com cc/
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurIPS code and data submission guidelines (https
/
wwwp://wweeeeeeeeeeeeeeeee Sawteees//ooSSmmiiiiia SowpoddeummsssiooPPe.. wwiiiisssiiiicyiyioo for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The settings of parameters for our algorithm provided in the supplementary material.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer:[Yer
The only experiment with variability in outcomes is the computational re- sources experiment, for which we run our method 10 times and report 99% Cls, which are extremely narrow because the runtime is quite consistent.
Justification:
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emoo bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
30

---

The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall run with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative еπτог rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: Provided in Section
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate type of compute workers CPU or GPU, internal cluster, cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted the paper conform, in every respect, with the NeurlPS Code of Ethics atttps//nnuure httt:::://euriigp...cp/bll////// wttppubllc/hhiccsiieeeeleeeeeeld
Answer: [Yes]
Justification:
We have reviewed the Code of Ethics and ocnfirm our paper conforms to the
Code.
Guidelines:
The answer NA means that the authors have not reviewed the NeurlPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the motivations for using randomized algorithms in socially impactful settings throughout the paper.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
31

---

If the authors answer NA or No, they should explain why their work has no societal impact or why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), faimess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms to monitor how a system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: No such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards challenging, and many papers do not require this, but encourage authors to take this into account and make best. faith effort.
12. Licenses for existing assets
Question: Are the creators or original owners of assets (e.g., code, data, models), used in the paper. properly credited and are the license and terms of use explicitly mentioned and properly respected?
Answer: [Yes]
Justification: We credit OpenReview and the Swiss NSF for the data they have openly released that we use in this paper.
Guidelines:
The answer NA means that the paper does existing assets.
The authors should cite the original paper that produced the code package or dataset.
The authors should state which version of the asset used and, possible, include a URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website), the copyright and terms of service of that source should be provided.
32

---

If assets are released, the license, copyright information, and terms of use in the thcode has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
package should be provided. For popular datasets, paperswi
latasets.paperssit
For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
No new assets released.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moode as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create an anonymized URL or include an anonymized zip file.
14. Crowdsoureing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then much detail as possible should be included in the main paper.
According to the NeurIPS Code of Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country institution) were obtained?
Answer: [NA]
Justification: No human subjects.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing research with human subjects.
Depending the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
33

---

We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors adhere the NeurIPS Code of Ethics and the guidelines for their institution.
For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology. scientific rigorousness, or originality of the research, declaration is not required.
Question:
Answer: [NA]
Justification: LLMs are not used as component of the core methods in this research.
Guidelines:
The answer NA means that the method development in this research does not involve LLMs as any important, original, or non-standard components.
Please refer to our LLM policy (https: /ttp:s:///euups..... /0tps//eec.oooeeeeeee/22//////// for what should or should be described.
34


Reviews:
[{"review_id": "eLo1DmYwHW", "reviewer": "Reviewer_BoxL", "summary": "This paper studies the problem of selecting the top-$k$ candidates under uncertainty (e.g., in grant funding). The authors proposed a principled approach that relies on the ordering of the interval (that does not correspond to a probability measure over possible outcomes). The proposed approach, MERIT, is a polynomial-time algorithm that can approximately obtain the optimal solution.\n\nIn the absence of ground truth for evaluation, the authors propose an axiomatic approach to compare the randomized mechanisms, and demonstrate that the MERIT algorithm prevents the \"maximal instability\" and can achieve \"monotonicity in budget\" with some modifications, while existing mechanisms do not satisfy. The author further evaluated this algorithm on real-world dataset and show that the proposed algorithm can easily scales to thousands of candidates.", "questions": "See \"Weakness\" above.", "limitations": "Yes.", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n\n1. Novelty and significance. This paper studies an important problem of selecting top-$k$ candidates under noisy evaluation, which could bring great social benefits. This paper presents a novel algorithm that formalizes the idea of random selection for candidates that are statistically indistinguishable. The authors proposed ex-ante optimality and ex-post validity as two primary principles. While the induced problem is NP-hard, the authors proposed a polynomial algorithm that can obtain solution with $\\epsilon$ accuracy. It is empirically verified that MERIT can be applied to large-scale, real-world problems.\n\n2. Solid theoretical analysis and reasonable evaluation. The technical contribution is very solid, this paper formulating the selection as a maximin linear program that focuses on worst-case performance (as required by the ex-ante optimality). Solving this is technically non-trivial, and the authors derived a polynomial-time solution. Beyond the algorithm itself, the authors also proposed a way to conduct axiomatic comparison, and shown that the MERIT algorithm avoids the \"maximal instability\" and can be modified to satisfy \"monotonicity in budget\" -- for which other existing mechanisms failed to satisfy.\n\n3. Clear presentation. I enjoyed reading this paper. While the problems and techniques are complicated, the authors presented the problem and key insights clearly. Many technical details are properly deferred to appendix.\n\n\nWeakness\n\n1. More discussion on \"ex-ante optimality\". This is the first principle and used to derive the LP, but I think this principle is very conservative, and needs more motivation. As described in the paper, this principle focuses on the worst-case scenario (i.e., assuming an adversary chooses a ranking after top-k is selected) -- this doesn't match with the primary motivating example of funding grant, where it's more reasonable to assume there is an unknown ground truth ranking that doesn't change after the tok-$k$ is selected (unless funding grant somehow has negative impact on research that receives the grant, which hopefully is not the case). It would be good to have more discussion on why this is a reasonable principle to have; or if there is other principle that can better approximate the \"average case\" instead of the \"worst case\".\n\n2. Monotonicity v.s. ex-ante optimality. In theorem 5.1, the authors show that it is not possible to simultaneously satisfy ex-ante optimality and monotonicity in the budget. The monotonicity seems to be a more reasonable property, so this also makes me wondering if ex-ante optimality can be modified.\n\n3. It's not clear to me how to compare the numbers in Table 1. It would be great if the authors can add some more discussion in the main paper on the take-away message of this table.", "was_revised": true, "final_justification": "The author response clarifies the \"ex-ante optimality\" principle. I don't have further questions. I keep my original rating of 5.", "rebuttal": "Thank you for your careful reading of the paper and thoughtful comments on the problem formulation and axiomatic analysis. We address concerns below:\n\n1. **Motivating ex-ante optimality:** this is a great point, as the average case and worst case models are the two most common modeling assumptions in the ML/CS/Statistics literature for selection and estimation problems. An “average case model” would assume that the funder has a probabilistic model of the data (e.g., a probability distribution over candidates’ true qualities and reviewers’ errors in assessing candidate quality). Our ex-ante optimality formulation starts from the assumption that a funder lacks such a probabilistic model of their data and therefore operates under Knightian uncertainty. We believe this assumption that a funder lacks a probabilistic model is appropriate for settings where quality is determined by peer review for a number of reasons motivated by prior work. First, there are many proposed probabilistic models of reviewer errors, but these models have generally performed poorly in real deployments [Sha22, Section ‘Miscalibration’]. One possible reason for this poor performance is that in practice, human miscalibration is more complex than simple models [BGK05]. Further, decision makers generally lack ground truth data with which to evaluate whether a given probabilistic model of review data is appropriate in a given setting, making it difficult to rely on a probabilistic model. Additionally, even if a decision maker had a reasonable model of review scores, these probabilities may not represent meaningful probabilities to the funder, since review scales are arbitrary and likely do not map linearly to the utility of selecting a candidate. Finally, the future success of candidates in the settings we consider may be inherently hard to predict. In the context of scientific peer review, many works find that it is difficult to predict future citations and peer review scores are poor predictors [SWL+22, CL21, WDH+24]. In the related area of college admissions, recent work demonstrated that predictive models used to rank candidates are highly unstable with respect to their training data, resulting in arbitrariness in the ranking from any single model [LHZ+24]. For these reasons, we assume that the funder does not have a reliable probabilistic model of their data. \n\n    Thus, the funder operates in a setting of “Knightian uncertainty” where they estimate intervals that convey information about the relative quality of proposals but are not trusted as probabilistic estimates. We answer the question—how should they make decisions given this interval ordering? Given that any ordering consistent with the intervals is consistent with the decision maker’s information, taking the worst case over this set of orderings is a natural objective to optimize over their uncertainty. \n\n    We will expand on this motivation in the final version of the paper.\n\n    Additionally, one interesting direction along the lines you propose would be a smoothed analysis, which combines the average and worst case model. We conjecture that our MERIT algorithm would continue to be near optimal for that as well, but  leave a formal investigation to future work.\n\n2. **Monotonicity vs. ex-ante optimality:** We provide a simple algorithm to enforce monotonicity with MERIT in Appendix D, which allows for violation of ex-ante optimality, thereby prioritizing the monotonicity property. We agree that monotonicity is a natural property. Interestingly, our analysis shows that existing randomized mechanisms of the Swiss NSF and randomize-above-threshold (which do not satisfy ex-ante optimality) also violate monotonicity. \n\n3. **Results in Table 1:** Since there is no ground truth, we cannot say which algorithm is “better” at selecting high quality candidates based on this real world data. Therefore, our results in table 1 are meant to provide an interesting qualitative comparison of applying the Swiss NSF’s method and MERIT in real settings. The biggest takeaway in qualitative differences is that MERIT does in practice assign variable probabilities to different applicants (allowing for more nuanced lotteries), while Swiss NSF enforces a uniform lottery.\n\n## References\n\n[Sha22] Nihar B Shah. An overview of challenges, experiments, and computational solutions in peer review, June 2022.\n\n[BGK05] Lyle Brenner, Dale Griffin, and Derek J Koehler. Modeling patterns of probability calibration with random support theory: Diagnosing case-based judgment. Organizational Behavior and Human Decision Processes, 97(1):64–81, 2005.\n\n[CL21]  Corinna Cortes and Neil D Lawrence. Inconsistency in conference peer review: Revisiting the 2014 NeurIPS experiment. arXiv Preprint arXiv:2109.09774, 2021.\n\n[WDH+24] Aidan S Weitzner, Matthew Davis, Andrew H Han, Olivia O Liu, Anuj B Patel, Brian D Sites, and Steven P Cohen. How predictive is peer review for gauging impact? The association between reviewer rating scores, publication status, and article impact measured by citations in a pain subspecialty journal. Regional Anesthesia & Pain Medicine, 2024.\n\n[LHZ+24] Jinsook Lee, Emma Harvey, Joyce Zhou, Nikhil Garg, Thorsten Joachims, and Rene F Kizilcec. Algorithms for college admissions decision support: Impacts of policy change and inherent variability. arXiv preprint arXiv:2407.11199, 2024."}, {"review_id": "GXxM8ociHO", "reviewer": "Reviewer_XMKU", "summary": "The scope of this work is the design of a novel randomized selection mechanism that improves upon previous approaches under multiple aspects.\nMore precisely, the authors consider settings where a funder receives $n$ proposals and has to estimate the quality of each one based on the available data via intervals $[\\\\ell\\_i, u\\_i]$ for $i \\\\in [n]$.\nThe intervals essentially represent the inherent uncertainty of the funder, who then has to select $k \\\\le n$ of the received proposals.\nAfter reviewing some of the mainly adopted selection criteria and pointing out their shortcomings, such as the lack of maximal instability, the authors introduce a novel randomized selection algorithm called MERIT, whose distribution over proposals is computed by solving a linear program with combinatorially many constraints.\nThe authors then illustrate that it is possible to efficiently compute solutions to the same optimization problem in polynomial time, and provide experiments that confirm their theoretical findings.\nFinally, the work provides general results about randomized selection algorithms via an axiomatic approach that is reminiscent of results from social choice theory, as motivated by the authors themselves.", "questions": "- Could you expand a bit on the relationship with related work that is briefly discussed in the paragraph “Selection Under Uncertainty and Robust Optimization” at lines 332-340? In particular, could you provide more details as to why those techniques could not be adapted to the problem setting considered in this work?\n- The results in Section 5, and more precisely point (3) of Theorem 5.1, demonstrate the impossibility of guaranteeing both the ex-ante optimality and the monotonicity in budget properties. Do you think this impossibility result can be avoided by considering slight relaxations of the two axioms, namely approximate ex-ante optimality (given an additive/multiplicative approximation factor, or some other meaningful relaxation of the optimization problem) or $\\\\epsilon$-approximate monotonicity in budget (i.e., increasing the number of proposals cannot decrease the selection probability by more than $\\\\epsilon$, in place of just strict monotonicity)? What about other reasonable but slightly weaker axioms?\n- Theorem 4.3 is stated to hold only for ex-ante optimal solutions $p$. Does the statement actually hold for any solution $p$? At least, from the discussion below, it seems that the claim could hold for any feasible (and not necessarily optimal) ex-ante solution $p$, thus stating that the ex-ante objective does not worsen instead of simply stating that ex-ante optimality is preserved if it already holds. If so, then the result of Theorem 4.3 could be stated with more generality.", "limitations": "My only comment regarding limitations is that this work would benefit from a more thorough discussion on the fairness aspects of the proposed methodology, since it could impact critical selection processes.\nI believe there should be no ethical concern with respect to this work, but I would just prefer to see this point addressed in a more direct way, e.g., in the Discussion section (especially given the authors’ answer to point 2 of the Paper Checklist).", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 3, "originality": 4, "strengths_and_weaknesses": "The strengths of this paper lie in its fundamentally sound and principled approach to developing a novel randomized selection mechanism. One main contribution is the detailed overview of existing state-of-the-art selection criteria like randomize-above-threshold and the Swiss NSF one. This comprehensive analysis highlights the inherent limitations of these currently adopted methods, which supports the need for a novel randomized selection algorithm as addressed in this work with the proposed MERIT algorithm. This rationale is presented with clarity, demonstrating how randomization can address the shortcomings of deterministic approaches, and more importantly, why providing a selection criterion satisfying ex-ante optimality and ex-post validity is crucial in realistic scenarios. I also particularly appreciated the axiomatic analysis in Section 5, which adds more rigor and generality to the results provided in this work. Moreover, the experimental results show how the runtime of the proposed algorithm scales well as the acceptance rate $k/n$ increases. The outline of interesting potential questions for future investigations shows that the authors have a clear understanding of the implications of their work.\n\nDespite the significant strengths of the core methodology, some minor concerns warrant further elaboration. These concerns, as pointed out in the \"Questions\" below, primarily revolve around the broader implications and connections of the proposed mechanism to relevant topics. A more detailed discussion on how this new mechanism interacts with or differs from existing related notions and techniques would provide a richer context and demonstrate a deeper understanding of its place within the current research landscape. I believe addressing these points in a clearer way would strengthen the paper's overall contributions. In addition to this, I addressed some quite minor concents in the \"Limitations\" section that I would like to see reasonably addressed further compared to what is already present in the paper.\n\n**Typos:**\n- Line 155: should $n$ actually be $k$?\n- Line 156: “game-theoretic” instead of “game theoretic”\n- Line 261: missing period after “process”\n- Line 265: should “maximum” actually be “maximal”?\n- Line 272: “not” should be “no”\n- Line 273: “thorough” should be “through”\n- Line 287: missing “and” between “NeurIPS 2024” and “ICML 2025”\n- Line 325: missing space between “robustness,” and “but”\n- Line 1026: “ocnfirm” should be “confirm”", "was_revised": true, "final_justification": "The authors fully addressed my concerns and provided replies to my questions. Since they also committed to include the required changes in their paper, I am more confident that this submission should be considered for acceptance. Hence, I increased my score accordingly.", "rebuttal": "Thank you for your thoughtful comments on our theoretical results and detailed attention to the writeup. Below we clarify your questions:\n\n1. **Relation to existing work:** While our setting shares the high-level motivation of making robust decisions under uncertainty, classical robust and distributionally robust optimization techniques typically operate over convex uncertainty sets in continuous spaces (e.g., real-valued parameters that can take any value within a range). In contrast, our problem optimizes over a discrete uncertainty set of rankings: we optimize worst-case utility over a combinatorially defined set of feasible permutations consistent with interval constraints. This ranking-based uncertainty set leads to a non-convex inner problem that cannot be directly addressed by standard RO/DRO formulations.\n\n2. **Monotonicity vs. ex-ante optimality:** This is an interesting question — we propose a simple algorithm to enforce monotonicity for MERIT in Appendix D, which allows for violation of ex-ante optimality. Empirically, it seems like it does not lead to much worse performance with respect to worst-case ex ante optimality. In the modified algorithm, we obtain a locally optimal solution at each step (i.e., at budget $k$ we solve for the ex ante optimal vector of probabilities that is monotonic with respect to the solution obtained at budget $k-1$). It will be an interesting area for future work to try to prove that this algorithm (or a suitable modification) is near the global optimum when one wishes to maximize the ex ante objective with the additional constraint of monotonicity.\n\n3. **Generalizing Theorem 4.3:** Thanks for the great observation—yes the statement holds for any solution. Precisely, we could generalize the theorem statement as follows:\n\n   > Given a feasible solution $p$ to linear program (2), applying Algorithm 3 yields a post-processed solution $p′$ such that any sampling of $k$ proposals with marginal probabilities $p′$ satisfies ex post validity. Moreover, the objective value achieved by $p′$ is greater than or equal to that of $p$.\n\n   We intend the current statement of the Theorem to capture the usefulness of the post-processing algorithm to the MERIT algorithm, which takes as input an ex ante optimal solution to the problem. We will add the above statement as a remark to capture that the post-processing algorithm could be more generally useful.\n\n4. **Discussion of fairness:** Our algorithm satisfies two notions of \"fairness.\" First, one can think of the \"ex-post validity\" criteria as a type of fairness in randomized decisions that states that if one candidate is clearly better than another candidate, the stronger candidate should not be randomly rejected when the weaker candidate is accepted. Second, the motivation for our work starts from the premise that it is unfair that two candidates can have similar point estimates of quality, but large differences in outcomes under deterministic selection rules. Randomization is more \"fair\" in tie-breaking between these similar quality proposals than accepting one and rejecting another. In fact, existing literature [FC16,HTB22,GLM+25] argues that deterministic grant peer review processes introduce unfairness by tie-breaking between proposals on arbitrary, spurious signals like applicant seniority or preference for a certain subject area. We will add discussion of such fairness considerations to our Discussion section.\n\nWe will make sure to fix the typos you point out in the final version of the paper.\n\n## References\n\n[FC16] Ferric C Fang and Arturo Casadevall. Research funding: The case for a modified lottery, 2016.\n\n[HTB22] Serge PJM Horbach, Joeri K Tijdink, and Lex M Bouter. Partial lottery can make grant allocation more fair, more efficient, and more diverse. Science and Public Policy, 49(4):580–582, 2022.\n\n[GLM+25] Joanie Sims Gould, Anne M Lasinsky, Adrian Mota, Karim M Khan, and Clare L Ardern. Threats to grant peer review: A qualitative study. BMJ Open, 15(2):e091666, 2025."}, {"review_id": "cpGhTZcCck", "reviewer": "Reviewer_iwWb", "summary": "The paper formalises the problem of *top-\\\\( k \\\\)* selection under Knightian uncertainty by modelling each candidate with an estimated quality interval and requiring the decision-maker to maximise the *worst-case* expected number of true top-\\\\( k \\\\) items selected.  \nThis leads to a max--min linear program whose feasible set is defined by the partial order induced by the intervals.  \nThe formulation explicitly encodes two criteria: *ex-ante optimality* (robust utility maximisation) and *ex-post validity* (no dominated candidate may be selected).\n\nTo solve the intractable LP, the authors develop MERIT (Maximin Efficient Randomised Interval Top-\\\\( k \\\\)), which combines:\n\n- a polynomial-time separation oracle tailored to interval orders, and  \n- a practical cutting-plane implementation that converges in under 30 LP solves, even for \\\\( n > 10{,}000 \\\\) proposals.\n\nThe result is the first algorithm with provable polynomial complexity for this class of robust selection problems (Theorem 4.1).\n\nThe paper introduces two behavioural axioms: *monotonicity in budget* and *stability*, and proves that MERIT avoids \"maximal instability,\" a pathology observed in existing peer-review lotteries, while performing comparably on monotonicity (which is violated by all known randomised methods).\n\nExperiments on Swiss NSF grants and NeurIPS/ICLR conference data show that MERIT scales to real workloads and achieves higher worst-case utility than either deterministic top-\\\\( k \\\\) or the Swiss NSF rule, while enabling a more nuanced allocation of lottery probabilities.", "questions": "- The construction of the quality intervals could be elaborated further.  \n  In particular, I wonder whether the constructed intervals tend to be large, and whether the true quality distribution within the interval may be skewed, especially toward the lower end.", "limitations": "- MERIT is substantially more complex to communicate than a simple uniform lottery.  \n  Stakeholders may be reluctant to adopt a mechanism that requires explaining a max--min LP, cutting-plane optimisation, and post-processing steps.", "rating": 6, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 3, "significance": 4, "originality": 3, "strengths_and_weaknesses": "I found the paper very well-written and compelling.  \nIt presents a powerful optimisation framework with broad applicability to important real-world problems.  \nThe results are theoretically sound and empirically impressive.\n\n**weakness**\n\n- MERIT is substantially more complex to communicate than a simple uniform lottery.  \n  Stakeholders may be reluctant to adopt a mechanism that requires explaining a max--min LP, cutting-plane optimisation, and post-processing steps.\n\n- The construction of the quality intervals could be elaborated further.  \n  In particular, I wonder whether the constructed intervals tend to be large, and whether the true quality distribution within the interval may be skewed, especially toward the lower end.", "was_revised": true, "final_justification": "Dear AC and authors,\n\nI really like the paper. I would recommend this paper for an oral presentation, given the potential impact it could have across a range of applications.\n\nBest wishes, \nYour reviewer", "rebuttal": "Thank you for the thoughtful and detailed review, and for recognizing the theoretical and practical contributions of the work.\n\n1. **Constructing intervals:** Our approach is designed to not be too closely tied to any specific model, so there are a number of ways that a decision maker could generate intervals. We had limited space to describe approaches for interval generation in the main text but give more detail in Appendix F on methods for generating intervals in our experiments. For example, the funder could simply take intervals as the range from minimum and maximum reviewer score given to each proposal or they could compute bootstrapped estimates. Alternatively, they could accommodate subjectivity in the reviewer recommendations by allowing for different mappings from sub-criteria scores (\"quality\", \"clarity\", \"significance\", and \"originality\" in NeurIPS reviews) to an overall score.\n\n2. **Interval characteristics:** The width of the constructed intervals varies between methods and settings. Below, we provide the mean width of intervals for the methods we tested (after normalizing the units of all intervals to lie in [0,1]):\n\n    | Dataset | Average Width |\n    |---------|---------------|\n    | Swiss NSF | 0.0625 |\n    | Swiss NSF (Manski Bounds) | 0.0198 |\n    | NeurIPS LOO | 0.1259 |\n    | NeurIPS Gaussian | 0.1671 |\n    | NeurIPS Subjectivity | 0.0700 |\n    | ICLR LOO | 0.0917 |\n    | ICLR Gaussian | 0.1196 |\n    | ICLR Subjectivity | 0.0684 |\n\n    In terms of the skew of the intervals, because we do not know the ground truth for the \"true quality\" of each candidate in real-world datasets like ICLR and NeurIPS, it is not possible to measure where in the interval the true quality lies. This is the motivation for optimizing over intervals where the true quality could lie anywhere within each interval.\n\n3. **Complexity of communicating MERIT:** This concern is valid, as we discuss in the Limitations section, but we have reasons to be optimistic about adoption of MERIT. First, funders are using the current Swiss NSF method—this method has a simpler decision making part than MERIT, but a much more complex method of generating intervals (hence the overall method is complex.) In spite of the complexity of the Swiss NSF's interval generation method it has been put into use for real-world funding decisions. Second, we are currently speaking with organizers of such selection processes, and so far, complexity has not come up as an issue."}, {"review_id": "rWwx4KVXTD", "reviewer": "Reviewer_Api3", "summary": "This paper introduces MERIT, which is a randomized decision-making framework for evaluating and selecting items under uncertainty. The core idea is to use interval estimates of item quality rather than precise point estimates to model the inherent uncertainty in domains like peer review, hiring, and admissions. The authors propose a maximin optimization problem where the goal is to maximize the worst-case expected number of top candidates selected, considering all possible true rankings consistent with the given quality intervals (Knightian uncertainty). They develop a polynomial-time algorithm for MERIT, and demonstrate it scales reasonably on real-world datasets. Additionally, they provide an axiomatic comparison with existing ad hoc randomization methods and show how to ensure ex-post validity through a post-processing step.", "questions": "See weakness", "limitations": "Yes,", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n* The paper proposes a optimization-based framework for randomized selection, which seems much better than ad-hoc methods, and incorporates uncertainty through interval estimates and does not make strong model-based assumptions. \n* The maximin objective function, maximizing the worst-case expected number of top-k candidates, is interesting.\n* The paper demonstrates that the proposed algorithm is computationally practical.\n* Axiomatic Analysis against properties like \"monotonicity in budget\" and \"stability\" is interesting\n * Explicitly adding a post-processing step to guarantee ex-post validity (ensuring a superior item is selected if an inferior one is) seems like a reasonable thing to add to the proposed mechanism.\n\nWeaknesses:\n* I am not an expert in this area, but the core maximin LP formulation appears to be a re-invention of well known min-max or Stackelberg problems, which have been extensively studied in various related settings. The paper does not clearly position its maxmin formulation against existing works on Stackelberg/min‑max settings.\n\n* I'm a bit confused by lines (112-114): \"For example, in the Swiss NSF ... 91% pairs of reviewer-proposal pairs received scores .... reviewer miscalibration.\"\nDoes \"over 91% pairs of reviewer-proposal pairs received scores\" actually mean that, on average, each reviewer reviewed almost every proposal (like, say, 910 out of 1000 proposals)? Assuming it's not true, the usage for Manski-bound intervals seems rarely achievable in typical peer-review environments. Is this the they were not used for ICLR/Neurips experiments?\n\n* Axiomatic Criteria: One of the proposed axioms \"maximal instability\" (a tiny interval tweak causing a flip from fully deterministic to fully random) also appears to be a contrived, pathological case that may not reflect typical real-world intervals (as the example considers only three possible values of range for n proposals). More practically relevant axiomatic criteria like robustness against a range of adversarial behaviors like axioms limiting the opportunity for strategic manipulation by a single or subset of reviewers, would have been better. \n\n\n* Post-Processing: Ex-post validity processing could, in principle, be applied to other baseline mechanisms. Is MERIT intrinsically better than other mechanisms (in some way)?\n\n* Underutilization of Interval information: (Not a weakness in a real sense) While the algorithm is designed for Knightian uncertainty, the maximin approach seems to underutilize information within interval overlaps. For example, given intervals like [1,5] and [4,8], a human might intuitively feel B is \"more likely\" to be better,", "was_revised": true, "final_justification": "I did not give it a score of 5 because of my concern that the axiomatic criteria are based on a worst-case scenario, which is unlikely to happen in practice. I would have appreciated more practically relevant axiomatic criteria, which I believe can be studied under this model.", "rebuttal": "Thank you for the review and for your attention to the paper's comparison to existing methods. We are glad that you found the problem formulation and axiomatic analysis interesting. Below we address some of your concerns:\n\n1. **Novelty of the formulation:** While Stackelberg / min-max formulations are indeed well-established and widely used in many different applications, our paper makes two novel contributions:\n   - First, as we discuss in Section 4 of the paper, it is computationally challenging to solve for an equilibrium in the specific Stackelberg game we describe. Many related optimization problems—minimum weight k-ideal problem [FK94], maximum cut on interval graphs [ABMR23], interval graph completion [RAK91] and minimum weighted completion time of jobs subject to interval ordering precedence constraints [Woe03]—are known to be NP hard. Our work develops a novel, polynomial-time algorithm to solve for the Stackelberg equilibrium in our problem setting and this algorithm is also practically efficient.\n   - Second, to our knowledge, modeling uncertainty in selection decisions as a Stackelberg game has not been done previously and represents a new take on the problem.\n\n2. **Manski bounds:** The statement in the paper about the use of Manski bounds on the Swiss NSF data is indeed accurate. The Swiss NSF setting only has ~300 proposals and panels meet and discuss each proposal and then vote on all proposals. Hence, the only reason for missing evaluations is conflict-of-interest. Therefore, the Manski intervals indeed cover most review pairs.\n\n   For conference settings, like NeurIPS and ICLR you are correct that Manski bounds would result in meaningless intervals that cover the entire range of scores since there are few reviews per paper, so we do not use Manski bounds there. We test a number of methods for generating intervals in conference settings, detailed in Appendix F.\n\n3. **Axiomatic criteria:** Our interpretation of the maximum instability axiom is that all natural algorithms should obey even this very weak notion of stability. It was quite surprising to us that existing methods did not respect this axiom and could be \"maximally unstable.\"\n\n4. **Post-processing for monotonicity:** We agree that other algorithms could also be post-processed in order to enforce monotonicity. MERIT is intrinsically better with respect to the robust objective over interval uncertainty and also is guaranteed to meet the ex-post validity principle, which prior methods do not guarantee. Additionally, our algorithm to enforce monotonicity for MERIT empirically seems like it does not lead to much worse performance with respect to worst-case ex ante optimality. In the modified algorithm, we obtain a locally optimal solution at each step (i.e., at budget k we solve for the ex ante optimal vector of probabilities that is monotonic with respect to the solution obtained at budget k-1). It will be an interesting area for future work to try to prove that this algorithm (or a suitable modification) is near the global optimum when one wishes to maximize the ex ante objective with the additional constraint of monotonicity.\n\n## References\n\n[FK94] Ulrich Faigle and Walter Kern. Computational complexity of some maximum average weight problems with precedence constraints. Operations Research, 42(4):688–693, 1994.\n\n[ABMR23] Ranendu Adhikary, Kaustav Bose, Satwik Mukherjee, and Bodhayan Roy. Complexity of maximum cut on interval graphs. Discrete & Computational Geometry, 70(2):307–322, 2023.\n\n[RAK91] Ramamurthy Ravi, Ajit Agrawal, and Philip Klein. Ordering problems approximated: Single-processor scheduling and interval graph completion. In Automata, Languages and Programming: 18th International Colloquium Madrid, Spain, July 8–12, 1991 Proceedings 18, pages 751–762. Springer, 1991.\n\n[Woe03] Gerhard J Woeginger. On the approximability of average completion time scheduling under precedence constraints. Discrete Applied Mathematics, 131(1):237–252, 2003."}, {"review_id": "nCKtuGLaQh", "reviewer": "Reviewer_CFrv", "summary": "The paper is motivated by the need to select top choices from a large group of candidates, where the quality against some qualifying criteria is uncertain. Specifically, this issue is present in peer-review journals and funding proposals, where assessors can be uncertain about the quality of a submission. This paper proposes a polynomial time algorithm which, given estimated quality intervals, returns a quantity of top-proposals whilst maintaining some key assumptions. The paper proves these desired properties, compares with the current literature, and demonstrates practical applicability with peer review data.", "questions": "Can you give an intuition as to why the acceptance rate between the two algortihms is more varied for the ICLR dataset? (Table 1). \n \nDoes the edge-case of k=1 still maintain the stated properties? Specifically, its seems to me that the maximum instability for MERIT could occur in this instance, but would be keen to hear your thoughts.", "limitations": "Yes", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper is well-motivated with the limitations of related work explained, and the benefits of this apporach are clearly presented. There is a good level of discussion in the comparison between algorithms, which is grounded in both theoretical and empirical analysis.  \n \nIn Section 5, the discussion on violating the monotonicity in budget could be further developed, perhaps an intuition as to why the current algorithms do not hold this property. Appendix A, Figure 2 would benefit from axis labels to make it more understandable.", "was_revised": true, "final_justification": "No unresolved issues after discussion, I confirm my recommendation.", "rebuttal": "Thank you for your feedback and interesting questions about our axiomatic comparison. Below we address some of your questions:\n\n* Figure 2 in Appendix A gives the main intuition for why MERIT violates monotonicity. We will add clearer axis labels. The Figure has four intervals where interval 1 lies entirely above intervals 3 and 4, while intervals 2,3, and 4 overlap.  When k=1, the top 1 proposal (given the overlaps in the intervals) could either be proposal 1 or proposal 2 since these two intervals overlap, while 3 and 4 are strictly below 1. Hence, MERIT chooses uniformly at random between proposals 1 and 2. Meanwhile, when k=2, proposal 1 is always in the top 2 so it gets accepted with probability 1. This leaves 1 probability mass to allocate between proposals 2,3,4. Since any of these 3 proposals could be the other proposal in the top-2, they each receive probability 1/3 of acceptance. Therefore, proposal 2 is worse-off when selecting 2 proposals rather than 1 proposal.\nCan you give an intuition as to why the acceptance rate between the two algortihms is more varied for the ICLR dataset? (Table 1).\n\n* In our formal definition of “maximal instability” we specifically define the axiom for budget (k) ranging from 2 to n-2, because k=1 is not a particularly meaningful case to consider for the stability of selection algorithms. Just by definition when k=1, changing one proposal (highest one) can fully change the outcomes. This holds even choosing deterministically or for any other reasonable method. Hence k>1 is a more meaningful parameter range in which to consider the stability of an algorithm. We will clarify this in the main text."}]

Author rebuttal:

Meta-review: his paper proposes a principled and computationally efficient method for randomized selection under uncertainty, with applications to peer review, admissions, and funding decisions. The approach optimizes a maximin objective over interval uncertainty and satisfies ex-ante optimality and ex-post validity. It combines strong theoretical guarantees with a scalable algorithm and empirical evaluation.

The reviewers were consistently positive, highlighting the paper’s novelty, clarity, and practical relevance. The rebuttal addressed remaining concerns effectively, including motivation, modeling choices, and connections to prior work.
Decision: Accept (spotlight)