Strategic Hypothesis Testing
Authors: Yatong Chen
Safwan Hossain
Yiling Chen
Venue: NeurIPS 2025 spotlight ()
Keywords: Stackelberg Game; Hypothesis Testing; Strategic Machine Learning; Game Theory
Primary area: social_and_economic_aspects_of_machine_learning

Abstract: We examine hypothesis testing within a principal-agent framework, where a strategic agent, holding private beliefs about the effectiveness of a product, submits data to a principal who decides on approval. The principal employs a hypothesis testing rule, aiming to pick a p-value threshold that balances false positives and false negatives while anticipating the agent’s incentive to maximize expected profitability. Building on prior work, we develop a game-theoretic model that captures how the agent’s participation and reporting behavior respond to the principal’s statistical decision rule. Despite the complexity of the interaction, we show that the principal's errors exhibit clear monotonic behavior when segmented by an efficiently computable critical p-value threshold, leading to an interpretable characterization of their optimal p-value threshold. We empirically validate our model and these insights using publicly available data on drug approvals. Overall, our work offers a comprehensive perspective on strategic interactions within the hypothesis testing framework, providing technical and regulatory insights.

Strategic Hypothesis Testing
Yatong Chen
Safwan Hossain"
Max Planck Institute for Intelligent Systems
Harvard University shossaingg. harvard.
Tübingen Al Center
yatong www.@tuebineen..gg......
Yiling Chen
Harvard University
yilingêseas harvardd edu
Abstract
We examine hypothesis testing within a principal-agent framework, where a strate- gie agent, holding private beliefs about the effectiveness of a product, submits data to a principal who decides on approval. The principal employs a hypothesis testing rule, aiming pick p-value threshold that balances false positives and false neg- atives while anticipating the agent's 's incentive to maximize expected profitability. Building on prior work, we develop game-theoretic model that captures how the agent's participation and reporting behavior respond to the principal's statistical decision rule. Despite the complexity of the interaction, we show that the princi- pal's еπтогs exhibit clear monotonic behavior when segmented by an efficiently computable critical p-value threshold. leading to an interpretable characterization of their optimal p-value threshold. We empirically validate our model and these insights using publicly available data on drug approvals. Overall, our work offers a comprehensive perspective strategic interactions within the hypothesis testing framework, providing technical and regulatory insights.
1 Introduction
In data-driven decision making, the outcome assigned an agent often depends on data submitted by them, creating the potential for misaligned incentives. The role of strategic behavior in decision making systems has thus become burgeoning area of research in the machine learning community, from classification [ 2. regression [3 4 [S] and beyond [6]. However, limited literature exists on this perspective for a widely used and influential process in regulatory and scientific settings: hypothesis testing. Widely used in clinical trials, scientific research, and technological innovations [7] 8, 9 [O), hypothesis testing serves as a foundational method for assessing whether the evidence provided by participating agent is both statistically significant and convincing. Consider, for instance, the U.S. Food and Drug Administration (FDA), which oversees drug approvals by setting p-value threshold a for submitted clinical trials. Pharmaceutical firms incur substantial costs by participating in drug development and running trials in the hopes of being approved. Although falsifying results has high reputational and legal risks, firms free to decide if to participate and how large to run a trial. We argue these decisions are intimately shaped by the FDA's p-value threshold and the cost-beat best-benefit calculus of the overall process. We thus propose a principal-agent model to understand these nuanced strategic decisions surrounding participation and evidence collection for hypothesis test.
Equal contribution. Correspondence yatong. chen@tuebingen. mpp. de or shossaingg. war.iii...aarrr..... harvard. edu.
39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Bates et al. []]] initiated this literature by casting how agents respond to regulatory approval thresholds as a Stackelberg game. We extend this framework to capture additional real-world complexities that are abstracted away in their model. Specifically, we extend the analysis beyond binary notions of effectiveness and fixed trial costs, allowing agents to strategically choose their trial size based on the expected benefit relative to marginal per-sample cost. This extension not only enriches the analysis of agent incentives but also provides a more nuanced understanding of how regulatory policies can influence participation and decision outcomes. This richer model, however, leads to a more complex relationship between the regulator-specified p-value and the resulting error rates that they wish to control for instance, if marginal costs increase faster than the expected revenue gains, raising & could paradoxically reduce the likelihood of passing the hypothesis test. This is further complicated by the principal wishing to control any combination of Type I (false positive) and Type II (false negative) errors within our model. Indeed, while the broader literature focuses on only the former [1] 12 [3] 14], we argue that minimizing the rejection non-participation of effective products is also consequential to decision-makers especially when this is influenced by low revenue relative to costs, is the case for low-cost and orphan drugs [15] We outline our key contributions toward addressing these challenges below:
1.
Like Bates et al. M. we consider principal-agent (Stackelberg game) framework but relax several key assumptions to adhere more closely to real-world settings. We allow agents strategically choose their sample sizes as part of their best responses and account for the marginal cost of such samples. Further, we model the principal as aiming to minimize any combination of Type and II errors.
Framework:
2. Analysis of Component- Wise Losses: We rigorously analyze the principal's loss components as functions of the p-value threshold under agent best-responding behavior. Despite the complex interplay between several factors, the monotonicity of Type and Type II errors as a function of the p-value threshold o is preserved in piecewise manner. Specifically, there exists critical threshold á such that the error dynamics are monotonic within each segmented region: one for a < ā and one for a 2 à.
3. Empirical Validation and Policy Implications: We validate our theoretical findings through empirical analysis of drug approvals for three major drug categories and show that under our model, the commonly used p-value of 0.05 aligns well with median revenue thresholds in these markets. The results suggest our model captures the economic dynamics of this process and suggests new policy insights.
Our work is closely related the literature on economic aspects of statistical testing, p-hacking, contract theory, and FDA regulatory policies. A growing body of
Additional Related Works:
research examines the economic incentives in statistical decision-making 112 14
17). Shi 16
13
et al. [13] extends the work of Bates et al. [I]] (which is discussed above) by studying strategic hypothesis testing under general concave utility functions, providing bounds on the Bayes False Discovery Rate (FDR). Relatedly, Bates et al. [14] models the agent-principal interaction through a contract-theoredt lens for incentive alignment, differing from our Stackelberg game approach. Also related is the well-known issue of p-hacking. where researchers manipulate sample sizes or selectively report findings to artificially achieve statistical significance [88 10 19 g Note that the strategic behavior in our setting is not inherently malicious, but rather reflects rational decision decision-making based on cost, revenue, and expected trial outcomes. A good p-value threshold discourages agents with ineffective drugs from participating based on economic incentives: the costs of participation outweigh the potential benefits. This disincentives p-hacking insofar as data collection is costly. Our work also intersects with the literature contract theory, which examines incentive alignment in the presence of private information 20 21 23 24). Recent work by Min (25) applics contract theory to model FDA approval processes, analyzing how firms of different sizes choose between cheaper and more expensive trials. Further, Isakov et al. (26 employs Bayesian Decision Analysis (BDA) to optimize p-value thresholds by balancing Type I and Type II errors. Our empirical results leverage the statistical testing- besting-based framework used by the FDA. Typically, this is established through either two successful controlled trials (with a p-value < 0.05) or a single robust multicenter trial (with p-value < 0.005), as outlined in 77. We simplify this in our analysis by only considering the former.
2 Model
Preliminaries: Consider a principal who must decide between approving or rejecting a product manufactured by agent based on evidence provided by the agent. Let X e {0, 1) be Bernoulli
2

---

random variable indicating whether the product was effective on random instance. Let Po [[ denote the mean effectiveness of this product, with fo q. While μo is private (known only by the agent), the distribution is assumed public (known to both the principal and agent). Let (4b denote a baseline effectiveness (i.e. effectiveness of current products) and be treated as constant: the principal's s goal is to only approve products perceived to be at least as good as the baseline (no 2 14)), with (Ho - (46) denoted as effect size. An agent faces two possible decisions for their product. First, they can choose whether not to participate in the approval process and engage with the principal. Not participating, equivalent to n = O, incurs no cost and collects no revenue, If they participate. they must also decide on the number of samples E [Mmin Rmaar to collect and submit an evidence set Xr = (X1.. ..,) to the principal. This incurs a fixed cost co and a marginal per-sample cost Thus, cost(n) 1 [n 0] (co + en). If their product is approved, they carn revenue R. Agents are assumed rational and act to maximize their expected utility- expected revenue minus cost (see Definition 2.2 We assume the revenue and cost parameters are known to the principal.
Connecting this to our running example, the FDA (the principal) decides on the approval of new drugs from pharmaceutical companies (the agents) based whether the clinical trial data suggests them to be at least as effective as current alternatives on the market. If a firm follows through on the development of a new drug and participates in clinical trials, co denotes any fixed costs herein, and c, the per-subject marginal cost of the trial. An FDA approval means a lifetime revenue R for their new drug, while rejection means revenue. As such, the firm only participates if its expected profit (utility) is non-negative, and chooses the samples n to maximize this.
Hypothesis Testing: The agent's 's decision clearly hinges on how the principal uses the evidence set to evaluate effectiveness. We model this process on hypothesis testing given its ubiquity and relevance in settings like drug approval. manufacturing, public policy, and
7 g 10 Formally, let Ho = {uo I 14b} be the null hypothesis (the product is less effective than baseline) and denote the alternative hypothesis and approve the product. To expand, given an evidence set Xx, let j denote the empirical mean. Further, let Sn denote the random sum of n variables sampled from the baseline process with effectiveness (4g. Then the p-value is defined as the prohability of observing outcomes at least as good as the evidence, conditioned on the null hypothesis: p(uo, n2) = Pr[S,, 2 [H] P Observe that when the empirical mean of the evidence is worse than the baseline, the p-value will Tikely be higher than a, leading the principal to reject; otherwise, when p(uo. n) S a, the principal will accept. For an evidence set with n samples and p-value a, the critical region is the number of successes needed to reject the mull-hypothesis and thus be approved. Since the sample outcomes are Bernoulli and the sum random variable follows a Binomial distribution, it is common to use normal distribution @ to approximate the p-value and critical region:
7i, {Ho > [.... The principal uses p-value threshold
reject the null
Zo,n { 1 6anps, 11ps (1-1))) S a nus + + - a) mips11 - 14)))
where & is the CDF of the standard normal and -- its inverse. Formalizing the principal's s decision- making process means that we can now compile the probability that a product with effectiveness Po approved (the randomness is over the evidence set), and thereby the agent's 'ss utility:
Definition 2.1 (Pass Probability). For a p-value threshold and baseline effectiveness ps, a product with effectiveness oo and evidence set [x,1 = n is approved with probability:
Pass(a, (c, Ho: n) e Prip(uo, n) s a1 $
Za.n 17440
Vnppo[1 -10)
Non-participation is considered equivalent to = 0, and Pass(a, po. n2 0) 2 0, V yoo a. Definition 2.2 (Agent Utility). An agent with revenue R, cost parameters (co, c). and a product with effectiveness μo ~ q has the following utility for their participation/sample parameter n.
u(ax, no: n1) R Pass(a, Po, n) - 11 [n + 0] (co + cn)
While the agent aims to maximize their utility, the principal's 's goal is to choose a p-value threshold that minimizes some combination of their Type I and Type II errors, a standard desideratum in
While p value calculations depends on nob we drop this from function signatures since it constant.
3

---

hypothesis testing. In machine learning terminology, these are equivalent to minimizing the false positive (approving ineffective products) and false negative (not approving effective products) rates. These error components are defined with respect to the effectiveness distribution g. and we note that the principal may desire an arbitrary trade-off between the two error components. This principal loss and its objective are shortly defined in Definition 2.3
Game Theoretic Model: is evident that agents will choose n strategically to maximize their utility, while the principal selects a p-value threshold a to minimize a loss function, which we will define in Definition [.3 This naturally leads to a game-theoretic framework, In most regulatory settings (drug approval, manufacturing, etc), the principal must first communicate the acceptance criteria to all possible participants. Agents, on the other hand, can make their decision to participate and collect samples based on the revealed criteria. This interaction outlines a Stackelberg Game, an asymmetric model of strategic decision-making allowing agents to then make their optimal Decision/best-respnss (participation decisions and number of samples) thereafter" In such games, the core solution concept is the Stackelberg Equilibrium the optimal principal strategy, given that downstream agents will best respond. We formally define the game and its details below:
the principal first commits to the p-value threshold,
Definition 2.3 (Stackelberg Game in Strategic Hypothesis Testing). The principal-agen interactions in hypothesis testing setling outline Stuckelberg game defined by the tuple I = (g, R, C, Co, (b)) The best-response of an agent with effectiveness po ~ q when the principal commits to a p-value threshold is: Du (ax) arg max,, u(ax, Po, Po,
The principal in choosing a p value threshold a suffers the following loss when agents best respond (we denote Fail(-) = 1- - Pass(-) and App, Afa are constants that scale the respective loss terms):
C(ax = =Aso E [Pass(a, p (a))) < pb] po-aq
+Dfn E Ho0d
[Failla, μoo Ru (l (a)) 2 pb]
False Positive (Type I error) FP(c, g) Approval of ineffective products (pn))
False Negative (Type Il emor) & FN((x, q) Non-approval of effecttive products (10 (4a)
The principal optimal strategy and the Stackelberg Equilibrium is to choose a* = arg min, L(a, I).
In Section 3 and Section f
we will investigate the agent's best response behavior as well as how the principal chooses p-value threshold affects the loss defined in Definition 2.3
3 Agent Best Response
According our strategic model, after the principal releases p-value threshold a, the utility- maximizing agent must decide (1) if they want to develop the product and participate in the approval process, and if so, (2) how many samples they ought to include in their submitted evidence set. Notice that this decision-making process is ex-ante and relies on the agent's 's belief in their product's effectiveness (4g Intuitively, the agent intends to determine the optimal number of samples n to maximize their ex ex-ante expected utility; if this is negative, they have no incentive to participate. Mathematically, participation is defined by 1 [u(a, Mo, Pu (a)) 2 0), where u(a, Mo, n) is defined in Definition 2.2 and no (a) is the optimal number of samples. We now show that this can be efficiently computed by the agent. We sketch the proof below, with the full proof in Appendix B
Theorem 3.1. For an instance I and a released p-value a, an agent with effectiveness pio can compute their best-response (participation decision and number of samples) in Oglog Rmman).
Sketch. By leveraging properties of the normal CDF that define the pass probability, we first show that when Mo < Pbg, the optimal n = Tmmin- For 14 2 pbb we undertake a first and second-order analysis to partition the utility function into a constant number of intervals wherein it is convex or concave; the boundaries of these regions can be computed in constant time, and the optimal n within each region requires at-most a binary search. Once the optimal n is computed. it suffices compute the passing probability and participate if the corresponding expected utility is non-negative.
We now prove that despite the agent being strategic and their participation and sampling decisions dynamically changing in both a and Mo: several natural and intuitive results still hold. These are
Technically, our setting is a Bayesian Stackelberg Game since agents have hidden types (the effectiveness (o), but the principal knows the distribution q.
4

---

instrumental to the analyses of the principal's optimal/equilbrium p-value threshold. We first show that the participation behaviour monotonic in a (Lemma (.I) The full proof is in Appendix [B]
Lemma 3.1. For an instance I (q. R. c, co2 Hb) and p- p-value threshold a, if an agent with effective- ness Po parlicipates in the statistical test, then any agent with belief jm 2 po will also participate. Similarly, if an agent with belief (1 does not participate, neither will one with belief μ4 < 1/-
This monotonicity result immediately implies the existence of a particination threshold (r(()) for any given p-value a. That is, for any instance I and p-value a, agents with effectiveness μa 2 f4, (a) will always participate, and those below will not. As we subsequently show, the participation threshold is a crucial concept in understanding the principal's 's decision - determines the selection mechanism induced by the p-value threshold a, shaping the set of participating agents based on their effectiveness. Consequently, understanding how pr (a) changes as a function of the instance parameters R, c, a provides key insights into optimizing the principal's 's objective function. We now show that the participation threshold decreases as the p-value threshold a increases; further, it can be computed in log time and is agnostic to the effectiveness distribution q (full proof in Appendix (B).
Definition 3.1 (Participation Threshold). For an instance I and a p-value o, we denote (=((x) as the participation threshold if it is optimal for agents with effectiveness po 2 po((0) to participate, and for those with μoo < p=(a) to abstain.
Lemma 3.2. The participation threshold p- p=(a) decreases in the p-value o. Further, this threshold can be computed with e precision in O (log (t) log(Hmas)).
Proof Sketch. Since the passing probability of the hypothesis test increases with for fixed effect size, agents with lower beliefs are more likely to meet the passing condition as a increases. By the monotonicity of participation (Lemma 3.1). if an agent participates μ4, (a1) under a p-value a2 > Qt, then agents with beliefs greater than (1(a1) ) must also participate. This implies that the participation threshold under 02 must be or below the threshold under on, establishing that (o(()) is non-increasing in a. To determine (Ar (a), we can discretize the effectiveness space into 1 intervals and run binary search. The algorithm and correctness rely on Lemma 3.1 and Theorem 3.1
This formalizes the intuition that even under strategic sampling and participation decisions, as the principal increases the p-value threshold, namely when it's "casier" to pass the statistical test, agents with lower effectiveness (who can still be better than baseline) will find it beneficial to participate in the approval process. The monotonicity of the agent participation decision also means that the agent's 's utility is increasing in Pv- This is formalized below, with the proof in Appendix B
Lemma 3.3. For any given p-value a, the agent's utility under optimal number of samples u(a, Ho: no (a)) increases in their effectiveness Nn-
4 Principal Equilibrium and Loss Dynamics
The principal's strategic choice is to set the p-value threshold a. Given the agent's best response behavior, how should the principal select a to minimize their loss, as defined in Definition 2.3] The loss captures the trade-off between two risks: approving ineffective products (false positives) and failing to approve effective ones (false negatives). A product can fail to be approved either by explicit rejection after participating in the approval process, or by the agents opting not to participate. Since the p-value threshold a influences the participation decision (Lemma 3.2) this becomes implicit factor in the loss function. The principal's 's challenge is to set a optimally: a low a reduces false positives but may discourage worthy candidates from participation, increasing false negatives; conversely, a high a boosts participation but raises the risk of approving ineffective products. Optimizing a requires careful balance between these risks while accounting for agents' strategic behaviour.
We begin by re-expressing the cumulative loss conditioned on the participation decision induced by for given agent, a decision entirely captured by the condition μi 2 (r(a)::
L(a,
Aip porig
[Pass Ho, Ro (a)) < Pb: Ho 2 P(( 2 (r(o))mo S pb) (FP particip)
+ Ap pa
E [asss(a, Mo, po (a)
) Pn nb, pa < <)-(a)) < (=()))p S 11)
(FP aasstain
+ Ayn pa
E [Fail(o, no (a)) Pnn 2 Pbb 2 (1(()) 2 p=(a))pp 2 1b) (FN particip)
5

---

+ Ayn E Faill(a, Po. Ryu (a) 2 pbx po < { < <mr(a)loo 2 1b) (FNabstain)
= PPFPaatee particip (a. I) Fatssuun a, I) + FN particip (a, I) + FN abssaaii (a, I)
Note that the overall false positive FP I = FP particip (a, I) + FP abstaii (a, I) and similarly, the overall false negative FN(a, = FN particip (a, I) + FNabstain (a, 2). Further, observe that when the agent does not participate, their probability of passing the test is 0- therefore, Pa = 0 at all times. Thus, the cumulative loss is fully specified by L(a, I) FP particip + FTNpaticip + abstain We will now analyze the properties of these components as a function of a. Remarkably, despite the complex interplay between the agent's 's strategic participation and the principal' decision, the loss terms exhibit consistent monotonic behavior within regions segmented by a critical threshold intuitively, this is the p-value wherein the participation threshold Jar (r)(ā) Nbb (see Definition [...p.
Definition 4.1 (Critical p-value it. The critical p-value threshold ã is the p-value at which the participation threshold (Definition 3.1) is equal to the baseline effectiveness - pe (ā) = fb. This quantity is agnostic to the effectiveness distribution q and scaling parameters (A5pr A5n)-
Before our detailed analysis, we highlight that the forthcoming results depend on a core observation for effective agents (nO 2 (b)), the probability of passing the statistical test increases as the p-value increases. While this is intuitive when the participation and the number of samples used (n) are fixed, when agents are strategic and dynamically change these decisions, it trickier. For example, if an increased led the agent to use fewer samples due to high marginal cost relative to the increase in passing probability and thus revenue (this indeed common as the p-value becomes high), the pass probability could decrease. The lemma below highlights that despite the complicated dynamics of optimal samples, nuo (a), the passing probability is always non-decreasing a. The proof is technical and appears in Appendix but we sketch the key ideas below.
Lemma 4.1. For an instance I and an agent with effectiveness po 2 pbb the probability of passing- Pass(a, (a, Po, Ppo (a)) - is non-decreasing a when the agent participates and uses no (a) samples.
0 (ass(a, Mu. nun (a) 2 O. The total effect is Proof Sketch. We prove that for μo 2 pbb we have 8a decomposed into direct effect, capturing the change in pass probability purely due a, and an indirect effect, which accounts for changes in the agent's 's optimal sample size (a. For the direct effect, an increase in a reduces the critical threshold 4 1 - a). making it easier for agents to pass the test. For the indirect effect, the agent adjusts their optimal sample size Ruso (a) based on the value of a. We compute On which incorporates the agent's ss optimal behavior. The second derivative of the pass probability with respect to n is shown to be negative, ensuring the utility is concave with respect to sample size. This concavity guarantees unique optimal sample size nyo (a). which varies predictably with a. Finally, we show that the combination of the direct and indirect effects results in the total derivative being non-negative, confirming that the pass probability is monotonic with respect a.
da(a) 0Pass and apply the Implicit Function Theorem to determine de
4.1 False Positive Loss
Since FP Faasstuin O (non- pootnpaaiiiiiation means the probability of passing is 0), the overall false positive purely determined by purticpp- We now show below that for a S â, FP FPpuiicpp O, while for a this is increasing. We set the scaling factor Afp = =1 since it does not affect the analysis.
The overall false positive loss, FP(a, I)= = FP parricci (a, I). Further, this quantity is 0 for a < ā and non- non-decreasing a for a 2 â
Theorem 4.1.
Proof. It suffices to prove the property for FPputiccp First, consider an a < a. We know from Lemma 3.1 that the participation threshold is decreasing in a. Thus, since (=( -bb, it means that our given a, frr (a) > ph- In other words, any agents participating under this p-value are effective (above baseline) and do not contribute to the false positive.
Next, consider a 2 â. We know from above that the participation threshold now below Pbg Observe that for any a, we can write the false positive rate as follows:
(()+(a)= S no S mml]
FP particip
Pass(a, Po, po ((Ho)) po S polduo
Plpo S pbl (u-(a)

---

/ Pass(a, no, Blg (a))g(uo)ddo
(
Pluo S [b] pp((.)
Note that PlaoSus] is constant and as increases to a', (u=(a") < (r(a)), meaning we integrate over a larger region. The integrand itself always positive, and for every μo 2 max( (²{(a')) Mr (a:,, the pass probability has increased (Lemmal4) Thus, FP putticip is non-decreasing in a.
4.2 False Negative Loss
We next consider the components of the false negative loss: FN particip and FN We first establish the monotonicity properties of aNaastin which is exactly equal to the proportion of effective agents not participating since not participating means that Pass(.) implying Fail(-)
In other words, FNaastain P(uo < prr (a)]Po 2 pa).. Once more, segmenting the p-value space by ā is crucial to establishing this relationship. As before, we set X since does not affect the analysis.
Proposition 4.1. The con-participation false negative loss, FNabstain I) is non-increasing in a for a S ã, and is 0 for any a 2 a
Proof. Consider on, , 02 where 0: 0n, where both on, 02 S S. Thus, first need to show that: P(uo < Hr ((11)) Ho 2 1bb) 2 P( < (4-(02))0 2 /bb). We know from the monotonicity of the threshold belief (Lemma 3.1 pry (a2) S pr (a1). Let Q2n, denote the CDF of the effectiveness distribution, conditioned on the event po 2 ph- Then Q21a (a)(a)) = Plpo I [r((a)lo 2 po]- It is then immediate that Q (az(a2) S Q2n, (ar(aa)) Lastly, for a z a, we note that by definition of â and Lemma 3.1 (a) S pb. In other words, it is optimal for all effective agents to participate, meaning FN abattain 0 in this regime.
One might expect to show a similar result for FN FNpatticip and thereby conclude the cumulative behavior of the false negative loss. Indeed, when a 2 ÔG the participation threshold smaller than the baseline (a(aa) < (=(ā)) (bb, which means all agents whose prior is greater than /4a will participate, thus Pr[mo 2 max(μu, (a), po))Pp 2 [6] 1. The FN under participation loss simplifies
for 2 a: FNpurticip (a, 1)
E [1 Pass(a, No: Rug (a)) 2 pb] W
Following directly from the monotonicity of the passing probability (Lemma 4.1 this implies that purtiic decreases as a function of a when a 2 â. We formally state this below:
Proposition 4.2. FN FNparticp(( (a, I) non-increasing in a for all 2 â.
When a < ā, however, FN purticip displays much more nuanced behaviour indeed, in Section 5 we show that this component not monotonic and can be increasing, decreasing, or both in the a region. This may suggest that, unlike the false positive setting, little can be said about the combined false negative effect pFNNpttiicip + FNabstain) ). we show in Theorem 4.2 however, this is not true, and we directly establish the monotonicity of the combined false negative rate when the p-value domain is segmented by à. The result is technical and the full proof is given in Appendix C
Theorem 4.2. The overall false negative loss, FN(a, I) non-increasing in a on both sides of a, that is, it is not increasing for < â, for all a 2 â.
PNperticip II + FNabsuain (a, I). is
4.3 Characterizing a* The Optimal p-value Threshold
FP particii
FP Passtainn
Overall FP
FN particip
FN
Overall FN
S a
O
Non-Inc
Non-Ine
Non-Dec
O
Non-Dec
Non-Inc
0
Non-Inc
Table 1: Summaay of the loss dynamics as a increases. â is defined in Definition 4.1
We comment
on FN FNparicip in the S â regime in Section 5 See figure for this dynamic visualized on an instance.
We summarize our analysis of the different loss components in the table above. These results lead to a succinet characterization of how the principal should use their equilibrium a*. Observe that for any < â, the overall false positive is O, while the overall false negative rate is decreasing in a. This
7

---

FN particip increases and immediately implies that ã is weakly better than any a S a and that a" must be at least as large as à! This observation and the value of â are agnostic to both the effectiveness distribution q and the loss scaling constants App, Afn- This is powerful characterization, While the true optimal o* may be larger, the principal suffers additional false positives here while lowering the false negatives. In settings like drug approvals and manufacturing, where the cost of false-positives is very high App > Afn the optimal may indeed be close to â.
In recommending values close to â as optimal, the question of computation arises. Fortunately, since by definition a. (ā) pbb, and the participation threshold (r- can be computed to accuracy in (log g log((Imaz)) time, an â can be efficiently computed to E accuracy in (log" (oa()))z)), This follows from discretizing the p-value domain into intervals and running binary search, leveraging the monotonicity of the participation threshold Lemma 3.2
Theorem 4.3. For any instance I, the principal's 's optimal p-value a* satisfies a' 2 ā. where a (a) = pb. Further, an approximation to an ā such that (r(āa) [ML E) can by computed in Ollog" 2 log((max))
Experimental Studies
FN particip losses for different distributions
Loss components for g TruncNNrmm0.622 0.04)
0.7
TounNNorm/
TomnN 0:3)
0.4
TonseNenn o.t)
0.0-
Occrll EN
Tital Lowa
1.5
0.2
0.3
0.1
0.1
0.0
0.66
0:20
0.55
&
0.20
p-tilse
p-miue
Figure 1: On the left, we plot the FNparicip losses for different effectiveness distributions q. On the right, we plot all the different loss components for one of these distributions with Asp Xyn 1
5.1 False Negative Rate Among Participating Agents
Our technical analysis in the preceding section accounts for all components of the loss except one:
ENpatiicp for a I a. For a principal unaware of the strategic implications of their chosen significance level (a) particuuulll that it may discourage participation by agents with above-baseline effectiveness this may be the only false negative they observe. At first glance, it seems increasing should reduce this: the test becomes easier to pass, effective agents should be more likely to succeed. Theorem (.2r also aligns with this intuition since the overall FN rate decreases with a.
However, Papaticip behaves more intricately. We highlight this with example. Consider μ4, 0.5 and truncated normal distribution q between 0.4 and 0.7 with standard deviation 0.04. First, suppose the mean of y is 0.53, meaning large mass of effective agents are only slightly better than baseline. As a increases from a1 to Q2, the participation threshold falls (by Lemma (.T), causing large mass of these slightly-effective agents to switch from abstaining participating. However, due to their small effect size and samples being costly, these new participants pass with a lower probability than those who also participated under a1 Their inclusion in the FN metric at dz outweighs increased pass probability among previously participating agents, increasing the FN rate under 0:2-
This effect is not monotonic and depends on the distribution. If g's mean rises to 0.62, the same initial increase in FN rate occurs the small a regime. For higher values of a, however, most effective agents are already participating, and few agents remain that are both slightly better than baseline and could switch; as a increases, the effect is dominated by the increasing pass probability of those already participating, decreasing the FN Partticip. As shown in Figure
8

---

then decreases. In selecting a higher mean (0.67), PNputicpp is always decreasing. Overall, this rich behavior fundamentally due participation and evidence set size being a strategic decision.
5.2 Analysis of ā in Drug Approvals
We now conduct case study of our strategic hypothesis testing model by considering our running example: drug approvals. Despite much of the data in this setting being proprietary, we use public sources to capture relevant metries for three classes of drugs: oncology. vaccines, and cardiovascular ProRelix [27] mentions the per-participant expense (c) of vaccine testing to be ~ $50, 000, while oncology and cardiovascular trials being around $128, 000 and $136, 000 respectively. The fixed expenditure (co) for clinical trials, regulatory approvals, and R&D, as well as the lifetime revenue (Rt vary widely, even within drug categories. Prasad and Mailankody [28] highlights the median fixed expenditure to be around $650 million for oncology drugs, although in the extremes it can be as low $150 million or higher than a billion. The median four year revenue to around 1.6 Billion; data on lifetime revenue is limited, but [29 30) suggest it can be around 10-15 billion, with blockbuster drugs above 50 billion [[4]. Analysis from Bhatt et al. [16] gives ranges of $74 $183 million for fixed costs of Cardiovascular drugs with median of $141 million. Rashid and Chandel 31 suggests the corresponding revenue of approved drugs here to be between under a billion to over 10 billion, with median of around $3.5 billion. For vaccines, Sertkaya al. 32 outlines revenues between 6.9 billion to 36.9 billion for blockbusters, with median fixed costs around $886 million. In the extremes, it can be below $100 million above billion. We present this rough data in Section 5.2
Drug Category
Revenue if approved (R)
Fixed Cost (co)
Cost Per Sample (c)
Oncology
[1, 500 50, 000]
(648); [150- 1, 000
0.136
Cardiovascular
(3,560); [1, 000 10, 000]
(141); [74 183]
0.128
Vaccine
[6, 900 36, 900]
(886); [100 - 1, 000
0.05
Table 2: Rough costs and revenue by drug category. The median value, where available, is presented parentheses. All numbers millions USD, for US-based development.
Figure 2: vs Revenue Oncology
Figure 3: â vs Revenue Cardiovascular
6
6a MLU
0000
110.0
0.-
05-
1g
12
0.1
900.
2000
Criikal p
0.3
Crilioal p
0.3
011
01
.000
2000
300000
2500
50,00
1500
2500
3000
32.
Revenue (in Millions)
Revenue (in Millions)
Our goal is to understand what & the critical p-value that weakly dominates any lower p-value looks like in the drug approval setting. Due the variability of fixed costs on and revenue R as compared to per-sample-cost e, we fix and plot ō for different revenue and fixed costs. These ranges are centered around the median values (where available) gathered in the table. We plot the figures for oncology
and cardiovascular categories above (see Figure and 3); the vaccines figure is in Appendix E Each plot also displays the a 0.05 boundary, a commonly used p-value threshold by the FDA
The results suggest that our strategic model is roughly capturing the economic dynamics of drug approvals. The in practice used p-value of 0.05 is the ideal choice at around the median revenue for
'The experiments conducted using MacBook with only CPU resources and the NumPy package 33

---

FFrom the principal perspective, recall that they do not need to know the distribution g determine & each drug category. Within our leader-followe dynamic, the FDA is committing to this threshold, first suggests that pharmaceuticals are choosing their parameters (samples collected, price, etc) to ensure profitability. Conversely, if regulatory bodies wish for decreased drug prices with the fixed costs being immutable, they ought to specify a higher p-value threshold, as the current value may dissuade effective but low-revenue drugs. In oncology, for instance, the critical threshold ā for around S5 billion revenue is between 0. and 0.2. Given our analysis, such a value does not induce any more false positives than more stringent ones (Theorem 4.1. but lowers the false negative rate. However, one may argue that choosing higher p-value would imply that those with much higher revenue (say 25 billion) will be approved even when they are ineffective since their á is lower. Practically, this is unlikely to happen since it is rare that an ineffective drug, albeit FDA -aprrvedd would achieve such blockbuster revenue, the market and post-marke surveillance would generally expose inefficacy. Conversely, choosing p- p-values stricter than ā hamstrings low-revenue but effective drugs; this is a real concern for orphan drugs or low-cost therapeutics where margins are much slimmer.
Discussion
This work spiritually extends the growing literature on strategic machine learning to hypothesis testing, a ubiquitous method used in regulatory approvals. Given the costs of data collection, firms may strategically choose smaller trials opt out entirely if the expected revenue, shaped by approval probabilities under the chosen threshold, does not justify the expense. By modeling this dynamic within principal-agent framework, our work uncovers systematic connection between the regulator-speciffes p-value and the resulting firm decision. This allows us to give an interpretable characterization of the regulator's optimal p-value, which minimizes false positive and negative error rates under such strategic consideration. We validate the findings of our model with real cost, revenue, and approval data from the US pharmaceutical sector and derive two key policy insights. First, the commonly used FDA p-value of 0.05 is fairly strict given development costs, requiring firms to extract large revenue from approved drugs for economic viability. This matches empirical evidence from the US pharmaceutical market. Second, to decrease drug costs while maintaining minimal error rates, regulators can either selectively increase the p-value threshold or give subsidies for running trials; our model gives guidance on how to adjust either lever.
Our main results rely on all parties knowing the cost and revenue parameters (ca, C, Rt) this determines agent participation and the principal's optimal threshold de] While these quantities may in practice be estimated, our qualitative conclusions are robust to small estimation errors. In Appendix we discuss how small perturbations in the cost and revenue parameters affect the agents' participation thresholds and the principal's optimal choice of & and argue that these perturbations have only a limited impact and leave the qualitative behavior of the model unchanged.
Acknowledgments and Disclosure of Funding
Yiling Chen was partially supported by the National Science Foundation under grant IIS 2147187.
References
[1] Moritz Hardt, Nimrod Megiddo. Christos Papadimitriou, and Mary Wootters. Strategic classi- fication. In Proceedings of the 2016 ACM conference on innovations in theoretical computer science, pages 111-122, 2016.
[2] Jinshuo Dong, Aaron Roth, Zachary Schutzman, Bo Waggoner, and Zhiwei Steven Wu. Strategic classification from revealed preferences. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 55-70. 2018.
[3] Yiling Chen, Chara Podimata, Ariel D Procaccia, and Nisarg Shah. Strategyproof linear regression in high dimensions. In Proceedings of the 2018 ACM Conference on Economics and Computation, pages 9-26, 2018.
[4] Yatong Chen, Jialu Wang, and Yang Liu. Linear classifiers that encourage constructive adapta- tion. arXiv preprint arXiv:2011.00355, 2020.
10

---

[5] Safwan Hossain and Nisarg Shah. The effect of strategic noise in linear regression. Autonomous Agents and Multi-Agent Systems, 35(2):21, 2021.
[6] Keegan Harris, Chara Podimata, and Steven Z Wu. Strategic apple tasting. Advances in Neural Information Processing Systems, (16779118-79940 2023.
[7] Demonstrating substantial evidence of effectiveness for human drug and biological products.
Technical report, U.S. Food and Drug Administration, December 2019. URL httpp
7/www. fda fda.
fda- demonstrating http::/aiannutaaaaaaaaaaoaaaeeee ott.://aaeeeaeeaaaaaaaoaaaaaaaaa aata--uuuuaac---ccummeettttt
hawnss-hhhannaa- drug- and and-biologiiiooooooooooooooooooa productss
Draft Guidance for Industry.
[8]
Brian Wansink.
The food scientist who manipulated data to make head- Axios, 2018. -awedddaaaaaalepppeee ottaaaaaa-eatttaaaeoaaaatteeeeea
lines.
URL httpp::/wwww..xoossccomo/ www.: /ttp///.....c////////////1//////
[9] Abel Brodeur, Scott Carrell, David Figlio, and Lester Lusher. Unpacking p-hacking and publication bias. American Economic Review, 1 113(11))29743300 2023. doi: 10. 1257/aer. 20210795. URLhhtts:://we
aeaweb.
1257/ http 20210795
[10] Angelika Stefan and Felix D. Schönbrodt. Big little lies: compendium and simulation of p- hacking strategies. Royal Society Open Science, 10(2):200346, 2023. doi: 10. (0.1/0///ss.2200 220346.. wwt.lattge.//ooyylloceeeepp..bbp http://wwwwp..aiiioooooooooo//aa
1098/r9os 220346
[11] Stephen Bates, Michael I Jordan, Michael Sklar, and Jake A Soloff. theretie bayesian inference for collaborative science. arXiv preprint arXiv:2307.033743 2023.
[12] Aleksey Tetenov. An economic theory of statistical testing. CeMMAP Working Paper
CWP50/16, Institute for Fiscal Studies, 2016. URL https://iden repec. org/p/azt.
cemmap/50-16 html
[13] Flora C. Shi, Stephen Bates, and Martin J. Wainwright. Sharp results for hypothesis testing with httg/abs/24111
risk-sensitive agents, 2024. htttsstss:/aaxii
[14] Stephen Bates, Michael I. Jordan, Michael Sklar, and Jake A. Soloff. Principal-agent hypothesis testing, 2022. arXiv preprint arXiv:225.006812.
[15] U.S. Food and Drug Administration. Multiple endpoints in clinical trials: Guidance for industry. Technical report, U.S. Food and Drug Administration, October 2022. URL https
//www fda. ieteeeddiaaa12161111 121///441//////aa//
Accessed: 2025-05-16.
[16] Deepak L. Bhatt, Christopher P. Cannon, Roxana Mehran, Gregg W. Stone, Muthiah Vaduganathan, Marc S. Sabatine, David J. Moliterno, Robert M. Califf, and Eugene Braunwald. The imperative to enhance cost-effects for cardiovascular clinical trials. JACC: Basic to Translational Science, 98):1029-0040, 2024. doi: 10. 10.1016/j jacbsss.. 12.. URL https: http:://mccccccii.icem nwwi.cciii..... cecbi.nlm.miie www.aiiw......ri.lleeee.es Pttps//MMMMM110707
[17] Andrew McClellan. Experimentation and approval mechanisms. Econometrica, 90(5):2215- 2247, 2022.
[18] Megan L. Head, Luke Holman, Rob Lanfear, Andrew T. Kahn, and Michael D. Jennions. The extent and consequences of p-hacking in science. PLOS Biology. 1002)000 2015. doi: 10. /f.1m77/ooual...bboooooo 1002 URL https://journall plos wwtoooog//ooo.ooo / article?id=10 117.0 wtt.:l..............11111111..0.
[19] Jonas Moss and Riccardo De Bin. Modelling publication bias and p-hacking. arXiv preprint arXiv:1911.1 12445, 2021. URL httpt::tt//:arttii org/ atggabs///11l111 1911. 12445
[20] Oliver Hart and Bengt Holmström. The theory of contracts. Truman Bewley, editor, Advances in Economic Theory: Fifth World Congress, pages 71-156. Cambridge Univer- sity Press, Cambridge, 1987. URL https: /tww wwwwrrigbbbbb.c.oo
aop- Sopp-cmmbridg--coo cotpr/ee-ooor///oteeetttt http:///wwo.oooooo////.c////////
978113905205403__71 156_ /dt.ppf///e..///.ccppccoco /ttf:/tteeeeee thhory htcpoyycooo_oottattssssso
[21] Jean-Jacques Laffont and David Martimort. The Theory of Incentives: The Principal-Agent
Model. Princeton University Press, Princeton, NJ, 2002. ISBN 9780691091846 URL https
/www.aaazzaa www.Theoryo--eecoa
Inw.ooriiinnnnnnieeeeeee - htt.a//w.oo.oooooooooooaooaaoaar
[22] Bernard Salanié. The Economics of Contracts: A Primer. MIT Press, Cambridge, MA, 2nd
edition, 2005. ISBN 9780262195556 URL URI.https: httthhs:t//ttpeeeeeesesee mit. mit.ccchh/7700222155555

---

[23] Rafael Frongillo and Bo Waggoner. Recent trends in information elicitation. ACM SIGecom Exchanges, 22(1))4-18, 2023. URL https://sigocom. org/ /ww.eeagee/voolmeeeee...///////e 22/1/
FRONGILLO. pdf
[24]
David Danz, Lise Vesterlund, and Alistair J. Wilson. Belief elicitation: Limiting truth telling with information incentives. NBER Working Paper 27327, National Bureau of Economic Research, 2020. UtLhttps::/ hwwp:///wwwwbbw./../e.m.. wawber..oggggaaaaa
wwtp:rorg/pppp/////eee w27327
[25]
Dachong Min. Screening for experiments. Games and Economic Behavior, 142:73-100, 2023. doi: 10.1016fj.gcb.2223.0700000 URL https://dot..oooo org 1016/ geb 2023. 07.007.0000
[26] Leah Isakov, Andrew W. Lo, and Vahid Montazerhodjat. Is the fda too conservative or too aggressive?: A bayesian decision analysis of clinical trial design. Journal of Econometrics, 211 (1):117-136, 2019. doi: 10. j00mommm 12..009. URL htw.https:///ae org. /0. 1016/
j j.conoomm 2018. 009
[27] Research ProRelix. sor needs to know!. 2025.
by-ehs clinical trial costs: What every spon-
URL https://prorellireseearr com/
phase-by www.yy-hhhseeecaheee hhaa---y-ppaee--cmaaaa---------- whs.aaaallaaaalllossosaaa----aae Accessed: 2025-05-10.
needs-
[28) Vinay Prasad and Sham Mailankody. Research and development spending to bring a single cancer drug to market and revenues after approval. JAMA Internal Medicine, 177(11): 1575, 2017. doi: 10.1001//amaiinn 10000///mmaanemmmme0007717.7.7.0
URL https: //tp:/pm..c...// ncbi mlm.mmmmmm.. 1.lmm.n.. nih
wtt/aarticles//MM57110000
[29] Mark Teryy Report: Projected sales of 10 top-selling drugs through 2024. August 2019. URL
http:::/wwww.bbospeeoc cww/top- 10 daw.-druusaaallaaa saressallsspppree poalls--projeteedtthee
2024 Ac-
cessed: 2025-05-13.
[30] Ayesha Rashid, Soumya Shashikumar, and Vincent Xie. Top 10 cancer drugs to watch in 2024 based on recent sales data, November 2024. URL https://ttalks. com/
2an-2002--basee on sat--eeeett-alle---ttaaaaaaaaa
Accessed: 2025-05-13.
[31] Ayesha Rashid and Deepika Chandel. Top 15 cardiovascular discase drugs in 2023 by 2022 sales data, February 2024.
URL https://xtalks, com/ t0p- www.----arrdoovaaala-----iee drrase--rgsss 2y aa -02---2022----e-------111111111- Accessed: 2025-05-10.
[32] Aylin Sertkaya, Trinidad Beleche, and Amber Jessup. The cost of vaccine development. Technical report, U.S. Department of Health and Human Ser- vices, 2016. URL https://aspe hhs http./////siiteeeeeaaaaaa//////a hew.ll////ouuen///////
BCCe77766f66222699888677773333 2025-05-07.
costtssottsssoooo brief pdf Accessed:
[33] Charles R. Harris, K. Jarrod Millman, Stéfan van der Walt, Ralf Gommers, Pauli Vir- tanen, David Coumapeau,, Eric Wieser, Julian Taylor, Sebastian Berg, Nathaniel J. Smith, Robert Kern, Matti Picus, Stephan Hoyer, Marten van Kerkwijk, Matthew Brett, Allan Hal- dane, Jaime Fernández del Río, Mark Wiebe, Pearu Peterson, Pierre Gérard-Marchant, Kevin Sheppard, Tyler Reddy, Warren Weckesser, Hameer Abbasi, Christoph Gohlke, and Travis E. Oliphant. Array programming with NumPy. Nature, 585(7825)3357-363 September 2020. doi: 10 1038/3415866666 0200-002220000 -20-2649--22222 2649-2
00.1038/s41586-022 2266--22--44-. URL httpt::///ott
[34] Lee Kennedy-Shann Shaffer. When the alpha is the omega: v-vuuuss, "substantial evidence," and the 0.05 standard at fda. Food and drug law journal, 72(4):595, 2017.
12

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: All theoretical results have associated proof's that verified. This is through- out the paper.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made in the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made the paper and important assumptions and limitations. A No or NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
It is fine to include aspirational goals as motivation as long is clear that these goals are not attained by the paper.
2. Limitations
Question:
Does the paper discuss the limitations of the work performed by the authors?
Answer: [Yes]
Justification: We specify the setting within which this paper and its model are applicable. See the model section (Section 2)
Guidelines:
The answer NA means that the paper has limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section in their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (c.g.. independence assumptions, noiseless settings, model sell-speciiiatiinn asymptotic approximations only holding locally). The authors should reflect on how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution low or images are taken in low lighting. Or speech-to-tex, system might not be used reliably to provide closed captions for online lectures because it fails handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and faimesss.
While the authors might fear that complete honesty about limitations might be used by reviewers as grounds for rejection, a worse outcome might be that reviewers discover limitations that aren't 't acknowledged in the paper. The authors should use their best judgment and recognize that individual actions in favor of transparency play an impor- tant role in developing norms that preserve the integrity of the community, Reviewers will be specifically instructed to not penalize honesty concerning limitations.
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
13

---

Answer: [Yes]
Justification: We state our assumptions clearly and provide all proofs. Proofs are mostly in Appendix Bland
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proof' in the paper should be numbered and cross referenced.
All assumptions should be clearly stated or referenced the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors encouraged provide short proof sketch to provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Does the paper fully disclose all the information needed to reproduce the main ex- perimental results the paper to the extent that affects the main claims and/or conclusions of the paper (regardless of whether the code and data are provided not)?
Question:
Answer: [Yes]
Justification: We specify all the data used in the plots in the main body of the paper (see Table 2, Section 5.2)
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is novel architecture, describing the architecture fully might suffice, or if the contribution a specific model and empirical evaluation, it may be necessary to either make it possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, other means that appropriate to the research performed.
also be provided via detailed
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make it clear how to reproduce that algorithm.
(b) the contribution is primarily new model architecture, the paper should describe the architecture clearly and fully.
(c) If the contribution is a new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset or instructions for how to construct the dataset).
(d) We recognize that reprodueibility may be tricky in some cases, in which authors are welcome to describe the particular way they provide for reproducibility. In the case of closed-source models, it may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path reproducing or verifying the results.
5. Open access to data and code
14

---

Does the paper provide open access to the data and code, with sufficient instruc- tions faithfully reproduce the main experimental results, as described in supplemental material?
Question:
Answer: [Yes]
Justification:
Code for the experiments are in the supplemental materials.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurlPS code and data submission guidelines (https://nips.co
for more details.
While we encourage the release of code and data, we understand that this might not be possible, "No" an acceptable answer. Papers cannot be rejected simply for including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run to reproduce the results. See the NeurlPS code and data submission guidelines (https
/ praips.cc/publiccguuieeee CoodeSumeeieeeeee ssi iessonPolll molyicyy for more details.
The authors should provide instructions on data access and preparation, including how to access the raw data, preprocessed data, intermediate data, and generated data, etc.
The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing much information possible in supplemental material (appended to the paper) recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: This is not a deep learning paper. All relevant details for experiments are in the main body- Section 5.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to a level of detail that is necessary appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Question: Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Answer: [No]
Justification: There is no randomness in the experiments we supply. It is just plotting analytical results. So there are error bars to report
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by error bars, confi- dence intervals, or statistical significance tests, least for the experiments that support the main claims of the paper.
The factors of variability that the error bars are capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
15

---

The method for calculating the error bars should be explained (closed form formula, call a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
It should be clear whether the error bar is the standard deviation or the standard error of the mean.
OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that out of range (e.g. negative error rates).
If error bars are reported in tables or plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time execution) needed to reproduce the experiments?
Question:
Answer: [Yes]
Justification: We mentioned in Section 5 that all experiments were conducted on a CPU machine. Again, this is not deep learning paper; the experiments simply validate analytical results.
Guidelines:
The answer NA means that the paper does include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (e.g., preliminary or failed experiments that didn't make it into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurIPS Code of Ethics atttps://neuree http:rrippp..cppull/eeeeea httpp/bliccttticccccceellleseese
Answer:
[Yes]
Justification:
We conform to the Code of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No. they should explain the special cireumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., if there is special consid- eration due to laws or regulations their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer: [Yes]
Justification: We discuss the policy implications of work, especially relates drug approvals, in Section 5.
Guidelines:
The answer NA means that there is no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
16

---

Examples of negative societal impacts include potential malicious unintended uses (e.g., disinformation, generating fake profiles, surveillance), faireess considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
The conference expects that many papers will be foundational research and not tied to particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional unintentional) misuse of the technology.
If there are negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper deseribe safeguards that have been put in place for responsible release of data or models that have a high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification:
This is a theory paper not applicable.
Guidelines:
The answer NA means that the paper poses no such risks.
Released models that have high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [NA]
Justification:
We do not use any existing assets, beyond standard library packages (NumPy).
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version of the asset used and, possible, include URL.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (e.g., website). the copyright and terms of service of that source should be provided.
If assets are released, the license, copyright information, and terms of use in package should be provided. For popular datasets, paperswithcode com/ datasets has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
17

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
this information not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
We do not release new assets. We share code in the supplementary
materials.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moodl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well details about compensation (if any)?
Answer: [NA]
Justification:
This is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included the main paper.
According to the NeurIPS Code Ethics, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or an equivalent approval/review based on the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This is not applicable.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere the NeurIPS Code of Ethics and the guidelines for their institution.
18

---

For initial submissions, do not include any information that would break anonymity (if applicable), such as the institution conducting the review.
16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is important, original, or non-standard component of the core methods this research? Note that the LLM is used only for writing, editing, formatting purposes and does not impact the core methodology, scientific rigorousness, or originality the research, declaration is not required.
Answer: [NA]
Justification: LLMs not used in this paper.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non standtdddard components.
Please refer to our LLM policy for what should or should not be described.
19

---

A
Notation Table
Symbol
Usage
X (0,1)
Bernoulli random variable indicating
whether a product effective random instance
9
PDF of the effectiveness distribution
Q CDF of the effectiveness distribution Q.
140
e effectiveness
Ab
baseline effectiveness
variance of a product's effectiveness: du = v(1 - po)Ho
To
variance of a baseline product's effectiveness: v(1 14b
Ob
CO
fixed cost of running clinical trial
C
per-sample cost
R
revenue upon approval
a
p-value threshold used by the principal
P(Po, n; Ps)
the probability of observing outcomes at least
good as the evidence conditioned on the null hypothesis μ4 The critical region given samples and p-value the probability that product with effectiveness μo approved utility for agent with effective size μo and number of samples optimal number of samples given p-value threshold and effectiveness μo
Za,
Pass(a, po:
u(a, Mo,
no (a:
Per (a)
The participation threshold belief under p-value threshold
o²
The critical p-value threshold such that (1(aa) = 1b
Table Primary Notation
20

---

B Omitted proofs in Section
3
Proof of Theorem 3.1
Proof. We first define the following variables for brevity. Let م - Mu - 14 denote the effect size, To = V/ - ho) denote the standard deviation of the true process and
/ nb) the standard deviation of the bascline Bernoulli variable. We use o(z) to denote the standard normal, and (2) to denote its CDF. Lastly, let da -- (1 a)) denote the 1 percentile of the standard normal. Recall that we work under the normal approximation of the binomial distribution where the critical threshold is:
20, E / N(nu, mpp(11 - -aao(l1--)))]1 + mpp(1 (b)) Montd, v
(4)
Thus, we may write the pass probability as follows, from which the expected utility expression follows:
V/npoo11 - (V -mO)
#/.a 74400
1ith + da v //jua(1 - 146) - -1/400
Pass(.) = - +b
=1 -
Vniuol1 - (o)
=1 - ( do
00
O0
da
A p A/n
unnna Mo: pb) R R
Co
00
0
Observe that when the drug is not effective, i.e. Mu μbb then the argument to o() increases in Hence, the passing probability, and thus the expected revenue, decreases with increasing n. It is evident that the cost also increases in Thus, when the underlying drug is not effective, it is optimal for the agent to choose the smallest value Rmmin possible. Computing the pass probability oracle with Amin, they determine their maximum expected utility and choose to participate if this is positive. In other words, when 140 < a constant amount of computation suffices to determine the optimal participation decision.
We next turn to the more interesting case where μi 2 Pbb Our goal is to efficiently search the sample space by leveraging structural properties of the utility function. Letting 8a derivative of the utility function with respect to is as follows:
d. A
the
du O dv du Ōn R&p c
-
(5)
On
RA
do
Φ(v) 20
(6)
200
The first term is always positive, and since the image of <() is always above 0. As n increases, this first term tends and the derivative dominated by the second term and becomes negative. In other words, increasing only increases utility to point. To precisely understand this characteristic, consider the second derivative of the utility function:
pu -RA,
RA
vo(v) = RØ(v) An
A
(vv) +
Va
On
3/2
20
3/2
200
The sign of the second derivative, and thus the concavity/convexity properties of the utility function depend solely on
A v 1 since the term multiplying it always positive. We now expand by plugging in the definition of e::
20
A v
1
A n + 20
da da
A 20
da 20
Vn
n
1
(8)
200
20
where we substitute in n.. Observe that this is negative quadratic expression with only positive real root being meaningful (since vn cannot be negative). The roots n1, n2 can easily be computed by applying the quadratic formula to the instance parameters. We categorize the outcome follows:
21

---

1. No positive roots = the function is always concavee
2. One positive root at n1 = the function is convex over [0, n1] and concave over [mi, 00).
3. Two positive root at 71, R2 with n2 > R1 = that over [0, ni] the function is concave, over
[n,, 122] is is convex and over [m2, 00) it concave.
We can consider the problem of optimizing n over each of these convex/concave regions defined by the roots. Note that a convex function always attains its maximum on the boundary. Thus, for a the convex interval, it suffices to just check the boundary points, of which there are only two. At the optimal n for concave interval, the first derivative is always or it is boundary point. Since the latter case is similar to the first one, suffices to find where the first derivative is 0, if it exists. Since the first derivative is always increasing (monotonic) and we need find the z-intercept. binary search suffices: starting in the middle of the interval, increase n if the first derivative is positive, and decrease if negative. This makes at most log 7maz calls. It immediate that taking max over the best from each of the at most three yields the globally optimal Ruo (a). The agent can compute the corresponding passing probability and utility and decide to participate this is positive.
Proof of Lemma
Proof. We start with the first direction. If an agent with prior belief No participates with samples, then by individual rationality, they must have non-negative utility. That is, Pr[p(n. po; (b)) I a] R 2 (c. n + cu), where (c co) is the cost. Now consider an agent with prior belief μl 2 Mo and using the same number of samples-n. n. Observe that if Pr(p(n, (1: (.)) S a) 2 Pr(p(n', Mu: (b)) S a), then the agent will participate under belief f41 To compute Pr(p(n, (6) S a). let npa Σ 1=1 X; denote the observed outcomes of n samples drawn independently from the distribution with effectiveness /41- Observe that since the same number of samples
used here as when the belief was No. the critical region Za,n does not change since it depends only on a, n and ps. The probability that these samples, drawn with respect to belief μ1, will lie in this critical region:
Σ (1-) BE KE
m1)
Σ iE
i
n2
pá((1 - po)
where the inequality follows immediately since f1 2 140-
For the reverse, consider agent with effectiveness nig not participating. This means that for all n, we have Prip(n, pbi (b)) S a): Ro < (c- n + co). Then for an agent with belief μa < 1g the following holds:
Vn Σ
11
[1 (1-1)=')"
< Σ (3)oda-waa" m'o)
kEaa.n
iEFu..
In other words, for each n, the passing probability under of is worse than pá and the agent was already not participating under 1ó for any n.
Proof for Lemma 3.2
Proof. We first consider solving for (ax,, given To solve this upto some accuracy we can discretize the belief space into = intervals of size E, and use the mid-point of the interval as its representative belief. The monotonicity of beliefs implies that we can run binary search over these intervals. Starting with the middle interval belief, we need not search intervals larger than this. Similarly, if the agent does not participate, we need not search all intervals smaller than this. Clearly, this terminates in log (2) calls to the pass probability oracle. To check participation, we need to compute the optimal n at every belief. Assume
è if the agent participates at its representative
2,
the maximum number of sample is imann it will again take log( (1))))))) to search for the optimal number of samples. So the total complexity should be log (1) * log(Tema))
Next, we show that (a) is non-increasing as a increases. For any a, we know that at the corresponding threshold belief (=(aa) and its corresponding optimal sample size ny, (a) e n=(a). the utility is O. In other words: R. Pass((n, (a), He (a), no. a)) en, (a) + co. It is known that for
22

---

fixed effect size and number of samples, the passing probability of hypothesis test increases in a. In other words, for two p-values a,, 02 where an I a2 the following holds:
P (01): nn (01)) & Pass( M4y ((2)) n, (02)) and u(n,
02) 2 0
Since the agent with effectiveness (-(a1)) will have non-negative utility when using (a1) samples when the p-value is a22, this agent will participate at this p-value (but not necessarily using n- (1((1)
samples as that may not be optimal). The monotonicity of participation (Lemma[3.] implies that under Ogg, agents with effectiveness greater than Prr (a1) will also participation. Thus, the participation threshold under 02 by definition must be to the left of (a(a1) i.e. (r((22) S Mr (a(a)) as desired. C
Proof for Lemma 3.3
Proof. We prove by contradiction. Assume that there exists an a and μn 2 No such that u(a, Ho. Rag (a)) u(a, Dus (a)). First note that if the agent is not participating in either, then the utility is always If they participate in one and not the other, the monotonicity of participa- tion Lemma 3.T means it must be under [41 (giving positive utility), with μs being O. Thus, the only
situation where the initial claim could hold if the agent participates under both. We divide this into the following three cases:
No S 141 S /4b2 since both (40 and dl are less effective than the baseline drug, Theorem 3.1 implies that in both cases, Remin samples are used, and in Lemma 3.1 we know that in such settings, for fixed n, the pass probability increases in μ.. Thus the utility under μ4 cannot be lower than (40-
ni S Pb S ja and μb S po S. M1: We know from Lemma 3.1 for every fixed n, the pass probability in this regime increases in μ.. Thus, Vn, u(a, n) u(a, n), Since nug (a) is the optimal number of samples for μ1, we have
u(a, Po: py (a)) < u(a, PI, (a)) S u(a, No. (ao
The last step is according the contradiction statement. However, this cannot be since an Run (a) could not be the optimal number of samples for μn if it led to lower utility than nuo (a).
C Omitted Proof for Section 4
For the ease of notation, let Q denote the CDF of the effectiveness distribution q. Further, let Qups and Q2n, denote the CDF of the belief distribution conditioned on po < μbb and po 2 pbb
Proof of Lemma
Proof. We are interested in the total derivative of the pass probability with respect to a for any agent with effectiveness μо 2 146- Note that for this lemma, consider the agent to always be participating. This total derivative can be expanded using the multi-variable chain rule as follows:
d da
OPass
Pass
do (ax)
Pass(a;, ar, po: Roo (a))
(9)
da
On
da
Direct
indirect effect
To simplify notation, denote Wa = t 1p-1 (11 at,, and let
((n,a)= Φ Φ-1(1 a)ob 0b- Vn(po 1b)
Tb Wa
Vn(uo 14b))
00
σ0
6
Then we have: Pass(a, Ho: nuu (a)) = (a), We now separate the analysis based on the direct and indirect influence.
Direct effect: The partial derivative with respect is:
PPass d
(EE a)) = -E(E) - o(E) ah dwaa ab
ag
x{E oo da To Φ(=a)
da
= de
2
23

---

since higher a lowers the critical threshold Wa and makes the test easier to pass.
The indirect effect depends the optimal number of samples the agent for given That is, while having more samples increases the chance of passing @aass dn > 0), the agent might reduce this effort when a increases (since the test becomes easier). The product of these two terms is therefore unclear. Formally,
Indirect effect:
Pass Ôn dn (E((, = - a On
Po M0
+(E)
> 0
200yn
To compute drnng (a.) let F(n, a) be the first-order condition of the agent's utility function: a(a, Mo. n)) = RPPass(a, Mo. n) Cp. Since we know no (a) is the optimal number of samples,
dos
we claim that:
F(a, Ppo (a)) Pass
0.
(10)
On m=
This holds because, as we show immediately below, the second derivative of the Pass(-) function always strictly negative with respect n, meaning the utility function is always strictly concave. This is formalized below:
Lemma C.1. The second derivative of Pass with respect always negative
8 Pass
O.
On
Proof. To see this,
8 On dn
d ( ((E)
(Po (b6)
(-
nb)2 406m 4o
(NO 14)
=(E)
200
Since @(E) > O, Ho Nb > 0, ou > 0, and n > O, and since & can be either sign, the two terms in the parentheses are both negative (even if & < O, the negative sign in front ensures negativity). Thus, Intuitively this means that every new sample
the entire expression strictly negative: On increases the chance of passing, but each one helps less than the last.
Pass
The result above also means that the first derivative of F(a, n) always non-zero at rp (a). is also evident that F(a,n) is continuously differentiable. Thus, we can apply the Implicit Function Theorem. Differentiating both sides of F(a, Nuo (a:) = 0 with respect to a yields:
OF On (a, po
du (a)
OF
(a, po (a)) = do da
3F
da
(11)
da da
oF
On
Next we compute da and aF accordingly. Observe that:
or
ar Pass where aa
8
140 ub dΦ(()
Ro Ön da
On da
200 da
Since:
oo(E)
do(E) oE
{Φ(E) σb 1
= -E((E)
da
=
dg
da
da
oo Φ(wa)
we have
OF No μb
ab (2x))
Ro
CΦ(E)
du 20 v
We next turn to computing the derivative of F' with respect to n. Observe the following (where we
plug in d'Pss from above):
On
OF
Pass
(EolE)(Ho - mb) o(E)(Ho - -Mh) 4oōn
Ro
Ro
On
dn
40
Now plugging them back into the expression computed by the Implicit function theorem, have:
(Mo-Ps) d(E)
2 / (a)a,
drip (ax)
OF
Dcz
200 /roo//op Map (a) ((w.)
((w.)
da
OF
T (a)
So(E) (M40-M6) ma)²
oiEline Pb) 40 (a)3/2
E(HO 1b)
do
On
4d (a)
Tup (a)

---

Total derivative. Having computed the direct and indirect effects, we can directly compute the total derivative. We have:
d Pass(a, no. puo (a:) = da
Pass
Pass
du (a)
(12)
da
On
do
2o (0)
Direct effect
indirect effect
Ob Φ(E)
+ +(())
140 14b
du (a)
(13)
To Φ(za)
20 nuo (a)
da
ob Φ(E)
(ND -16 5
1-
(14)
σn ((za)
E(1o jub)
The sign of
(Ho-)Ma)
decides the monotonicity of the passing probability. As we
mo ((Ho-Pa)) Vana (o)
show below, when agent chooses the optimal number of samples n. = Tlp mo (ar), it will never be the case that E(PO - (bb)
1 (1-
(Lemma C.2 which implies that Vn(a)
do
(<(10-P)) (o))-Pa)---- Vapg tm)
always holds at the optimal nuo (a), thus the passing probability at the optimal number of samples poo (a) is always monotonically increasing. We finish the proof by proving Lemma
C.2
Lemma C.2. When nuo (a) is the optimal sample size chosen by the agent to maximize utility, then E(PO
1b) To
0 never holds.
Vn((a)
Proof. Recall the second derivative of the utility w.r.t n is:
Utilltyy Mu: pb, a.)
Pass
On²
On²
(Φ(E)(H - -m6)
( - 1b)
40ēn
40on3/2
(PE) - mb))
(E(HO ub) - 09.
4oōn
Vn
At the optimal number of sample n = nun (a). the second derivative of the utility must be negative, otherwise it implies that increasing the number of samples will increase the utility, which leads
to contradiction to the fact that Rpo (a) is an optimal number of samples, thus we have the
following always holds at n npo (p)(a)) E(poo - 14b) V nro (a)
8n S
Proof of Theorem 4.2
Proof. Recall that the false negative rate in our setting consists of components, conditioned on whether agents participate. Observing that when an agent does not participate - i.c. Ao < μyy [a(a)- their probability passing the statistical test is O, we can simplify the overall FN rate follows:
FN I) E [Fail(a,,o, Po (a)) Z pb] Ha
[aill(a, Po, no (a (P) Z pb, No (=(a)) p-(a)lmo mb)
pa
PNNpassip
Piuo pr(a))mo 2 pb
FNaam
25

---

Consider first any a1 S â. We can explicitly express each component of the false negative as follows, since we are guaranteed that (-(a1) 2 ubi
FNpatticip (a1,I)= 1 - ((1+((01)) (-(as)
S('
g(w)) Fail(a1. Po. Ruo (a)) 1 Q(vr(a)))
duo
(15)
- Q(4b)
FN abstain II Q(1)((1)) - Q(Hu))
(16)
1 1 Q(w))
Let us focus on FN PNarticip- To simplify this, the following result is helpful. For a positive function g(x)) and function fxx with minimum and maximum values fmin and fmax over an interval [a, b). the following holds:
fmin J[ (())) [ g(=)f(z)dr= S fiaz S[ g(x) smin S Looa)saa)dz S fmas
f0g(z)
Since the expression in the middle is between fiin and and fmar. by the intermediate value theorem, there exists an E [a, b] such that: f(c) = L f(x)) which means: there exists c such that 1 a(a)
((c) la ((x) = _a [()))()) This can be interpreted as an integral mean value theorem. Using this, there exists value 3 E (pr(ar)) 1] such that the FN (Nparticsp(a1) can be expressed follows:
1 - Q (ur((a))) Fail(a1 Pho nyg (a)) Paaaticcp II
g(Hu)
m.((s)) 1 ((P((((())) duo
1 Q(ms)
1
((wo) duo
1 ((m)) Fail(an pe mug may (a))
pr(as)) 1- Q(u((an))
1 Qur (ai)) Fail(an, Q(1s)
me M mpt (a))
Therefore, we can write:
1 Qur((a1))
Pail-a Pail(an, "o (a1)(a)))-Q1)
for some ne E (I)() FN(a), I) 1 Q(16)
-QQ(14)
Now consider an such that o <a2 <², Note that due to Appendix B can thus write the FN particip of this instance follows:
Mer (az) S pt (a1). We
1 (()(()2)) f'd
7(10) Q( (az))
FN Partticip (a22, I)
Fail(a2 po: no (a)) Ju-(az)
duo
1 Q(H))
(al())
1
Fail(ay, Ho, nao (az))a(Ho)dpo) + (r()2)
Fail (22, Ho, Pupo (az)) (ts)
1- ((mb)
((((aw) guvo)dpo + Fail(az, 22 n n2 (a()))
1
Fail(az, p2 n (a))
g(p))no
1- [(p()))
(12)))
(1s)
- ((ur(a2)) + Fail(az, 22,, 22(2) 1 1 1(p())
Q(u((an))
= Fail( (21 , n
2.1 (a))
1 1-4(()
where in the third transition, we use the integral mean value theorem as in the previous αi case. Note that for the integral between (ar(a)), 1] in the d22 setting, we know that for every μi0 E Juz (ax1,, 1], Fail( p)) < Fail(an, и²) since from Lemma 4.1 we know that the pass probability increases as alpha increases and this set of agents participated under both a1 and ag. This immediately means that Fail(a2, 222 n, (a)) < Fail( me, n. (a)). In the [ur (az), My (a1)) )) the failure probability is at most 1. Hence, Fail(az, v221 .,,z. < 1. Thus, we can upper bound the FN participation loss at 02 as follows:
FN "Nputticip (az) S (()+(a1)) ((p+(02)) + Fail(a1 m Rug (a)) 1 Q(u-(a1)) - Q(1))
(17)
- Q(1))
26

---

We now express the abstain loss for ag follows:
Nabsstain (a2) = Q(u+((22) - Q(4))
= Q(u((ar)) - Q(ub) Q(u+(an)) - Q((+((2))
1 - 1((())
1 1 1(p())
1 ((6())
Then we have that:
FN(02, = FN particip + FNasstain
(18)
Q((((an)) 1 - Q(pb)
+ Q(v+(an)) - -((u)) - Q(1)) = FN (19)
s Fail(a uer muy (a))
Lastly, for any 2 o, know from Proposition 4.2 that FN particip (az, 1) 0 and from Proposi- tion 4.1 that aNaastai (a, I) O.
D Discussion on Sensitivity Analysis
In this section, we discuss how our framework can be naturally extended to accommodate small misspecifications on either the agent principal side, and why our main insights remain stable under such uncertainty.
Both the agent's participation decision and the principal's ss optimal p-value threshold á depend only on the cost and revenue parameters (co, c, Rt.. In realistic settings such drug development, these parameters are estimated from historical data or business forecasts and may contain some error. We therefore model parameter uncertainty by assuming that the agent and principal each have slightly perturbed estimates of the true values. In particular, we assume the agent has access to approximate parameters (êu, é, R) within of the true values, and let the principal have their own estimates (Co, 0, It) within 8 of the true values. The magnitudes of and á can differ across parameters.
Agent-side uncertainty. We first examine how such c-level errors affect the agent's 's decision for given belief μo. Extending Theorem 3.1 recall that the utility function u(n; a)) can be partitioned by indices (m,, n2) where it is either convex or concave. As noted in Equation (8) of Appendix B, these indices are invariant to small parameter perturbations. In the convex region, the optimal occurs at an endpoint and therefore unchanged. the x--intereppt of Ou/ /n. A perturbation of size shifts this derivative by Oee), implying that the optimal n moves by at most O(e)/ L, where L the inverse Lipschitz constant of 0u/ôn F
the concave region, the optimum corresponds to
no (a) no (a)l S OL))L.
A similar argument bounds the difference in participation thresholds:
(ā=(a) - (+((a) S (ce/k,
where k is the inverse Lipschitz constant (with respect po) of u(pa, ((aa,po))
Principal-side uncertainty. If the principal's perceived parameters (Co, é, R) differ from those of the agent, they will infer a different threshold belief (i-(a), Without observing the agent's c, the principal optimizes a under their own model, obtaining â such that (ā) μ4g- However, because e 6, the agent's ' actual threshold A (ā) may not equal μ4. This deviation can again be bounded by the inverse Lipschitz constant l:
|ar ((()) S 5((e-88)1
(ale-8))1 81)/k.
The agent's deviation depends only on the difference between their perceived parameters and the true ones, whereas the principal's deviation depends on the gap between their perception and the agent's 's. These uncertainties can thus be analyzed independently. Overall, small estimation errors cost or revenue induce proportionally small and bounded shifts in the optimal participation and threshold decisions.
"by inverse lipschitz constant, we mean (f(x1) - f(xz)) 2 L/m Ø2). allowing to bound change in the domain given change in range
27

---

Stability of our results Building on the above observations, we now explain why the main insights of our model remain stable under such uncertainty. This robustness follows from two structural properties of the model: monotonicity and Lipschitz continuity. First, both the participation threshold |r (a) and the principal's optimal choice are continuous and monotone functions of the cost and revenue parameters (Co, R). Second, their derivatives with respect to these parameters are bounded by Lipschitz constants &s and L. ensuring that small perturbations induce at most proportional changes:
J (o:) < k (ACo, Ac, AR)II,
| v L (((Co, Ac, AR)II-
These bounds imply that the relative ordering of participation and approval thresholds is preserved: The curves may shift, but they do not cross or invert. As a result, the qualitative behavior of the model-for for instance, that higher costs discourage participation and looser increases it remains unchanged.
E
Plot for Vaccine Drugs
Figure 4: â vs Revenue Vaccines
0.6
Og 600.0
700.
0.5
8000
Critical p-value 0
0.1
On 1000.0
0.3
0.05
0.0
7500
10,000
15,000
25,000
40,000
Revenue (in Millions)
28


Reviews:
[{"review_id": "cfd1puann1", "reviewer": "Reviewer_s2e9", "summary": "This paper studies a very interesting model in which a principal and an agent are involved in hypothesis testing. The principal must decide whether to accept a new product (say a new drug) based on how effective it is compared to an existing baseline $\\mu_b$. The agent knows how effective their sample is $\\mu_0$ which is drawn from a distribution $q$. This is a Stackleberg game in which the principal first decides a threshold $\\alpha$ for acceptance, which is decided by seeing some number of Bernoulli samples. The agent, upon knowing this, will decide whether to participate and, if so, choose and collect some number of samples $n \\in [n_{\\min}, n_{\\max}]$. Each collection will cost $c_0 + cn$, where $c_0$ is a fixed cost and $c$ is a unit cost. If the product is accepted, the agent receives a reward of\n$R - c - cn$. The main goal is to design the threshold $\\alpha$ to minimize the loss due to false positives and false negatives (appropriately weighted).\n\nIn terms of techniques, the analysis is fairly intuitive but getting the details right takes care. The authors first show that for a given alpha, there is a particular threshold $\\mu_T$ such that the agent will only participate if their value is above it. Furthermore it is computable efficiently and that as their value $\\mu$ increases so does their utility. Now they show that they can bound the loss due to false positives and negatives for any given choice of alpha by breaking down the losses in each component. \n\nFinally, they give a show case for drug companies and approvals for the FDA. They show that the their model roughly captures the economic dynamics of drug approvals in the United States. One interesting recomendation is that to decrease drug prices, the current threshold might be too low to incentivize low revenue but highly effective drugs.", "questions": "N/A", "limitations": "N/A", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "The paper studies a very natural variant of the principal agent model that is highly relevant in FDA drug trials. They show that the game has a natural equilibrium that is computable for the principal and the agents to minimize the total rate of false positives and false negatives. The paper is well written and the technical sections are thorogh.\n\n The main weakness is that the model is natural and nice but the results are not entirely surprising. In fact, technically the paper is quite standard and doesn't innovate in any meaningful way. \n\n Nonetheless, I find the model useful and the practical application compelling. Perhaps EC would be a better conference for this work but I recommend acceptance.", "was_revised": false, "final_justification": "", "rebuttal": "Thank you for your helpful review and feedback! We present a natural, tractable model, grounded in real‑world concerns, for studying hypothesis testing in the presence of strategic participants. Our analysis builds on a sequence of subtle observations to produce clear, interpretable results that give valuable insights on incentives and behaviours in this setting. \n\nDepending on the perspective, the resulting conclusions can be surprising. The fact that a lower bound on the optimal p-value emerges from our work is distinct from the classic thinking of upper-bounding the p-value to control Type I errors. Our analysis is also flexible, highlighting that if context dependent p-values are unpalatable, regulators can instead subsidize the fixed and marginal costs to achieve similar results. We will highlight the latter point more in the revision. \n \nWe view our work as part of the broader research direction on data-driven decision-making under strategic behavior. Strategic classification, strategic regression, and incentive-aware machine learning have received significant attention in the past decade, with many contributions appearing in NeurIPS and other leading machine learning venues. We hope our work encourages further exploration into strategic aspects of statistical analysis, an area that remains relatively under-explored."}, {"review_id": "nn25ImGDvW", "reviewer": "Reviewer_uvQJ", "summary": "The paper studies the problem of principal-agent hypothesis testing, expanding upon the framework proposed by Bates et al. (2023). More specifically, they extend the framework to include considerations of per-sample cost and thus model the agents’ (i.e. pharmaceutical firms) ability to select trial sizes in response to the principal’s (e.g. the FDA) declared significance level $\\alpha$. Additionally, they incorporate false negative considerations in the principal’s loss (which is a combination of the expected incidence of false positives and of false negatives).\n\nBy analyzing the Stackleberg game arising out of the agent maximizing their utility function in response to the published significance level and the corresponding loss landscape achievable by different settings of $\\alpha$, they are able to identify a critical threshold $\\widehat{\\alpha}$ defined as the level for which the participation threshold is exactly the baseline effectiveness. They show that for $\\alpha \\leq \\widehat{\\alpha}$: (i) the overall false positive loss is $0$ and (ii) the overall false negative loss is non-increasing, and for $\\alpha > \\widehat{\\alpha}$: (i) the overall false positive loss is non-decreasing and (ii) the overall false negative rate is non-increasing. This suggests that the true optimal significance level $\\alpha^\\star$ is \\emph{at least} $\\widehat{\\alpha}$ -- therefore using the critical value enables lower loss than a predefined threshold such as $0.05$ and the loss in optimality relative to $\\alpha^\\star$ is due to over-conservativeness rather than allowing more false positives.\n\nFinally, they complement their theoretical insights with experimental studies where they compute the critical value $\\widehat{\\alpha}$ as a function of the revenue $R$, fixed trial cost $c_0$, and per-sample cost $c$ -- where the ranges for these variables are extracted from actual real trial data. Their experiments show that the critical value for lower revenue interventions is actually higher than the FDA imposed 0.05 (and thus the optimal $\\alpha^\\star$ as well and therefore simply shifting towards $\\widehat{\\alpha}$ would already be an improvement \\emph{under the framework’s assumptions}).", "questions": "My main question is how do you think your framework should impact actual practice and regulatory policy?\n\nFor example, with respect to my first note in the “Limitation” section below, how seriously should the takeaway that “if the regulatory bodies wish for decreased drug prices with fixed costs being immutable, they ought to specify a higher p-value threshold” be interpreted?\n\nAnd with respect to my second note, how do you think of the meaning the results in situations where $q$ and $\\mu_0$ are actually not known? For $\\mu_0$ could it be changed to be in terms of “belief” and could the “belief” be a bit more realistic such as uniform over a range? (and is there anything similar that could make the assumption on knowledge of $q$ more reasonable?)\n\nAdditionally, what is additional work (if any) that should be done to ensure such a framework can be actually relevant to regulatory bodies? Finally, do you have any thoughts on experiments or tests that could be done that could assess welfare effects of such theory-based policy suggestions?", "limitations": "I would expand the discussion on the take-away that a stricter $\\alpha$ might dissuade effective but low-revenue drugs from participating without significantly increasing the possibility of false positives for very high revenue drugs. As is, the possible limitations of the framework and how the resulting take-away might not materialize is not really discussed.\n\nGiven that if the framework is to be taken seriously in a policy setting (which I am guessing would be the ideal goal, even if hard to achieve) this take-away is a very striking one that would change the outcomes of the current process significantly, I believe that a more nuanced discussion (with more pointed discussion and even ideally relevant related FDA policy papers discussing their current choice) is warranted.\n\nAnother limitation that would warrant further discussion is the assumption that the distribution $q$ is known to the principal and the agent and that the agent knows the mean effectiveness $\\mu_0$. It would be good to acknowledge this is not realistic and how this affects the applicability of the results in real contexts.", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "A key strength is that the proposed extension of allowing agents to strategically choose their sample size in response to the principal’s announced significance level is very well aligned with practice, both in aim and in the concrete way this is added to the model. The incorporation of a false negative loss component is also appealing, though here I would say that it would be more realistic for this to scale with the effect size (for example the FDA cares less about false negatives of very marginally effective drugs than of highly effective drugs).\n\nI also really appreciate the empirical analysis as it is grounded in actual data with appropriate citations to plug in ranges for the revenue $R$, the fixed cost $c_0$ and the per-sample costs $c$. Finally, I think it works well that proof sketches are included and that the take-aways are well-presented and it is easy to follow the logic and the results.\n\nA potential weakness is that the distribution $q$ is assumed to be known to both the principal and the agent, and, realistically, I think neither would have even coarse knowledge of it. Likewise (though possibly less problematically), I don’t think the agents really would have knowledge of the mean effectiveness of the proposed compound $\\mu_0$.\n\nFinally, some small typos/writing considerations:\n- On line 7, I think it’s meant to be “requested”\n- On line 27, I am not sure what is meant by “high reputation”\n- On line 139 I think it should be “the principal”\n- On line 139, the “loss” was not really defined or discussed previously to an extent comparable to the utility so it is a bit abstract/hazy to say that the principal is minimizing it\n- On line 156, it should be just “principal” not “principal’s”\n- On line 296, there is no Theorem reference – it appears as “[]”\n-\tThe first ‘sentence’ in Theorem 4.3 is not really a sentence\n-\tIn Figure 1 in the RHS plot I think it was meant to be “TruncNorm”", "was_revised": true, "final_justification": "My recommendation is for acceptance. The reason for giving a 4 rather than a 5 is based upon the descriptions of the score which state that a 5 should have \"high impact on one sub-area or moderate-to-high impact on multiple areas,\" and to me the paper is a technically strong paper that has moderate impact on one sub-area.", "rebuttal": "Thank you for your detailed review and comments, which gave us new insights into our contributions and setting! We will fix all typos in the revision; please see below a discussion on the more specific points raised:\n\n---\n\n> **Noisy and imperfect observations of distribution $q$ and effectiveness $\\mu_0$**\n\nWe presently assume that the effectiveness of a product (drug) $\\mu_0 \\sim q$, where $\\mu_0$ is the privately observed effectiveness and $q$ is the originating distribution. $q$ can be seen as the historical performance of drugs in a given category, data that the FDA typically maintains. As for $\\mu_0$, we interpret this as the evidence the manufacturer has based on internal testing and trials, which typically precede a public trial. You are absolutely right that there can be noise and uncertainties about these parameters in practice. However, we note a key strength of our work/model: **neither the agent participation decision nor the p-value threshold $\\hat{\\alpha}$ requires access to q**. Specifically, \n1. Knowledge of q is not required by the agent: the agent observes $\\mu_0 \\sim q$, and then reacts (decides to participate and chooses samples if so) based on this realization.\n2. Our core result, the optimal p-value lower bound $\\hat{\\alpha}$, is agnostic to the distribution $q$. This is of major operational importance since it means the principal also does not need to know $q$ to choose $\\hat{\\alpha}$. That said, $q$ does affect the loss incurred by the principal at this chosen p-value. We detail this below. \n\n> **Quantify the impact of uncertainty due to estimation errors in $q$ on loss calculations**\n\nComputing the FN/FP loss at any chosen p-value, including $\\hat{\\alpha}$, requires access to $q$. Suppose the principal has an erroneous version of the distribution $q$ called $q_{\\text{approx}}$. Then for any $\\alpha$, we can bound the difference between the true FN/FP as calculated under $q$, and the approximate one calculated under $q_\\text{approx}$ using TV distance. In particular, if distributions $q$ and $q_{\\text{approx}}$​ differ by at most $\\delta$ in TV distance, then for any bounded measurable function $f \\in [0,1]$, we can show the difference in conditional expectations over any event with non-negligible probability (at least $\\gamma$) is bounded by:\n$$|E_{\\mu \\sim q}[f(\\mu)|A] - E_{\\mu \\sim q_{approx}}[f(\\mu)|A]|\\leq \\frac{2\\delta}{\\gamma}.$$ \nSince $\\gamma$ is purely from the agent's decision, it is unaffected by the noise level $\\delta$. This allows us to quantify the change in FN and FP losses due to errors in $q$. In particular, both the FN and FP losses take the form of conditional expectations over events defined by thresholds (e.g.,$\\mu_0 \\leq \\mu_b$​), multiplied by ratios of marginal probabilities. We can show that both terms change by at most $\\frac{2\\delta}{\\gamma}$ if the total variation distance between $q$ and $q_{\\mathrm{approx}}$​ is at most $\\delta$, and the relevant conditioning events have probability at least $\\gamma$ under both distributions. In short, we can give an approximation error bound of the form ($\\text{Loss}$ can be either FP or FN): $|\\text{Loss}(q) - \\text{Loss}(q_{\\text{approx}})| \\leq \\frac{4\\delta}{\\gamma}$. We will include the formal proof of this result and a discussion on the effect of noisy/imperfect observations in the revised version.\n\n> **Agent's imperfect observation of $\\mu_0$**\n\nAs for the agent (drug manufacturer), suppose that instead of exactly observing $\\mu_0$, they have some unbiased estimator of it, $\\hat{\\mu}$. In extending our results to this unbiased estimate setting, the agent participation decision is no longer a threshold/deterministic; instead of observing $\\mu_0$ and taking the participation decision $1[\\mu_0 \\geq \\mu_{\\tau}(\\alpha)]$, now they observe $\\hat{\\mu}$ and their participation decision is given by $P[\\hat{\\mu} > \\mu_{\\tau}(\\alpha)]$ – a softer decision boundary. While key properties like monotonicity in $\\mu_0$ still hold if the estimator, the analysis needs to be re-done with this change, and will now include parameters relating to the distribution of the estimator (like variance).  \n\n---\n\n> **Interpreting our conclusion on cost and revenue dependent p-value threshold**\n\nOur results highlight that when agents strategically choose their trial samples and participation based on private beliefs and cost-revenue calculus, the p-value decision must also consider such dynamics. There are two ways to interpret this:\n1. As pointed out in the review, one perspective is to set the p-value based on the expected revenue-cost calculus. This implies that for low-revenue high-development-cost drugs, the p-value threshold should be more generous; otherwise, it is not in the best interest of developers to participate. This can be interpreted as setting a domain/market-specific p-value.\n2. In many cases, the first interpretation may not be palatable, and agencies may be reluctant to change p-values. In such cases, our model suggests that the principal can still derive optimal results against strategic agents by adjusting the other parameters. Specifically, we imagine regulators providing subsidies for development costs and trials to ensure optimal participation on the desired p-value. This is not far-fetched since governments often subsidize crucial drug developments (the COVID-19 vaccine, for example). From this perspective, our result can be interpreted as giving an accurate and interpretable optimal “cost subsidy” needed for drugs to pass under a given confidence interval and charge a given price.\n\nThank you for bringing this up, and we think the latter interpretation is especially insightful. We aim to discuss this in the revised version.\n\n---\n\n> **Additional work to convince regulatory bodies**\n\nOur initial results suggest that the current pricing/revenue of drugs in the US under the FDA p-value of 0.05 is consistent with our model of agent strategic behaviour. That said, from a regulatory perspective, more precise and fine-grained empirical work would be needed to further substantiate this.\n\nA more detailed discussion on the mitigation strategies, whether through adjusting p-values or subsidizing development costs (as mentioned above), is also warranted. Policy decisions like this naturally have broader externalities, hidden or explicit, and they need to be carefully assessed. As for the welfare of such strategies, it is possible to define a Price of Anarchy for Stackelberg games where the optimal welfare corresponds to maximizing the utility for all agents. Conceptually, however, this may be a bit odd since it is not clear why any objective beyond minimizing total error (the current principal objective being optimized) should be socially desirable. It may, however, be prescient to consider whether subsidizing development costs to encourage worthy participation is indeed optimal if there are budget constraints, or what the right trade-offs here are."}, {"review_id": "HeXX87XXuB", "reviewer": "Reviewer_vgRW", "summary": "The paper studies a principal-agent setting applied to hypothesis testing. The goal of the principal is to set a p-value in order to control the rate of false positive and false negative. The paper characterizes how the agent behavior changes depending on the p-value. Moreover, the authors characterize the dependence of the errors on the set p-value. The main contribution of the paper is the definition of a critical threshold $\\hat \\alpha$ such that the agent's error is monotonic for both $\\alpha<\\hat \\alpha$ and $\\alpha>\\hat \\alpha$. This implies that setting the $p$-value to $\\hat \\alpha$ is a good choice in practice. Finally, the theoretical claims are validated by an empirical analysis.", "questions": "Do you have any insight on the structure of the optimal p-value in more general settings?", "limitations": "None.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 4, "significance": 2, "originality": 3, "strengths_and_weaknesses": "Strengths:\nThe problem is interesting and well-motivated. Moreover, the paper is well-written and easy to follow. Finally, the paper provides a clear characterization of the problem.\n\nWeaknesses:\nThe claim that the paper characterizes the optimal $p$-value is false, at least for the general problem introduced in the preliminaries (line 153). The only theoretical result is that the optimal value is larger than $\\hat \\alpha$. This is (approximately) optimal only when $\\lambda_{f_p}>>\\lambda_{f_n}$. It would be great to study also the general problem or change the focus of the paper to a more restrict scenario in which false positive should be avoided.\n\nThe technical contribution is quite weak, and limited to some calculations to characterize whether the rates of error increase or decrease with the increase of the  $p$-value.  \n\n\n\n\nMinor:\n\nLine 158, a a \n\nLine 192, to for those with\n\nLine 198, missing space\n\nLine 222, $\\lambda_1$ non consistent with the notation. \n\nLine 296, Theorem []\n\nLine 300, the statement of the theorem is a bit confusing, the function is always non increasing\nLine 982, $\\alpha^*-> \\hat \\alpha$?", "was_revised": true, "final_justification": "The authors adressed my questions. My evaluation of the paper is unchanged.", "rebuttal": "Thank you for your thoughtful review and comments. We are glad you found the work well-motivated and interesting. Indeed, given the breadth and scope of hypothesis testing and statistical decision-making in the wild (e.g., food, drugs, manufacturing), we hope this work opens up a broader research direction on designing optimal decision rules that account for agent incentives and strategic manipulation. Please see below detailed answers to questions/comments in the review:\n\n---\n\n> **Regarding the optimal p-value:**\n\nYou are right that the value $\\hat{\\alpha}$ we rigorously characterize is not the global optimum for the general problem; rather, it is a lower bound on the optimal p-value threshold $\\hat{\\alpha} \\: \\alpha^* \\geq \\hat{\\alpha}$. \n1. Practicality: In most settings (like drug approvals), the natural desire is to choose a low p-value as this bounds the false positive rate. As a result, bounding the optimal p-value from below may be sufficient in practice, as regulators/principals are generally adverse to higher p-values. Moreover, since our problem is always constant-dimensional (effectiveness is a scalar), if computing the exact $\\alpha^*$ is desired, this can be easily done empirically, by plotting the cumulative loss and noting the minimum (see Figure 1 for example). We include code for this computation.\n2. Interpretability: As our analysis shows, $\\hat{\\alpha}$ dominates any p-value below it in the sense that both FN and FP are monotonically worse for such values. Above $\\hat{\\alpha}$, the relative strengths of FN and FP dictate this trade-off. Since the value of $\\hat{\\alpha}$ is agnostic to both the underlying distribution $q$ and the hyper-parameters $\\lambda_{fn}$ and $\\lambda_{fp}$ (parameters which may be noisy/heuristic driven), this is a much easier decision to justify. Note that the global optimal $\\alpha^*$ will depend on all these parameters – a formal analysis of this would compare the derivatives $\\frac{d}{d\\alpha} \\text{FN}(\\alpha)$ and $\\frac{d}{d\\alpha} \\text{FP}(\\alpha)$ which depend on parameters (q, $\\lambda_{fn}$, $\\lambda_{fp}$).\n\nIn the revised version, we will highlight these operational advantages and interpretability of $\\hat{\\alpha}$ and comment on the properties and comparisons to $\\alpha^*$. \n\n---\n\n> **Regarding the technical contribution:**\n\nOur goal at the onset was to propose a realistic and principled model of strategic interaction in the hypothesis testing framework. Obtaining clear and interpretable conclusions seemed difficult here; unlike standard settings, the principal must control both statistical error and agent participation using only the p-value threshold $\\alpha$. The challenge is further compounded by the rich interplay among the prior, passing probabilities, sample sizes, and agents’ strategic behavior. Some of these relations cannot be explicitly characterized and are not smooth. Yet a sequence of subtle observations and technical insights results in a set of clean structural and practically relevant results. The clarity and easy interpretability of such results are, in our view, a key strength of the framework, especially since decision-makers in these contexts, such as regulators, must justify their choices to a broad audience.\n\nMoreover, while the results are interpretable, the conclusions can be surprising. The fact that a lower bound on the optimal p-value emerges from our work is distinct from the classic thinking of upper-bounding $\\alpha$ to control Type I errors. Our analysis is also flexible, highlighting that if context-dependent p-values are unpalatable, regulators can instead subsidize the fixed and marginal costs to achieve similar results. We will highlight the latter point more in the revision."}, {"review_id": "CWOEhHdEh8", "reviewer": "Reviewer_87c8", "summary": "Summary:\n\nThis submission develops a game-theoretic framework for hypothesis testing in a regulatory environment where an agent (who has private information about a product’s effectiveness) strategically responds to a p-value threshold set by a principal (e.g. a regulatory body like the FDA). This submission extends prior work in this direction by extending the analysis beyond binary notions of effectiveness, and allowing the agents to choose trial sample sizes. The principal’s goal is to minimize some combination of Type I and Type II errors while accounting for the agent’s strategic behavior. \n\nThis strategic interaction is modeled as a Stackelberg game, where the principal commits to a p-value, and the agent best-responds by deciding (1) whether to participate in the trial and (2) how many samples to collect if they choose to participate. In their analysis, the authors show that error rates behave monotonically, which allows them to compute the optimal p-value threshold given this strategic behavior. \n\nEmpirically, the authors apply their methods to drug approval data in oncology, cardiovascular, and vaccine settings by using publicly available data to estimate trial costs and revenue. They find that the FDA’s common p-value threshold of 0.05 closely aligns with the critical threshold for median-cost drugs. They also highlight how overly conservative thresholds may disproportionately discourage participation from low-revenue but effective drugs.", "questions": "How sensitive is your model to unknown revenue and cost parameters from (1) the principal’s side and (2) the agent’s side?", "limitations": "Yes", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 4, "clarity": 4, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths:\n\nThe theoretical model is well-motivated by practice, since in reality agents (1) have a strong incentive to strategize and (2) have the ability to select their own sample size. The monotonicity properties are technically interesting, with proof sketches in the main body and full proofs in the appendices. More broadly, the paper is clearly written and well-structured. The experimental results are also quite interesting, with potential real-world policy implications, which is nice as it is often difficult to run meaningful experiments outside of simulation for these types of strategic learning models. \n\nWeaknesses:\n\nThe authors’ model assumes that the principal has perfect knowledge of the agent’s cost and revenue parameters. While this assumption may be reasonable for after-the-fact analyses like the ones presented in the experimental results of this paper, the agents themselves may not even know these parameters before the trial is run and the product is sold, even putting aside the assumption that the principal knows these parameters. It would have been nice to see some discussion around this, either justifying these assumptions or providing some sort of sensitivity analysis to the equilibrium if the players have estimates of these parameters.", "was_revised": true, "final_justification": "After the rebuttal, my opinion on the paper remains positive. The theoretical model and policy implications are interesting, and my concern about knowledge of model parameters has been addressed.", "rebuttal": "Thank you for your review and insightful comments. We are glad that you found the problem interesting and well-motivated, and the results technically interesting. We hope our work can lead to further empirical and technical works exploring strategic behaviour for hypothesis testing. Below, we include a detailed response to your question: \n\n---\n\n> **Uncertainties about cost and revenue parameters:**\n\nIndeed, both the agent participation decision and the principal’s optimal p-value lower-bound $\\hat{\\alpha}$ depend on the cost and revenue parameters; in fact, these are the only parameters required for this decision. In drug development, past data on trials of similar drugs/genres can be used to estimate costs, whereas projected revenue must be computed for any business decision prior to development and trials. That said, you are correct that in practice, these parameters can only be calculated approximately/with some error. Below is a way to extend our work to such scenarios.\n\nIn the most general case, we can consider the agent having access to parameters $\\hat{c_0}, \\hat{c}, \\hat{R}$ that are within $\\epsilon$ of the true values (note that epsilon can be different for each of the three; this is just for ease of exposition); similarly, the principal has access to $\\tilde{c_0}, \\tilde{c}, \\tilde{R}$ that are within $\\delta$ of the true values. \n\n**1. Agent-side approximation error:**\nWe can first ask how different (due to $\\epsilon$ error) the agent's decision would be upon observing the effectiveness $\\mu_0$. In generalizing Theorem 3.1, we note that the analysis depends on the properties of $u(n; \\alpha)$ and that it can be partitioned by indices $n_1, n_2$ such that within each partition, the function is convex or concave. As we see in equation 8 in Appendix B, the indices $n_1, n_2$ are not affected by $\\epsilon$; neither is the decision in the convex region since the minimum is at the endpoint. For the concave region, the optimal corresponds to the x-intercept of $\\partial u/\\partial n$. The error $\\epsilon$ will change the value of the utility function derivative by $O(\\epsilon)$; thus the change in x-intercept (and thus the optimal choice of $n$) will shift by at most $O(\\epsilon)/L$, where $L$ is the inverse Lipschitz constant of $\\partial u/\\partial n$ (by inverse lipschitz constant, we mean $|f(x_1) - f(x_2)| \\geq L|x_1 - x_2|$, allowing us to bound change in the domain given change in range). That is, $|\\hat{n} (\\alpha; \\mu_0) - n(\\alpha; \\mu_0)| \\leq \\frac{O(\\epsilon)}{L}$. For the function $\\partial u/\\partial n$ specifically, this inverse lipschitz constant can be expressed in terms of  $n_1, n_2, n_{min}, n_{max}$.\n\nUncertainty about cost and revenue also affects the participation threshold $\\mu_\\tau(\\alpha)$, defined as the minimum belief where an agent will participate for a given p-value $\\alpha$. Again, since the utility deviates by $O(\\epsilon)$, the corresponding change in threshold belief can again be bounded by the inverse lipschitz constant $k$ (in terms of $\\mu_0$) of the function $u(\\mu_0, n(\\alpha, \\mu_0))$. In short, this approach can bound $\\mu_{\\tau}(\\alpha)$ and $\\hat{\\mu}_{\\tau}(\\alpha)$.\n\n**2. Principal side approximation error:**\nAs for the principal, their perception of the cost/revenue, $\\tilde{c_0}, \\tilde{c}, \\tilde{R}$, is different than the agents. So for any $\\alpha$, they will perceive a threshold belief $\\tilde{\\mu}_{\\tau}(\\alpha)$. As the principal does not know the agent's error $\\epsilon$, they can do no better than follow the conclusions of section 4 and choose a p-value of at least \n$\\hat{\\alpha}$ such that \n\n$\\tilde{\\mu}_{\\tau}(\\hat{\\alpha}) = \\mu_b.$\n\nBut under this $\\hat{\\alpha}$, since the agent's estimated parameter is different from the principal's estimated parameter ($\\epsilon \\neq \\delta$), we know that $\\hat{\\mu}_{\\tau}(\\hat{\\alpha}) \\neq \\mu_b$, which means that the agent will not behave exactly as the principal expects.\n\nHowever, it is possible to compute how far away $\\hat{\\mu}_{\\tau}(\\hat{\\alpha})$ is from $\\mu_b$ by using the inverse Lipschitz constant $k$ of function $u(\\mu_0, n(\\alpha, \\mu_0))$ mentioned above. \n\nImportantly, note that the principal's uncertainty error depends only on how far their cost/revenue estimate $\\tilde{c_0}, \\tilde{c}, \\tilde{R}$  was from the agent's estimate $\\hat{c_0}, \\hat{c}, \\hat{R}$ (and not the true value), since this is what the agent uses to make their decision. The agent's uncertainty error, on the other hand, only depends on how far their estimate $\\hat{c_0}, \\hat{c}, \\hat{R}$ was from the true values $c_0, c, R$. One can analyze them separately."}]

Author rebuttal:

Meta-review: **Summary**

The paper studies problems where a principal must decide between approving or rejecting a product manufactured by an agent based on evidence provided by the agent. The paper tackles the problem by using game-theoretic tools in contract theory, by providing an interesting characterization of the principal’s optimal p-value. The paper also empirically validates the model and results by using publicly available data on drug approvals. 

**Strengths**

The paper is very well written, and the results presented are of broad interest. The model is a natural variant of the classical principal-agent problem that may be relevant in many application scenarios. 

**Weaknesses**

The results are *not* entirely surprising. However, the model is natural and worth studying.

**Decision**

All the Reviewers agree that this is a good paper for NurIPS, and thus I recommend its **acceptance**.
Decision: Accept (spotlight)