Title: Strategic Hypothesis Testing
Venue: NeurIPS 2025
Accepted: yes
Keywords: Stackelberg Game; Hypothesis Testing; Strategic Machine Learning; Game Theory
Abstract: We examine hypothesis testing within a principal-agent framework, where a strategic agent, holding private beliefs about the effectiveness of a product, submits data to a principal who decides on approval. The principal employs a hypothesis testing rule, aiming to pick a p-value threshold that balances false positives and false negatives while anticipating the agent’s incentive to maximize expected profitability. Building on prior work, we develop a game-theoretic model that captures how the agent’s participation and reporting behavior respond to the principal’s statistical decision rule. Despite the complexity of the interaction, we show that the principal's errors exhibit clear monotonic behavior when segmented by an efficiently computable critical p-value threshold, leading to an interpretable characterization of their optimal p-value threshold. We empirically validate our model and these insights using publicly available data on drug approvals. Overall, our work offers a comprehensive perspective on strategic interactions within the hypothesis testing framework, providing technical and regulatory insights.


Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nSummary:\n\nThis submission develops a game-theoretic framework for hypothesis testing in a regulatory environment where an agent (who has private information about a product’s effectiveness) strategically responds to a p-value threshold set by a principal (e.g. a regulatory body like the FDA). This submission extends prior work in this direction by extending the analysis beyond binary notions of effectiveness, and allowing the agents to choose trial sample sizes. The principal’s goal is to minimize some combination of Type I and Type II errors while accounting for the agent’s strategic behavior. \n\nThis strategic interaction is modeled as a Stackelberg game, where the principal commits to a p-value, and the agent best-responds by deciding (1) whether to participate in the trial and (2) how many samples to collect if they choose to participate. In their analysis, the authors show that error rates behave monotonically, which allows them to compute the optimal p-value threshold given this strategic behavior. \n\nEmpirically, the authors apply their methods to drug approval data in oncology, cardiovascular, and vaccine settings by using publicly available data to estimate trial costs and revenue. They find that the FDA’s common p-value threshold of 0.05 closely aligns with the critical threshold for median-cost drugs. They also highlight how overly conservative thresholds may disproportionately discourage participation from low-revenue but effective drugs.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\n\nThe theoretical model is well-motivated by practice, since in reality agents (1) have a strong incentive to strategize and (2) have the ability to select their own sample size. The monotonicity properties are technically interesting, with proof sketches in the main body and full proofs in the appendices. More broadly, the paper is clearly written and well-structured. The experimental results are also quite interesting, with potential real-world policy implications, which is nice as it is often difficult to run meaningful experiments outside of simulation for these types of strategic learning models. \n\nWeaknesses:\n\nThe authors’ model assumes that the principal has perfect knowledge of the agent’s cost and revenue parameters. While this assumption may be reasonable for after-the-fact analyses like the ones presented in the experimental results of this paper, the agents themselves may not even know these parameters before the trial is run and the product is sold, even putting aside the assumption that the principal knows these parameters. It would have been nice to see some discussion around this, either justifying these assumptions or providing some sort of sensitivity analysis to the equilibrium if the players have estimates of these parameters.\n\nQUESTIONS:\nHow sensitive is your model to unknown revenue and cost parameters from (1) the principal’s side and (2) the agent’s side?\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-06-24"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies a principal-agent setting applied to hypothesis testing. The goal of the principal is to set a p-value in order to control the rate of false positive and false negative. The paper characterizes how the agent behavior changes depending on the p-value. Moreover, the authors characterize the dependence of the errors on the set p-value. The main contribution of the paper is the definition of a critical threshold $\\hat \\alpha$ such that the agent's error is monotonic for both $\\alpha<\\hat \\alpha$ and $\\alpha>\\hat \\alpha$. This implies that setting the $p$-value to $\\hat \\alpha$ is a good choice in practice. Finally, the theoretical claims are validated by an empirical analysis.\n\nSTRENGTHS AND WEAKNESSES:\nStrengths:\nThe problem is interesting and well-motivated. Moreover, the paper is well-written and easy to follow. Finally, the paper provides a clear characterization of the problem.\n\nWeaknesses:\nThe claim that the paper characterizes the optimal $p$-value is false, at least for the general problem introduced in the preliminaries (line 153). The only theoretical result is that the optimal value is larger than $\\hat \\alpha$. This is (approximately) optimal only when $\\lambda_{f_p}>>\\lambda_{f_n}$. It would be great to study also the general problem or change the focus of the paper to a more restrict scenario in which false positive should be avoided.\n\nThe technical contribution is quite weak, and limited to some calculations to characterize whether the rates of error increase or decrease with the increase of the  $p$-value.  \n\n\n\n\nMinor:\n\nLine 158, a a \n\nLine 192, to for those with\n\nLine 198, missing space\n\nLine 222, $\\lambda_1$ non consistent with the notation. \n\nLine 296, Theorem []\n\nLine 300, the statement of the theorem is a bit confusing, the function is always non increasing\nLine 982, $\\alpha^*-> \\hat \\alpha$?\n\nQUESTIONS:\nDo you have any insight on the structure of the optimal p-value in more general settings?\n\nLIMITATIONS:\nNone.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 4, "CLARITY": 4, "IMPACT": 2, "ORIGINALITY": 3, "DATE": "2025-07-02"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper studies the problem of principal-agent hypothesis testing, expanding upon the framework proposed by Bates et al. (2023). More specifically, they extend the framework to include considerations of per-sample cost and thus model the agents’ (i.e. pharmaceutical firms) ability to select trial sizes in response to the principal’s (e.g. the FDA) declared significance level $\\alpha$. Additionally, they incorporate false negative considerations in the principal’s loss (which is a combination of the expected incidence of false positives and of false negatives).\n\nBy analyzing the Stackleberg game arising out of the agent maximizing their utility function in response to the published significance level and the corresponding loss landscape achievable by different settings of $\\alpha$, they are able to identify a critical threshold $\\widehat{\\alpha}$ defined as the level for which the participation threshold is exactly the baseline effectiveness. They show that for $\\alpha \\leq \\widehat{\\alpha}$: (i) the overall false positive loss is $0$ and (ii) the overall false negative loss is non-increasing, and for $\\alpha > \\widehat{\\alpha}$: (i) the overall false positive loss is non-decreasing and (ii) the overall false negative rate is non-increasing. This suggests that the true optimal significance level $\\alpha^\\star$ is \\emph{at least} $\\widehat{\\alpha}$ -- therefore using the critical value enables lower loss than a predefined threshold such as $0.05$ and the loss in optimality relative to $\\alpha^\\star$ is due to over-conservativeness rather than allowing more false positives.\n\nFinally, they complement their theoretical insights with experimental studies where they compute the critical value $\\widehat{\\alpha}$ as a function of the revenue $R$, fixed trial cost $c_0$, and per-sample cost $c$ -- where the ranges for these variables are extracted from actual real trial data. Their experiments show that the critical value for lower revenue interventions is actually higher than the FDA imposed 0.05 (and thus the optimal $\\alpha^\\star$ as well and therefore simply shifting towards $\\widehat{\\alpha}$ would already be an improvement \\emph{under the framework’s assumptions}).\n\nSTRENGTHS AND WEAKNESSES:\nA key strength is that the proposed extension of allowing agents to strategically choose their sample size in response to the principal’s announced significance level is very well aligned with practice, both in aim and in the concrete way this is added to the model. The incorporation of a false negative loss component is also appealing, though here I would say that it would be more realistic for this to scale with the effect size (for example the FDA cares less about false negatives of very marginally effective drugs than of highly effective drugs).\n\nI also really appreciate the empirical analysis as it is grounded in actual data with appropriate citations to plug in ranges for the revenue $R$, the fixed cost $c_0$ and the per-sample costs $c$. Finally, I think it works well that proof sketches are included and that the take-aways are well-presented and it is easy to follow the logic and the results.\n\nA potential weakness is that the distribution $q$ is assumed to be known to both the principal and the agent, and, realistically, I think neither would have even coarse knowledge of it. Likewise (though possibly less problematically), I don’t think the agents really would have knowledge of the mean effectiveness of the proposed compound $\\mu_0$.\n\nFinally, some small typos/writing considerations:\n- On line 7, I think it’s meant to be “requested”\n- On line 27, I am not sure what is meant by “high reputation”\n- On line 139 I think it should be “the principal”\n- On line 139, the “loss” was not really defined or discussed previously to an extent comparable to the utility so it is a bit abstract/hazy to say that the principal is minimizing it\n- On line 156, it should be just “principal” not “principal’s”\n- On line 296, there is no Theorem reference – it appears as “[]”\n-\tThe first ‘sentence’ in Theorem 4.3 is not really a sentence\n-\tIn Figure 1 in the RHS plot I think it was meant to be “TruncNorm”\n\nQUESTIONS:\nMy main question is how do you think your framework should impact actual practice and regulatory policy?\n\nFor example, with respect to my first note in the “Limitation” section below, how seriously should the takeaway that “if the regulatory bodies wish for decreased drug prices with fixed costs being immutable, they ought to specify a higher p-value threshold” be interpreted?\n\nAnd with respect to my second note, how do you think of the meaning the results in situations where $q$ and $\\mu_0$ are actually not known? For $\\mu_0$ could it be changed to be in terms of “belief” and could the “belief” be a bit more realistic such as uniform over a range? (and is there anything similar that could make the assumption on knowledge of $q$ more reasonable?)\n\nAdditionally, what is additional work (if any) that should be done to ensure such a framework can be actually relevant to regulatory bodies? Finally, do you have any thoughts on experiments or tests that could be done that could assess welfare effects of such theory-based policy suggestions?\n\nLIMITATIONS:\nI would expand the discussion on the take-away that a stricter $\\alpha$ might dissuade effective but low-revenue drugs from participating without significantly increasing the possibility of false positives for very high revenue drugs. As is, the possible limitations of the framework and how the resulting take-away might not materialize is not really discussed.\n\nGiven that if the framework is to be taken seriously in a policy setting (which I am guessing would be the ideal goal, even if hard to achieve) this take-away is a very striking one that would change the outcomes of the current process significantly, I believe that a more nuanced discussion (with more pointed discussion and even ideally relevant related FDA policy papers discussing their current choice) is warranted.\n\nAnother limitation that would warrant further discussion is the assumption that the distribution $q$ is known to the principal and the agent and that the agent knows the mean effectiveness $\\mu_0$. It would be good to acknowledge this is not realistic and how this affects the applicability of the results in real contexts.", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-03"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a very interesting model in which a principal and an agent are involved in hypothesis testing. The principal must decide whether to accept a new product (say a new drug) based on how effective it is compared to an existing baseline $\\mu_b$. The agent knows how effective their sample is $\\mu_0$ which is drawn from a distribution $q$. This is a Stackleberg game in which the principal first decides a threshold $\\alpha$ for acceptance, which is decided by seeing some number of Bernoulli samples. The agent, upon knowing this, will decide whether to participate and, if so, choose and collect some number of samples $n \\in [n_{\\min}, n_{\\max}]$. Each collection will cost $c_0 + cn$, where $c_0$ is a fixed cost and $c$ is a unit cost. If the product is accepted, the agent receives a reward of\n$R - c - cn$. The main goal is to design the threshold $\\alpha$ to minimize the loss due to false positives and false negatives (appropriately weighted).\n\nIn terms of techniques, the analysis is fairly intuitive but getting the details right takes care. The authors first show that for a given alpha, there is a particular threshold $\\mu_T$ such that the agent will only participate if their value is above it. Furthermore it is computable efficiently and that as their value $\\mu$ increases so does their utility. Now they show that they can bound the loss due to false positives and negatives for any given choice of alpha by breaking down the losses in each component. \n\nFinally, they give a show case for drug companies and approvals for the FDA. They show that the their model roughly captures the economic dynamics of drug approvals in the United States. One interesting recomendation is that to decrease drug prices, the current threshold might be too low to incentivize low revenue but highly effective drugs.\n\nSTRENGTHS AND WEAKNESSES:\nThe paper studies a very natural variant of the principal agent model that is highly relevant in FDA drug trials. They show that the game has a natural equilibrium that is computable for the principal and the agents to minimize the total rate of false positives and false negatives. The paper is well written and the technical sections are thorogh.\n\n The main weakness is that the model is natural and nice but the results are not entirely surprising. In fact, technically the paper is quite standard and doesn't innovate in any meaningful way. \n\n Nonetheless, I find the model useful and the practical application compelling. Perhaps EC would be a better conference for this work but I recommend acceptance.\n\nQUESTIONS:\nN/A\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 3, "CLARITY": 3, "IMPACT": 3, "ORIGINALITY": 3, "DATE": "2025-07-15"}