Auction-Based Regulation for Artificial Intelligence

## Abstract

Abstract In an era of “moving fast and breaking things” , regulators have moved slowly to pick up the safety, bias, and legal pieces left in the wake of broken Artificial Intelligence (AI) deployment.
Since AI models, such as large language models, are able to push misinformation and stoke division within our society, it is imperative for regulators to employ a framework that mitigates these dangers and ensures user safety.
While there is much-warranted discussion about how to address the safety, bias, and legal woes of state-of-the-art AI models, the number of rigorous and realistic mathematical frameworks to regulate AI safety is lacking.
We take on this challenge, proposing an auction-based regulatory mechanism that provably incentivizes model-building agents (i) to deploy safer models and (ii) to participate in the regulation process.
We provably guarantee, via derived Nash Equilibria, that each participating agent’s best strategy is to submit a model safer than a prescribed minimum-safety threshold.
Empirical results show that our regulatory auction boosts safety and participation rates by 20 % percent 20 20\% 20 % and 15 % percent 15 15\% 15 % respectively, outperforming simple regulatory frameworks that merely enforce minimum safety standards.
Code can be found on GitHub at https://github.com/marcobornstein/AI-Regulatory-Auctions .

## 1 Introduction

Current Artificial Intelligence (AI) models are powerful and have revolutionized a wide swath of industries.
The recent large-scale deployment of Large Language Models (LLMs) has simultaneously boosted human productivity while sparking concern over safety (e.g., hallucinations, bias, and privacy).
We have seen industry leaders, such as Google, Meta, and OpenAI, embroiled in controversy surrounding bias and misinformation (Brewster, ; Robertson, ; White, ), safety (Jacob, ; Seetharaman, ; White, ), as well as legality and ethics (Bruell, ; Metz et al., ; Moreno, ) in their development and deployment of LLMs.
Furthermore, irresponsible deployment of LLMs runs the risk of allowing adversaries the ability to spread misinformation or propaganda (Barman et al., ; Neumann et al., ; Sun et al., ).
Unfortunately, a consistent and industry-wide solution to oversee safe AI deployment remains elusive.

Naturally, one such solution to mitigate these dangers is for increased governmental regulation over AI deployment.
In the United States, there have been some strides, on federal (House, ) and state levels (Information, ), to regulate the safety and security of large-scale AI systems (including LLMs).
While these recent executive orders and bills highlight the necessity to develop safety standards and enact safety and security protocols, few details are offered.
This follows a consistent trend of well-deserved scrutiny towards the lack of AI regulation without the development of rigorous and realistic mathematical frameworks to regulate.
Our work sets out to solve this disconnect by proposing a novel regulatory framework that a regulator can follow to not only strictly enforce the safety of deployed AI models, but simultaneously incentivize the production of safer AI models.

The goal of our work is to formulate the AI regulatory process mathematically, and subsequently develop a mechanism to incentivize safer development and deployment of AI models.
Specifically, we formulate the AI regulatory process as an all-pay auction, where agents (companies) submit their models to a regulator.
The regulator’s job is twofold: (a) prohibit deployment of models that fail to meet prescribed safety thresholds, and (b) incentivize safe model production and deployment by providing additional rewards to agents that submit safer models than their peers.
We design an auction-based regulatory mechanism, with a novel reward-payment protocol, that emits Nash Equilibria at which agents develop and deploy models safer than the prescribed safety threshold.

Figure: Figure 1: Step-by-Step Sira Schematic. (Step 0) The regulator sets a safety threshold, $\epsilon$, having a corresponding price, $p_{\epsilon}$, required to achieve $\epsilon$. (Step 1) Agents evaluate their total value, $V_{i}$, from model deployment value ($v_{i}^{d}$) and potential regulator compensation ($v_{i}^{p}$). Agents only participate if their total value exceeds $p_{\epsilon}$. (Step 2) Participating agents submit their models to the regulator, accompanied by their bid $b_{i}$, which reflects the amount spent to improve their model’s safety level. Models with bids below $p_{\epsilon}$ are automatically rejected. (Step 3) The submitted models are randomly paired, and the safer model (i.e., the higher bid) in each pair wins the comparison. In this example, agent 3 wins since $b_{3}>b_{1}$. (Step 4) Winning models receive both a premium and deployment value (i.e., agent 3 in this example wins premium $v_{3}^{p}$ and deployment value $v^{d}_{3}$), while losing models receive only the deployment value (i.e., agent 1 in this example wins deployment value $v^{d}_{1}$ only).
Refer to caption: extracted/5896309/figures/sira_schematic_new.png

Summary of Contributions.
In summary, the main contributions of our paper are as follows.

- (AI Regulation):
We propose a Safety-Incentivized Regulatory Auction (Sira) mechanism, offering a simple yet practical approach to AI regulation.
- (Safety-First):
We establish, through derived Nash Equilibria, that agents are incentivized to submit models that surpass the required safety threshold.
- (Effective):
Empirical results demonstrate that Sira increases model safety by over 20% and boosts participation rates by 15% compared to baseline regulatory mechanisms.

## 2 Related Works

#### Regulation Frameworks for Artificial Intelligence.

A handful of work focuses on regulation frameworks for AI (de Almeida et al., ; Jagadeesan et al., ; Rodríguez et al., ; Yaghini et al., ).
First, de Almeida et al. () details the need for AI regulation and surveys existing proposals.
The proposals are ethical frameworks detailing specific ethical decisions to make and dilemmas to address.
These proposals lack a mathematical framework to incentivize provably safer models.
Rodríguez et al. () utilize AI models to detect collusive auctions.
This work is related to our own but in reverse: AI is applied to regulate auctions and ensure that they are not collusive.
In contrast, our work aims to use auctions to regulate AI deployment.
Jagadeesan et al. () focuses on reducing barriers to entry for smaller companies who are competing against larger and more established incumbent companies.
A multi-objective high-dimensional regression framework is proposed to capture “reputational damage” for companies who deploy unsafe AI models.
This work allows varying levels of safety constraints, where newer companies face less severe constraints in order to spur their entry into the market, which is unrealistic in many settings and only considers simple linear-regression models.
The closest related work to ours, Yaghini et al. (), proposes a regulation game for ensuring privacy and fairness that is formulated as a Stackelberg game.
This game is a multi-agent optimization problem that is also multi-objective (for fairness and privacy).
An equilibrium-search algorithm is presented to ensure that agents remain on the Pareto frontier of their objectives (although this frontier is estimated algorithmically).
Notably, Yaghini et al. () considers only one model builder (agent) and multiple regulators that provide updates to the agent’s strategy.
We consider a more realistic setup, where there are multiple agents and a single regulator whose goal is to incentivize safer model deployment.
We do not believe that it is the regulator’s job to collaborate with agents to optimize their strategy.
Furthermore, our mechanism is simple and efficient; we do not require Pareto frontier estimation or multiple rounds of optimization.

#### All-Pay Auctions.

Compared to the dearth of literature in regulatory frameworks for AI, all-pay auctions are well-researched (Amann and Leininger, ; Baye et al., ; Bhaskar, ; DiPalantino and Vojnovic, ; Gemp et al., ; Goeree and Turner, ; Siegel, ; Tardos, ).
These works formulate specific all-pay auctions and determine their equilibria.
Some works consider settings where agents have complete information about their rivals’ bids (Baye et al., ) while others consider incomplete information, such as only knowing the distribution of agent valuations (Amann and Leininger, ; Bhaskar, ; Tardos, ).
One major application of all-pay auctions are crowd-sourcing competitions.
Many agents participate to win a reward, with those losing still incurring a cost for their time, effort, etc.
DiPalantino and Vojnovic () is one of the first works to model crowd-sorucing competitions as an all-pay auction.
Further research, such as Gemp et al. (), have leveraged AI to design all-pay auctions for crowd-sourcing competitions.
Instead of crowd-sourcing, our work formulates the AI regulatory process as an asymmetric and incomplete all-pay auction.
We leverage previous analysis in this setting (Amann and Leininger, ; Bhaskar, ; Tardos, ) to derive equilibria.

## 3 The Regulatory AI Setting

There exists a regulator $R$ with the power to set and enforce laws and regulations (e.g., U.S. government regulation on lead exposure).
The regulator wants to regulate AI model deployment, by ensuring that all models meet a given safety threshold $\epsilon\in(0,1)$, e.g., the National Institute for Occupational Safety and Health regulates that N95 respirators filter out at least 95% of airborne particles.
If a model does not reach the safety criteria $\epsilon$, then the model is deemed unsafe and the regulator bars deployment.
On the other side, there are $n$ rational model-building agents.
Agents seek to maximize their own benefit, or utility.

Bidding & Evaluation.
By law, each agent $i$ must submit, or bid in auction terminology, its model $w_{i}\in\mathbb{R}^{d}$ for evaluation to the regulator before it can be approved for deployment.
Let $S(w;x):\mathbb{R}^{d}\rightarrow\mathbb{R}_{+}$ be a safety metric that outputs a safety level (the larger the better) for model $w$ given data $x$.
In effect, each agent, given its own data $x_{i}$, bids a safety level $s_{i}^{A}:=S(w_{i};x_{i})$ to the regulator.
Subsequently, the regulator, using its own data $x_{R}$, independently evaluates the agent’s safety level bid as $s_{i}^{R}:=S(w_{i};x_{R})$.
We assume that agent and regulator evaluation data is independent and identically distributed (IID) $x_{i},x_{R}\sim\mathcal{D}$.

Agent and regulator evaluation data comes from the same distribution $x_{i},x_{R}\sim\mathcal{D}$.

This assumption is often realistic in regulatory settings, because both agents and regulators typically rely on standardized or widely accepted datasets for model evaluation, ensuring a fair and unbiased assessment of safety levels.
For instance, when assessing LLMs, common datasets like benchmarks for toxicity or bias are employed consistently across evaluations, reflecting real-world data distributions. Therefore, it is reasonable to define agent $i$’s safety level bid as $s_{i}:=\mathbb{E}_{x\sim\mathcal{D}}[S(w_{i};x)]$.
We address scenarios where evaluation data may not follow the IID assumption in Section 7.

The Price of Safety.
We assume that there exists a strictly increasing function $M:(0,1)\rightarrow(0,1)$ that determines the “price of safety” (i.e., maps safety into cost).
Simply put, safer models cost more to attain.
As a result, we define the price of attaining $\epsilon$ safety as $p_{\epsilon}:=M(\epsilon)$.
{assumption}
There exists a strictly increasing function $M$ that maps safety to cost.
The assumption that a strictly increasing function $M$ maps safety to cost is realistic, because achieving higher safety levels typically requires greater resources. Safer models often demand more data, advanced tuning, and extensive validation, all of which increase costs.
Thus, defining the price of safety as $p_{\epsilon}:=M(\epsilon)$, where $M$ is strictly increasing, reflects the practical trade-off that safer models cost more to develop.

Agent Costs.
Unfortunately for agents, training a safer model comes with added cost.
Consequently, each agent $i$ must decide how much money to bid, or spend, $b_{i}$ to make its model safer.
By Assumption 3, the resulting safety level of an agent’s model will be $s_{i}=M^{-1}(b_{i})$.

Agent Values.
(1) Model deployment value $v_{i}^{d}$.
While it costs more for agents to produce safer models, they gain value from having their models deployed.
Intuitively, this can be viewed as the expected value $v_{i}^{d}$ of agent $i$’s model.
The valuation for model deployment varies across agents (e.g., Google may value having its model deployed more than Apple).
(2) Premium reward value $v_{i}^{p}$.
Beyond value for model deployment, the regulator can also offer additional, or premium, compensation valued as $v_{i}^{p}$ by agents (e.g., tax credits for electric vehicle producers or Fast Track and Priority Review of important drugs by the U.S. Food & Drug Administration).
The regulator provides additional compensation to agents whose models demonstrate safety levels exceeding the prescribed threshold.
However, the value of this compensation varies across agents due to differing internal valuations.
It is unrealistic for the regulator to compensate all agents meeting the safety threshold due to budget constraints. Therefore, we limit the additional rewards to a top-performing half of agents who surpass the threshold, ensuring that compensation targets those contributing the most to enhanced safety while maintaining feasibility for the regulator.

Value Distribution.
We define the total value for each agent $i$ as $V_{i}:=v_{i}^{d}+v_{i}^{p}$, which represents the sum of the deployment value and premium compensation.
Although these values can vary widely in practice, we normalize $\{V_{i}\}_{i=1}^{n}$ for all $n$ agents to be between 0 and 1 for analytical tractability, allowing us to work within a standardized range.
Consequently, the price to achieve the safety threshold
$\epsilon$ is also normalized to fall within the $(0,1)$ interval, i.e., $p_{\epsilon}\in(0,1)$.

The proportion of total value allocated to deployment versus compensation is determined by a scaling factor $\lambda_{i}\sim\mathcal{D}_{\lambda}(0,1/2)$.
Therefore, the deployment value is $v_{i}^{d}:=(1-\lambda_{i})V_{i}$, and the premium compensation value is $v_{i}^{p}:=\lambda_{i}V_{i}$.
Both $V_{i}$ and $\lambda_{i}$ are private to each agent, though the distributions $\mathcal{D}_{V}$ and $\mathcal{D}_{\lambda}$ are known by participants.
We set the maximum allowable factor at $\lambda_{i}=1/2$, reflecting the realistic constraint that compensation should not exceed deployment value. Although our results primarily consider $\lambda_{i}\leq 1/2$, theoretical extensions can be made for scenarios where $\lambda_{i}>1/2$.

All-Pay Auction Formulation.
Overall, agents face a trade-off: producing safer models garners value, via the regulator, but incurs larger costs.
Furthermore, in order to attain the rewards detailed in Section 3, agents must submit a model with safety level at least as large as $\epsilon$.
We can formulate this problem as an asymmetric all-pay auction with incomplete information (Amann and Leininger, ; Bhaskar, ; Tardos, ).
The problem is an all-pay auction since agents incur an unrecoverable cost, safety training costs, when submitting their model to regulators.
The problem is asymmetric with incomplete information since valuations $V_{i}$ are private and differ for each agent.

Agent Objective.
The objective for each model-building agent $i$, is to maximize its own utility $u_{i}$.
Namely, each agent seeks to determine an optimal safety level to bid to the regulator $b_{i}$.
However, depending upon the all-pay auction formulation, agents would need to take into account all other agents’ bids $\bm{b}_{-i}$ in order to determine their optimal bid $b_{i}^{*}$,

$$ $b_{i}^{*}:=\operatorname*{arg\,max}_{b_{i}}u_{i}(b_{i};\bm{b}_{-i}).$ (1) $$

A major portion of our work is constructing an auction-based mechanism, thereby designing the utility of each agent, such that participating agents maximize their utility when they bid more than “the price to obtain the safety threshold”, i.e., $b_{i}^{*}>p_{\epsilon}$.
We begin by providing a simple mechanism, already utilized by regulators, that does not accomplish this goal, before detailing our auction-based mechanism Sira that provably ensures that $b_{i}^{*}>p_{\epsilon}$ for all agents.

## 4 Reserve Thresholding: Bare Minimum Regulation

The simplest method to ensure model safety is for the regulators to set a reserve price, or minimum acceptable safety.
We term this mechanism a multi-winner reserve thresholding auction, where the regulator awards a deployment reward, $v_{i}^{d}$, to each agent whose submitted model meets or exceeds the safety threshold $\epsilon$.
Within this auction, each agent $i$’s utility is mathematically formulated as,

$$ $u_{i}(b_{i};\bm{b}_{-i})=\begin{cases}-b_{i}\text{ if $b_{i}<p_{\epsilon}$},\\ v_{i}^{d}-b_{i}\text{ if }b_{i}\geq p_{\epsilon}.\end{cases}$ (2) $$

The formulation above, however, is ineffective at incentivizing agents to produce models that are safer than the $\epsilon$ threshold.
{theorem}[Reserve Thresholding Nash Equilibrium]
Under Assumption 3, agents participating in Reserve Thresholding (2) have an optimal bidding strategy and utility of,

$$ $b_{i}^{*}=p_{\epsilon},\quad u_{i}(b_{i}^{*};\bm{b}_{-i})=v_{i}^{d}-p_{ \epsilon},$ (3) $$

and submit models with the following safety level,

$$ $s_{i}^{*}=\begin{cases}\epsilon&\text{ if }u_{i}(b_{i}^{*};\bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else.}\end{cases}$ (4) $$

When a regulator implements reserve thresholding, as formally detailed in Theorem 2, agents exert minimal effort, submitting models that just meet the required safety threshold $\epsilon$.
While this approach ensures that all deployed models satisfy the minimum safety requirements, it fails to encourage agents to build models with safety levels exceeding $\epsilon$.
Additionally, agents whose deployment rewards are less than the cost of achieving the safety threshold, i.e., $v_{i}^{d}<p_{\epsilon}$, have no incentive to participate in the regulatory process, leading to reduced participation rates, as detailed in Remark 4.

[Lack of Incentive]
Each agent is only incentivized to submit a model with safety $s_{i}^{*}=\epsilon$.
Our goal is to construct a mechanism that incentivizes agents to build models reaching past minimum requirements: $s_{i}^{*}>\epsilon$.

## 5 Safety-Incentivized Regulatory Auctions ( Sira )

Figure: Algorithm 1 Sira: Safety-Incentivized Regulatory Auction

To alleviate the lack of incentives within simple regulatory auctions, such as the one in Section 4, we propose a regulatory all-pay auction that emits an equilibrium where agents submit models with safety levels larger than $\epsilon$.

Algorithm Description.
The core component of our auction is that agent safety levels are randomly compared against one another, with the regulator rewarding those having the safer model with premium compensation.
Only agents with models that achieve a safety level of $\epsilon$ or higher are eligible to participate in the comparison process; models that do not meet this threshold are automatically rejected.
The detailed algorithmic block of SIRA is depicted in Algorithm 1.

Agent Utility. The utility for each agent $i$ is therefore defined as in Equation (5).

$$ $u_{i}(b_{i};\bm{b}_{-i})=\big{(}v_{i}^{d}+v_{i}^{p}\cdot 1_{(\text{if $i$ wins comparison})}\big{)}\cdot 1_{(\text{if }b_{i}\geq p_{\epsilon})}-b_{i}.$ (5) $$

Per regulation guidelines, the safety criteria of an accepted model must at least be $\epsilon$.
Equation (5) dictates that values are only realized by each agent if their model has a bid larger than the required cost to reach $\epsilon$ safety, $1_{(\text{if }b_{i}\geq p_{\epsilon})}$.
Furthermore, agents only realize additional compensation value $v_{i}^{p}$ from the regulator if their safety level outperforms a randomly selected agent, $1_{(\text{if $i$ wins comparison})}$.
Any agent that bids $b_{i}=1$ will automatically win and realize both $v_{i}^{p}$ and $v_{j}^{w}$.
It is important to note that the cost that every agent incurs when building its model is sunk: if the model is not cleared for deployment, the cost $-b_{i}$ is still incurred.
We rewrite the agent utility in a piece-wise manner below,

$$ $u_{i}(b_{i};\bm{b}_{-i})=\begin{cases}-b_{i}\text{ if $b_{i}<p_{\epsilon}$},\\ v_{i}^{d}-b_{i}\text{ if $b_{i}\geq p_{\epsilon}$ and }b_{i}<b_{j}\text{ randomly sampled agent bid }b_{j},\\ v_{i}^{d}+v_{i}^{p}-b_{i}\text{ if $b_{i}\geq p_{\epsilon}$ and }b_{i}>b_{j} \text{ randomly sampled agent bid }b_{j}.\end{cases}$ (6) $$

By introducing additional compensation, $v_{i}^{p}$, and, crucially, conditioning it on whether an agent’s model is safer than that of another random agent, we seek to make it rational for agents to bid more than the price to obtain the minimum safety threshold (unlike Theorem 2).

Figure: Figure 2: Validation of Uniform Nash Bidding Equilibrium. Agent utility is maximized when agents follow the theoretically optimal bidding function shown in Equation (8). Across varying safety prices, $p_{\epsilon}=0.25$ (left), $0.5$ (middle), $0.75$ (right), agents attain less utility when they deviate from the optimal bid (red line) derived in Corollary 2.
Refer to caption: extracted/5896309/figures/uniform/regauc_equilibrium_deviation_pe0.25_lam0.5_uniform.png

Incentivizing Agents to Build Safer Models.
We establish a guarantee that agents participating in Sira maximize their utility with an optimal bid $b_{i}^{*}$ that is larger than “the price required to attain $\epsilon$ safety” (i.e., $b_{i}^{*}>p_{\epsilon}$) in Theorem 5.
Further, agents bid in proportion to the additional compensation value $v_{i}^{p}$ that the regulator awards for extra safe models.

Agents participating in Sira (6) will follow an optimal bidding strategy $\hat{b}_{i}^{*}$ of,

$$ $\hat{b}_{i}^{*}:=p_{\epsilon}+v_{i}^{p}F_{v}(v_{i}^{p})-\int_{0}^{v_{i}^{p}}F_ {v}(z)dz>p_{\epsilon},$ (7) $$

where $F_{v}(\cdot)$ denotes the cumulative density function of the random premium reward variable $v_{i}^{p}=V_{i}\lambda_{i}$.
Theorem 5 applies to any distribution for $V_{i}$ and $\lambda_{i}$ on $[0,1]$ and $[0,1/2]$, i.e., $V_{i}\sim\mathcal{D}_{V}(0,1)$ and $\lambda_{i}\sim\mathcal{D}_{\lambda}(0,1/2)$, respectively.
Determining specific optimal bids, utility, and model safety levels requires given distributions for $V_{i}$ and $\lambda_{i}$.
Analysis of all-pay auctions (Amann and Leininger, ; Bhaskar, ; Tardos, ), as well as many other types of auctions, often assume a Uniform distribution for valuations.
Therefore, our first analysis of Sira, below in Corollary 2, presumes Uniform distributions for $V_{i}$ and $\lambda_{i}$.

(Special Case 1) Uniform $V_{i}$ and $\lambda_{i}$: Optimal Agent Strategy.
Corollary 2 determines that a participating agent’s optimal strategy to maximize its utility is to submit a model with safety levels larger than $\epsilon$ when their values $V_{i}$ and $\lambda_{i}$ come from a Uniform distribution.

[Uniform Nash Bidding Equilibrium]
Under Assumption 3, for agents having total value $V_{i}$ and scaling factor $\lambda_{i}$ both stemming from a Uniform distribution, with $v_{i}^{d}=(1-\lambda_{i})V_{i},$ and $v_{i}^{p}=\lambda_{i}V_{i}$, their optimal bid and utility participating in Sira (6) are,

$$ $b_{i}^{*}:=\min\{\hat{b}_{i}^{*},1\},\quad\hat{b}_{i}^{*}=\begin{cases}p_{ \epsilon}+\frac{(v_{i}^{p})^{2}\ln(p_{\epsilon})}{p_{\epsilon}-1}&\text{ if }0 \leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}(\ln(2v_{i}^{p})-1/2)+p_{\epsilon}^{2}}{8(p _{\epsilon}-1)}&\text{ if }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2} ,\end{cases}$ (8) $$

$$ $u_{i}(b_{i}^{*};\bm{b}_{-i})=\begin{cases}\frac{2(v_{i}^{p})^{2}\ln(p_{ \epsilon})}{p_{\epsilon}-1}+v_{i}^{d}-b_{i}^{*}&\text{ if }0\leq v_{i}^{p}\leq \frac{p_{\epsilon}}{2},\\ \frac{2(v_{i}^{p})^{2}(\ln(2p_{\epsilon})-1)+p_{\epsilon}}{p_{\epsilon}-1}+v_{ i}^{d}-b_{i}^{*}&\text{ if }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2 }.\end{cases}$ (9) $$

Agents participating in Sira submit models with the following safety,

$$ $s_{i}^{*}:=\begin{cases}M^{-1}(b_{i}^{*})>\epsilon&\text{ if }u_{i}(b_{i}^{*}; \bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else}.\end{cases}$ (10) $$

(Special Case 2) Beta $V_{i}$ and Uniform $\lambda_{i}$: Optimal Agent Strategy.
In many instances, a more realistic distribution for $V_{i}$ is a Beta distribution with $\alpha,\beta=2$.
This choice of distribution looks more Gaussian, with the bulk of the probability density centered in the middle, and as such is realistic when agent values do not congregate amongst one another and outliers (near 0 or 1) are rare.
We analyze the performance of Sira under this more realistic scenario in Corollary 3.
Corollary 3 states that, under a Beta(2,2) distribution for $V_{i}$, agent $i$ maximizes its utility with an optimal bid $b_{i}^{*}$ larger than the price to attain $\epsilon$ safety, $b_{i}^{*}>p_{\epsilon}$, resulting in a model safer than the $\epsilon$ threshold.
Furthermore, Corollaries 2 and 3 surpass the baseline optimal bid $b_{i}^{*}=p_{\epsilon}$ for Reserve Thresholding (Theorem 2).

Figure: Figure 3: Validation of Beta Nash Bidding Equilibrium. Akin to the Uniform results, agent utility is maximized when agents follow the theoretically optimal bidding function shown in Equation (11). Across varying prices of safety values, $p_{\epsilon}=0.25$ (left), $0.5$ (middle), $0.75$ (right), agents attain less utility when they deviate from the optimal bid (red line) derived in Corollary 3.
Refer to caption: extracted/5896309/figures/beta/regauc_equilibrium_deviation_pe0.25_lam0.5_beta.png

[Beta Nash Bidding Equilibrium]
Under Assumption 3, let agents have total value $V_{i}$ and scaling factor $\lambda_{i}$ stem from Beta ($\alpha=\beta=2$) and Uniform distributions respectively, with $v_{i}^{d}=(1-\lambda_{i})V_{i}$ and $v_{i}^{p}=\lambda_{i}V_{i}$.
Denote the CDF of the Beta distribution on $[0,1]$ as $F_{\beta}(x)=3x^{2}-2x^{3}$.
The optimal bid and utility for agents participating in Sira (6) are,

$$ $b_{i}^{*}:=\min\{\hat{b}_{i}^{*},1\},\quad\hat{b}_{i}^{*}=\begin{cases}p_{ \epsilon}+\frac{3(v_{i}^{p})^{2}(p_{\epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta }(p_{\epsilon})}&\text{ if }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}\big{(}6(v_{i}^{p})^{2}-8v_{i}^{p}+3\big{)} +p_{\epsilon}^{3}(3p_{\epsilon}-4)}{8(1-F_{\beta}(p_{\epsilon}))}&\text{ if } \frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2},\end{cases}$ (11) $$

$$ $u(b_{i}^{*};\bm{b}_{-i})=\begin{cases}v_{i}^{d}+\frac{6(v_{i}^{p})^{2}(p_{ \epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}-b_{i}^{*}\;&\text{ for }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ v_{i}^{d}+\frac{v_{i}^{p}\big{(}8(v_{i}^{p})^{3}-12(v_{i}^{p})^{2}+6v_{i}^{p}+ p_{\epsilon}^{2}(2p_{\epsilon}-3)\big{)}}{1-F_{\beta}(p_{\epsilon})}-b_{i}^{*} \;&\text{ for }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq 1/2.\end{cases}$ (12) $$

Agents participating in Sira submit models with the following safety,

$$ $s_{i}^{*}=\begin{cases}M^{-1}(b_{i}^{*})>\epsilon&\text{ if }u_{i}(b_{i}^{*}; \bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else}.\end{cases}$ (13) $$

[Improved Model Safety]
As shown in Corollaries 2 and 3, participating agents will submit models that are safer than the regulator’s safety threshold, $s_{i}^{*}=M^{-1}\big{(}b^{*}_{i}\big{)}>\epsilon$.

[Improved Utility & Participation]
Through the introduction of regulator’s premium compensation, agent utility is improved, in Equations (9) and (12), versus Reserve Thresholding in Equation (3).
As a result, more agents break the zero-utility barrier of entry for participation, boosting both overall agent utility and participation rate.
Due to space constraints, we place the proofs of Theorems 2 and 5 as well as Corollaries 2 and 3 within Appendix B.
We note that since the premium compensation value $v_{i}^{p}$ is a product of two random variables, the PDF and CDF of $v_{i}^{p}$ becomes a piece-wise function (as shown within Appendix B).
As a result, the optimal bidding and subsequent utility also becomes piece-wise in both Corollaries 2 and 3.
We empirically verify the correctness of our computed PDF and CDFs within Appendix C.

## 6 Experiments

Our Section 5 results demonstrate that this safety-incentivized regulatory auction, Sira, creates incentives for any agents to submit safer models and to participate at higher rates than the baseline Reserve Thresholding mechanism in Section 4.
Below, we validate these theoretical results empirically.

Experimental Setup.
We simulate a regulatory setting with $n=100,000$ agents.
Each agent $i$ receives a random total value $V_{i}$ from either a Uniform (Corollary 2) or Beta(2,2) (Corollary 3) distribution.
Each agent also receives a scaling factor $\lambda_{i}$ that splits the total value into deployment $v_{i}^{d}=(1-\lambda_{i})V_{i}$ and premium compensation $v_{i}^{p}=\lambda_{i}V_{i}$ values.
Once the private values are provided, agents calculate their bid according to the optimal strategies in Theorem 2 and Corollaries 2 & 3.

Figure: Figure 4: Improved Safety and Participation with Uniform Values. When total value stems from a Uniform distribution $V_{i}\sim U(0,1)$, agents participate at a higher rate in Sira than Reserve Thresholding (left) and spend more to train safer models (right).
Refer to caption: extracted/5896309/figures/uniform/regauc_participation_rates_uniform_0.5.png

Lack of Baseline Regulatory Mechanisms.
To the best of our knowledge there are no other comparable mechanisms for safety regulation in AI.
As a result, we compare against the Reserve Threshold mechanism that we propose in Section 4.
While simple, the Reserve Threshold mechanism is a realistic baseline to compare against; there exist regulatory bodies, like the Environmental Protection Agency (EPA), that follow similar steps before clearing products (e.g., the EPA authorizes permits for discharging pollutants into water sources once water quality criteria are met).

Verifiable Nash Bidding Equilibria.
Our first experimental goal is to validate that the theoretical bidding functions found in Corollaries 2 and 3 constitute Nash Equilibria.
That is, agents receive worse utility if they deviate from this bidding strategy if all other agents abide by it.
To test this, we compare the optimal bid for a single agent versus $100,000$ others.
We vary the single agent’s optimal bid on a range up to $\pm 50\%$.
We note that comparisons only occur if the other agent’s bid is at least $p_{\epsilon}$, in order to accurately reflect how the auction mechanism in Algorithm 1 functions.

Figure: Figure 5: Improved Safety and Participation with Beta Values. When total value stems from a Beta distribution $V_{i}\sim\text{Beta}(\alpha=\beta=2)$, agents participate at a higher rate in Sira than Reserve Thresholding (left) and spend more to train safer models (right).
Refer to caption: extracted/5896309/figures/beta/regauc_participation_rates_beta_0.5.png

In Figures 2 and 3, we plot the average utility over all $100,000$ comparisons.
We find that both our Uniform and Beta optimal bidding functions maximize agent utility and thus constitute Nash Equilibria.
Interestingly, utility decays much quicker when reducing the bid, since agents are (i) less likely to win the premium reward and (ii) at risk of losing the value from deployment if the bid does not reach $p_{\epsilon}$.
At a certain point, utility increases linearly once the agent continuously fails to bid $p_{\epsilon}$.
The linear improvement stems from the agent saving the cost of its bid, $-b_{i}$, shown in Equation (6).

Improved Agent Participation and Bid Size.
We find that for both Uniform and Beta(2,2) distributions, shown in Figures 4 and 5, our mechanism (Sira) increases participation rates and average bids by upwards of 15% and 20% respectively.
At the endpoints of possible price thresholds, $p_{\epsilon}=0\text{ and }1$, we find that both mechanisms perform similarly.
The reason is that at a low safety threshold price $p_{\epsilon}\approx 0$, agents are highly likely to have a total value $V_{i}$ larger than a value close to zero.
The inverse is true for $p_{\epsilon}\approx 1$, where it is unlikely that agents will have a total value $V_{i}$ larger than a value close to 1.
Our mechanism shines when safety threshold prices are in the middle; the premium compensation offered by the regulator incentivizes agents to participate and bid more for the chance to win.

## 7 Conclusion and Future Work

As AI models grow, the risks associated with their misuse become increasingly significant, particularly given their often opaque, black-box nature.
Establishing robust algorithmic safeguards is crucial to protect users from unethical, unsafe, or illegally-deployed models.
In this paper, we present a regulatory framework designed to ensure that only models deemed safe by a regulator can be deployed for public use.
Our key contribution is the development of an auction-based regulatory mechanism that simultaneously (i) enforces safety standards and (ii) provably incentivizes agents to exceed minimum safety thresholds.
This approach encourages broader participation and the development of safer models compared to baseline regulatory methods.
Empirical results confirm that our mechanism increases agent participation by 15% and raises agent spending on safety by 20%, demonstrating its effectiveness to promote safer AI deployment.

Future Work. While this work addresses key challenges in regulating AI safety, several directions remain open for future exploration:

(1) Model Evaluation: Creating a realistic protocol for the regulator to evaluate submitted model safety levels is important to ensure agents do not skirt around safety requirements. While we leave this problem for future work, one possible solution is that agents can either provide the regulator API access to test its model or provide the model weights directly to the regulator. Truthfulness can be enforced via audits and the threat of legal action.

(2) Extension to Heterogeneous Settings: Extending our mechanism to heterogeneous scenarios, where evaluation data for agents and regulators differs, is a critical next step.
Real-world data distributions often vary across contexts, and understanding how these variations affect both model safety and agent strategies will create a more robust regulatory mechanism.
While explicit protocols or mathematical formulations are left as future work, we have a few ideas.
One idea could be establishing a data-sharing framework between agents and the regulator, where each participating agent must contribute part of (or all of) its data to the regulator for evaluation.
If data can be anonymized, then this would be a suitable solution.
Another idea could be that the regulator collects data on its own, and can compare its distribution of data versus each participating agents’ data distribution.
If distributions greatly differ, then the regulator could collect more data or resort to the previous data-sharing method.

## Acknowledgements

Bornstein, Che, and Huang are supported by DARPA Transfer from Imprecise and Abstract Models to Autonomous Technologies (TIAMAT) 80321, National Science Foundation NSF-IIS-2147276 FAI, DOD-ONR-Office of Naval Research under award number N00014-22-1-2335, DOD-AFOSR-Air Force Office of Scientific Research under award number FA9550-23-1-0048, DOD-DARPA-Defense Advanced Research Projects Agency Guaranteeing AI Robustness against Deception (GARD) HR00112020007, Adobe, Capital One and JP Morgan faculty fellowships.

## Appendix A Notation Table

**Table 1: Notating and Defining all Variables Listed Within Sira.**
| Definition | Notation |
| --- | --- |
| Regulator | $R$ |
| Number of Agents | $n$ |
| Safety Threshold | $\epsilon$ |
| Safety-to-Cost Function | $M$ |
| Price of Attaining Safety | $p_{\epsilon}$ |
| Agent $i$ Bid | $b_{i}$ |
| All Other Agents Bids | $\bm{b}_{-i}$ |
| Agent $i$ Utility | $u_{i}$ |
| Agent $i$ Model Safety | $s_{i}$ |
| Total Value for Agent $i$ | $V_{i}$ |
| Total Value Distribution | $\mathcal{D}_{V}$ |
| Agent $i$ Scaling Factor | $\lambda_{i}$ |
| Scaling Factor Distribution | $\mathcal{D}_{\lambda}$ |
| Deployment Value for Agent $i$ | $v_{i}^{d}$ |
| Premium Compensation Value for Agent $i$ | $v_{i}^{p}$ |
| Probability Density Function for Premium Compensation | $f_{v}$ |
| Cumulative Distribution Function for Premium Compensation | $F_{v}$ |

## Appendix B Theoretical Proofs

Below, we provide the full proofs of our Theorems and Corollaries presented within our work.

### B.1 Proof of Theorem 1

###### Theorem 2 (Restated) .

Under Assumption 3, agents participating in Reserve Thresholding (2) have an optimal bidding strategy and utility of,

$$ $b_{i}^{*}=p_{\epsilon},\quad u_{i}(b_{i}^{*};\bm{b}_{-i})=v_{i}^{d}-p_{ \epsilon},$ $$

and submit models with the following safety level,

$$ $s_{i}^{*}=\begin{cases}\epsilon&\text{ if }u_{i}(b_{i}^{*};\bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else.}\end{cases}$ $$

###### Proof.

From agent $i$’s utility within Reserve Thresholding, Equation (2), it is clear that $u_{i}(0)=0$.
We proceed to break the proof up into cases where agents have (1) a deployment value equal to or less than the price of safety $p_{\epsilon}$ and (2) a deployment value larger than $p_{\epsilon}$.

Case 1: $v_{i}^{d}\leq p_{\epsilon}$.
From Equation (2), if $v_{i}^{d}\leq p_{\epsilon}$ then an agent will never attain positive utility,

$$ $\displaystyle\max_{b_{i}\in(0,1]}v_{i}^{d}\cdot 1_{b_{i}\geq p_{\epsilon}}-b_{ i}\leq\max_{b_{i}\in(0,1]}p_{\epsilon}\cdot 1_{b_{i}\geq p_{\epsilon}}-b_{i}= \max_{b_{i}\in[p_{\epsilon},1]}p_{\epsilon}-b_{i}=p_{\epsilon}-p_{\epsilon}=0.$ (14) $\displaystyle\operatorname*{arg\,max}_{b_{i}\in(0,1]}u_{i}(b_{i})=p_{\epsilon}.$ (15) $$

For an agent with deployment value at most equal to $p_{\epsilon}$, the upper bound on attainable utility when it participates, i.e., $b_{i}\in(0,1]$, is zero (Equation (14)).
This maximum utility is attained when bidding $b_{i}=p_{\epsilon}$ (Equation (15)).
Thus, agents have nothing to gain by participating, as they already start at zero utility $u_{i}(0)=0$.
As a result, agents will not submit a model, $s_{i}^{*}=M(0)=0$.

Case 2: $v_{i}^{d}>p_{\epsilon}$. Similar steps to Case 1 above,

$$ $\displaystyle\max_{b_{i}\in(0,1]}v_{i}^{d}\cdot 1_{b_{i}\geq p_{\epsilon}}-b_{ i}>\max_{b_{i}\in(0,1]}p_{\epsilon}\cdot 1_{b_{i}\geq p_{\epsilon}}-b_{i}=\max _{b_{i}\in[p_{\epsilon},1]}p_{\epsilon}-b_{i}=p_{\epsilon}-p_{\epsilon}=0.$ (16) $\displaystyle b_{i}^{*}=\operatorname*{arg\,max}_{b_{i}\in(0,1]}u_{i}(b_{i})=p _{\epsilon}\longrightarrow u_{i}(b_{i}^{*})=v_{i}^{d}-p_{\epsilon}>0.$ (17) $$

An agent with deployment value larger than $p_{\epsilon}$ will have a maximal utility that is non-negative when it participates (Equation (16)).
Maximal utility is attained when bidding $b_{i}^{*}=p_{\epsilon}$ (Equation (17)).
Furthermore, at this optimal bid, the corresponding safety level is $s_{i}^{*}=M^{-1}(p_{\epsilon})=\epsilon$.

∎

###### Theorem 2 (Restated) .

###### Proof.

### B.2 Proof of Theorem 2

###### Theorem 5 (Restated) .

Agents participating in Sira (6) will follow an optimal bidding strategy $\hat{b}_{i}^{*}$ of,

$$ $\hat{b}_{i}^{*}:=p_{\epsilon}+v_{i}^{p}F_{v}(v_{i}^{p})-\int_{0}^{v_{i}^{p}}F_ {v}(z)dz>p_{\epsilon},$ $$

where $F_{v}(\cdot)$ denotes the cumulative density function of the random variable corresponding to the premium reward $v_{i}^{p}=V_{i}\lambda_{i}$.

###### Proof.

Before beginning our proof, we note that each agent $i$ cannot alter its own valuation $v_{i}^{p}$ for winning the all-pay auction.
Each valuation is private (unknown by other agents) and predetermined: total reward $V_{i}$ and partition factor $\lambda_{i}$ are randomly selected from a given distribution $\mathcal{D}$ on $[0,1]$ and $[0,1/2]$ respectively by “nature”.
We define the cumulative distribution function for the auction reward $v_{i}^{p}=V_{i}\lambda_{i}$ as $F_{v}(\cdot)$ and the probability distribution function as $f_{v}(\cdot)$.

From Equation (6), we find that an agent $i$ that does not participate (i.e., $b_{i}=0$) receives no utility,

$$ $u_{i}(0)=0.$ (18) $$

An agent receives negative utility if its bid does not reach the price of safety $p_{\epsilon}$,

$$ $\max_{b_{i}\in(0,p_{\epsilon})}u_{i}(b_{i})<0.$ (19) $$

Consequently, rational agents will either opt not to participate (notated as the set of agents $N$) or participate (notated as the set of agents $P$) and bid at least $p_{\epsilon}$.
We define these groups as,

$$ $\displaystyle N=\{i\in[n]\;|\;\max_{b_{i}\in[0,1]}u_{i}(b_{i})\leq 0\},$ (20) $\displaystyle P=\{i\in[n]\;|\;\max_{b_{i}\in[0,1]}u_{i}(b_{i})>0\}.$ (21) $$

From here, we only focus on agents $i\in P$ which participate (i.e., have utility to be gained by participating).
As a result from Equations (18) and (19), Equation (21) transforms into,

$$ $P=\{i\in[n]\;|\;\max_{b_{i}\in[p_{\epsilon},1]}u_{i}(b_{i})>0\}.$ (22) $$

The result of (22) is that participating agents bid at least $p_{\epsilon}$.
This is important, as every participating agent knows that all rival agents $j$ they will possibly be compared against have $b_{j}\in[p_{\epsilon},1]$.
Agents can dictate how much they bid, and we design our auction to ensure that agents bid in proportion to their valuation.

Following previous literature [Amann and Leininger, , Bhaskar, , Tardos, ], we desire a monotone increasing bidding function $b(\cdot):[0,1/2]\rightarrow[p_{\epsilon},1]$ that each agent follows.
We will prove that each agent $i$’s best strategy is to bid its own valuation $b(v_{i}^{p})$ irrespective of other agent bids (Nash Equilibrium).
Using a bidding function transforms agent utility,

$$ $\displaystyle u_{i}(b_{i})$ $\displaystyle=\big{(}v_{i}^{d}+v_{i}^{p}\cdot 1_{(\text{if $i$ wins auction})} \big{)}\cdot\underbrace{1_{(\text{if }b_{i}\geq p_{\epsilon})}}_{\text{ satisfied for agents }i\in P}-\;b_{i},$ $\displaystyle=\mathbb{P}\big{(}b(b_{i})>b(b_{j})\big{)}v_{i}^{p}-b(b_{i})+v_{i }^{d},\quad b_{j}\sim\text{randomly sampled agent bid}.$ (23) $$

Since $b(x)$ is monotone increasing up to 1, agents bidding $b=1$ automatically win, the utility function above can be simplified as,

$$ $\displaystyle u_{i}(b_{i})$ $\displaystyle=v_{i}^{p}\mathbb{P}\big{(}b_{i}>b_{j}\big{)}-b(b_{i})+v_{i}^{d}, \quad b_{j}\sim\text{randomly sampled agent bid},$ $\displaystyle=v_{i}^{p}F_{v}(b_{i})-b(b_{i})+v_{i}^{d}.$ (24) $$

Taking the derivative and setting it equal to zero yields,

$$ $\frac{d}{db_{i}}u_{i}(b_{i})=v_{i}^{p}f_{v}(b_{i})-b^{\prime}(b_{i})=0.$ (25) $$

As agents bid in proportion to their valuation, we solve the first-order equilibrium conditions at $b_{i}=v_{i}^{p}$,

$$ $b^{\prime}(v_{i}^{p})=v_{i}^{p}f_{v}(v_{i}^{p}).$ (26) $$

Integrating by parts, and knowing $\epsilon$ is the minimum bid ($b(0)=p_{\epsilon}$), reveals our optimal bidding function,

$$ $\displaystyle b(v_{i}^{p})-b(0)$ $\displaystyle=\int_{0}^{x}v_{i}^{p}f_{v}(v_{i}^{p})dv_{i}^{p},$ $\displaystyle b(v_{i}^{p})-p_{\epsilon}$ $\displaystyle=v_{i}^{p}F_{v}(v_{i}^{p})-\int_{0}^{v_{i}^{p}}F_{v}(z)dz,$ $\displaystyle\hat{b}_{i}^{*}=b(v_{i}^{p}):$ $\displaystyle=p_{\epsilon}+v_{i}^{p}F_{v}(v_{i}^{p})-\int_{0}^{v_{i}^{p}}F_{v} (z)dz.$ (27) $$

∎

###### Theorem 5 (Restated) .

###### Proof.

### B.3 Proof of Corollary 1

###### Corollary 2 (Restated) .

Under Assumption 3, for agents having total value $V_{i}$ and scaling factor $\lambda_{i}$ both stemming from a Uniform distribution, with $v_{i}^{d}=(1-\lambda_{i})V_{i},$ and $v_{i}^{p}=\lambda_{i}V_{i}$, their optimal bid and utility participating in Sira (6) are,

$$ $b_{i}^{*}:=\min\{\hat{b}_{i}^{*},1\},\quad\hat{b}_{i}^{*}=\begin{cases}p_{ \epsilon}+\frac{(v_{i}^{p})^{2}\ln(p_{\epsilon})}{p_{\epsilon}-1}&\text{ if }0 \leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}(\ln(2v_{i}^{p})-1/2)+p_{\epsilon}^{2}}{8(p _{\epsilon}-1)}&\text{ if }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2} ,\end{cases}$ $$

$$ $u_{i}(b_{i}^{*};\bm{b}_{-i})=\begin{cases}\frac{2(v_{i}^{p})^{2}\ln(p_{ \epsilon})}{p_{\epsilon}-1}+v_{i}^{d}-b_{i}^{*}&\text{ if }0\leq v_{i}^{p}\leq \frac{p_{\epsilon}}{2},\\ \frac{2(v_{i}^{p})^{2}(\ln(2p_{\epsilon})-1)+p_{\epsilon}}{p_{\epsilon}-1}+v_{ i}^{d}-b_{i}^{*}&\text{ if }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2 }.\end{cases}$ $$

Agents participating in Sira submit models with the following safety,

$$ $s_{i}^{*}:=\begin{cases}M^{-1}(b_{i}^{*})>\epsilon&\text{ if }u_{i}(b_{i}^{*}; \bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else}.\end{cases}$ $$

###### Proof.

Let $v_{i}^{p}:=V_{i}\lambda_{i}$, where $V_{i}\sim U[p_{\epsilon},1]$ and $\lambda_{i}\sim U[0,1/2]$.
The reason that $V_{i}$ is within the interval $[p_{\epsilon},1]$, is that all participating agents must have a value of at least $p_{\epsilon}$ or else they would not have rationale to bid.
The smallest value of $V_{i}$ such that this is possible is $p_{\epsilon}$, so it is the lower bound on this interval.
Our first goal is to find the PDF of $v_{i}^{p}$, $f_{v_{i}^{p}}(\cdot)$.

We begin solving for $f_{v_{i}^{p}}(\cdot)$ by using a change of variables.
For the product of two random variables $v=x_{1}\cdot x_{2}$, let $y_{1}=x_{1}\cdot x_{2}$ and $y_{2}=x_{2}$.
Thus, we find inversely that $x_{2}=y_{2}$ and $x_{1}=y_{1}/y_{2}$.
Since $x_{1}$ and $x_{2}$ are independent and both uniform, we find that,

$$ $f_{y_{1},y_{2}}(x_{1},x_{2})=(\frac{1}{1-p_{\epsilon}})(\frac{1}{1/2-0})=\frac {2}{1-p_{\epsilon}}.$ (28) $$

When using the change of variables this becomes,

$$ $f_{y_{1},y_{2}}(y_{1},y_{2})=f_{y_{1},y_{2}}(x_{1},x_{2})|J|=\frac{2}{(1-p_{ \epsilon})y_{2}},\quad|J|=\bigg{|}\begin{pmatrix}1/y_{2}&-y_{1}/y_{2}^{2}\\ 0&1\end{pmatrix}\bigg{|}=1/y_{2}$ (29) $$

Marginalizing out $y_{2}$ (a non-negative value) yields,

$$ $f_{y_{1}}(y_{1})=\int_{0}^{\infty}\frac{2}{(1-p_{\epsilon})y_{2}}dy_{2}.$ (30) $$

The bounds of integration depend upon the value of $y_{1}$.
The change of variable to the $(y_{1},y_{2})$ space, where $0\leq y_{1},y_{2}\leq 1/2$, results in a new region of possible variable values.
This region is a triangle bounded by the three vertices: $(0,0)$, $(p_{\epsilon}/2,1/2)$, and $(1/2,1/2)$.
Thus, the bounds of marginalization depend upon the value of $y_{1}$.
For $0\leq y_{1}\leq p_{\epsilon}/2$ we have,

$$ $f_{y_{1}}(y_{1})=\int_{y_{1}}^{y_{1}/p_{\epsilon}}\frac{2}{(1-p_{\epsilon})y_{ 2}}dy_{2}=\frac{2}{(1-p_{\epsilon})}[\ln(y_{2})\big{|}_{y_{1}}^{y_{1}/p_{ \epsilon}}]=\frac{2\ln(p_{\epsilon})}{(p_{\epsilon}-1)}.$ (31) $$

For $p_{\epsilon}\leq y_{1}\leq 1/2$ we have,

$$ $f_{y_{1}}(y_{1})=\int_{y_{1}}^{1/2}\frac{2}{(1-p_{\epsilon})y_{2}}dy_{2}=\frac {2}{(1-p_{\epsilon})}[\ln(y_{2})\big{|}_{y_{1}}^{1/2}]=\frac{2\ln(2y_{1})}{(p_ {\epsilon}-1)}.$ (32) $$

Thus, as a piecewise function the PDF is formally,

$$ $f_{y_{1}}(y_{1})=\begin{cases}\frac{2\ln(p_{\epsilon})}{(p_{\epsilon}-1)}\quad &\text{ for }0\leq y_{1}\leq\frac{p_{\epsilon}}{2},\\ \frac{2\ln(2y_{1})}{(p_{\epsilon}-1)}\quad&\text{ for }\frac{p_{\epsilon}}{2} \leq y_{1}\leq 1/2.\end{cases}$ (33) $$

Now, the CDF is determined through integration,

$$ $F_{y_{1}}(y_{1})=\int_{0}^{y_{1}}f_{y_{1}}(y_{1})dy_{1}=\begin{cases}\frac{2y_ {1}\ln(p_{\epsilon})}{(p_{\epsilon}-1)}\quad&\text{ for }0\leq y_{1}\leq\frac{ p_{\epsilon}}{2},\\ \frac{2y_{1}(\ln(2y_{1})-1)+p_{\epsilon}}{(p_{\epsilon}-1)}\quad&\text{ for } \frac{p_{\epsilon}}{2}\leq y_{1}\leq 1/2.\end{cases}$ (34) $$

We can integrate the CDF to get,

$$ $\int_{0}^{y_{1}}F_{y_{1}}(y_{1})=\begin{cases}\frac{y_{1}^{2}\ln(p_{\epsilon}) }{(p_{\epsilon}-1)}\quad&\text{ for }0\leq y_{1}\leq\frac{p_{\epsilon}}{2},\\ \frac{4y_{1}^{2}(2\ln(2y_{1})-3)+8y_{1}p_{\epsilon}-p_{\epsilon}^{2}}{8(p_{ \epsilon}-1)}\quad&\text{ for }\frac{p_{\epsilon}}{2}\leq y_{1}\leq 1/2.\end{cases}$ (35) $$

Plugging all of this back into Equation (7) yields,

$$ $\displaystyle\hat{b}^{*}_{i}$ $\displaystyle=\begin{cases}p_{\epsilon}+v_{i}^{p}\frac{2v_{i}^{p}\ln(p_{ \epsilon})}{p_{\epsilon}-1}-\frac{(v_{i}^{p})^{2}\ln(p_{\epsilon})}{p_{ \epsilon}-1},\\ p_{\epsilon}+v_{i}^{p}\frac{2v_{i}^{p}(\ln(2v_{i}^{p})-1)+p_{\epsilon}}{(p_{ \epsilon}-1)}-\frac{4(v_{i}^{p})^{2}(2\ln(2v_{i}^{p})-3)+8v_{i}^{p}p_{\epsilon }-p_{\epsilon}^{2}}{8(p_{\epsilon}-1)},\end{cases}$ $\displaystyle=\begin{cases}p_{\epsilon}+\frac{(v_{i}^{p})^{2}\ln(p_{\epsilon}) }{p_{\epsilon}-1}&\text{ if }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}(\ln(2v_{i}^{p})-1/2)+p_{\epsilon}^{2}}{8(p _{\epsilon}-1)}&\text{ if }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2} .\end{cases}$ (36) $$

Since $b_{i}$ cannot be larger than 1, we cap the bidding function at one via,

$$ $\displaystyle b_{i}^{*}:=\min\{\hat{b}^{*}_{i},1\}.$ (37) $$

The utility gained by agent $i$ for using such a bidding function is,

$$ $u(b_{i}^{*})=\begin{cases}v_{i}^{d}-b_{i}^{*}+\frac{2(v_{i}^{p})^{2}\ln(p_{ \epsilon})}{p_{\epsilon}-1}\;&\text{ for }0\leq v_{i}^{p}\leq\frac{p_{\epsilon }}{2},\\ v_{i}^{d}-b_{i}^{*}+\frac{2(v_{i}^{p})^{2}(\ln(2v_{i}^{p})-1)+p_{\epsilon}}{(p _{\epsilon}-1)}\;&\text{ for }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq 1/2. \end{cases}$ (38) $$

When this utility is larger than 0, the agent will participate otherwise the agent will not submit a model to the regulator.
Finally, we can find the optimal safety level by using Assumption 3,

$$ $\displaystyle s_{i}^{*}:=M^{-1}\big{(}b_{i}^{*}\big{)}.$ (39) $$

∎

###### Corollary 2 (Restated) .

###### Proof.

### B.4 Proof of Corollary 2

###### Corollary 3 (Restated) .

Under Assumption 3, let agents have total value $V_{i}$ and scaling factor $\lambda_{i}$ stem from Beta ($\alpha=\beta=2$) and Uniform distributions respectively, with $v_{i}^{d}=(1-\lambda_{i})V_{i}$ and $v_{i}^{p}=\lambda_{i}V_{i}$.
Denote the CDF of the Beta distribution on $[0,1]$ as $F_{\beta}(x)=3x^{2}-2x^{3}$.
The optimal bid and utility for agents participating in Sira (6) are,

$$ $b_{i}^{*}:=\min\{\hat{b}_{i}^{*},1\},\quad\hat{b}_{i}^{*}=\begin{cases}p_{ \epsilon}+\frac{3(v_{i}^{p})^{2}(p_{\epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta }(p_{\epsilon})}&\text{ if }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}\big{(}6(v_{i}^{p})^{2}-8v_{i}^{p}+3\big{)} +p_{\epsilon}^{3}(3p_{\epsilon}-4)}{8(1-F_{\beta}(p_{\epsilon}))}&\text{ if } \frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2},\end{cases}$ $$

$$ $u(b_{i}^{*};\bm{b}_{-i})=\begin{cases}v_{i}^{d}+\frac{6(v_{i}^{p})^{2}(p_{ \epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}-b_{i}^{*}\;&\text{ for }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ v_{i}^{d}+\frac{v_{i}^{p}\big{(}8(v_{i}^{p})^{3}-12(v_{i}^{p})^{2}+6v_{i}^{p}+ p_{\epsilon}^{2}(2p_{\epsilon}-3)\big{)}}{1-F_{\beta}(p_{\epsilon})}-b_{i}^{*} \;&\text{ for }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq 1/2.\end{cases}$ $$

Agents participating in Sira submit models with the following safety,

$$ $s_{i}^{*}=\begin{cases}M^{-1}(b_{i}^{*})>\epsilon&\text{ if }u_{i}(b_{i}^{*}; \bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else}.\end{cases}$ $$

###### Proof.

Similar to Corollary 2, we begin solving for $f_{v_{i}^{p}}(\cdot)$ using a change of variables.
For the product of two random variables $v=x_{1}\cdot x_{2}$, let $y_{1}=x_{1}\cdot x_{2}$ and $y_{2}=x_{2}$.
Inversely, $x_{2}=y_{2}$ and $x_{1}=y_{1}/y_{2}$.
While $x_{1}$ and $x_{2}$ are independent, $x_{1}$ comes from a Beta distribution and $x_{2}$ from a Uniform one.
The PDF and CDF of a Beta distribution, with $\alpha=\beta=2$, on $[0,1]$ are defined as,

$$ $\displaystyle f_{\beta}(x):=6x(1-x),$ (40) $\displaystyle F_{\beta}(x):=3x^{2}-2x^{3}.$ (41) $$

Now, the PDF over $y_{1},y_{2}$ is defined as,

$$ $f_{y_{1},y_{2}}(x_{1},x_{2})=(\frac{6x_{1}(1-x_{1})}{1-F_{\beta}(p_{\epsilon}) })(\frac{1}{1/2-0})=\frac{12x_{1}(1-x_{1})}{1-F_{\beta}(p_{\epsilon})}.$ (42) $$

When using the change of variables this becomes,

$$ $f_{y_{1},y_{2}}(y_{1},y_{2})=f_{y_{1},y_{2}}(x_{1},x_{2})|J|=\frac{12y_{1}(1- \frac{y_{1}}{y_{2}})}{(1-F_{\beta}(p_{\epsilon}))y_{2}^{2}},\quad|J|=\bigg{|} \begin{pmatrix}1/y_{2}&-y_{1}/y_{2}^{2}\\ 0&1\end{pmatrix}\bigg{|}=1/y_{2}$ (43) $$

Marginalizing out $y_{2}$ (a non-negative value) yields,

$$ $f_{y_{1}}(y_{1})=\frac{12y_{1}}{1-F_{\beta}(p_{\epsilon})}\int_{0}^{\infty} \frac{1}{y_{2}^{2}}-\frac{y_{1}}{y_{2}^{3}}dy_{2}.$ (44) $$

The bounds of integration depend upon the value of $y_{1}$.
The change of variable to the $(y_{1},y_{2})$ space, where $0\leq y_{1},y_{2}\leq 1/2$, results in a new region of possible variable values.
This region is a triangle bounded by the three vertices: $(0,0)$, $(p_{\epsilon}/2,1/2)$, and $(1/2,1/2)$.
Thus, the bounds of marginalization depend upon the value of $y_{1}$.
For $0\leq y_{1}\leq p_{\epsilon}/2$ we have,

$$ $\displaystyle f_{y_{1}}(y_{1})$ $\displaystyle=\frac{12y_{1}}{1-F_{\beta}(p_{\epsilon})}\int_{y_{1}}^{y_{1}/p_{ \epsilon}}\frac{1}{y_{2}^{2}}-\frac{y_{1}}{y_{2}^{3}}dy_{2}=\frac{12y_{1}}{1-F _{\beta}(p_{\epsilon})}[-\frac{1}{y_{2}}+\frac{y_{1}}{2y_{2}^{2}}\big{|}_{y_{1 }}^{y_{1}/p_{\epsilon}}]$ $\displaystyle=\frac{12y_{1}}{1-F_{\beta}(p_{\epsilon})}[-\frac{p_{\epsilon}}{y _{1}}+\frac{p_{\epsilon}^{2}}{2y_{1}}+\frac{1}{y_{1}}-\frac{1}{2y_{1}}]=\frac{ 6(p_{\epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}.$ (45) $$

For $p_{\epsilon}\leq y_{1}\leq 1/2$ we have,

$$ $\displaystyle f_{y_{1}}(y_{1})$ $\displaystyle=\frac{12y_{1}}{1-F_{\beta}(p_{\epsilon})}\int_{y_{1}}^{1/2}\frac {1}{y_{2}^{2}}-\frac{y_{1}}{y_{2}^{3}}dy_{2}=\frac{12y_{1}}{1-F_{\beta}(p_{ \epsilon})}[-\frac{1}{y_{2}}+\frac{y_{1}}{2y_{2}^{2}}\big{|}_{y_{1}}^{1/2}]$ $\displaystyle=\frac{12y_{1}}{1-F_{\beta}(p_{\epsilon})}[-2+2y_{1}+\frac{1}{y_{ 1}}-\frac{1}{2y_{1}}]=\frac{6(4y_{1}^{2}-4y_{1}+1)}{1-F_{\beta}(p_{\epsilon})}.$ (46) $$

Thus, as a piecewise function the PDF is formally,

$$ $f_{y_{1}}(y_{1})=\begin{cases}\frac{6(p_{\epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{ \beta}(p_{\epsilon})}\quad&\text{ for }0\leq y_{1}\leq\frac{p_{\epsilon}}{2}, \\ \frac{6(4y_{1}^{2}-4y_{1}+1)}{1-F_{\beta}(p_{\epsilon})}\quad&\text{ for } \frac{p_{\epsilon}}{2}\leq y_{1}\leq 1/2.\end{cases}$ (47) $$

Now, the CDF is determined through integration,

$$ $F_{y_{1}}(y_{1})=\int_{0}^{y_{1}}f_{y_{1}}(y_{1})dy_{1}=\begin{cases}\frac{6y_ {1}(p_{\epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}\quad&\text{ for }0\leq y_{1}\leq\frac{p_{\epsilon}}{2},\\ \frac{2y_{1}(4y_{1}^{2}-6y_{1}+3)+p_{\epsilon}^{2}(2p_{\epsilon}-3)}{1-F_{ \beta}(p_{\epsilon})}\quad&\text{ for }\frac{p_{\epsilon}}{2}\leq y_{1}\leq 1/ 2.\end{cases}$ (48) $$

We can integrate the CDF to get,

$$ $\int_{0}^{y_{1}}F_{y_{1}}(y_{1})=\begin{cases}\frac{3y_{1}^{2}(p_{\epsilon}^{2 }-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}\quad&\text{ for }0\leq y_{1} \leq\frac{p_{\epsilon}}{2},\\ \frac{8y_{1}\big{(}2y_{1}^{3}-4y_{1}^{2}+3y_{1}+p_{\epsilon}^{2}(2p_{\epsilon} -3)\big{)}+p_{\epsilon}^{3}(4-3p_{\epsilon})}{8(1-F_{\beta}(p_{\epsilon}))} \quad&\text{ for }\frac{p_{\epsilon}}{2}\leq y_{1}\leq 1/2.\end{cases}$ (49) $$

Plugging all of this back into Equation (7) yields,

$$ $\displaystyle\hat{b}^{*}_{i}$ $\displaystyle=\begin{cases}p_{\epsilon}+v_{i}^{p}\frac{6v_{i}^{p}(p_{\epsilon} ^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}-\frac{3(v_{i}^{p})^{2}(p_{ \epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})},\\ p_{\epsilon}+v_{i}^{p}\frac{2v_{i}^{p}(4(v_{i}^{p})^{2}-6v_{i}^{p}+3)+p_{ \epsilon}^{2}(2p_{\epsilon}-3)}{1-F_{\beta}(p_{\epsilon})}-\frac{8v_{i}^{p} \big{(}2(v_{i}^{p})^{3}-4(v_{i}^{p})^{2}+3v_{i}^{p}+p_{\epsilon}^{2}(2p_{ \epsilon}-3)\big{)}+p_{\epsilon}^{3}(4-3p_{\epsilon})}{8(1-F_{\beta}(p_{ \epsilon}))},\end{cases}$ $\displaystyle=\begin{cases}p_{\epsilon}+\frac{3(v_{i}^{p})^{2}(p_{\epsilon}^{2 }-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}&\text{ if }0\leq v_{i}^{p}\leq \frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}\big{(}6(v_{i}^{p})^{2}-8v_{i}^{p}+3\big{)} +p_{\epsilon}^{3}(3p_{\epsilon}-4)}{8(1-F_{\beta}(p_{\epsilon}))}&\text{ if } \frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2}.\end{cases}$ (50) $$

Since $b_{i}$ cannot be larger than 1, we cap the bidding function at one via,

$$ $\displaystyle b_{i}^{*}:=\min\{\hat{b}^{*}_{i},1\}.$ (51) $$

The utility gained by agent $i$ for using such a bidding function is,

$$ $u(b_{i}^{*})=\begin{cases}v_{i}^{d}-b_{i}^{*}+\frac{6(v_{i}^{p})^{2}(p_{ \epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}\;&\text{ for }0\leq v _{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ v_{i}^{d}-b_{i}^{*}+\frac{v_{i}^{p}\big{(}8(v_{i}^{p})^{3}-12(v_{i}^{p})^{2}+6 v_{i}^{p}+p_{\epsilon}^{2}(2p_{\epsilon}-3)\big{)}}{1-F_{\beta}(p_{\epsilon})} \;&\text{ for }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq 1/2.\end{cases}$ (52) $$

When this utility is larger than 0, the agent will participate otherwise the agent will not submit a model to the regulator.
Finally, we can find the optimal safety level by using Assumption 3,

$$ $\displaystyle s_{i}^{*}:=M^{-1}\big{(}b_{i}^{*}\big{)}.$ (53) $$

∎

###### Corollary 3 (Restated) .

###### Proof.

## Appendix C Additional Experiments

Within this section, we verify empirically that our computed PDF and CDFs in Corollaries 2 and 3 are correct.
To accomplish this, we randomly sample and compute the product of $V_{i}$ and $\lambda_{i}$ fifty million times.
We then plot the PDF and CDF of the resultant products and compare it with our theoretical PDF and CDF.
The theoretical PDF and CDF for Corollary 2 are defined in Equations (33) and (34), while those for Corollary 3 are found in Equations (47) and (48).
The results of these simulations, which validate our computed PDFs and CDFs, are shown in Figures 6 and 7.
To ensure correctness, we perform testing on different values of $p_{\epsilon}$.
As expected, our theory lines up exactly with our empirical simulations for both Corollaries as well as across varying $p_{\epsilon}$.

Figure: Figure 6: Numerical validation of our derivations for $f_{v}(v_{i}^{p})$ and $F_{v}(v_{i}^{p})$, where $v_{i}^{p}:=V_{i}\lambda_{i}$, for $V_{i}$ and $\lambda_{i}$ coming from Uniform distributions (Corollary 2). The price of attaining $\epsilon$ is set as $p_{\epsilon}=1/4$ (top row) and $p_{\epsilon}=1/2$ (bottom row).
Refer to caption: extracted/5896309/figures/distributions/regauc_pdf_uniform_pe_0.25.png

Figure: Figure 7: Numerical validation of our derivations for $f_{v}(v_{i}^{P})$ and $F_{v}(v_{i}^{P})$, where $v_{i}^{p}:=V_{i}\lambda_{i}$, for $V_{i}$ coming from a Beta distribution and $\lambda_{i}$ from a Uniform distributions (Corollary 3). The price of attaining $\epsilon$ is set as $p_{\epsilon}=1/4$ (top row) and $p_{\epsilon}=1/2$ (bottom row).
Refer to caption: extracted/5896309/figures/distributions/regauc_pdf_beta_pe_0.25.png

## Appendix D Repeating Sira Auctions

The current auction structure (Algorithm 1) expects agents to submit a single model trained solely for the upcoming auction.
There is no expectation that the model will be reused for a future auction, or indication that the model has been submitted to a previous auction.
Looking towards the future, we would like to design Sira to fit a repeatable auction structure, in which approved or rejected models may be resubmitted in subsequent auctions.

Repeated Agent Utility.
Previously, in Algorithm 1,
agents start the regulatory process with zero cost and value (i.e., they are building their models from scratch).
In repeating Sira auctions, agent cost and value are accumulated across all previous auction submissions.
For example, if an agent trains its already-accepted model further to attain a higher safety level $s_{i}$, its total accumulated training cost is $M(s_{i})$.
This agent’s total value becomes the value its model gained from previous auction submissions plus any value gained from the current auction.

By allowing repeated Sira auctions, an agent is able to repeatedly submit its model for regulatory review. We note that repeated submissions decrease the value of model deployment; once an agent earns the reward for deploying their model, subsequent deployments of the same model with improved safety levels can be realistically expected to earn less value than the initial deployment.
We characterize this loss in value for repeated submissions with an indicator function in the utility function that only allows deployment value to be obtained once, on initial acceptance of a model.
While we allow agents to win premium rewards across multiple auctions, we note that a regulator can curb this by either limiting the number of auction submissions per agent or the number of auctions held per year.
We now define the repeated Sira auction utility of agent $i$, who has participated in $a-1$ previous auctions, as:

$$ $u_{i,a}(b_{i})=\left(\sum_{n=1}^{a}\nu_{i}^{n}\right)-b_{i},$ (54) $$

where $\nu_{i}^{n}$, the value gained at the $n^{th}$ auction model $i$ was submitted to, is formulated as:

$$ $\nu_{i}^{n}=\begin{cases}v_{i}^{d,n}\ \cdot 1_{\text{(if $\nu^{n-1}_{i}=0$)}}& \text{if $b_{j}^{n}\geq p_{\epsilon}^{n}$ and }b_{i}^{n}<b_{j}^{n}\text{ randomly sampled bid }b_{j}^{n},\\ v_{i}^{d,n}\cdot 1_{\text{(if $\nu^{n-1}_{i}=0$)}}+v_{i}^{p,n}&\text{if $b_{i} ^{n}\geq p_{\epsilon}^{n}$ and }b_{i}^{n}>b_{j}^{n}\text{ randomly sampled bid }b_{j}^{n},\\ 0&\text{if n $\leq$ 0}.\end{cases}$ (55) $$

The repeated Sira auction setup creates a unique property for models in training.
If an agent intends to obtain a high safety level, but an auction takes place mid-training, the agent is actually incentivized to submit their model early if they have a chance at winning the premium reward.
Though the model may have a lower likelihood of earning the reward, there is no consequence for models failing to attain the premium reward.
Gaining value is strictly beneficial to agents, and accumulated value helps offset the costs of training a model.
This property only exists for the premium reward; the deployment reward can only be obtained once, thus there is no incentive to submit early to earn it.

Repeated Optimal Bidding Function.
Using the same assumptions for single-auction Sira, namely Assumptions 3 and 3 along with private values, we can derive the bidding function for a rational agent under a repeated Sira auction setting.
We follow an equivalent setup to Lemma 1 with regards to the valuation of rewards, giving us the cumulative distribution function for $v^{p}_{i}=V_{i}\lambda_{i}$ as $F_{v}(\cdot)$ and the probability distribution function as $f_{v}(\cdot)$.

From our definition of utility $u_{i,a}(b_{i})$, we find that an agent $i$ that does not participate (i.e., submitting $b_{i}=0$) receives utility equal to $\nu^{a}_{i}$.
However, since $b_{i}=0$ will never be larger than $p_{\epsilon}$ (by definition), it must be true that $\nu^{a}_{i}=0$ as well, since the model will never meet the required safety threshold.
Therefore, a non-participating agent will always receive non-negative utility.

$$ $u_{i,a}(0)=0.$ (56) $$

Following closely to the proof of Theorem 5 in Appendix B, we find that participating agents $i\in P$ (with $P$ defined in the previous proof) will now have a utility of,

$$ $\displaystyle u_{i,a}(b_{i})$ $\displaystyle=\nu^{a}_{i}+v^{d}_{i}\cdot 1_{\text{($\nu^{a}_{i}$ = 0)}}+v_{i}^ {p}\mathbb{P}\big{(}b_{i}>b_{j}\big{)}-b(b_{i}),\quad b_{j}\sim\text{randomly sampled agent bid},$ $\displaystyle=\nu^{a}_{i}+v^{d}_{i}\cdot 1_{\text{($\nu^{a}_{i}$ = 0)}}+v_{i}^ {p}F_{v}(b_{i})-b(b_{i}).$ (57) $$

Taking the derivative and setting it equal to zero yields,

$$ $\frac{d}{db_{i}}u_{i,a}(b_{i})=v_{i}^{p}f_{v}(b_{i})-b^{\prime}(b_{i})=0.$ (58) $$

As agents bid in proportion to their valuation, we solve the first-order conditions at $b_{i}=v_{i}^{p}$,

$$ $b^{\prime}(v_{i}^{p})=v_{i}^{p}f_{v}(v_{i}^{p}).$ (59) $$

Note, at this point in the proof the bidding function calculation is now equivalent to the calculations found in Lemma 1. We can thus follow the same steps to reveal our optimal bidding function,

$$ $\displaystyle b(v_{i}^{p}):$ $\displaystyle=p_{\epsilon}+v_{i}^{p}F_{v}(v_{i}^{p})-\int_{0}^{v_{i}^{p}}F_{v} (z)dz,$ (60) $$

which is equivalent to the optimal bidding function derived in Lemma 1.

As the optimal bidding function is equivalent, calculations for the Nash Bidding Equilibrium are also equivalent to those found in Corollary 2 and Corollary 3. The optimal bid and utility participating in Sira (6) under the assumptions of Corollary 2 will thus be,

$$ $b_{i}^{*}:=\min\{\hat{b}_{i}^{*},1\},\quad\hat{b}_{i}^{*}=\begin{cases}p_{ \epsilon}+\frac{(v_{i}^{p})^{2}\ln(p_{\epsilon})}{p_{\epsilon}-1}&\text{ if }0 \leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}(\ln(2v_{i}^{p})-1/2)+p_{\epsilon}^{2}}{8(p _{\epsilon}-1)}&\text{ if }\frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2} ,\end{cases}$ $$

$$ $u_{i,a}(b_{i}^{*};\bm{b}_{-i})=\begin{cases}\nu^{a}_{i}+v^{d}_{i}\cdot 1_{ \text{($\nu^{a}_{i}$ = 0)}}+\frac{2(v_{i}^{p})^{2}\ln(p_{\epsilon})}{p_{ \epsilon}-1}-b_{i}^{*}&\text{ if }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ \nu^{a}_{i}+v^{d}_{i}\cdot 1_{\text{($\nu^{a}_{i}$ = 0)}}+\frac{2(v_{i}^{p})^{ 2}(\ln(2p_{\epsilon})-1)+p_{\epsilon}}{p_{\epsilon}-1}-b_{i}^{*}&\text{ if } \frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2}.\end{cases}$ $$

Agents participating in Sira under Corollary 2 submit models with the following safety,

$$ $s_{i}^{*}:=\begin{cases}M^{-1}(b_{i}^{*})>\epsilon&\text{ if }u_{i}(b_{i}^{*}; \bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else}.\end{cases}$ $$

The optimal bid and utility participating in Sira (6) under the assumptions of Corollary 3 will be,

$$ $b_{i}^{*}:=\min\{\hat{b}_{i}^{*},1\},\quad\hat{b}_{i}^{*}=\begin{cases}p_{ \epsilon}+\frac{3(v_{i}^{p})^{2}(p_{\epsilon}^{2}-2p_{\epsilon}+1)}{1-F_{\beta }(p_{\epsilon})}&\text{ if }0\leq v_{i}^{p}\leq\frac{p_{\epsilon}}{2},\\ p_{\epsilon}+\frac{8(v_{i}^{p})^{2}\big{(}6(v_{i}^{p})^{2}-8v_{i}^{p}+3\big{)} +p_{\epsilon}^{3}(3p_{\epsilon}-4)}{8(1-F_{\beta}(p_{\epsilon}))}&\text{ if } \frac{p_{\epsilon}}{2}\leq v_{i}^{p}\leq\frac{1}{2},\end{cases}$ $$

$$ $u_{i,a}(b_{i}^{*};\bm{b}_{-i})=\begin{cases}\nu^{a}_{i}+v^{d}_{i}\cdot 1_{ \text{($\nu^{a}_{i}$ = 0)}}+\frac{6(v_{i}^{p})^{2}(p_{\epsilon}^{2}-2p_{ \epsilon}+1)}{1-F_{\beta}(p_{\epsilon})}-b_{i}^{*}\;&\text{ for }0\leq v_{i}^{ p}\leq\frac{p_{\epsilon}}{2},\\ \nu^{a}_{i}+v^{d}_{i}\cdot 1_{\text{($\nu^{a}_{i}$ = 0)}}+\frac{v_{i}^{p}\big{ (}8(v_{i}^{p})^{3}-12(v_{i}^{p})^{2}+6v_{i}^{p}+p_{\epsilon}^{2}(2p_{\epsilon} -3)\big{)}}{1-F_{\beta}(p_{\epsilon})}-b_{i}^{*}\;&\text{ for }\frac{p_{ \epsilon}}{2}\leq v_{i}^{p}\leq 1/2.\end{cases}$ $$

Agents participating in Sira under Corollay 3 submit models with the following safety,

$$ $s_{i}^{*}=\begin{cases}M^{-1}(b_{i}^{*})>\epsilon&\text{ if }u_{i}(b_{i}^{*}; \bm{b}_{-i})>0,\\ 0\text{ (no model submission) }&\text{ else}.\end{cases}$ $$

## Peer Reviews

confidence: 3
contribution: 3
initial_rating: 6
presentation: 3
questions: * What is the rationale of choosing the Beta and Uniform distribution (beyond what is described in line 323-324). Are there any related works that you could cite to support this choice of distributions?

* What is the scaling of complexity and cost (such as evaluation and communication) as the number of the agents increase? Are there any risks of agents colluding to achieve a suboptimal safety level?
soundness: 3
strengths: The paper is well-written and well-supported by both theoretical proofs and empirical results. It addresses the important area of AI regulatory via a multi-agent economic, game-theory type framework. There are a few assumptions to simplify the mechanism but they appear to be acceptable/realistic such as i) the regulator and the participating agents use data from the same distribution to evaluate and submit the safety level, and ii) safer models cost more to develop. These assumptions perhaps need more clarification/grounding or adjustment to become more applicable and feasible in practice. A safer model can tend to cost more to develop but perhaps cost and safety might not always be strictly increasing. The paper has help enhance the current AI regulatory work with a well-formulated framework and has a potential to have some significance in this domain.
summary: This paper proposes a novel framework of auction-based regulatory mechanism as an asymmetric and incomplete all-pay auction. The mechanism is described mathematically and also shows good empirical results of enhancing safety and participation rates. The framework consists of a regulator and multiple participating agents. Overall, this is an interesting framework with good potentials to explore and create safer and more robust AI regulatory.
weaknesses: While the paper is well-supported in the mathematical formulation and proofs, it perhaps could have provided more evidence on the experiments and empirical data. More description of how this framework can be applied in AI regulatory or in practice might help ground it further and make it relevant to a wider group of audiences.

---

confidence: 3
contribution: 2
initial_rating: 3
presentation: 3
questions: **Q1.** Is there a reasonable mechanism for estimating the market value ($v_i^d$) of a model before it is submitted to the regulator or even before the training phase begins?

**Q2.** Considering that SIRA’s performance deteriorates at high safety thresholds, would a simple increase in the threshold serve as a better incentive in such cases, as it may more directly encourage safer model development?

**Q3.** The authors mention that safety evaluations rely on IID assumptions for both agent and regulator data. How would the proposed mechanism adapt to non-IID settings, where the agent's training data might be maliciously poisoned, or where the regulator's evaluation data is collected through other means?

**Q4.** Is the random comparison fair for all competitive agents? For example, if we have utility values such that $u_A > u_B > u_C > u_D$, and A and B are grouped together while C and D are grouped together, then B and D cannot receive the policy bonus. However, since $u_B > u_C$, this situation could be considered unfair to B.
soundness: 2
strengths: **Originality.** The approach presents a unique use of all-pay auction mechanisms in AI regulation, where each agent's utility is linked to model safety levels (training cost), model value (market returns), and premium (policy compensation), creating an incentive for improved safety compliance.

**Quality.** The paper theoretically derives Nash Equilibria to back the proposed incentive structure, demonstrating that agents' rational behavior leads them to exceed the minimum safety threshold. The experimental results align with the theoretical model.

**Clarity.** This paper is well-written and easy to follow. The authors provide clear descriptions of the auction-based model and detailed steps in the algorithmic design of SIRA, supported by both theoretical and empirical validation.

**Significance.** This paper tries to tackle an essential issue in AI regulation by encouraging safer model deployment.
summary: This paper presents a new AI regulatory framework known as the Safety-Incentivized Regulatory Auction (SIRA), designed as an all-pay auction. SIRA aims to motivate model-building agents to prioritize safety beyond a minimum threshold by formulating the AI regulatory process as an asymmetric all-pay auction with incomplete information. In this framework, agents submit their models to a regulator, and those that meet or exceed a specified safety level become eligible for deployment and may also receive additional rewards based on the randomized pair comparison result. The authors theoretically prove that under some assumptions and when all agents adopt a certain strategy, the system reaches a Nash Equilibrium. Empirical results indicate that when safety threshold prices are in the middle (0.2~0.8), SIRA enhances safety compliance and agent participation by 20% and 15%, respectively compared with the basic regulatory method.
weaknesses: **Rationality of the auction framework.** Considering the regulation process as an all-pay auction is not reasonable, at least in my opinion. Intuitively, safety-driven regulation establishes a minimum cost for the model-building agent. Every model-building agent must incur this cost, regardless of whether it can successfully meet the regulatory requirements. This represents an unavoidable exploration process within the model space. Even if we assume that all competitive agents know how to train their models to meet the safety threshold, accurately estimating the value of deployment remains a challenge. Thus, the framework may be overly simplistic in its approach to "safety" regulation.

**Feasibility of Assumptions 1 and 2.** Assumption 1 fails when a model-building agent maliciously injects backdoor triggers into the model by altering the training dataset. Assumption 2 is also not straightforward. More cost (e.g., computational resources) does not necessarily equate to better safety. Safety also depends on other factors, such as the learning paradigm, model architecture, loss function design, and hyperparameter selection.

**Performance at high thresholds.** As highlighted in the experiments, SIRA demonstrates limited advantages when safety thresholds approach the upper range (e.g., above 0.8), where its performance is similar to that of simpler reserve threshold models.

1. Evan Hubinger, et al., Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training, arXiv:2401.05566.

---

confidence: 3
contribution: 2
initial_rating: 5
presentation: 4
questions: 1. What is the technical challenge in the considered auction problem for AI models, compared to classic auction problems?

2. Practical AI models are often very large. How can the safety of these model be evaluated? Given that the auction is done in a one shot setting, probably it is fine even if the model is large.

3. I am more concerned about the compensation $v_i^p$, which needs to be provided by a regulator to implement the proposed auction algorithm. Why is this practical for existing AI models? How large does the compensation need to be? According to bidding equilibrium in Theorem 2,  $v_i^p$ needs to be large for safer models. How could this be made up to compensate what the commercial AI models could achieve?
soundness: 3
strengths: 1. The topic considered in this paper is interesting and important. Regulations are needed to ensure AI safety.

2. Theoretical results are provided whose proofs can be found in the appendix. I didn't check all the mathematical proofs.

3. The paper is overall well-written and well motivated.
summary: The paper addresses the challenges regulators face, particularly with the deployment of large language models that can amplify misinformation and societal division. It highlights the urgent need for effective regulatory frameworks to mitigate these risks and enhance user safety. Observing a gap in the availability of rigorous and realistic mathematical frameworks for AI regulation, the authors propose an innovative auction-based regulatory mechanism. This mechanism is designed to incentivize the development and deployment of safer AI models and encourage active participation in the regulatory process. It demonstrates through derived Nash Equilibria that the proposed auction mechanism effectively ensures that each participating agent’s optimal strategy aligns with submitting a model that exceeds a set minimum-safety threshold.
weaknesses: 1. The way used by the paper to model the safety may not be realistic. It is assumed to be some safety level $s_i$ of a model $w_i$, which is expected to be less than $\epsilon$. How is the safety measured for AI models using the metric mapping $S$ in practice? For common foundation models and LLMs, it might be hard to evaluate $S$ for $w_i$, especially given the size of $w_i$. What if a model provider take advantage of the inaccuracy of the safety evaluation to benefit itself?

2. The proposed auction algorithm, together with the theoretical results and analysis seem quite standard. How does it differ from the classic all-pay auction results (for instance, Amann et al. 1996) in the setting for AI models? It is worth highlighting the technical novelty and emphasize why the proposed method is needed for AI models, given that it is claimed in Line 398-399 that "To the best of our knowledge there are no other comparable mechanisms for safety regulation in AI."

---

confidence: 4
contribution: 2
initial_rating: 5
presentation: 2
questions: Beyond the questions listed in the Weakness section, here are some additional questions I have:
- The framework assumes that the cost $M$ is the same across agents. This assumption seems unrealistic in practice, given that different agents may have varying models, training procedures, and resources, which makes the cost of aligning the safety levels different. If $M$ differs across agents, is there a way to adapt the framework to accommodate heterogeneous costs while maintaining its theoretical properties? 
- The paper didn't mention incentive compatibility, a key issue in auction literature. Is truthful report of $b_i$ guaranteed?
soundness: 2
strengths: - A novel and important question, and strong motivation
- Sound theoretical analysis
- Genrally well-written
summary: The authors provides a formulation of the AI regulatory process as an all-pay auction, and design an auction-based regulatory mechanism that produces Nash Equilibria that induces safety considerations.
weaknesses: The authors' formulation of the regulatory process and safety components appears to be somewhat simplified and may diverge from current AI developments in a few key ways:
- (Minor) The authors assume a fixed safety threshold, denoted as $\epsilon$, for model development. While this may hold in domains such as drug approvals or medical equipment (as illustrated by the authors' N95 mask example), applying a similar framework to AI models is more challenging and complex.
- (Minor) The model assumes that the test set used by regulators is drawn from the same distribution as the agent’s evaluation data. However, in the specific context of language models, techniques such as fine-tuning and reinforcement learning from human feedback (RLHF) can easily improve performance metrics if the evaluation distribution remains consistent. This weakens the argument that a single scalar value would sufficiently capture the intricacies of regulatory inspection.
- The authors propose a strictly increasing relationship between safety and cost, arguing that "safer models cost more to develop." However, they do not explicitly account for the trade-off between safety and the model's quality or usefulness in their framework. This omission raises questions, particularly since existing alignment approaches (e.g., RLHF) are often designed to balance helpfulness and harmlessness. In practice, a model could be made extremely safe (e.g., by providing only generic responses), but this could significantly reduce its usefulness without necessarily increasing development costs. In fact, under the authors' framework, one could submit a trivial model (e.g., one that always responds with "Thank you for your input"), bid the highest possible value, and meet the safety threshold $\epsilon$ to claim the regulator's compensation. This suggests that achieving safety in some cases may not necessarily be costly unless the model’s quality or usefulness is held constant.
- This issue could be exacerbated by the presence of open-source models like LLaMA, which may further incentivize the "gaming" of the regulatory system. Agents could enter the competition with low-cost variants of open-source models that prioritize safety at the expense of quality, primarily to secure the regulator’s compensation. Put it in a different way, low-quality models (which are safe but not useful) could flood the regulatory system, making it easier to claim compensation without delivering valuable AI products. This could distort incentives, where participants optimize for regulatory approval rather than producing high-quality, well-rounded models.

For the mechanism itself, a minor concern include the use of randomization, which introduces envy into the mechanism. With development costs potentially huge, this might lead to issues and discontent and distrust with the mechanism after the outcome is realized.

Meta-review: This paper looks at the problem of regulating AI models, specifically for safety. This is approached through theoretical framework, through an all-pay auction with companies and a regulator. The authors find Nash equilibria and have theoretical results. 

Reviewers agree that this is a very important problem, and that the approach taken is novel and interesting. Reviewers also agree that the paper is well-written, which I agree with. I agree that these are all key strengths of the paper. 

Unfortunately, all reviewers agree on a key limitation: the restrictiveness of the assumptions, specifically Assumption 2. There is of course a fine line between have realistic assumptions and those that allow for the kind of theory this paper does. The authors have defended their assumptions in rebuttal and with additional text in the paper (and an ablation study). Upon further discussion with reviewers, however, we all find that this is still a key limiting factor of this paper. I encourage the authors to take these concerns into account for a future version of the paper, actively acknowledging and tackling these issues even earlier in the paper (eg by giving specific examples of where Assumption 2 is both realistic *and not realistic*).

Decision: Reject