Interference Among First-Price Pacing Equilibria: A Bias and Variance Analysis
Authors: Luofeng Liao
Christian Kroer
Sergei Leonenkov
Okke Schrijvers
Liang Shi
Nicolas Stier Moses
Congshan Zhang
Venue: ICLR 2025 Poster (2025)
Keywords: First-price auctions
Pacing equilibrium
interference bias
Primary area: other topics in machine learning (i.e., none of the above)
TL;DR: We propose a method to remove bias caused by cross-market bidding in auction markets.

Abstract: A/B testing is widely used in the internet industry. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers have budget constraints, as budget consumption in one arm of the experiment impacts performance of the other arm. 
This is often addressed using a budget-split design. Yet such splitting may degrade statistical performance as budgets become too small in each arm.
We propose a parallel budget-controlled A/B testing design where we use market segmentation to identify submarkets in the larger market, and we run parallel budget-split experiments in each submarket.
We demonstrate the effectiveness of this approach on real experiments on advertising markets at Meta.
Then, we formally study interference that derives from such experimental designs, using the first-price pacing equilibrium framework as our model of market equilibration.
We propose a debiased surrogate that eliminates the first-order bias of FPPE, and derive a plug-in estimator for the surrogate and establish its asymptotic normality. We then provide an estimation procedure for submarket parallel budget-controlled A/B tests. Finally, we present numerical examples on semi-synthetic data, confirming that the debiasing technique achieves the desired coverage properties.

Published as a conference paper at ICLR 2025
INTERFERENCE AMONG FIRST-PRICE PACING EQUI- LIBRIA: A BIAS AND VARIANCE ANALYSIS
Luofeng Liao
Christian Kroer
Sergei Leonenkov Ads Online Experimentation, Meta 1eomenkov@met. com
Columbia University
(113530, ck2945) @co lumbi a edu
Okke Schrijvers Liang Shi Central Applied Science, Meta {okke, liangshi, nstier, eszhang) @meta. com
Nicolas Stier-Moses
Congshan Zhang
ABSTRACT
A/B testing is widely used in the internet industry. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers have budget constraints, as budget consumption in one arm of the experiment impacts performance of the other arm. This often addressed using budget-split design. Yet such splitting may degrade statistical performance as budgets become too small in each arm. We propose a parallel budget-controlled market segmentation to identify submarkets in the
A/B testing design where larger market, and we run parallel budget-split experiments in each submarket. We demonstrate the effectiveness of this approach on real experiments on advertising markets at Meta. Then, we formally study interference that derives from such experimental designs, using the first-price pacing equilibrium framework as our model of market cquilibration. We propose a debiased surrogate that eliminates the first-order bias of FPPE, and derive a plug-in estimator for the surrogate and establish its asymptotic normality. We then provide an estimation procedure for submarket parallel budget-controoled A/B tests. Finally, we present numerical examples semi-synthetic data, confirming that the debiasing technique achieves the desired coverage properties.
INTRODUCTION
Online A/B testing is widely used in the internet industry to inform decisions on new feature roll- outs. For online marketplaces (such as advertising markets), standard approaches to A/B testing may lead to biased results when buyers operate under a budget constraint, as budget consumption in one arm of the experiment impacts performance of the other arm. To counteract this interference, one can use budget-split design where the budget constraint operates on per-arm basis and each arm receives an equal fraction of the budget, leading to "budget-controlled A/B testing." " see e.g. Basse et al. (2016); Liu et al. (2021).
Despite clear advantages of budget-controlled A/B testing, companies are extremely constrained by the number of such experiments they can run. While it's possible to create more budget splits, this will lower the budget per group substantially, which could lead different equilibrium outcomes and may disproportionately affect smaller buyers. Additionally, a common approach to increase experimentation throughput to run orthogonal experiments (with their own orthogonal random- ization), but this would either suffer from the same interference the vanilla A/B test setup, also require further budget splits.
In this paper, we propose parallel budget-controlled A/B test design where use market seg- mentation to identify submarkets in the larger market, and we run parallel experiments on each sub- market. When the overall market can be divided into several relatively isolated submarkets, budget- controlled A/B tests can be conducted in parallel within these submarkets. However, this method also presents some challenges. First, submarkets rarely completely isolated; certain items may

---

Published as a conference paper at ICLR 2025
attract buyers from multiple submarkets, resulting in interference across submarkets when conduct- ing tests in parallel. Second, submarkets differ in terms of buyer (and user) composition, which might cause the local treatment effect estimates not be representative of the global treatment ef- fect where all buyers are included in the market. The second challenge is relatively easy to address in practice by imposing balancing constraints in the clustering algorithm used define submarkets, while the first challenge is more fundamentally important and requires deeper understanding.
Before the theoretical exposition, we consider comparison of results for paired experiments be- tween parallel budget-controlled A/B test setup, and that of a traditional budget-split design; where the latter is considered the gold standard. Sec. on the left shows comparisons of 99 experiments where the point estimate and CIs are plotted on the vertical axis for the parallel design, and on the horizontal axis for the budget-split design. The most important feature whether ments agree between (negative, neutral, positive), as change would result in launch reversal. The two experiment designs agree 75% of cases (at 90% confidence level, hence the optimal agree- ment is 81.5%) which increases to 79% after the introducing guardrail metric, Sec. on the right. These results are quite satisfactory, but do point at the existence remaining interference bias. In the remainder of this paper, correcting the interference bias is the main objective.
two experi-
Contributions:
1. We formally define market interference in first-price auction markets using the first price pacing equilibrium (FPPE) framework (Conitzer al., 2022a) (Sec. 3)
2. We propose a debiased surrogate that eliminates the first-order bias FPPE, and derive a plug-in in estimator for the surrogate and establish its asymptotic normality. (Sec. 4)
3. We semi-synthetic experiments, confirming that the debiasing technique achieves the desired coverage properties. (Sec 5).
1.1 PARALLEL A/B TESTING IN PRACTICE
In this section we describe the real-worle problem of A/B testing with congestion that we wish to model, and our proposed solution of parallel A/B tests in carefully balanced submarkets. We start by describing the market environment. There is set of advertisers, and each advertiser i has budget 6, Whenever a user shows up on the platform an impression opportunity occurs, and an auction is conducted in order to determine which ad will be shown to the user. Each advertiser i has some stated value (f) of being shown to a particular impression opportunity 8. The advertiser submits bid which is determined based (O), well the expenditure of the advertiser so far. For example, in multiplicative pacing Balseiro et al. (2017); Conitzer et al. (2022b), the platform adaptively learns pacing multiplier 3, [0, 1] such that the bid formed (0). system then adaptively controls 5 over time, in order to ensure the correct rate of budget expenditure on behalf of the advertiser. We consider first-price auctions, which the predominant way display advertising is sold online.
The
1.00
1.03
Sub-market Test
1.05
Lub-market A/B Test
1.04
3.00
1.53
162
3.01
1.01
1.00
1.00
0.59
0.59
0.58
1.00
1.02
1.06
0.59
1.00
1.01
1.07
1.03
Standard A/B Test
Standard A/B Test
Figure 1: Parallel standard budget-controlled A/B test, daily treatment effect. We denote neutral treatment effects with a value of 1.0. Red crosses indicate instances of sign inconsistencics. Left are all datapoints, the right, datapoints that fail a guardrail metric are removed.
2

---

Published as a conference paper at ICLR 2025
Now that have discussed budget management, describe the A/B testing problem. Suppose that a platform wants to run A/B tests, which may affect, e.g., the valuations that advertisers have for impression slots, revenue, etc. We construct the market-segmented experimental setup as follows: We first define a bipartite graph between advertisers and users based on targeting criteria. Subsequently, cluster the advertisers into K clusters, where the objective is minimize the of weighted edges between clusters subject to traffic balancing constraints to make the resulting clusters similar to the whole market as possible. The edge weight between pair of advertisers is the number of impressions (or users) where they are both within the top-k bids. The choice of a parameter that must be chosen based on experience with the specific application setting. the clustering achieves small objective function value, then each cluster is mostly isolated submarket, the sense that each user will mostly receive bids from advertisers in a single cluster. Then, we run an A/B test within each of the K submarkets. Every user is randomly assigned to either "A" or "B" each submarket. The main challenge is that while submarkets are relatively isolated, there is remaining interference from users who are targeted by advertisers from different submarkets, leading to a slightly different equilibrium. Our main contribution to define a framework for analyzing such interference, and giving estimator that removes the bias from these users. We survey related works in App. D.
REVIEW OF FPPE THEORY
Notation, For a measurable space (0, d0), we let L" (and L resp.) denote the set of (nonnegative, resp.) LP functions θ w.r. the base measure d0 for any p [1, oo] (including t E L and E L', we let (u, x) = So v(0)x(0) d0. We treat all functions that agree on all but measure-zero set as the same. For a sequence of random variables {X,,), we say X, = Op if for any there exists finite M, and finite N, such that P([ say X, = (1) if X, converges to zero in probability. For a subset @' c θ, let (.): e {0, be the indicator function of O'. Convergence in distribution and probability is denoted by 4 and
oo). Given
M,) < for all 2 No. We
D. Given vector a [ai...., a,]", let Diag(a) denote the diagonal matrix with (i, i)-th entry being a; sometimes we write Diag(a:) when is convenient define each a; inline, Let A denote the Moore-Penrose inverse of the matrix A, ej the j-th unit vector, and [x]] (1,....n.}
s(
b,
bad
Figure Left: Finite FPPE (left) and limit FPPE (right). a finite FPPE, there are a finite number of items; in limit FPPE, the item set is continuum. Right: The interference model Left (-aa): the observed market where interference present among submarkets. Middle (.Ma)): the limit market with interference from bad item set. Right (.M)): the limit market with perfectly separated submarkets. We use data from the left panel to make inferences about the market the right panel.
Limit FPPE. We first introduce notion of limit market and two regularity conditions the market, which models the underlying market structure that we sample from. We have n buyers and a possibly continuous set of items ④ with integrating measure d0. For example, could take O

---

Published as a conference paper at ICLR 2025
[0, 1] and dØ = the Lebesgue measure on [0. 1]. Each buyer has budget b;; let b (b1, (01,...,, br). The valuation for buyer is a function V/ E + such that buyer has valuation (0) for one unit of item 0 E θ; let θ - R" v(0) [e) (8) (f)..... .. vn (f)]. We assume ö max; sup, (0) < The supplies of items are given by a function 8 E L.P i.e., item E θ has s(0) units of supply. Without loss of generality, we assume a unit total supply So s dd = 1. Given g : θ + R, we let E[g] = f (0))=(0)))))))))))))))))))) d0 and Varlg] = Elg²] - (E(g))². Given / i.i.d. draws {0r ( 0² from 8, let - P= =
Next we introduce the market equilibrium concept that is the foundation of our study. For that we leverage the first-price pacing equilibrium (FPPE) (Conitzer et al., 2022a). FPPE models equi- librium outcomes under budget-managemen systems employed in several practical settings. Each buyer is assigned a pacing multiplier 8r, which is used control their budget expenditure. For each individual auction 0, the buyer bids (0). which can be seen as their adjusted valuation after factoring in their budget constraint (in practice, the valuation (0) may not be the buyer's true val- uation, but instead their unscaled bid reported to the platform). The goal of the budget management system is achieve an equilibrium, meaning that must ensure that buyers spend their budget ex- actly by appropriately choosing B, If the budget cannot be fully spent, then no pacing must occur (i.e.. 3 1). Below we formally define the pacing equilibrium concept in the continuous setting (see Fig 2, right), and give the finite version in the next section (Fig 2, left).
Definition (Limit FPPE, Gao Kroer (2022); Conitzer et al. (2022a)). limit FPPE, denoted as FPPE(b, v, 8, θ), is the unique tuple (3, (()) E 1) x (O) such that there exist an allocation T; : θ - [0, for all i E [n] such that,
1. (First-price) Prices and allocations are determined by first price auctions: for all items 9 E O. p(0)= = max; B,u, and only the highest bidders obtain items: for all and 8, 00( > 0 implies Beu; (0) max Bav (0)
2. (Feasiblity, market clearing) Budget are respected: for all i, (zz((0)(()ss())) do S bi. There is no overselling: for all 6. -= S 1. Items with nonzero price are fully allocated: for all 0, p(0) > 0 implies (0)= 1.
3. (No unnecessary pacing) For all fz:(())(())s) de < b, implies B, = 1.
Let 3a and p* be the equilibrium pacing multipliers and prices. Revenue the limit FPPE is REV* = f (0)s(0) d0 It measures the profitability of the auction platform. The leftover budgets for buyers are denoted by o = b; - f (0(())))))))))))())) (0)= (0) do.
The first two conditions simply describe the possible outcomes of a first-price auction system that uses pacing as the budget management strategy. The last condition, no unnecessary pacing, ensures that we only scale down a buyer's bids case their budget constraint binding. FPPE has many nice properties, including that they are competitive equilibria and that they are revenue-maximizing among budget-fcasible pacing multipliers (Conitzer et al., 2022a).
In a pacing auction market M FPPE(b, s,s, θ) the following two regularity conditions important for the study of its statistical properties.
Definition 2 (SMO). We say the smoothness condition holds if the map (3 + E.[max, (.), is swice continuously differentiable a neighborhood of B
Definition 3 (SCS). We say strict complementary slackness holds if, whenever buyer is unpaced (3,- = 1). then her leftover budget is strictly positive (8; >> 0).
The condition SMO ensures that in the limit market, items that incur a tie are measure zero The condition SCS rules out degenerate buyers that spend their budget exactly at 3r. See Liao & Kroer (2023) and Liao et al. (2023) for extensive discussion about these conditions.
Finite FPPE. Next we introduce the finite FPPE, which models the auction data we observe in practice. Let 7 (0) ..... a E R4, units. A finite FPPE is limit FPPE where the supply is a discrete measure supported on the observed items 7. Let of (f").
..... 0²) be sequence of items. Assume each item has the same supply of
Definition 4 (Fintee FPPE, informal). A finite FPPE, FPPE(b, v. o, 7), is a limit FPPE where the item set is the finiee set of observed items 7, See App. C the full definition.
4

---

Published as a conference paper at ICLR 2025
In Liao & Kroer (2023), it is shown that if consists of t i..dd. draws from distribution s, and one takes = 1/t, then the pacing multiplier in FPPE(b, v, 1/t,7) converge to the pacing multiplier in FPPE(b, v, s. ))) in probability. Also, note that the FPPE FPPE(tb, v, 1, 1.7) converges to the same limit FPPE FPPE(6, v, s, )) because the pacing multipliers of finite FPPE do not change when budgets and supplies are multiplied by the same scalar.
The Eisenberg-Gale Program. Both the limit FPPE and the finite FPPE have convex program characterizations (Chen et al., 2007; Conitzer al., 2022a; Gao & Kroer, 2022). We define the dual Eisenberg- Gale (EG) objective for a single item as
F(e, B) f(0,3)
Σ b, log f(0. 3)
max (0)
(1)
iEIn]
The population and sample (dual) EG objectives are then defined as
H(3) = E[F(0 H(( = Σ F(00,)))
(2)
We say H is the Hessian of market A when H V2 BB I F(O, B)s d0
The equilibrium pacing multipliers 3r in FPPE(b, v, )) can be recovered through the population dual EG program
8 = arg min H(3)
()(
The pacing multiplier vector 3 is the unique solution to Eq. (3). Let 3 be the equilibrium pacing multiplier in FPPE(b, v, 1/4,)). Then 3 solves the sample analogue of Eq. (3):
B = arg min (3)
(4)
BE(0,1)
Let us briefly comment on the differential structure of f. since it plays role in later sections. The function f(3, 0) is convex function of [3 and its subdifferential 0 0) is the convex hull of (u.e, index i such that (0) max; 8q (0)), with e; being the base vector IR. When max; (0) attained by a unique i* the function f(- 0) differentiable. In that case, all entries of V (B, 0) are zero expect that the i"-th entry is filled with the value Up (0).
3 INTERFERENCE AS CONTAMINATION
In this section, we discuss how to estimate market equilibria when there is contamination in the supply, meaning that items generated from mixture of two distributions, when in reality wish to estimate equilibrium quantities from one of the two distributions. Then, we show that interference from other markets can be viewed form of contamination, and the problem of removing interference bias can be analyzed via our contamination framework.
3.1 FPPE WITH CONTAMINATED SUPPLY
We assume that we are in the same FPPE setting as before: there are n buyers, each with budget b, and an item set θ which is now partitioned in to Obad and Ogood. However, now we assume that the supply & is contaminated by s', another supply distribution. We define the a-contaminated market M. FPPE(b. U, Sa. θ) and the uncontaminated market as Mo FPPE(b, v, +), where 8, = as + (1 a)ss, distribution s' is supported on Obad. and on @oodd Our goal is to perform inference about FPPE properties in the limit FPPE with the supply However,
given access to finite FPPES sampled from 8a instead. In particular, let 7 be f i.i.d. draws from 8, and let
M.. ( v, 1/t,)). We assume is known throughout the paper. In practice, this often be estimated from historical data; in the parallel A/B test setting, this can be estimated directly from the sampled set of items, since we know whether item drawn from
s'. Let 3 and 3 be the limit pacing multipliers in My and Ma. respectively. Let 8 be the pacing multipliers in the
sampled market Mn, and let H (B) ! F(0", ,3) be the sample EG objective.
5

---

Published as a conference paper at ICLR 2025
If we wanted to make inferences about Ma then could use existing statistical inference theory how to use data in finite FPPE Mr make inferences about the limit FPPE Ma (Liao & Kroer, 2023). However, the supply contamination prevents the application of these tools to our problem.
Our central research question is then on how use data in the finite contaminated market Ma to make inferences about the uncontaminated limit market Mo-
In Sec. 4 propose estimator for this problem and derive its properties. The results there apply to general item space @ and supplies s and s'. By imposing structure on O s and s', we show that the contamination model captures the interference among FPPEs.
3.2 APPLICATION: MODELING INTERFERENCE AMONG FPPES
Now we show how the contamination model from the previous section can be used to model inter- ference. Consider K separate auction markets, which together form global market. In the global market there are n buyers, each with budget b; and an item set θ, partitioned into Q @ood and O Let CI...., CK be a partition and si.... Sx be set of supply functions, supported on i,.... Ox respectively. The k-th sub- market consists of buyers in Ck. the item set O₁ and supply $g. Let s = K Σx 8/. be the average mixture and s' be supply supported on Obd Let the contaminated supply be 8,, ass' + - ax88.
the buyers, .I...., Og be a partition of the good item set good.
By imposing structure on O good and Oad the contamination model can capture interference among auction markets. We assume that submarkets no interference. A buyer i E C2 is only interested in items from the submarket he belongs to: (e) 0 for 9 Ow, k' * k. Next, submarkets; see the green edges in Fig 2 left panel. An item referred to as bad it has positive values for buyers from at least two different submarkets. Formally, 0 E Obad if there exist i Cs. j E Ck., h * K,, such that 0; (0) > O and (0) > 0. Combining these assumptions, we have that buyer from submarket & has positive values only for items from the θ₂ and possibly Obad. Now we have fully specified contaminated market setup: we wish to make inferences on the market consisting of only ggood (which is really K fully separate submarkets), but we observe an actual market containing items from O @ 0 With this setup, we can use the results developed in the following section to model interference in parallel submarkets.
separated, which models the ideal case where there
let Oad represent items that cause outbound edges from
In Fig 2 we present an example of interference among K 3 submarkets. The market of interest the perfectly separated market (right). This is because, in parallel A/B testing, submarkets are explicitly created such that each submarket resembles the global market. Then when submarket receives a treatment, the observed quantities in that submarket, such as revenues and social welfare, considered surrogates for the treatment effect in the global market. However, in practice we only observe the interfered finite market (left), which converges to the interfered limit market (middle). In App. F we show how to analyze parallel A/B testing using this framework.
4 A DEBIASED ESTIMATOR AND ITS PROPERTIES
This section develops a methodology for making inferences about the uncontaminated limit FPPE. Since the interference setting is a special case of the contamination setting, we develop theories for the latter. We introduce surrogate for pacing multipliers, based on notion of directional derivatives, and establish its debiasing property in Sec. 4.1. Then, we focus on estimating this surrogate quantity in Sec. 4.2, and develop asymptotic normality results in Sec. 4.3. Secondly, we consider estimating revenue, which can be thought of as smooth function of pacing multipliers. We discuss debiased revenue estimation and inference based on our pacing multiplier results in App. E.
4.1 A DEBIASED SURROGATE FOR PACING MULTIPLIERS
If view 3 as function of the level of contamination a, then one can imagine that under sufficient regularity conditions, the pacing multipliers in the perfectly separated market, Bg can be approximated by some form of Taylor expansion of + B at This can be made rigorous by the

---

Published as a conference paper at ICLR 2025
notion of directional derivatives. We define
Ba_c 13
dB (a) = lim
(5)
<-
the limit exists. We will show in Thm. 1 that 3 + a (a) serves as a good approximation to G--
3
Thanks to the convex program characterization of FPPE, the directional derivative d,3" (ax) has a closed-form expression under certain regularity conditions (the conditions are given in Thm. 1; the full proof is given in the appendix). We need a few notations for this expression. Define 8, d0. Let Ho V f ((0,8%)ss d0 be the Hessian matrix in the market Mr and P. (1 < 1)). Then, under the regularity conditions given in Thm. 1 below,
( (s s')d0.
(a) = - (Pa Po)
(6)
We present a heuristic derivation in App. G. Given the closed-form expression of d,3 (a), we define the following debiased pacing multiplier
3 = t (-
Theorem (Analysis of Bias). Suppose that the market Mu conditions SMO and SCS hold, and assume that 3 + V F)), do is twice continuously differentiable in neighborhood of f Then the directional derivative (.) is well-defined in a neighborkood of zero, and is given by Eq. (6), Moreover, $ O.
| - 12 o(a).
The proof is given in App. B.1. Thm. indicates that the debiased surrogate 3 removes first-order bias caused by contamination. The limit pacing multipliers 3r of the contaminated market Ma will have bias of order 3% 3" O(a). In contrast, Thm. 1 shows that the debiased surrogate only incurs a bias of order o(a).
4.2
THE ESTIMATOR
In this section introduce a plug-in estimator for the debiased surrogate 3 and introduce a con- sistency theorem. The next section discusses constructing confidence intervals.
To estimate d (a) in Eq. (6) need estimates of its three components: the Hessian Ha V [ F(0, 3r) d6, the diagonal matrix P. and the vector 8, = (e, - s') d8.
The Hessian. For simplicity in our theoretical results, we will simply assume a generic Hessian estimator H, such that for some 7)t O we have H H, Op(m). Hong et al. (2015) discuss the estimation of the derivative in detail. Different kinds of statistical guarantees require different rate conditions Mh: see Theorems 2 and 3. We then introduce two Hessian estimators: one is applicable for general FPPE, while the other requires an extra market regularity condition. The first Hessian estimator is the finite difference method. Let e;, e; be basis vectors and be step-size. Then the estimator
Habi -Ha, Ha..(87.)) Ha.((B (8..((82-11)
where B- B% + e;c, + ejet, and H. (B) = F with {0r), }- being the items in M., In practice, diagonal approximation of the Hessian suffices. The second method relies additional regularity condition, in which case derive simplified formula for the Hessian, thereby enabling simpler estimation procedure (see Thm. 3).
The vector 5,, Let be the Radon-Nikodym ratio g()) = (d(s -
- - gma (0) 1 Le (0). With the ratio g. the true vector 8, can be written as 6., = v (0,3')), s. d0, which is easy to estimate given i.i.d. draws from 8a. In particular, our estimator is then 8... t (00 JN Here μ [xTe] ..... "T, subgradient of f (X)) w.r.t. B.
The diagonal matrix P.. Recall P. = Diag(1 < 1)). So natural estimator is P.
Diag(1(82, 1 44),, where the slackness 4

---

Published as a conference paper at ICLR 2025
With all three components estimated, using P is the pacing multiplier in the market Ma we define the plug-in estimator for dl3" (a) in Eq. (6) as
3 3% a P.. P.H.P.)
(8)
Theorem 2 (Consisteney). Suppose that the market Ma conditions SMO and scs hold. If the Hessian estimation error satisfies = o(1), then 3 2 B". The proof is in App. B.2.
4.3 ASYMPTOTIC NORMALITY AND INFERENCE
We present two asymptotic normality results. In the first result, we require a stronger condition on the Hessian error T/-- In particular, as will be shown in Thm. 3, the rate condition f/- o(1/v() is sufficient for normality. One could use a separate large historical dataset to obtain a good estimate of the Hessian matrix. In the second result, we impose an additional condition on market structure which simplifies the Hessian expression and facilitates efficient Hessian estimation. To describe the additional market structure, we define the gap between the highest and the accondhiighest bid for an item / under pacing 3 by (idgap(33,5 8) max [ax(3;u;;0) (0)) secondmax ( (0)} where sccondmax the second- highesth entry potentially equal to the highest: e.g.. sccondmax(11, 1, 2))
1. When there is a tie for item 0 under pacing B, we have bidgap(3, 0) O. When there is no tie for an item 0, the gap bidgap(33, 0) strictly positive.
For any + R, let [o [g] = g%a d0 and Cova(g) = E.I(g Lg Exl")) Recall 3 the debiased surrogate in Eq. (7) and [ is its estimator defined in Eq. (8).
We need to introduce few more notations to describe the normality results. First, let d, = - (.) market converge to the limit counterpart and have the representation
As mentioned previously, the pacing multipliers in the contaminated
Vi (;)
(d) - E. [d. o,)
In the statisties literature, the function d. (-) - E[d,) called the influence function (van der Vaart, 2000). For our debiased estimators, we need the following (uncentered) influence functions.
(0) = ( (0) (0), (2 = di (0) - (3 (0)
Theorem 3. Let SCS and SMO hold in Mr.
1. Asymptotic Normality in General Market. It holds that 3- Op()) op (2))
where Vtz, 6..((0 (0, En) with Eg (C)) and is the Hessian estimation error.
Asymptotic Normality under a Bid Gap Condition. If in addition Le[L/bidgap( 0)] < co, then H. = (iag(b:/(B2..)) Suppose we estimate H. with H. = Diag(b,/ (b, (2).))).. Then
Vi(@- 3-) < (0, Σ2) where E₂ Cova(d2). The proof is in App. B.3.
Thm. 3 part shows how the error of the Hessian estimate affects the distribution of the estimator G. If th = o(1/VD), then the decomposition becomes vi( 3³) = Vizi + (1), implying asymptotic normality, i.e. Vi( 3)) 3.1 S. ), in which case one can construct an ellipsoidal confidence region for " Thm. 3 part 2 shows direct asymptotic normality under the extra condition, with a simpler Hessian estimator that avoids finite differences.
To perform inference, we need to construct consistent estimate of the covariance ma- trix. Now we describe a plug-in estimate of Σ,. Let the estimator di be di where ft' = ...., 2., ", and P., H. have been de-
-(1-. 10 fined in Sec. 4.2. The plug-in estimator is Σ +Σ:1 (d) dd) - d with di
"E;=dd"' dn' By similar arguments as in Liao et al. (2023), the plug-in estimates of Σ, and Eg are consistent. Algorithm 1 summarizes the debiasing procedure.
In App. E we present a similar debiased estimator for revenue and its bias and variance propertics.
In App. F we specialize the debiased estimator parallel budget-controlled A/B testing.
8

---

Published as a conference paper at ICLR 2025
Norma lred bte
20
0.0
025
0.30
Figure Normalized bias (in percent of true value) as a function of semi-synthetic experiments. 3 and REV are the debiased surrogates for pacing multiplier and revenue in the limit market with interference Ma.
5 SEMI-SYNTHETIC EXPERIMENT
To evaluate our proposed framework and debiased estimator, we run semi-synthetic simulations to check if the proposed estimator for beta and revenue are indeed less biased, and test the coverage of the proposed estimator. Fully synthetic experiments are presented App. H.
In the semi- remi-synthetic experiments, simulate 40 buyers and 10000 good items in two submarkets, with a varying number of bad items (up to 5000) in order to study the effect of the contamination parameter a. For each a, we randomly sample a budget for each buyer, and compute 3 and REV" from the limit pure market Mo with a value function in each submarket. Both the budget and values are sampled from historical bidding data, making the budget and value distributions heavy-tailed as in the real-world applications. More specifically, we first sample a certain number of auctions. For each auction, we sample given number of advertisers with their per-impression bids. Advertisers that are sampled across different auctions are treated as the same buyers and their budgets are determined by aggregating their values over auctions up to a scalar to calibrated to get the percentage of budget-constrained buyers equal to what was observed in the real-world auction market, along the same lines as the experiments of Conitzer et al. (2022b).
To check if the debiased surrogate reduces bias, compute 3G and REV* from the limit market with interference Ma and their surrogates, 3 and REV (@² is defined Eq. (7), REV is defined in App. E. We look the normalized bias for the surrogate, defined as | --00--"'1 1121111111110101 for pacing multipliers and REEV /REV* 1| for revenue, and similarly defined for the limit quantities. Fig 3 shows the normalized bias curves as function of a. The magnitude of the bias increases with o, for both the variables in the limit market with interference Mr and their debiased surrogates. The bias of the debiased surrogates is indeed much smaller than the contaminated limit quantities.
Next, we check the coverage of the proposed variance estimator. For each and each budget sample, we run 100 simulations in the following way: We sample items (or their values for each buyer) considering two submarkets and bad items. We then run the finite FPPE with bad items and obtain a baseline estimate for pacing multiplier and revenue without applying the debiasing procedure. Then, we apply the debiasing procedure compute the debiased estimates. For each simulation, we check if the debiased surrogate is within the confidence interval of the debiased estimator. Finally, we aggregate them to compute the estimated coverage of the estimator. The results both pacing multiplier and revenue are shown in Table 1. For the coverage 3, we first compute the coverage of each component and report only the average in the table. For revenue, construct the CI using the two approaches as mentioned in App. E: one based on Eq. (22) and the other using parametric bootstrap based on the estimated asymptotic distribution of 3 (with "(b)" in the column names).
Firstly, Fig shows that both CIs converges to the true value from the limit market with interfrence Ma as the number of items goes infinity. Then, in Table we show that the coverage for 3 is slightly smaller than the nominal level (95%), well as the coverage of the bootstrap CI of

---

Published as a conference paper at ICLR 2025
Thue value ane yt
boond mp
Sevenue
0.35
0.20
0.35
500
15:00
2000 2300
3000
Number of koms
Figure 4: Revenue confidence intervals as a function of the number of items in semi-synthetic experiments. The analytic CI comes from Eq. (22). The true value is the debiased surrogates for revenue in the limit market with interference Mr.
revenue. The under-coverage for mainly driven by the under-estimation of the variance of 3. while the under-coverage of the bootstrap CI for revenue can also be partially attributed to the higher dimensionality (with 40 buyers), making the bootstrap resampling harder to explore the whole space.
Although the proposed variance estimator has good asymptotic properties, the results from our syn- thetic experiments suggest that it can perform badly, in either direction, for finite markets. Construct- ing more accurate variance estimators for our debiased estimator in finite settings would certainly mitigate the over- or under-coverage issues that we observe here and deserve more future research. One promising alternative is construct the CI for 3 and REV by directly bootstrapping the ob- served value matrix, though this might work best for independent valuations across buyers.
3
REV
coverage
CI width
CI width (b)
coverage
coverage (b)
1000/1 1000
0.877
0.244
0.044
1.0
0.95
2000/1 2000
0.849
0.225
0.043
0.87
3000/1 1 3300
0.852
0,210
0.041
1.0
0.81
4000/14008 4000
0,828
0,200
0,040
1.0
0.90
5000/15000
0.826
0.191
0.039
1.0
0.90
Table 1: Coverage of 9 and revenue estimates in the semi-synthetic experiments. All quantities averaged over 100 simulations for each a (the ratio the number of bad items and the total items). The coverage of / is averaged over all components of 3. For revenue estimates, the columns with "(b)" represent the quantities from the bootstrap CI, while the columns without "(b)" are for the from Eq. (22). The CI widths are normalized by the revenue from the limit market M...
CONCLUSION
We have proposed a practical experimental design for performing concurrent A/B tests large-scale ad auction markets, using a submarket clustering approach, and showed that in production exper- iments, this submarket clustering approach leads strong sign consistency performance, as com- pared to A/B testing on the full market, while allowing bggnficanntl--ighhee A/B test throughput. In order to model the potential for interference between submarket A/B tests, we introduced a theo- retical model of statistical inference in first-price pacing equilibrium problems, under settings with supply contamination. We showed how one can perform statistical inference in such setting using debiased estimation procedure, and studied the statistical properties of this procedure. We then showed how our model of statistical inference in FPPE with contamination can used to model the submarket clustering parallel A/B test design, and gave theoretical performance guarantees. Finally, we presented numerical experiments on fully synthetic and semi-synthetic data derived from Meta ad auctions. The experiments showed that our proposed debiased estimator achieves smaller biases and its statistical coverage on realistic data is generally in line with the predictions from our theory.
10

---

Published as a conference paper at ICLR 2025
ACKNOWLEDGEMENTS
We would like to thank the anonymous reviewers for their useful comments. Christian Kroer and Lu- ofeng Liao were supported by the Office of Naval Research awards 100014-22-1-2533 and N00014- 23-1-2374, and the National Science Foundation awards [1S-214736] and IIS-2238960.
REFERENCES
Gagan Aggarwal, Ashwinkumar Badanidiyuru, and Aranyak Mchta. Autobidding with constraints. In Web and Internet Economics: 15th International Conference, WINE 2019, New York, NY, USA, December 10-12, 2019, Proceedings 15, pp. 17-30. Springer, 2019.
Peter M. Aronow and Cyrus Samii. Estimating average causal effects under general interfer- ence, with application to a social network experiment. The Annals of Applied Statisties, 11 (4):1912 1947, 2017. doi: 10.1214/16-0AAS000: URL wtw.ttts///cl..or////// 16 AOAS1005.
Susan Athey, Dean Eckles, and Guido W. Imbens. Exact p-values for network interference. Journal of the American Statistical Association, 1136521):23--240, 2018. doi: 00.180001621459.2000 1241 URL httph:tt// //tp:i org/ 10. .
621459 201.5922000000 6.121112.8
Moshe Babaioff, Richard Cole, Jason Hartline, Nicole Immorlica, and Brendan Lucier. Non-quasi- linear agents in quasi-linear mechanisms. In 12th Innovations in Theoretical Computer Science Conference (ITCS 2021), volume 185, pp. 84, 2021.
Santiago Balseiro, Anthony Kim, Mohammad Mahdian, and Vahab Mirrokni. Budget management strategies in repeated auctions. In Proceedings of the 26th International Conference on World Wide Web, pp. 15-23, 2017.
Santiago Balseiro, Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song Zuo. Robust auction design in the auto-bidding world. Advances in Neural Information Processing Systems, 14:77777-,7788, 2021a.
Santiago Balseiro, Anthony Kim, Mohammad Mahdian, and Vahab Mirrokni. Budget-manngement strategies in repeated auctions. Operations research, pp. 859-876 2021b.
Santiago R Balseiro and Yonatan Gur. Learning in repeated auctions with budgets: Regret mini- mization and equilibrium. Management Science, 65(9):3952-330) 2019.
Santiago Balseiro, Omar Besbes, and Gabriel Y Weintraub. Repeated auctions with budgets in ad exchanges: Approximations and design. Management Science, 61(4):864-88) 884, 2015.
Guillaume Basse, Hossein Azari Soufiani, and Diane Lambert. Randomization and pernicious effects of limited budgets auction experiments. In Arthur Gretton and Christian C. Robert (eds.), Proceedings of the 19th International Conference on Artificial Intelligence and Statistics, volume 51 of Proceedings of Machine Learning Research, pp. 1412 1412-1420, Cadiz, Spain, 09-11 May 2016. PMLR. URL https://p/// httpp:://pooceeed.eeeeee// httpppooccinngeeeee mlr wrw.srreepeeee... /ttess/v55// htt////1/////eleeeeee 6b. html.
Christian Borgs, Jennifer Chayes, Nicole Immorlica, Kamal Jain, Omid Etesami, and Mohammad Mahdian. Dynamics of bid optimization in online advertisement auctions. In Proceedings of the 16th international conference on World Wide Web, pp. 531-540, 2007.
Lihua Chen, Yinyu Ye, and Jiawei Zhang. A note equilibrium pricing convex optimization. In International Workshop on Web and Internet Economics, pp. 7-16. Springer, 2007.
Xi Chen, Christian Kroer, and Rachitesh Kumar. The complexity of pacing for second price auc- tions. Muthematics of Operations Research, 2023.
Alex Chin. Regression adjustments for estimating the global treatment effect in experiments with interference. Journal of Causal Inference, 7, 05 2019. doi: 10.1515/jci-2018-0000

---

Published as a conference paper at ICLR 2025
Richard Cole, Nikhil R Devanur, Vasilis Gkatzelis, Kamal Jain, Tung Mai, Vijay V Vazirani, and Sadra Yazdanbod. Convex program duality, fisher markets, and Nash social welfare. In 18th ACM Conference on Economics and Computation, EC 2017. Association for Computing Machinery, Inc, 2017.
Vincent Conitzer, Christian Kroer, Debmalya Panigrahi, Okke Schrijvers, Nicolas E Stier-Moses, Eric Sodomka, and Christopher A Wilkens. Pacing equilibrium in first price auction markets. Management Science, 2022a.
Vincent Conitzer, Christian Kroer, Eric Sodomka, and Nicolas E Stier-Moses. Multiplicative pacing equilibria in auction markets. Operations Research, 70(2)963-989, 2022b.
Yuan Deng, Jieming Mao, Vahab Mirrokni, and Song 7uo. Towards efficient auctions in an auto- bidding world. In Proceedings of the Web Conference 2021, pp.
2021.
Yuan Deng, Jieming Mao, Vahab Mirrokni, Yifeng Teng, and Song Zuo. Non-uniform bid-scaling and equilibria for different auctions: An empirical study, arXiv preprint arXiv:2311.10679 2023.
Dean Eckles, Brian Karrer, and Johan Ugander. Design and analysis of experiments in networks:
Reducing bias from interference. Journal of Causal Inference, 5(1):20150021, 2017. doi: doi:
10. 10.1515/jei-2015-000 URL https: /ooi..ioo//o org /1/g/10.. 1515/ /ei////ciia
Jason Gaitonde, Yingkai Li, Bar Light, Brendan Lucier, and Aleksandrs Slivkins. Budget pacing in repeated auctions: Regret and efficiency without convergence. In 14th Innovations in Theoretical Computer Science Conference (ITCS 2023), volume 251, pp. 52, 2023.
Yuan Gao and Christian Kroer. Infinite-dimensional fisher markets and tractable fair division. Op-
erations Research (forthcoming), 2022.
Yuan Gao, Christian Kroer, and Alex Peysakhovich. Online market equilibrium with application to fair division. arXiv preprint arXiv:2103. 12936, 2021.
Christopher Harshaw, Fredrik Sävje, David Eisenstat, Vahab Mirrorkni, and Jean Pouget-Abadie. Design and analysis of bipartite experiments under linear exposure-response model. arXiv, 1..3.06392v3, 12 2021.
Han Hong, Aprajit Mahajan, and Denis Nekipelov. Extremum estimation and numerical derivatives. Journal of Econometrics, 188(1)2250-663 2015.
Brian Karrer, Liang Shi, Monica Bhole, Matt Goldman, Tyrone Palmer, Charlie Gelman, Mikael Konutgan, and Feng Sun. Network experimentation at scale. In Proceedings of the 27th acm sigkdd conference on knowledge discovery & data mining, pp. 3106-3116, 2021.
Shuangning Li and Stefan Wager. Random graph asymptotics for treatment effect estimation un- der network interference. The Annals of Statistics, 50(4):2334 - 2358, 2022. doi: 10.1214/
22-AOS2191. URL https: //tt://dooiiie// org 10. 121 / 22-A0S2191.
Luofeng Liao and Christian Kroer. Statistical inference and a/b testing for first-price pacing equilib- ria. International Conference on Machine Learning. 2023.
Luofeng Liao and Christian Kroer. Bootstrapping fisher market equilibrium and first-price pacing equilibrium. In International Conference on Machine Learning, 2024.
Luofeng Liao, Yuan Gao, and Christian Kroer. Nonstationary dual averaging and online fair alloca- tion. Advances in Neural Information Processing Systems, 155:37159-3771 2022.
Luofeng Liao, Yuan Gao, and Christian Kroer. Statistical inference for fisher market equilibrium. In The Eleventh International Conference on Learning Representations, 2023. URL https:
/poerrvview. net. forum? d=KemSRWOYJC.
Min Liu, Jialiang Mao, and Kang Kang. Trustworthy and powerful online marketplace experi- mentation with budget-split design. In Proceedings of the 27th ACM SIGKDD Conference on
Knowledge Discovery & Data Mining, KDD '21, pp. 33193329, New York, NY, USA, 2021. Association for Computing Machinery, ISBN 978 4781433333 doi: 01.1145/344754834477775 URL https::t///. /oo. wwt./oorgo...... 11.00..115/3444 0884483.446799 3467193
12

---

Published as a conference paper at ICLR 2025
David Rolnick, Kevin Aydin, Jean Pouget-Abadie, Shahab Kamali, Vahab Mirrokni, and Amir Na- jmi. Randomized experimental design via geographic clustering. In Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data Mining, pp. 2745- 2753, 2019.
Paul R Rosenbaum. Interference between units in randomized experiments Journal the American Statistical Association, 102(477):111-200, 2007. doi: 00.119800162145566000000 112. URL https: /tt.////.//////////////o/ www./oo./....o...ooo.oo 01w111///011140000000000
Alexander Shapiro. Sensitivity analysis of nonlinear programs and differentiability properties of
metric projections. SIAM Journal on Control and Optimization, 26(3):628-645, 1988.
Alexander Shapiro. On differential stability in stochastic programming. Mathematical Program- ming, 47:107-116, 1990.
Alexander Shapiro, Darinka Dentcheva, and Andrzej Ruszczynski. Lectures on stochastic program- ming: modeling and theory. SIAM, 2021.
Gilbert W Stewart. On the perturbatioo of pscudo-inverses, projections and linear least squares problems. SIAM review, 19(4)6334662, 1977.
Johan Ugander, Brian Karrer, Lars Backstrom, and Jon Kleinberg. Graph cluster randomization: Network exposure to multiple universes. In Proceedings of the 19th ACM SIGKDD Interna- tional Conference Knowledge Discovery and Data Mining, KDD '13, pp. 329337, New York, NY, USA, 2013. Association for Computing Machinery. ISBN 9781450321747. doi: 10.1145/2487555.248766 URL https: /twi.iiooo// http//ii..oroog////// 111.001111/////155112.1 248777522477988
Aad W van der Vaart. Asymptotic statistics, volume 3. Cambridge university press, 2000.
Davide Viviano, Lihua Lei, Guido Imbens, Brian Karrer, Okke Schrijvers, and Liang Shi. Causal clustering: design of cluster experiments under network interference. arXiv preprint arXiv:2310.14983, 2023.
Stefan Wager and Kuang Xu. Experimenting in equilibrium. Management Science, 67(11): 66946715, 2021. doi: 10.1287/mns:.2220.34444 URL https://dci... .orgloorg//// orr/100.187/ mnsc. 2020.3844 3844.
Corwin M. Zigler and Georgia Papadogeorgou. Bipartite causal inference with interference. arXiv, 1807.08660v1, 07 2018
A TECHNICAL LEMMAS
A.1 A PERTURBATION RESULT FOR CONSTRAINED STOCHASTIC PROGRAMS
We introduce a theorem from Shapiro (1990). Theorem in that paper handles the case where the constraints are also defined as expectations some random functions, and constraints are also perturbed in the analysis. For simplicity we specialize the theorem to deterministic constraints and do not perturb constraints. Note that Theorem 7.27 from Shapiro et al. (2021) can also used to prove our Thm. 1.
Let θ be a probability space equipped with an appropriate o-algebra. Consider F F:Ox x R" and set B C R", given by
R
B= (3: (B) 0, I: (g) < 0
where 7 and J are finite index sets. Let P and Q be probability measures on O Let 6(B, a) (P + - ((,3)) ()) and ((33 = ((33,) 0). Here ming & is the main program of interest and Q perturbation measure. The amount of perturbation is measured by E [0, 1).
Let 3 be the unique minimizer of ((B3 over B. Let J C J be the inequality constraints active at 3", meaning ())) for all E J". Let (3* (a) be the unique optimal solution ming Φ(;a).
13

---

Published as a conference paper at ICLR 2025
Lemma 1 (Theorem I from Shapiro (1990)). Define the Lagrangian function by L(B. X, a) 6B3 a) + Eieus Agg((B and A)) L(3, X, 0). Let Ao be the set of optimal Lagrangian multipliers Define the critical cone C = {u u' Vgg(B²) = 0, E 1; t;u'Vgi Vgr( S 0 for i E ("""'vv((") S O}. Define Ag arg Aa Ligrus>igi(3 Assume the following conditions.
1. Differentiability. The functions o(, 0), o(, 1), and gi, E J.J, are continuously differen- tiable in neighborhood of 8
2. Constraint Qualification. The gradients (3) E are linearly independent. And there exists b E IR" such that b" V. = 0 E I and b bVVg;((")) < f i E J
3. Differentiability, The functions ( 0), 9. i J twice continuously differentiable in a neighbourhood of 3
4. Second-order sufficient condition. maxxeλ; u v' 2 (B" A)u O.
Assume for all
nonzero u
E C,
Define the function and the set E by
1 z" Vi Lo A)u
b(u) max u V2 V Lo , #2
AEAG
Σ min min Vp((3)
(10)
s.t. u -,
S0,i
Then (i) there exists positive K such that (a) B S Ka for all positive in neigh- borhood of zero. (ii) If. in addition, the function has a unique minimizer u over E. then the limit lim (3t (ox) 3 exists and equals u.
B
PROOFS
B. PROOF OF FLOSED-FORM EXPRESSION FOR BIAS AND THM.
Now we prove Thm. 1.
To begin with, define directional derivatives. For probability measure P the item set O, let B(P) be the unique optimal solution to the Hisenberg- program: 3(P) minge(0.1)" F(O ()),)())(()) Let P and Q be two measures. When exists, the directional derivative
is defined as
B(P + 1(Q - P)) 3(P)
d - P) lim t
Step 1. Show there exist K 0 and a 0 such that ||: - B1 I Ka for [0,āg]. And d0(s; - 8') = ,HH, P)
We apply lemma with F(B, () max, (,(3), Σ, b, log(3;), (B( B, - B (0, 1]", dP s d8, d(Q - P) = (s s') dA.
By SCS in the market My, the set E (defined in Eq. (10)) becomes the plane {u u, =, E 1+} where we recall T. = (i: 3 = 0}. By SMO Ao. we know the Lagrangian multiplier of the EG program Mo unique, and thus b(-) defined in Eq. (9) becomes b(u) = u uTo + + Hou. We
conclude the directional derivative ds 8¹) = lim (36 - -
where P = (1, < 1)) and 5, = VS(. 3 (s s')do do
Step 2. For all 0 small enough, it holds I, where recall Ia {i 8 1} and
T= (i: 87 0).
First we show I C I., i.e., (i: , = 1) C (i: R. = 1} for a small enough. This is saying, if constraint 8; < is strongly active in the EG program of Mo. then it is also strongly active
'A Ao iff Lo (Bo, A) and A, 0, and for E J Jr
14

---

Published as a conference paper at ICLR 2025
in the EG program of the market Ma. This holds by Lemma 2.2 from Shapiro (1988) and the SCS condition in My. Next, we show, for small a, Ia - by showing [n]] \ S [n]] las ie.,
{i: f7 < 1] S {i: P < 1}. This holds by ||B 8 Ka for all a S ar.
Step 3. For all
0 small enough, the directional derivative dB(ssa; s') limm (Ba-e / exists and a + (s 8 s') is continuous.
Applying lemma with dP- sa d0 and d(Q - P) (s- 5') d0 shows that the directional derivative exists for all a small enough. Now we find an expression for dB(s); - x'). By sMO in Mo. twice continuously differentiability of B - V f F(B,0)s 0)s do in neighborhood of f and the Lipschitzness result from step 1, we know 3 + V2 F(8, g)sa d6 is twice continuously differentiable at 8% which implies uniqueness of Lagrangian multiplier in the market Ma. for all small enough (Lemma 2.2 from Shapiro (1988)). And thus b(-) defined in Eq. (9) becomes b(u) = u 5, + 5 u. Next, the SCS condition in the market Mo implies SCS in Mr for all a small enough. The E (defined in Eq. (10)) becomes the plane {u : u, 0, E Ta)- The quadratic program miny b has a closed-form solution: = - (P H.. d' where Pa = ( 0,4 < 1)), 8,
f (s) s') dO. And we apply La I for all a small enough from step 2, and so Pa = P. We conclude
dB(sni 8 - s') =')=-(PH,PP))
for all a small enough.
Next, we show that the map a + P) is continuous for all a small enough. To see this, note H. 2 f F(0, f3, S., d0 = (1-a) ((, +0+ F(, d0 is continuous in a. The P matrix is fixed, and so a PH, P is continuous. Without loss of generality, suppose Ie = {i: 3 0} = [k] are the first k buyers. Then PH, P creates matrix with upper left k-by-k block equal to the upper left k-by-k block of H denoted H Ha,1e and zeros everywhere else. Then (PH,P) 1)ii is matrix with upper left block being (Ha, and zeros everywhere else. Since Ha is positive definite for all small a, the submatrix H- must also be positive definite. And so a + (Ha, is continuous, implying continuity of a + (PH, P)t, Finally, 6_ = fV (s - s') d0 is continuous in a.
Step 4. The desired claim: 3 3 ola). Note d33 (a) defined in Eq. (6) is exactly dB(s.i 8-s'). And so
3 g =², a a,3 sai 8) ) + o(a)) + aB s's) 8r
(g" a dB s')
addB(ssa: dB(so; s')) + ola)) o(a). where the last line uses continuity in a of the directional derivative. This completes the proof of Thm. 1.
B.2 PROOF OF THM.
Recall the estimator 3 = 33 - a -P H. to, and the debiased surrogate is 3 = 3% a(P, H, P.)t p d.. By results from Liao & Kroer (2023) know that SCS and SMO in the mar- ket Ma and 7/ = (1) imply 83 3 and (P.H.PP) & (PaHaPa) Finally, 8_ 3 8, holds by the law of large numbers. We complete the proof of Thm. 2.
B.3 PROOF OF THM.
8a and the debiased surrogate is 3 B Recall the estimator 3 = 33. - a(P. P.) o
a(Pa Ha.HaPa)
Pa) 6 Define
2, 3 - Br, a oP..H..P.) - d..) &. al (P.H.... PP).H...))
(11)
(12)
S. = - al (P H. P.y - H.. (3 5,a)
(13)
Then clearly
3 3- 2t E, + Gt.
15

---

Published as a conference paper at ICLR 2025
Step 1. We show vizi converges to normal distribution. By Liao & Kroer (2023), it holds
Vilag Ba)
1 Vi
de (0') E(da] 0p(1)
(14)
where - V.) is the influence function. Next, we define the likeli- hood ratio function g = (d(s 5 (() = a 1e (e (0). Then da VV(-B%)( d') = [ (3)8 (-)d0. Then by definition 6,, can be written as 8. = P:(g(-)V(((.B33) where P.h(-) = h((0))((0)) and are i.id, draws from sa. Also let 14h = vt(Pth- hss d0). By the decomposition,
Vi(ō.- -8))
(15)
-Vi -V ((((()V((.B)))) | (()) d0
(16)
-V P( / (())Vf(,,8))s d0 +M(0(VVf((08) vf(.
(17)
-V(o()VV[(( (1))
(18)
where the last line follows by stochastic equicontinuity argument as in Liao & Kroer (2023).
Finally, do (-) ag))) H P.) theorem.
Pa) /V is exactly the influence function di (-) defined in the
Step 2. Show & Op(nt).
We need a matrix perturbation result.
Lemma 2 (Theorem 2.2 from Stewart (1977)). Let A and B A A E be nonsingular square matrices, and |||||||0 < 1. Here ||.I] is the operator norm. Then
1
|B- 1 - A <
E
(19)
E A
By Liao & Kroer (2023), we know P(P.. = Pa) + 1. Without loss of generality, suppose I (i : B7 < 0} = [k] are the first buyers. Then (P, H...,.) creates matrix with upper left k-by-k block equal to the upper left k-by-k block of H., denoted Ho, and zeros everywhere left left k-by-k block being upper left b-by-k block Ha. denoted Ho.1-1. and zeros everywhere else. Now let A = Ha.1'f and Ha. It clear that || l2 S a||5_ || A-1 B-1 11 = =Op(ne by lemma 2.
else. Under the event (Pa }. the matrix H, Pa) is one with upper
Step 3. Show C: = (p( From previous derivation, 8, - 8, Op(1/Vt) and (P.H.P.))
H.) = (n) = (1). We conclude Ç: = (L) (L/V3)
We complete the proof of Thm. part
Now we prove part 2. The claim that H., Diag(b;/ ((xx,)² (B/, )2 follows from Liao et al. (2023). Next derive the asymptotic distribution. We only need to handle the & term defined as before. Under
the event { (P. = Pa}. we have & -Diag(ada, 189% ()bb=)((8)))) 3% 3 B3) By Liao & Kroer (2023) know Vi(g -B:) - Id (dT) [al)) (1). So by the delta method,
1
Vi(gg o .3% -3, o +3. 12ag(2
(d, E., E,(dd)) (1)
Vi
Summarizing,
1 Σ
Vis. D8, < <1),,1)
(d. EE(dal)) (1)
16

---

Published as a conference paper at ICLR 2025
Note that for i such that 8%, 1, the i-th entry of da will be almost surely, the indicator can be dropped in the diagonal matrix. Now let d2(0) = -Diag -Diag(2as (2.122, da + d1, where
we recall d1 is defined in Thm. 3. Then under the condition that Ea bidgap(333.)) 05 < 00, we have
ViB - 3*) 4 N ((0, Eal(d2 - Ea (dld2))(d2 - Ea (2))))))
B.4 PROOF OF THM. 4
Proof. Note
/ max mxxx(v((0))))s(().) / max((;((0)8))s)))) do
max ma (O)x) - -m)x(((()))))((0)))
x maxsup u: (0) 3| -Bc = ((a)
as &. O.
Let P.h(-) = h(0 and or are i.dd. draws from Sa. and Ph(-) = f hsa. Let g(-) = ds/ dsa
1-11g 10 Also let vuh vi(Pth f hs. d0). We change the measure and write REV = 5 [(, (0, = ((0,)3))(00) do. For the claim regarding asymptotic normality, note
1-a
Vi(REV - REV)
-V ())(())))))))))))))))))))) P)((,
-V -P)(, (V)()))))++((()) - PS(,3*))())
+ - -V)F-PP)(((.B300 -
= + =V(BB +
where the last line follows by the delta method and stochastic equicontinuity argument. We com- plete the proof the theorem.
C OMMITED MAINTEXT
C.I FORMAL DEFINITION OF FINITE FPPE
Definition 5 (Finite FPPE, Conitzer et al. (2022a)). Given (b, the finite FPPE,
FPPE(b, v. a, 7) the unique tuple (3.p) E [0, R. such that there exist x. E [0. 1]: (First- price) For all T, p" = max, 3:o For all and r, xz. > 0 implies B,'g = max; BLUY. (Supply and budget feasible) For all o o. < b; For all t, Ei
1. (Market clearing) For all p² > implies ET12 1. (No unnecessary pacing) For all o - b, implies 3, = 1.
D RELATED WORK
Pacing equilibrium. Pacing is budget management strategy where bids are scaled down by a mul- tiplicative factor order to smooth out and control spending. In the first-price setting, Borgs et al. (2007) study first price auctions with budget constraints in perturbed model, where the limit prices converge to those of an FPPE, Building on that work, Conitzer al. (2022a) introduce the FPPE framework to model autobidding in repeated auctions, and discover several properties of FPPE such as proofnessss monotonicity properties, and close relationship between FPPE and the quasilin- Fisher market equilibrium (Chen et al., 2007; Cole et al., 2017). Gao & Kroer (2022) propose an dnfinite-dimension variant of the quasilinear Fisher market, which lays the probability founda- tions of the current paper. Gao al. (2021) and Liao al. (2022) study online computation of the
17

---

Published as a conference paper at ICLR 2025
dufinite-dimnnsione Fisher market equilibrium, and utilize their methods to compute equilib- ria. In the second-price setting, a variety of models have been proposed for modeling the outcome of budget management: Balseiro al. (2015) study budget-management in second-prics auctions through a fluid mean-field approximation; Balseiro & Gur (2019) investigate adaptive pacing strat- egy from buyers' perspective in stochastic continuous setting; and Balseiro et al. (2021b) study several budget smoothing methods including multiplicative pacing in stochastic context. A second- price analogue of FPPE was explored by Conitzer et al. (2022b). Modeling statistical inference in the second-pris price setting is most likely harder, due to equilibrium multiplicity issues (Conitzer et al., 2022b), and hardness of even computing equilibria (Chen al., 2023).
Recently, a series of papers (c.g., Aggarwal et al. (2019); Deng et al. (2021); Balseiro et al. (2021a)) investigated an "autobidding" model for galuuemaximizingg bidders, who aim to maximize their value subject to budget or return-on-spend constraints, which may include buying individual ad impres- sions that have negative utility. Several extensions provide results for more general settings, such as the quality of equilibria for generalized rationality models for bidders Babaioff et al. (2021), non-uniform bidding strategies Deng et al. (2023), and liquid welfare guarantees when budget- constrained buyers fail to reach equilibrium Gaitonde al. (2023). Statistical inference for parallel A/B testing under such models represent an interesting future direction of work.
Interference in A/B Testing for Auction Markets. Experimentation under interference has been extensively studied in the past decade within the context of social networks Ugander et al. (2013); Aronow & Samii (2017); Eckles et al. (2017); Athey et al. (2018); Li & Wager (2022) and two- sided marketplaces Zigler& & Papadogeorgou (2018); Wager & Xu (2021). Under interference, the outcome of an experiment unit, either a user in social network or buyer in an online marketplace, can be affected by the treatment status of other units in the networks, violating the Stable Unit Treatment Values Assumption (SUTVA) assumption commonly employed in analyzing online A/B tests Rosenbaum (2007). To take into account this type of interference, novel designs and analysis approaches have been proposed, e.g., cluster randomization Ugander et (2013); Karrer et al. (2021), regression-basee estimators Chin (2019), and bipartite analysis Harshaw et al. (2021). A/B testing in the presence of seller-side budget constraints has been discussed by Basse et al. (2016) and Liu et al. (2021) where the budget-splitting design spans the full market. However, most of the above works do not consider the particular type of interference caused by competitive equilibrium effects. Recent work has also addressed equilibrium effects (Wager & Xu, 2021; Liao et al., 2023; Liao & Kroer, 2024). Most relevant to our results, Liao & Kroer (2023) consider A/B testing via budget splitting in Fisher markets and FPPE. However, none of these works consider the effects caused by parallel A/B tests that interfere with each other. Our method builds on top of the budget- splitting design; our approach additionally segments the full market into submarkets, where different submarkets receive different treatment. In the formal sections of our paper we assume that the submarket structure is known. In practice, the submarkets may not be known; Rolnick et al. (2019) and Viviano et al. (2023) present methods to identify submarkets through clustering approaches that aim to limit the interference between clusters.
E RESULTS FOR REVENUE
Algorithm 1: Debiasing procedure
Input. Budgets [b,,...,bi, ]. Values vf. First-price allocation xi E [0, 1). The observed equilibrium pacing multipliers G = =B [B,...,,,77 Proportion of bad items a. Desired confidence level (1 c) E (0, 1).
Estimate the Hessian matrix H., e.g. via H. = Diag(b:/(A2,))) or using finite differences.
2. Compute relevant quantities Pa. 8.. μ²" H."
3. Compute the debiased estimator 3 and covariance matrix Σ.
4. Output confidence region: 3 + (Xx,e/ (t).e// where x2 is the (1 -q-uuntelle of chi- chi-square distribution with degree n, and B is the unit ball in R
18

---

Published as a conference paper at ICLR 2025
The revenue in the good market is defined REV* (e0)) d8. We define the debiased revenue surrogate and its estimator
REV
max (())()))))))))
(20)
REV 1
eod (0) ua
(0
(21)
In words, we approximate REV* with the revenue generated in the market where buyers bid ac- cording the pacing profile 3 and then we estimate this via plug-in estimation on the subset of items that are good.
Theorem 4. Let the conditions from Thm. 1 and Thm. 3 hold. The debiased revenue removes first-order bias: RFV. = RFV + o(or). Assume that viB 33) has influence function d, ie., VIß - -3) oREV = Cov (dREV)) dREV()) = - (10gmo (*((3))) + E. ( (/0,,3" 1 d())- The proof is in App. B.4.
-(--)d(0)) + op(1), then V(RFVV REV S N (0, o <REV) where
Examples of influence functions compatible with Thm. 4 the influence functions in Thm.
We have two methods to construct Cls for the debiased revenue surrogate in Eq. (7). The first method is based on the asymptotic normality results in Thm. 4. In the second method, revenue CI is constructed based on a CI for Br. Suppose CI is some confidence region for 3 Then 1e (07) max; (0, B E ClI) serves a CI for revenue. Another way to utilize a pacing multiplier CI is based on the observation that revenue is monotone function of the pacing multiplier. Let 3: min / CI] and B, max (3, B ClI.. Then another natural
(potentially wider) CI for REV is IEEV, REV,, J, where
1
1
1e (07)
18 (0r)
REV
max e (0T )3:
REV
max 2 (07),,
(22)
F PARALLEL A/B TEST UNDER INTERFERENCE
In this section, we use the theory from the previous sections to formulate a statistical inference theory for parallel A/B tests under interference.
Consider auction markets in which we want to run parallel A/B tests. There are n buyers, each with budget b, corresponding to one unit supply of items, and an item set θ, partitioned into θ good and Orad. Let Ci,..., Cr partition of the buyers, .I,..., Ox be partition the good item set θ good. and $1.... be set of supply functions, supported on OI,..., Ox respectively. The k-th submarket consists of buyers in Ct. the item set O₂ and supply Let = 1 Σx 8g be the average mixture and be a supply supported on O
To model treatment application we introduce the potential value functions
(0) = (0, (..,,),..,,,lll (.(0,,)), }), u(1) = (1,(1,,,,.... ",,,,,))
item exposed to treatment w E 1). then value to buyer will be v;(w,
We extend the structure on Obad and + good introduced in Sec. 3.2 to A/B testing here. We assume these submarkets are separated. This is to model the ideal case where there no interference. A buyer i E Oş is only interested in items from the submarket he belongs to: wy 0) 0 for E C # kt and w E {0, 1}. An item is called a bad item if it has positive values for buyers from different submarkets. Formally, 0 E Obdd if there exist E {0, 1), i Ck. j E Ce. & K. so that U; (ww 8) > 0 and (w, f) > 0. A buyer from submarket has a (potentially) positive value only for items in O bad-
Step 1.
The treatment effects. Let A M(,, k) = F hec, (us(., w) Wiec, Sk, k-th limit submarket under treatment E {0, 1}, and let revenue be RFV; { (w). Then the treatment
., O be the
19

---

Published as a conference paper at ICLR 2025
effect in the submarket lo is T REV REV?(0). Equivalently, all K submarkets can be formed simultancously, M that buyer in C₁ has budget b;/ K bid for items in θg whose supply in M (w) * Sg. The k-th component of A (w) corresponds to M (w, k).
FPPE ( (w)), s, @) The scaling 1/K in budgets ensures
Step 2. The experiment. practical parallel A/B test scheme is as follows.
Step 2.1. Budget splitting. Decide on a budget split ratio No, #i satisfying To + #1 = 1, and split budgets accordingly. Choose A/B test size t, the total number of impressions in each submarket summed across treatments. Then, the budget of buyer for treatment O will be Toth, and T titb; for treatment This because normalized to correspond one unit of total supply.
Step 2.2. Observe markets with interference. Draw items/impressions from as' + (1 - a)s. Draw [Tot] items for each submarket under treatment 0 and [ for treatment 1. A total of tu [Ttt]] items are drawn for the whole market under treatment w. Since items come from the contaminated distribution, roughly small fraction of the bo items cause interference.
The data observed following the described A/B test scheme can be compactly represented by two finite FPPEs (one for each treatment) (Def. 5) (u) FPPEE tb, v(w), 1, where y θ consists of ta [Tt i..dd. draws from the mixture as + (1 - atss.
It can be shown that the observed market Ma(w) FPPE ( v(w), Sa. θ).
converges
Ma(w)
Step 3. Debiasing B. Following previous section, we can debias (a(w to approximate M (w).
Let ((w) be the debiased pacing multiplier.
Step Revenue estimator. Estimate REV & (w) by
tm Σ le, (0r) 7=1
max v; (0
Tw (1 att
iECz
Then cx REV - (REVv((0) and variance estimate can be obtained by either asymptotic normality, confidence region B, or bootstrap.
G HEURISTIC DERIVATION OF THE DEBIASED SURROGATE
We present a heuristic argument for Eq. (6) here; for a rigorous treatment we need a perturbation theory result which is given in lemma 1 in the appendix. Define the contaminated EG objective (Q( F(3, 0)sa, d0. Then H. = (²Ha( The map (3,a) a) + (3) has gradient -8,00) at (3,, at). Then we have the [VH(3"), F(8 (s' - s) dØ) and Hessian H.. 6a: -8,,
following quadratic approximation of H (
B (B-B:)
(1 [FB () d0)
VH.(
Ha) = Ha (B%)
+
[ B%
Ha
-a
B 3%
+ 2 -t
8 (-8T
-6
And so
B:_E = arg min (a)
arg min (H) + B
_E10,1
BE(0,1i"
+ B:) H (3 B%)
where we dropped terms that do not involve B.
Next we need two observations. For small enough, we have 3 3 if 3 = 1. Intuitively, buyers in the market M. with 8 to small changes in supply, and thus their pacing multipliers remain one. Next, we also note 3 + v H, (8%) 33 is constant the set (B : if 8a, 1} by complementary slackness. Then
have some nonzero leftover budget, which acts as a buffer
20

---

Published as a conference paper at ICLR 2025
17.5
REV. REV
15.0
8
Norma lized bias (%)
12.5
4-
10.0
3-
Normalzzer
7.5
2.5
0.1
0.5
Figure Normalized bias (in percent of true value) as a function fully synthetic experiments. 3 and REV are the debiased surrogates for pacing multiplier and revenue in the limit market with interference Ma.
further simplify and obtain
3
arg min A
c 2 3, -3%)
The right-hand side is just linearly constrained quadratic optimization problem, which admits closed-form solution 3% - c(P Pa)
H FULLY SYNTHETIC EXPERIMENTS
In the fully synthetic experiments, we simulate 10 buyers and 1000 good items in two submarkets, with a varying number of bad items (up 500) in order study the effect of the contamination parameter For each we randomly sample a budget from uniform distribution for each buyer, and compute 3 and REV" from the limit pure market Mo with uniformly distributed value function in each submarket. Fig shows the normalized bias curves function of a for the fully synthetic experiment, which similar that from semi-synthetic experiments Fig 3.
We also check the coverage of the proposed variance estimator in fully synthetic experiments. The coverage of [3 and revenue estimated in the same way in the semi-synthetic ones. It is shown in Fig that both Cls converges to the true value from the limit market with interfrence Ma, as the number of items goes infinity. Then, in Table 2, show that the bootstrap CI width is much smaller compared to that from Eq. (22). However, unlike in the semi-synthetic experiments, both CIs are over-covered, higher than the nominal level (95%). It is expected though that the CI from Eq. (22) is relatively large and hence conservative since it uses the min and max of each Bi, hence likely living outside of the confidence region (1-dimension ellipsoid) for f and leading to over-coverage. For the bootstrap CI, requires enough bootstrap samples to fully explore the whole boundary of the confidence region of B3, as well as an accurate estimate of the asymptotic distribution of B.. In our fully synthetic simulation, our hypothesis is that the over-coverage of the bootstrap CI mainly due to the over-estimation of the variance for 3.
21

---

Published as a conference paper at ICLR 2025
REV
coverage
CI width
CI width (b)
coverage
coverage (b)
100/1 100
0.993
0.132
0.043
1.0
1.0
200/ 200
0.994
0.141
0.059
1.0
1.0
300/1300
0.980
0.107
0.059
1.0
1.0
400/1400
0.983
0.098
0.064
1.0
500/1500
0.971
0.108
0.069
1.0
1.0
Table 2: Coverage of 3 and revenue estimates in fully synthetic experiments. All quantities are averaged over 100 simulations for each a. The coverage of 3 is averaged over all components of 3. For revenue estimates, the columns with "(b)" represent the quantities from the bootstrap CI, while the columns without "(b)" for the CI from Eq. (22). The CI widths normalized by the revenue from the limit market Ma. a is expressed as the ratio the number of bad items and the total items.
True value
analytical C
bootstrap CI
Revenue
500
1000 1500
2000
2500
3000
Number
Figure 6: Revenue confidence intervals function of the number of items in fully synthetic
experiments. The analytic CI comes from Eq. (22). The true value the debiased surrogates for revenue in the limit market with interference M...
22


Reviews:
[{"review_id": "zipMdO9qBG", "reviewer": "Reviewer_5WoS", "summary": "This paper considers A/B testing in online marketplaces, where interference arises because items can be recommended to advertisers in both the control and the treatment groups. The paper adopts the first-price pacing equilibrium (FPPE) framework from prior work [Conitzer et al., 2022], and analyzes how the equilibrium (i.e., pacing parameters \\beta and the total revenue) changes as a function of the level of contamination/interference. The paper proposes a first-order bias correction by Taylor expansion and proves consistency and asymptotic normality guarantees under certain regularity conditions. The paper conducts semi-synthetic experiments to demonstrate the effectiveness of the proposed estimators (where the distributions come from real data).", "questions": "1. The theoretical guarantees heavily rely on the parameter \\eta for the error in estimating the Hessian. Could the authors comment on the the rate \\eta for the proposed finite differencing Hessian estimator proposed in the paper? \n\n2.  Does the guarantee on the revenue in Theorem 4 generalize to estimating other quantities that are a functional of the parameters \\beta?\n\n3. Providing a discussion on limitations and addressing my minor comment #4 above would be helpful.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "Interference is a well-known, important, and practical problem in A/B testing of online marketplaces. The paper meaningfully extends prior work on pacing equilibria by considering how interference affects the results. The paper provides rigorous theoretical guarantees, complemented by preliminary semi-synthetic experiments.", "weaknesses": "While I find the formulation and assumptions proposed in the paper reasonable, I think the paper can benefit from a discussion on limitations. For example, a natural alternative model is not to partition items into a good set and a bad set, but instead assume an item can be \"good\" for some groups but \"bad\" (interference) for other groups. The paper may also discuss other types of interference that the proposed approach does not cover, etc.\n\n===\nAdditional minor comments:\n\n1. I think the claim in the abstract on demonstrating \"the effectiveness of this approach on real experiments on advertising markets at Meta\" is an overstatement. I think the paper should make it clear that only semi-synthetic experiments are performed.\n\n2. While I understand citing unpublished work is optional, I think the paper can benefit from citing the following paper:\nZhu, Cai, Zheng, and Si. \"Seller-Side Experiments under Interference Induced by Feedback Loops in Two-Sided Platforms\". arXiv, 2024.\n\n3.  I find the term \"buyer\" confusing, as \"buyers\" can be understood as users who make purchases or sellers who bid for ads. I prefer the terminology of advertisers vs. users introduced in other parts of the paper.\n\n4.  I don't understand details of the experiment in Fig 1. What're the axes? Are these ratios of the two experimental designs? How are two experiments paired? What is the guardrail metric? Also a 79% agreement compared to the 81.5% optimal agreement appears pretty good to me. Is the paper trying to address the remaining 2.5% of the cases?\n\n5.  In Paragraph L87-118, two graphs are introduced. One is a bipartite graph between advertisers and users, from which a graph for advertisers is derived for clustering (\"the edge weight between a pair of advertisers...\"). It would be helpful to clarify they are not the same graph.\n\n6.  Paragraph L54-63: \"Sec\" -> \"Fig\"", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "1TveI8J9WF", "reviewer": "Reviewer_Ms53", "summary": "This paper studies how to estimate market equilibria in each submarket when there is interference across submarkets. The authors consider the first-price pacing equilibrium (FPPE) model where each buyer uses a single pacing multiplier to shade values. For each submarket, some items outside would also attract the buyers in this submarket, which is unavoidable since we are unable to find completely isolated submarkets. From the modeling point of view, the supply is contaminated by another distribution. The authors first propose a debiased surrogate, which can approximate the limit FPPE in the uncontaminated market based on the limit FPPE in the contaminated market, and then prove that it has only a small error due to the removal of the first-order bias. Then, since the finit FPPE will converge to the limit FPPE in probability, the above estimator can be applied to the actual dynamic market with a slight modification. The authors also present two asymptotic normality results to further demonstrate the superiority of the estimator. Experiments on semi-synthetic data show that the proposed estimator is indeed less biased in terms of pacing multipliers and revenue.", "questions": "NA", "limitations": "", "rating": 5, "confidence": 3, "soundness": 3, "presentation": 1, "contribution": 2, "strengths": "1. I can understand that A/B test design is indeed a very important issue in the industry. Since buyers will use strategic behaviors to counter the platform's strategy or mechanism, a/b testing is the most effective way to test the platform's strategy/mechanism. However, the vanilla a/b test setup will bring about the issue of mutual influence between a/b groups, which may interfere with the experimental results.\n\n2. The theoretical proof in the paper is non-trivial and very solid. The authors have proved from many aspects that the estimator can indeed converge well to the limit FPPE in the uncontaminated market, which the platform hopes to observe.", "weaknesses": "Overall, this paper is hard to parse. The flow of this paper is not clear enough to assist readers well. The abstract and introduction of this paper both start with a/b testing, but the contribution part does not mention it. It was not until page 6 that I began to realize what the authors are doing. After reading the paper several times, I finally understood the authors' real contribution. \n\nAccording to my understanding, the real question is how to study the impact of different strategies/mechanisms on market FPPE. The method proposed by the authors is to find relatively isolated submarkets as a/b groups, and then construct an estimator to eliminate cross-market interference. As far as I know, the former is a method that has been adopted in the industry (of course, the issue of cross-market interference remains), while the latter is tailor-made for the FPPE model. Therefore, the focus of this paper's contribution should actually be on FPPE rather than ab test design. In fact, even if all ab test parts are removed, this will still be a complete paper that studies how to approximate the limit FPPE of an uncontaminated market using contaminated data.\n\nSince the estimator in this paper cannot be directly extended to other scenarios, I suggest that the authors should not emphasize the proposal of a novel ab test framework, but use the research on FPPE as the background to introduce challenges and solutions.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8Uw1fu3YpN", "reviewer": "Reviewer_pX7g", "summary": "The paper proposes an estimation technique for A/B test revenue by modeling budget interference between two experiments as a mixture of distributions.\n\nAt first, the paper is hard to follow for readers unfamiliar with the FPPE formalism, such as myself. Even though I am familiar with advertising systems and budget pacing mechanisms. A lot of symbols and concepts to grasp. However, once I familiarized myself with the relevant cited background, the explanation of modeling of budget interference using a mixture of distribution makes sense, and I believe it's an important concept that this paper introduces.\n\nThe experimental section is quite shallow, but I believe that since the paper is focused on rigorous theory, extensive experiments are less important in such a paper.\n\nHowever, there are many weaknesses of this paper, described below in the weaknesses section, that make me believe this paper requires additional work to make it ready for publication. The largest problem is, I believe, that the theoretical framework of an equilibrium does not appear to model real-world marketplaces with ever-changing pacing factors. However, the paper presents the technique as something applicable in practice, and no attempt is made to close this gap. More weaknesses appear in the weakness section.", "questions": "- is the decomposition into submarkets part of the contribution or not? It appears as something important, but it's in the introduction, rather than being in the main paper. Why? Please clarify this. If this is a contribution, it should be explicitly evident and not appear in the introduction. If it is not, this should be explicit in the paper, maybe with proper citations.\n- Please explain - why is the equilibrium framework applicable in practice, given the changing nature of pacing factors?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "- aims to solve an extremely important practical problem\n- Introduces an interesting theoretical framework of modeling \"budget interference\" using a mixture of distributions.\n- makes a good job explaining *why* such a modeling is reasonable.", "weaknesses": "- real budget pacing systems, from my experience, do not operate with constant pacing factors. But the FPPE theory assumes each buyer has one specific pacing factor defined by the equilibrium. There is no explanation in the paper for how this framework actually models the real world of dynamically changing pacing factors, and this limits the usefulness of the estimators in practice. If it is possible to apply the framework to such a dynamic system, there is no explanation in the paper for *how* to do it. So overall, there is no motivation explaining why the equilibrium framework is useful.\n- the notation is extremely hard to follow. Typically there is a difference between how vectors, scalars, sets, random variables, and matrices are denoted. For example, vectors by boldface, matrices as upcase-bold, and so on.. It requires a lot of mental effort to follow all the symbols and concepts in the paper, which makes it practically unreadable to audience unfamiliar with the FPPE framework\n- a paper aims to show us how to debias the revenue metrics in A/B tests, but it does *not* derive a debiased estimator for the revenue. only for the budget pacing factors. For such an important concept, I'd assume there should be at least a corollary for how the revenue estimator can be computed from the estimated pacing factors, and an explanation for why this estimator is also debiased (it's not obvious that a function of a de-biased estimator is also de-biased in the same sense).\n\n\n## Update\nMost of the weaknesses have been addressed by the rebuttal and the revised version.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)