Title: Fair Adaptive Experiments
Venue: NeurIPS 2023
Accepted: yes
Keywords: Adaptive Randomized Experiment; Adaptive Design; Causal Inference
Abstract: Randomized experiments have been the gold standard for assessing the effectiveness of a treatment, policy, or intervention, spanning various fields, including social sciences, biomedical studies, and e-commerce. The classical complete randomization approach assigns treatments based on a pre-specified probability and may lead to inefficient use of data. Adaptive experiments improve upon complete randomization by sequentially learning and updating treatment assignment probabilities using accrued evidence during the experiment. Hence, they can help achieve efficient data use and higher estimation efficiency. However, their application can also raise fairness and equity concerns, as assignment probabilities may vary drastically across groups of participants. Furthermore, when treatment is expected to be extremely beneficial to certain groups of participants, it is more appropriate to expose many of these participants to favorable treatment. In response to these challenges, we propose a fair adaptive experiment strategy that simultaneously enhances data use efficiency, achieves an ``envy-free'' treatment assignment guarantee, and improves the overall welfare of participants. An important feature of our proposed strategy is that we do not impose parametric modeling assumptions on the outcome variables, making it more versatile and applicable to a wider array of applications. Through our theoretical investigation, we characterize the convergence rate of the estimated treatment effects and the associated standard deviations at the group level and further prove that our adaptive treatment assignment algorithm, despite not having a closed-form expression, approaches the optimal allocation rule asymptotically. Our proof strategy takes into account the fact that the allocation decisions in our design depend on sequentially accumulated data, which poses a significant challenge in characterizing the properties and conducting statistical inference of our method. We further provide simulation evidence and two synthetic data studies to showcase the performance of our fair adaptive experiment strategy.

Fair Adaptive Experiments

Abstract

Randomized experiments have been the gold standard for assessing the effectiveness of a treatment, policy, or intervention, spanning various fields, including social sciences, biomedical studies, and e-commerce. The classical complete randomization approach assigns treatments based on a pre-specified probability and may lead to inefficient use of data. Adaptive experiments improve upon complete randomization by sequentially learning and updating treatment assignment probabilities using accrued evidence during the experiment. Hence, they can help achieve efficient data use and higher estimation efficiency. However, their application can also raise fairness and equity concerns, as assignment probabilities may vary drastically across groups of participants. Furthermore, when treatment is expected to be extremely beneficial to certain groups of participants, it is more appropriate to expose many of these participants to favorable treatment. In response to these challenges, we propose a fair adaptive experiment strategy that simultaneously enhances data use efficiency, achieves an 'envy-free' treatment assignment guarantee, and improves the overall welfare of participants. An important feature of our proposed strategy is that we do not impose parametric modeling assumptions on the outcome variables, making it more versatile and applicable to a wider array of applications. Through our theoretical investigation, we characterize the convergence rate of the estimated treatment effects and the associated standard deviations at the group level and further prove that our adaptive treatment assignment algorithm, despite not having a closed-form expression, approaches the optimal allocation rule asymptotically. Our proof strategy takes into account the fact that the allocation decisions in our design depend on sequentially accumulated data, which poses a significant challenge in characterizing the properties and conducting statistical inference of our method. We further provide simulation evidence and two synthetic data studies to showcase the performance of our fair adaptive experiment strategy.

1 Introduction

1.1 Motivation and contribution

Randomized experiments are considered gold standards for evaluating the effectiveness of public policies, medical treatments, or advertising strategies [32, 34, 35]. They involve randomly assigning

participants to different treatment groups, allowing for rigorous causal conclusions and robust evidence for decision-making and policy implementation. However, classical randomized experiments, which maintain fixed treatment assignment probabilities, often do not optimize data utilization. This limitation is problematic due to the high costs associated with conducting such experiments. Consequently, maximizing information gain is crucial, but classical randomized experiments do not prioritize this objective [25, 46].

Compared to classical randomized experiments, adaptive experiments provide enhanced information gain and improved statistical efficiency. As a result, adaptive experiments have gained popularity in diverse domains such as field experiments, online A/B testing, and clinical trials [26, 56, 63, 64]. The information gain of adaptive experiments stems from their ability to iteratively adjust treatment allocations based on refined knowledge obtained from accumulated data during the experiment. This iterative process often favors the treatment arm that offers more informative or beneficial outcomes, maximizing the information gained from each participant and optimizing the overall statistical efficiency of the experiment [45]. Moreover, adaptive experiments, thanks to their adept utilization of data resources, often exhibit greater statistical testing power when practitioners employ the collected data to assess the null hypothesis of zero treatment effect upon experiment completion [14].

Despite their appealing benefits in improving data use efficiency and boosting statistical power, adaptive experiments potentially bring fairness concerns in applications. This issue is neither sufficiently explored nor fully addressed in the existing literature. Below, we shall concretely discuss the fairness concerns under a scenario where the study population can be divided into distinct groups based on demographic information or biomarkers - a scenario frequently encountered in field experiments or clinical trials. The first fairness concern in adaptive experiments arises when there are significant disparities in treatment allocations among different participant groups [14]. This is because when the treatment is potentially beneficial, it is crucial to ensure a fair chance for each group to receive beneficial treatment. Similarly, it is important to avoid disproportionately burdening any specific group with unfavorable treatment. However, conventional adaptive experiments prioritizing efficiency gains may inadvertently result in unfair treatment allocations. For example, if the outcome of a particular group of participants exhibits a higher variance in response to the treatment, more participants in the group will be allocated to the treatment arm. Consequently, this group would have a significantly higher treatment assignment probability than the others, regardless of the sign and magnitude of the treatment effect. This may lead to an unfair allocation of treatments among participants. Completely randomized experiments with fixed one-half treatment assignments would avoid this challenge, but they suffer from information loss. The second fairness concern arises when the adaptive treatment allocation does not adequately account for the overall welfare of experimental participants. This is crucial as a fair experiment is expected to not only assign a large proportion of participants to a beneficial treatment arm but also assign a small proportion of participants to a harmful treatment to avoid adverse effects.

There are evident challenges in addressing fairness concerns while optimizing information gain in adaptive experiments due to the potential trade-off among fairness concerns, welfare improvement, and information gain. For example, if most of the participants are assigned to the beneficial treatment to maximize welfare, then there would be insufficient sample size in the control arm, resulting in imprecisely estimated treatment effect and hence reduced statistical efficiency for conducting inference. To overcome these challenges, we propose a fair adaptive experimental design strategy that balances competing objectives: improving fairness, enhancing overall welfare, and gaining efficiency. By rigorously demonstrating the effectiveness of our approach and providing statistical guarantees, we offer a practical solution that is both grounded in theory and reconciles fairness concerns with the requirement for robust information gain in adaptive experiments. Our contributions can be summarized as follows:

First , in comparison to existing adaptive experiments, our proposed strategy integrates fairness and welfare considerations while optimizing information gain. As a result, the treatment allocation probability generated by our method avoids extreme values and exhibits minimal necessary variations across different groups. These desirable characteristics are supported by our simulation studies and empirical illustration using synthetic data. It is important to note that due to the additional constraints of welfare and fairness, the optimal treatment allocation probability does not have a closed-form expression, which brings additional technical challenges to studying the theoretical properties of our design. Despite this challenge, we demonstrate that the constructed treatment allocation rule for each group of our design converges to its oracle counterpart (Theorem 2). This implies that our proposed

designs in Section 2.3, despite not relying on prior knowledge about the underlying data distribution before the start of the experiment, can allocate treatments in a similar manner to a scenario where perfect knowledge of the data distribution is available.

Second , we do not impose any specific parametric modeling assumptions on the outcomes beyond mild moment conditions. We instead estimate the mean and variance of potential outcomes at the group level, which are further incorporated into our algorithm. The nonparametric nature of our procedure delivers an efficient and accurate estimation of the average treatment effect. As an important theoretical contribution, we prove that those group-level estimates are asymptotically consistent (Theorem 1).

Third , our theoretical framework addresses the challenges and complexities associated with adaptive experiment design, where data are sequentially accumulated, and treatment allocation decisions are adaptively revised, resulting in non-independently and non-identically distributed data. By leveraging the martingale methods, we demonstrate that the estimate of the average treatment effect is consistent and asymptotically normally distributed (Theorems 1 and 3). An important methodological and practical innovation of our framework is that it does not require the number of participants enrolled in the first stage to be proportional to the overall sample size. This flexibility allows researchers to allocate more participants in later stages of the experiment, enabling a truly adaptive approach to experiment design and implementation. This innovation has significant implications for the methodology and practical application of adaptive experiments.

1.2 Related literature

Our proposed fair adaptive experiment strategy has a natural connection with the response adaptive randomization (RAR) design literature. The early work develops the randomized play-the-winner rule in clinical trial settings based on urn models [46, 48, 60]. Theoretical properties of urn models are investigated in [6] and [29]. Another conventional response adaptive design is the doubly adaptive biased coin (DBCD) design [15, 26, 27, 53]. However, to our best knowledge, many existing works on response adaptive designs do not take fair treatment allocations into account [24, 47]. An insightful work in [33] proposes an efficient RAR design to minimize the variance of the average treatment effects and discusses some directions for fair experimental design. Compared with [33], our method does not require estimating outcome models, which can be challenging in the presence of correlated data in RAR designs. In addition, our design centers around 'group' fairness, aiming to enhance participants' well-being while avoiding extra fairness complications among distinct individuals. Furthermore, RAR designs that further incorporate covariate information are known as covariateadjusted response adaptive (CARA) designs [7, 9, 37, 49, 57, 71, 72]. Some early work proposes to balance covariates based on the biased coin design [44, 69]. Later work considers CARA designs that account for both efficiency and ethics [28] and extends the CARA design framework to incorporate nonparametric estimates of the conditional response function [1]. It is worth mentioning that another strand of literature focuses on ethical designs using Bayesian frameworks. Some recent work proposes to use the Gittins index to improve participants' welfare [55, 58]. A later work develops a Bayesian ethical design to further improve statistical power [62]. Some other ethical designs are discussed in [18, 52, 68].

Our manuscript also relates to the literature on semiparametric efficiency and treatment effect estimation [20, 42, 54]. Our algorithm adaptively allocates participants to treatment and control arms, with the aim of not only minimizing the variance of the estimated average treatment effect but also incorporating constraints on fairness and welfare. There is also a large literature on efficient estimation of treatment effects and, more broadly, on estimation and statistical inference in semiparametric models. See, for example, [8, 10, 11, 12, 16, 22, 38, 39, 59, 61] and references therein. Our algorithm takes the group structure as given. Another strand of literature studies stratified randomization. Some recent contributions in this area include [5, 51].

Lastly, our proposed design is connected to the multi-armed bandit (MAB) literature. [50] studies the trade-off between regret and statistical estimation efficiency by formulating a minimax multiobjective optimization problem and proposing an effective Pareto optimal MAB experiment. They provide insightful theoretical results on the sufficient and necessary conditions for the Pareto optimal solutions. Our procedure attains the minimax lower bound for fair experiment design problems. Our work has a different focus on uncovering the underlying causal effect by providing an adaptive procedure for efficient treatment effect estimation while incorporating fairness and welfare consid-

erations. Furthermore, in the theoretical investigations, we focus on the asymptotic normality of the proposed estimator and its variance estimator, which enables valid statistical inference. Our work broadly connects with fair contextual bandit literature [13, 17, 19, 30, 40, 43]. [65] and [66] propose algorithms under subpopulation fairness and equity requirements for the tasks of best arm identification and ranking and selection. The work in [31] characterizes fairness under the contextual bandit setting by bridging the fair contextual bandit problems with 'Knows What It Knowns' learning. While [31] defines fairness metric on the individual level, we focus on group-level fairness and further incorporate a welfare constraint.

2 Fair adaptive experiment

2.1 Problem formulation and notation

In this section, we formalize our adaptive experiment framework and introduce necessary notations. In adaptive experiments, participants are sequentially enrolled across T stages. We denote the total number of enrolled participants as N = ∑ T t =1 n t , where n t is the number of participants in Stage t , t = 1 , . . . , T . In line with the existing literature [24, 25, 28], we assume T →∞ , and n t is small relative to the overall sample size N , meaning that we have many opportunities to revise the treatment allocation rule during the experiment (see Assumption 3 below). At Stage t , we denote participant i 's treatment assignment status as D it ∈ { 0 , 1 } , i = 1 , . . . , n t , with D it = 1 being the treatment arm and D it = 0 being the control arm. Denote participant i 's covariate information as X it ∈ R p and the observed outcome as Y it ∈ R .

Next, we quantify causal effects under the Neyman-Rubin potential outcomes framework. Define Y it ( d ) as the potential outcome we would have observed if participant i receives treatment d at Stage t , d ∈ { 0 , 1 } . The observed outcome can be written as

In accordance with classical adaptive experiments literature, we assume that the outcomes are observed without delay, and their underlying distributions do not shift over time [25]. The average treatment effect (ATE) is the mean difference between the two potential outcomes:

In our proposed fair adaptive experiment strategy, to protect the participant's welfare (more discussions in Section 2.2), we also consider the group-level treatment effects. We assume the study population can be partitioned based on demographics or biomarkers, which is frequently seen in clinical settings or social science studies [3, 36, 67]. More concretely, by dividing the sample space X of the covariate X it into m non-overlapping regions, denoted as {S j } j m =1 , we define the treatment effect in each group as

We further denote the total number of participants enrolled in the group j as N j = ∑ T t =1 n tj .

In adaptive experiments, as we aim to adaptively revise the treatment assignment probabilities based on the evidence accrued during the experiment to meet our fairness and efficiency goals, we define treatment assignment probability (or propensity scores) for participants in groups j at stage T as

The goal of our experiment is to dynamically revise e tj for efficiency improvement, fairness guarantee, and welfare enhancement.

2.2 Design objective in an oracle setting

Classical adaptive experiments, aimed at reducing variance (or, equivalently, efficiency improvement), often assign treatment using Neyman allocation for participants in each group, that is

where σ 2 j ( d ) = V [ Y it ( d ) | X it ∈ S j ] , d ∈ { 0 , 1 } , denotes the variance of the potential outcome under treatment arm d in group j . Although Neyman allocation improves the estimation efficiency of the ATE, it brings two critical fairness concerns. First , different groups of participants may have substantially different probabilities of receiving treatments. The form of Eq (5) implies that the treatment assignment probabilities solely rely on group-level variances under different arms. More specifically, the group of participants with a larger variance under the treatment arm will have a higher probability of being treated, which may lead to disproportionate treatment allocations across different groups. Second , some participants' welfare could be harmed under the adaptive experiment strategy in Eq (5). To see this, assume a group of participants exhibits a large variance yet rather negative responses under the treatment arm. However, more participants in this group will be assigned to the treatment arm to improve the estimation efficiency of ATE despite the impairment of those participants' welfare.

To address fairness concerns and facilitate the introduction of our experimental goals, we begin with an infeasible 'oracle' setting, where we possess knowledge of the true underlying data distribution before the experiment begins. Since adaptive experiments naturally allow for sequential learning of unknown parameters and adjustment of treatment allocations during the experiment, we will present our adaptive experimental design strategy in the following section (Section 2.3), which attains the same theoretical guarantee for estimating the ATE as in the oracle setting (see Theorems 2 and 3 for justification).

In the oracle setting, given we have perfect knowledge of the underlying data distribution (thus τ j and σ 2 j ( d ) are known to us), our goal is to find optimal treatment allocations e ∗ = ( e ∗ 1 , . . . , e ∗ m ) ⊺ that solve the following optimization problem:

̸

where c 1 ∈ (0 , 1) and c 2 ∈ (0 , 1 / 2) . Here, the objective function captures the goal of improving information gain from study participants, which is formalized as minimizing the asymptotic variance of the inverse probability weighting estimator of the ATE (c.f. Theorem 3). The 'feasibility' constraint restricts that the treatment assignment probability in each group is bounded away from 0 and 1 by a positive constant c 1 .

To ensure fair treatment assignment and mitigate significant disparities in treatment allocations among participant groups, we introduce the 'envy-freeness' constraint. This constraint limits the disparity in treatment assignment probabilities across different groups in an acceptable pre-specified range. The concept of 'envy-freeness' originates from game theory literature and ensures that agents are content with their allocated resources without envying their peers [2, 4, 23, 41]. By incorporating this envy-freeness constraint, we address the first fairness concern and promote equitable treatment allocation.

To enhance the overall welfare of experiment participants, we introduce the 'welfare' constraint. This constraint ensures that a group of participants is more likely to receive the treatment if their treatment effects are positive and less likely to receive the treatment otherwise. Specifically, when the group-level treatment effect τ j ≥ 0 , indicating that group j benefits from the treatment, we want the treatment assignment probability e j to be larger than 1 2 . The welfare constraint achieves this by ensuring that the sign of log( e j 1 -e j ) aligns with the sign of τ j . When incorporating the welfare constraint, we effectively address the second fairness concern by providing more treatment to beneficial groups.

2.3 Learning oracle strategy in adaptive experiments

In this section, we present our fair adaptive experimental design strategy for realistic scenarios where we lack prior knowledge about the underlying data distribution, and our approach achieves the same desirable properties as in the oracle setting (refer to Section 3 for justification).

We present our proposed fair adaptive experiment strategy in Algorithm 1.

Algorithm 1 Fair adaptive experiment

Stage 1 (Initialization) :

1: Enroll n 1 participants, and assign treatments in group j according to e 1 j = 1 2 ;

2: Compute ˆ τ 1 j , ˆ σ 2 1 j ( d ) , and ˆ p 1 j as in Eq (6). Also, see the Supplementary Materials. Stage t (Fully-adaptive experiment) :

3: for t → 2 to T do

4: With ˆ τ t -1 ,j , ˆ σ 2 t -1 ,j ( d ) , and ˆ p t -1 ,j , solve Problem B to find ˆ e ∗ tj ;

5: Enroll n t participants and assign treatment with probability ˆ e ∗ tj ;

6: Update ˆ τ tj , ˆ σ 2 tj ( d ) , and ˆ p tj as in Eq (6).

7: end for

Stage T (Inference):

8: Compute ˆ v 2 j and ˆ v 2 as in Eq (7).

9: Construct two-sided confidence intervals for ˆ τ j and τ as in Eq (8).

Concretely, in Stage 1 (line 1-2), because we have no prior knowledge about the unknown parameters, we obtain initial estimates of the group-level treatment effect ˆ τ 1 j and the associated variances ˆ σ 2 1 j ( d ) . Then, in Stage t (line 4-6), our design solves the following sample analog of Problem A at each experimental stage:

Problem B

Here, we define

One important feature of Problem B is that we introduce a relaxation of the welfare constraint through δ ( N t -1 ) . From a theoretical perspective, the function δ ( · ) should be strictly positive, and satisfies lim x →∞ δ ( x ) = 0 and lim x →∞ √ xδ ( x ) = ∞ . For implementation, we recommend using δ ( N t -1 ) = √ log( N t -1 ) /N t -1 . It is also possible to incorporate the standard error of the estimated

̸

subgroup treatment effects into the welfare constraint, which motives the more sophisticated version:

where ˆ v t -1 ,j is the adaptively estimated standard deviation defined in the Supplementary Materials. Scaling the welfare constraint by ˆ v t -1 ,j , a measure of randomness in ˆ τ t -1 ,j , delivers a more clear interpretation: the above now corresponds to a t-test for the subgroup treatment effect τ j with a diverging threshold, and the specific choice stems from Schwarz's minimum BIC rule.

After the final Stage T , we have the group-level treatment effect estimates ˆ τ j := ˆ τ Tj , the variance estimates ˆ σ 2 j ( d ) := ˆ σ 2 Tj ( d ) , and the group proportions ˆ p j := ˆ p Tj (that is, we omit the time index T for estimates obtained after the completion of the experiment). Together with valid standard errors, one can conduct statistical inference at some pre-specified level α . To be precise, the estimated ATE is ˆ τ = ∑ m j =1 ˆ p j ˆ τ j , and we define the following

Lastly, we can construct the two-sided confidence intervals for ˆ τ j and ˆ τ as

3 Theoretical investigations

In this section, we investigate the theoretical properties of our proposed fair adaptive experiment strategy, and we demonstrate that our approach achieves the same desirable properties as in the oracle setting. We work under the following assumptions:

Assumption 1 For t = 1 , . . . , T and i = 1 , . . . , n t , the covariates and the potential outcomes, ( X it , Y it (0) , Y it (1)) , are independently and identically distributed; the potential outcomes have bounded fourth moments: E [ | Y it ( d ) | 4 ] < ∞ for d = 0 , 1 .

Assumption 2 The group proportions p j are bounded away from 0: there exists δ > 0 such that p j ≥ δ for all j = 1 , 2 , . . . , m .

Assumption 3 The sample size for each stage, n t , are of the same order: there exists c ≥ 1 such that N cT ≤ n t ≤ cN T .

Assumption 1 imposes a mild moment condition on the potential outcomes over different stages. Assumption 2 assumes that the proportion of each group is nonzero. Assumption 3 requires that the sample size in each stage are of the same order. We remark that this assumption can be easily relaxed.

Theorem 1 (Consistent treatment effect and variance estimation) Assume Assumptions 1-3 hold. Then, the estimated group-level treatment effects and the associated variances are consistent:

where N t = ∑ t s =1 n s . As a result, after stage T ,

Theorem 1 shows consistency of the group-level treatment effects and the associated variance estimators. This further implies the consistency of the average treatment effect estimator. The proof of Theorem 1 leverages the martingale methods [21]. Building on Theorem 1, we can establish the theoretical properties of the actual treatment allocation under our design strategy.

Theorem 2 (Convergence of actual treatment allocation) Assume Assumptions 1-3 hold. Then the actual treatment allocation, defined in Eq (7) , converges to the oracle allocation: ˆ e j -e ∗ j = o p (1) .

Theorem 2 is a key result. It suggests that despite having minimum knowledge regarding the distribution of the potential outcomes at the experiment's outset, we are able to adaptively revise the treatment allocation probability using the accrued information, and the actual treatment probabilities under our proposed fair adaptive experiment strategy converge to their oracle counterparts. Building on Theorem 1 and Theorem 2, we are able to establish the asymptotic normality results of our proposed estimators and show that the standard errors are valid.

Theorem 3 (Asymptotic normality and valid standard errors) Assume Assumptions 1-3 hold. Then, the estimated group-level treatment effects and the estimated ATE are asymptotically normally distributed:

where

In addition, the standard errors in Eq (7) are consistent: ˆ v 2 j -v 2 j ( e ∗ j ) = o p (1) and ˆ v 2 -v 2 ( e ∗ ) = o p (1) .

Theorem 3 shows the asymptotic normality results of the estimated treatment effects under our proposed adaptive experiment strategy. In addition, Theorem 3 verifies that the constructed confidence intervals in Eq (8) attain the nominal coverage thanks to the consistency of standard errors. The proof of Theorem 3 relies on the convergence of the actual treatment allocation in Theorem 2 and and the martingale central limit theorem [21].

4 Simulation evidence

In this section, we evaluate the performance of our proposed fair adaptive experiment strategy through simulation studies. We summarize the takeaways from the simulation studies as follows. First, our proposed fair adaptive experiment strategy achieves higher estimation efficiency than the complete randomization design. Second, compared to a classical adaptive experiment strategy, our method avoids disproportionate treatment assignment probabilities across different groups of participants and accounts for participants' welfare.

Our simulation design generates the potential outcomes under two data-generating processes. DGP 1 : Continuous potential outcomes Y i ( d ) | X i ∈ S j ∼ N ( µ d,j , σ d,j ) , where µ 1 = (1 , 4) ⊺ , µ 0 = (4 , 2) ⊺ , σ 1 = (2 . 5 , 1 . 2) ⊺ , and σ 0 = (1 . 5 , 3 . 5) ⊺ . The group proportions are p = (0 . 5 , 0 . 5) ⊺ . The group-level treatment effects are τ = ( -3 , 2) ⊺ . DGP 2 : Binary potential outcome: Y i ( d ) | X i ∈ S j ∼ Bernoulli ( µ d,j ) , where µ 1 = (0 . 6 , 0 . 2 , 0 . 3 , 0 . 4 , 0 . 1) ⊺ , µ 0 = (0 . 1 , 0 . 5 , 0 . 3 , 0 . 4 , 0 . 6) ⊺ . The group proportions are p = (0 . 15 , 0 . 25 , 0 . 2 , 0 . 25 , 0 . 15) ⊺ . To mimic our first case study (in Supplementary Materials), consider the log relative risk as the parameter of interest: log E [ Y (1)] -log E [ Y (0)] . The group-level treatment effects are τ = (1 . 79 , -0 . 92 , 0 , 0 , -1 . 79) ⊺ .

We compare three experiment strategies for treatment assignment: (1) our proposed fair adaptive experiment strategy, (2) the doubly adaptive biased coin design (DBCD) [70], and (3) the complete randomization design, which fixes the treatment allocation probability to be 1 / 2 throughout the experiments. To mimic the fully adaptive experiments, we fix stage 1 sample size at n 1 = 40 and n t = 1 for t = 2 , . . . , T , where the total number of stages ranges from T ∈ { 40 , . . . , 400 } . We evaluate the performance of each strategy from two angles. First, we compare the standard deviation of the ATE estimates to evaluate the estimation efficiency. Second, we compare the fraction of participants assigned to the treatment arm in each group to evaluate the fairness in treatment allocation. The simulation results are summarized in Figure 1.

We first focus on (A) and (B), which correspond to DGP 1. Panel (A) depicts standard deviations of the treatment effect estimates under the three experiment designs. It clearly demonstrates that our proposed method achieves higher estimation efficiency compared to complete randomization.

(A) Standard Deviation

(B)% Allocated to Treatment Arm

Complete randomization

Doubly biased coin design

Proposed fair adaptive experiment

0.60

Standard Deviation

0.40

Allocation ratio

0.40-

0.30

0.20

0.00

100

200

300

400

1

2

1

2

2

SampleSize

Groups

(C) Standard Deviation

(D) % Allocated to Treatment Arm

0.25

Complete randomization

Doubly biased coin design

Proposed fair adaptive experiment

0.60

0.20

Standard Deviation

H

0.20

3

0.20

0.15

0.10

0.00

100

200

300

400

1

2

3

4

5

1

2

3

4

5

1

2

3

4

Sample Size

Groups

AllocationCompleterandomization

Doublybiased coin design

★F

Proposed fair adaptive experiment

Panel (B) shows the treatment assignment probabilities produced by the three experiment design strategies. Not surprisingly, complete randomization allocates 50% participants to the treatment arm regardless of their group status. On the other hand, the DBCD design may produce extreme treatment allocations. In addition, participants in different groups may receive drastically different treatment allocations, which can raise fairness concerns. Encouragingly, our approach generates treatment assignment probabilities that not only are closer to 50% (i.e., less extreme) but also exhibit less variation across groups. Panel (C) and (D) summarize the simulation evidence for DGP 2 in which the outcome variable is binary. A similar pattern emerges: our fair adaptive experiment design approach improves upon complete randomization, delivering more precise treatment effect estimates. It also accounts for fairness and participants' welfare in assigning treatments.

The simulation results demonstrate the clear trade-off between fairness/welfare and statistical efficiency by adopting our proposed fair adaptive experiment strategy. Although it involves a minor sacrifice in estimation efficiency when contrasted with the DBCD design, our approach delivers more fair treatment allocations and safeguards participant well-being. As our proposed method does not restrict each group to have exactly the same treatment assignment probabilities as in the complete randomization design, it improves the estimation efficiency of ATE. We provide additional simulation results and synthetic data analyses in the Supplementary Materials.

5 Discussion

In this work, we propose an adaptive experimental design framework to simultaneously improve statistical estimation efficiency and fairness in treatment allocation, while also safeguarding participants' welfare. One practical limitation of the proposed design is that its objective mainly aligns with the experimenter's interests in estimating the effect of a treatment, as opposed to the interests of the enrolled participants. This aspect offers opportunities for future research exploration.

References

Giacomo Aletti, Andrea Ghiglietti, and William F Rosenberger. Nonparametric covariateadjusted response-adaptive design based on a functional urn model. The Annals of Statistics , 46(6B):3838-3866, 2018.

Christian Arnsperger. Envy-freeness and distributive justice. Journal of Economic Surveys , 8(2):155-186, 1994.

Susan F Assmann, Stuart J Pocock, Laura E Enos, and Linda E Kasten. Subgroup analysis and other (mis) uses of baseline data in clinical trials. The Lancet , 355(9209):1064-1069, 2000.

Haris Aziz, Bo Li, Hervé Moulin, and Xiaowei Wu. Algorithmic fair allocation of indivisible items: A survey and new questions. ACM SIGecom Exchanges , 20(1):24-40, 2022.

Yuehao Bai. Optimality of matched-pair designs in randomized controlled trials. American Economic Review , 112(12):3911-40, 2022.

Zhi-Dong Bai and Feifang Hu. Asymptotics in randomized urn models. The Annals of Applied Probability , 15(1B):914-940, 2005.

Uttam Bandyopadhyay and Atanu Biswas. Allocation by randomized play-the-winner rule in the presence of prognostic factors. Sankhy¯ a B , pages 397-412, 1999.

Alexandre Belloni, Victor Chernozhukov, Iván Fernández-Val, and Christian Hansen. Program evaluation and causal inference with high-dimensional data. Econometrica , 85(1):233-298, 2017.

Federico A Bugni, Ivan A Canay, and Azeem M Shaikh. Inference under covariate-adaptive randomization with multiple treatments. Quantitative Economics , 10(4):1747-1785, 2019.

Matias D. Cattaneo. Efficient semiparametric estimation of multi-valued treatment effects under ignorability. Journal of Econometrics , 155(2):138-154, 2010.

Matias D Cattaneo, Michael Jansson, and Xinwei Ma. Two-step estimation and inference with possibly many included covariates. The Review of Economic Studies , 86(3):1095-1122, 2019.

Xiaohong Chen, Oliver Linton, and Ingrid Van Keilegom. Estimation of semiparametric models when the criterion function is not smooth. Econometrica , 71(5):1591-1608, 2003.

Yifang Chen, Alex Cuellar, Haipeng Luo, Jignesh Modi, Heramb Nemlekar, and Stefanos Nikolaidis. Fair contextual multi-armed bandits: Theory and experiments. In Proceedings of the 36th Conference on Uncertainty in Artificial Intelligence (UAI) , volume 124, pages 181-190. PMLR, 2020.

Isabel Chien, Nina Deliu, Richard Turner, Adrian Weller, Sofia Villar, and Niki Kilbertus. Multi-disciplinary fairness considerations in machine learning for clinical trials. In Proceedings of the 2022 ACM Conference on Fairness, Accountability, and Transparency , pages 906-924, 2022.

Jeffrey R Eisele. The doubly adaptive biased coin design for sequential clinical trials. Journal of Statistical Planning and Inference , 38(2):249-261, 1994.

Max H Farrell. Robust inference on average treatment effects with possibly more covariates than observations. Journal of Econometrics , 189(1):1-23, 2015.

Stephen Gillen, Christopher Jung, Michael Kearns, and Aaron Roth. Online learning with an unknown fairness metric. In Proceedings of the 32nd International Conference on Neural Information Processing Systems , volume 31, pages 2605-2614, 2018.

Alessandra Giovagnoli. The Bayesian design of adaptive clinical trials. International Journal of Environmental Research and Public Health , 18(2):530, 2021.

Riccardo Grazzi, Arya Akhavan, John IF Falk, Leonardo Cella, and Massimiliano Pontil. Gairness in learning: Classic. In Proceedings of the 36th Conference on Neural Information Processing Systems , volume 35, pages 24392-24404, 2022.

Jinyong Hahn. On the role of the propensity score in efficient semiparametric estimation of average treatment effects. Econometrica , 66(2):315-331, 1998.

Peter Hall and Christopher C Heyde. Martingale Limit Theory and Its Application . Academic Press, 2014.

Keisuke Hirano, Guido W Imbens, and Geert Ridder. Efficient estimation of average treatment effects using the estimated propensity score. Econometrica , 71(4):1161-1189, 2003.

Safwan Hossain, Andjela Mladenovic, and Nisarg Shah. Designing fairly fair classifiers via economic fairness notions. In Proceedings of The Web Conference 2020 , pages 1559-1569, 2020.

Feifang Hu and William F Rosenberger. Optimality, variability, power: Evaluating responseadaptive randomization procedures for treatment comparisons. Journal of the American Statistical Association , 98(463):671-678, 2003.

Feifang Hu and William F Rosenberger. The Theory of Response-adaptive Randomization in Clinical Trials . John Wiley & Sons, 2006.

Feifang Hu and Li-Xin Zhang. Asymptotic properties of doubly adaptive biased coin designs for multitreatment clinical trials. The Annals of Statistics , 32(1):268-301, 2004.

Feifang Hu, Li-Xin Zhang, and Xuming He. Efficient randomized-adaptive designs. The Annals of Statistics , 37(5A):2543-2560, 2009.

Jianhua Hu, Hongjian Zhu, and Feifang Hu. A unified family of covariate-adjusted responseadaptive designs based on efficiency and ethics. Journal of the American Statistical Association , 110(509):357-367, 2015.

Svante Janson. Functional limit theorems for multitype branching processes and generalized pólya urns. Stochastic Processes and their Applications , 110(2):177-245, 2004.

Matthew Joseph, Michael Kearns, Jamie Morgenstern, Seth Neel, and Aaron Roth. Meritocratic fairness for infinite and contextual bandits. In Proceedings of the 2018 AAAI/ACM Conference on AI, Ethics, and Society , pages 158-163, 2018.

Matthew Joseph, Michael Kearns, Jamie H Morgenstern, and Aaron Roth. Fairness in learning: Classic and contextual bandits. In Advances in Neural Information Processing Systems , volume 29, pages 325-333, 2016.

Dean S Karlan and Jonathan Zinman. Credit elasticities in less-developed economies: Implications for microfinance. American Economic Review , 98(3):1040-68, 2008.

Masahiro Kato, Takuya Ishihara, Junya Honda, and Yusuke Narita. Efficient adaptive experimental design for average treatment effect estimation. arXiv preprint arXiv:2002.05308 , 2020.

Eugene Kharitonov, Aleksandr Vorobev, Craig Macdonald, Pavel Serdyukov, and Iadh Ounis. Sequential testing for early stopping of online experiments. In Proceedings of the 38th International ACM SIGIR Conference on Research and Development in Information Retrieval , pages 473-482, 2015.

Ron Kohavi and Roger Longbotham. Online controlled experiments and A/B testing. Encyclopedia of Machine Learning and Data Mining , 7(8):922-929, 2017.

K Kubota, Y Ichinose, G Scagliotti, D Spigel, JH Kim, T Shinkai, K Takeda, S-W Kim, T-C Hsia, RK Li, et al. Phase iii study (monet1) of motesanib plus carboplatin/paclitaxel in patients with advanced nonsquamous nonsmall-cell lung cancer (nsclc): Asian subgroup analysis. Annals of Oncology , 25(2):529-536, 2014.

Yunzhi Lin, Ming Zhu, and Zheng Su. The pursuit of balance: An overview of covariate-adaptive randomization techniques in clinical trials. Contemporary Clinical Trials , 45(Pt A):21-25, 2015.

Xinwei Ma, Yuya Sasaki, and Yulong Wang. Testing limited overlap. working paper , 2023.

Xinwei Ma and Jingshen Wang. Robust inference using inverse probability weighting. Journal of the American Statistical Association , 115(532):1851-1860, 2020.

Blossom Metevier, Stephen Giguere, Sarah Brockman, Ari Kobren, Yuriy Brun, Emma Brunskill, and Philip S Thomas. Offline contextual bandits with high probability fairness guarantees. In Advances in Neural Information Processing Systems , volume 32, pages 14922-14933, 2019.

Hervé Moulin. Fair division in the internet age. Annual Review of Economics , 11:407-441, 2019.

Whitney K Newey. Semiparametric efficiency bounds. Journal of Applied Econometrics , 5(2):99-135, 1990.

Vishakha Patil, Ganesh Ghalme, Vineet Nair, and Yadati Narahari. Achieving fairness in the stochastic multi-armed bandit problem. Journal of Machine Learning Research , 22(1):78857915, 2021.

Stuart J Pocock and Richard Simon. Sequential treatment assignment with balancing for prognostic factors in the controlled clinical trial. Biometrics , 31(1):103-115, 1975.

David S Robertson, Kim May Lee, Boryana C López-Kolkovska, and Sofía S Villar. Responseadaptive randomization in clinical trials: From myths to practical considerations. Statistical Science , 38(2):185-208, 2023.

William F Rosenberger. Randomized Urn Models and Sequential Design . Taylor & Francis, 2002.

William F Rosenberger and Feifang Hu. Maximizing power and minimizing treatment failures in clinical trials. Clinical Trials , 1(2):141-147, 2004.

William F Rosenberger and John M Lachin. The use of response-adaptive designs in clinical trials. Controlled Clinical Trials , 14(6):471-484, 1993.

William F Rosenberger, AN Vidyashankar, and Deepak K Agarwal. Covariate-adjusted responseadaptive designs for binary response. Journal of Biopharmaceutical Statistics , 11(4):227-236, 2001.

David Simchi-Levi and Chonghuan Wang. Multi-armed bandit experimental design: Online decision-making and adaptive inference. In Proceedings of the 26th International Conference on Artificial Intelligence and Statistics , volume 206, pages 3086-3097, 2023.

Max Tabord-Meehan. Stratification trees for adaptive randomization in randomized controlled trials. Review of Economic Studies , 90(5):2646-2673, 2023.

Peter F Thall and J Kyle Wathen. Practical bayesian adaptive randomisation in clinical trials. European Journal of Cancer , 43(5):859-866, 2007.

Yevgen Tymofyeyev, William F Rosenberger, and Feifang Hu. Implementing optimal allocation in sequential binary response experiments. Journal of the American Statistical Association , 102(477):224-234, 2007.

Aad W. van der Vaart. Asymptotic Statistics . Cambridge University Press, New York, 1998.

Sofía S Villar, Jack Bowden, and James Wason. Multi-armed bandit models for the optimal design of clinical trials: Benefits and challenges. Statistical Science , 30(2):199, 2015.

Sofía S Villar, Jack Bowden, and James Wason. Response-adaptive designs for binary responses: How to offer patient benefit while being robust to time trends? Pharmaceutical Statistics , 17(2):182-197, 2018.

Sofía S Villar and William F Rosenberger. Covariate-adjusted response-adaptive randomization for multi-arm clinical trials using a modified forward looking gittins index rule. Biometrics , 74(1):49-57, 2018.

Sofía S Villar, James Wason, and Jack Bowden. Response-adaptive randomization for multi-arm clinical trials using the forward looking gittins index rule. Biometrics , 71(4):969-978, 2015.

Stefan Wager and Susan Athey. Estimation and inference of heterogeneous treatment effects using random forests. Journal of the American Statistical Association , 113(523):1228-1242, 2018.

LJ Wei and S Durham. The randomized play-the-winner rule in medical trials. Journal of the American Statistical Association , 73(364):840-843, 1978.

Waverly Wei, Maya Petersen, Mark J van der Laan, Zeyu Zheng, Chong Wu, and Jingshen Wang. Efficient targeted learning of heterogeneous treatment effects for multiple subgroups. Biometrics , 79(3):1934-1946, 2023.

S Faye Williamson, Peter Jacko, Sofía S Villar, and Thomas Jaki. A bayesian adaptive design for clinical trials in rare diseases. Computational Statistics & Data Analysis , 113:136-153, 2017.

Yuhang Wu, Zeyu Zheng, Guangyu Zhang, Zuohua Zhang, and Chu Wang. Adaptive a/b tests and simultaneous treatment parameter optimization. arXiv preprint arXiv:2210.06737 , 2022.

Yuhang Wu, Zeyu Zheng, Guangyu Zhang, Zuohua Zhang, and Chu Wang. Non-stationary a/b tests. In Proceedings of the 28th ACM SIGKDD Conference on Knowledge Discovery and Data Mining , pages 2079-2089, 2022.

Yuhang Wu, Zeyu Zheng, and Tingyu Zhu. Best arm identification with fairness constraints on subpopulations. Proceedings of the Winter Simulation Conference , 2023.

Yuhang Wu, Zeyu Zheng, and Tingyu Zhu. Selection of the best policy under fairness and equity constraints. Working Paper , 2023.

Yanxun Xu, Lorenzo Trippa, Peter Müller, and Yuan Ji. Subgroup-based adaptive (suba) designs for multi-arm biomarker trials. Statistics in Biosciences , 8(1):159-180, 2016.

Guosheng Yin, Nan Chen, and J Jack Lee. Phase ii trial design with bayesian adaptive randomization and predictive probability. Journal of the Royal Statistical Society: Series C (Applied Statistics) , 61(2):219-235, 2012.

Marvin Zelen. The randomization and stratification of patients to clinical trials. Journal of Chronic Diseases , 27(7-8):365-375, 1994.

Lanju Zhang and William F Rosenberger. Response-adaptive randomization for clinical trials with continuous outcomes. Biometrics , 62(2):562-569, 2006.

Li-Xin Zhang, Feifang Hu, Siu Hung Cheung, and Wai Sum Chan. Asymptotic properties of covariate-adjusted response-adaptive designs. The Annals of Statistics , 35(3):1166-1182, 2007.

Wanying Zhao, Wei Ma, Fan Wang, and Feifang Hu. Incorporating covariates information in adaptive clinical trials for precision medicine. Pharmaceutical Statistics , 21(1):176-195, 2022.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper investigates adaptive experimental design with a focus on fairness. In this context, fairness refers to ensuring that the probabilities of treatment assignment do not significantly differ across various groups. The authors propose a fair adaptive experimental design that simultaneously enhances data use efficiency, achieves an “envy-free” treatment assignment guarantee, and improves the overall welfare of participants.\n\nSTRENGTHS:\n1.\tFairness is gaining increasing importance in various fields, including experimental design.\n2.\tThe technical parts (Section 2 and Section 3) are solid, well organized and clearly presented. \n3.\tBoth simulation studies and a case study based on real data are provided.\n\nWEAKNESSES:\n1.\tRegarding the definition of 'fairness,' this paper considers fairness as the requirement for treatment adoption probabilities to be similar across different groups. However, in many scenarios, particularly in clinical trials, such a definition of fairness may not be entirely convincing. The reason why we want to consider the covariate $X_{it}$ and group $\\mathcal{S}_j$ is because the treatment effect may be very different among different groups. For instance, in the case of patients grouped by biomarkers into $\\mathcal{S}_1$ and $\\mathcal{S}_2$, the new treatment yields a highly positive effect in $\\mathcal{S}_1$ but a strongly negative effect in $\\mathcal{S}_2$. Consequently, enforcing close treatment probabilities in these two groups might be deemed unfair. It would be beneficial if the authors could provide motivating examples in the introduction that align well with the current formulation.\n2.\tRegarding the consideration of 'welfare,' I have some uncertainty about its direct relevance to the main topic of 'fairness.' While I understand that enforcing fairness constraints may potentially impact welfare, I'm unsure about the specific implications if we were to remove the welfare constraint in Problem A. In lines 66-70, the authors briefly touch upon this question, but the points made are not entirely clear to me. It would be beneficial if the authors could further elaborate on why “the second fairness concern arises when the adaptive treatment allocation does not adequately account for the overall welfare of experimental participants.”\n3.\tAbout the feasibility of the optimization Problem A and Problem B. In my view, it seems that Problem A could be infeasible, especially when $c_1$ is very small or $c_2$ is very close to ½. I am curious about whether the authors once met the infeasibility issue in the simulation studies and the case study.\n4.  Regarding the length of the first stage, it would be beneficial to include additional comments in the paper on how to select the value of $n_1$. Furthermore, I would appreciate a clearer explanation of how exactly $n_1$ influences the main results, as this aspect is important for my understanding. Additionally, in lines 107-108, the authors mention that 'An important methodological and practical innovation of our framework is that it does not require the number of participants enrolled in the first stage to be proportional to the overall sample size.' However, more supporting comments on this claim within the main text are expected. Providing some additional explanation or evidence for this statement would strengthen the paper's argument.\n5. Regarding the objectives of the experiment, in lines 76-78, the authors state that 'we propose a fair adaptive experimental design strategy that balances competing objectives: improving fairness, enhancing overall welfare, and gaining efficiency. Our strategy strikes a delicate balance between these trade-offs.' However, it remains unclear what trade-off means among these three objectives. While I can comprehend the trade-off between enhancing overall welfare and gaining efficiency, as it has been previously investigated in [1][2]. I find it unlikely that improving fairness could simultaneously trade off with two objectives that already have a trade-off between them.  I believe it would be highly beneficial if the authors could provide further insights into the nature and rationale behind the trade-off that exists among these objectives. Expanding on this aspect would provide a deeper understanding of the decision-making process and the underlying considerations in the proposed strategy.\n\nReference:\n\n[1] Erraqabi, A., Lazaric, A., Valko, M., Brunskill, E., & Liu, Y. E. (2017, April). Trading off rewards and errors in multi-armed bandits. In Artificial Intelligence and Statistics (pp. 709-717). PMLR.\n\n[2] Simchi-Levi, D., & Wang, C. (2023, April). Multi-armed bandit experimental design: Online decision-making and adaptive inference. In International Conference on Artificial Intelligence and Statistics (pp. 3086-3097). PMLR.\n\nQUESTIONS:\nApart from the major points above, I have several minor points.\n\n1)\tThe paper is well organized in general. However, I think the writing could be further improved. For the technical parts (Sections 2 and 3), after each lemmas and theorems, it will be better to provide more insights and discussions. \n\n2)\tIn Lemma 1, it is more appropriate to denote the oracle treatment assignment probablity for group j under the proposed welfare constraint as $e^∗ _{j,proposed}$ instead of $e^∗ _{j,alternative}$.\n\n3)\tFor a conference paper, I believe the abstract could be more concise as it is currently a bit long.\n\nLIMITATIONS:\nSee previous comments.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-20"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe authors delve into an examination of adaptive experimental design subject to the constraints of fairness. They portray a scenario where a researcher, within a sequential experimental context, adapts the allocation between control and treatment groups among a cohort of subjects to optimize the statistical efficiency of the average treatment effect estimation. At the same time, the researcher strives to maintain a balanced proportion of treatments across various demographic groups and maximize the total welfare of these treatments, a condition set forth to uphold fairness. The main contribution of this manuscript lies in the formulation of a novel strategy for treatment allocation that simultaneously adheres to the constraints of fairness. The authors provide a theoretical analysis of their devised method, validating that the average treatment effect estimated from their approach exhibits asymptotic consistency and adheres to a central limit theorem-like convergence toward a particular Gaussian distribution. Empirical studies, leveraging synthetic simulations, demonstrate their method's competitive statistical efficiency against existing approaches while ensuring fairness is upheld. Furthermore, the efficacy of the developed method is also confirmed via empirical simulations that mimic real data, further solidifying the reliability and practicality of their method.\n\nSTRENGTHS:\nThis paper investigates the fairness problem in the adaptive experimental design. This problem is well-motivated and of particular interest to NeurIPS's community. The experimental results adequately demonstrate the proposed method's competitive efficiency and guarantee of fairness.\n\nWEAKNESSES:\nThis paper lacks consideration of essential existing contributions, specifically those of \n- V. Hadad et al. Confidence Intervals for Policy Evaluation in Adaptive Experiments. Proceedings of the National Academy of Sciences, 2021.\n- D. Simchi-Levi et al. Multi-armed Bandit Experimental Design: Online Decision-making and Adaptive Inference. AISTATS, 2023.\n- M. Kato et al. Adaptive Experimental Design for Efficient Treatment Effect Estimation. NeurIPS 2020 Workshop on Causal Discovery & Causality-Inspired Machine Learning.\n\nNotably, the findings put forth by Hadad et al. warrant particular attention, given their methodological relevance to this research. Hadad et al. introduce an off-policy estimation of the average treatment effect, a method that works regardless of the treatment assignment mechanism. They also validated the asymptotic consistency and normality of their approach. While Hadad et al.'s method operates without covariates, unlike the scenario in the present paper, this discrepancy does not diminish the applicability of their method. This paper's research problem can be recast into a non-covariate one by transforming the experimental process into group-wise processes. Furthermore, with an appropriately selected set of parameters, the method discussed by Hadad et al. can be reduced to this paper's proposed solution. Consequently, the asymptotic consistency and normality results offered by Hadad et al. would translate to the circumstances of this paper, which seamlessly results in Theorems 1 and 2.\n\nAdditional insightful points can be gleaned from the work of Kato et al., who propose a treatment assignment mechanism to minimize the variance of the average treatment effect. While unconstrained, their method presents intriguing similarities with the constrained approach of the current paper. Appendix F of Kato et al., particularly noteworthy, introduces an ethical consideration in the treatment assignment process by incorporating a fairness constraint.\n\nThe omission of these existing methods from the current discussion and the direct applicability of these approaches to the circumstances of this paper leads me to question the originality of the proposed research. The paper would benefit from a comparative evaluation with these significant contributions to ensure that it advances the related field.\n\n\nThe present paper is unfortunately beset by a number of quality issues that necessitate substantial revisions. To begin with, the paper neglects to outline the assumptions required for causal inference. Particularly, the authors' methodology employs the inverse propensity score weighting technique. This approach necessitates the fulfillment of certain assumptions, encompassing consistency, no unmeasured confounders, and positivity. The lack of explaining these assumptions calls into question the validity of the resultant findings and negatively impacts the overall quality of the paper.\n\nMoving to Section 2.2, it is observed that quality concerns exist. Specifically, Lemma 1's statement introduces the concept of an alternative constraint without providing its rigorous definition. This omission makes us hard to ascertain the correctness of the lemma. Additionally, the section contains considerable redundancy, particularly evident between the contents of Lines 209-219 and Lines 220-230. Such repetition impairs the overall readability of the paper and dilutes the impact of the information conveyed. Cumulatively, these quality issues have contributed to a lower assessment of the paper's presentation score in this review.\n\nQUESTIONS:\nI suggest that the authors elucidate a specific scenario wherein fairness issues arise and illustrate how the applied fairness constraint alleviates these problems. The fairness constraints utilized by the authors appear to diverge from conventional ones; thus, it is essential for the authors to rationalize their choice to implement the proposed fairness constraint. Illustrating a specific scenario will be instrumental in justifying the use and effectiveness of the fairness constraint.\n\nThe authors state that their formulated fairness constraints are based on the principles of envy-freeness and total welfare. Nevertheless, there may be a potential misapprehension of these principles by the authors as their derived fairness constraints deviate from the standard definitions established from the above notions. Both principles are rooted in the game theory literature and are presented in a context where each game participant possesses a distinct utility function. This utility function is exploited to evaluate the benefits derived from a specific treatment. Envy-freeness and total welfare can be viewed as inherent characteristics of the participants' utility contingent upon their treatments. In essence, the constructs of envy-freeness and total welfare are fundamentally anchored on utility functions. Consequently, it is recommended that the authors shed light on the nature of the utility functions leveraged in their investigation. By doing so, they ensure a more precise understanding of the fairness constraints.\n\nIn this paper, the authors endeavor to validate Theorems 1 and 2 by invoking Lemma A.1. However, the assertion of Lemma A.1 appears to be flawed, particularly in the analysis of the conditional expectation of $Z_t$ given $\\mathcal{F}{t-1}$. This is because $E[Z_t|\\mathcal{F}{t-1}]$ is not a random variable and hence cannot equate to $Z_{t-1}$, evident by $E[D_tY_t1(X_t \\in \\mathcal{S}j)|\\mathcal{F}{t-1}] = p_j\\hat{e}^*{t,j}E[Y_t|D_t=1,X_t \\in \\mathcal{S}j]$. However, despite this ostensible flaw in Lemma A.1, it is plausible that the statements of Theorems 1 and 2 might be correct. The validity of these theorems may be independently corroborated by employing the known fact that $Z_t$ possesses a common mean. This subsequently implies that the process of $Z_t$ minus the common mean satisfies the definition of a martingale difference sequence.\n\nLIMITATIONS:\nThere are no specific limitations or potential negative social impacts that require discussion in relation to the current method.", "TITLE": "Official Review", "RECOMMENDATION": 2, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-28"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper is concerned with the problem of experimental design for adaptive experiments under a fairness constraint, i.e., the restriction that some individuals not be disproportionately more likely to receive treatment. The authors approach this problem by modifying the usual Neyman allocation with constraints for envy, welfare, and feasibility of the problem. Update rules are provided for the problem to provide treatment probabilities at each round. The authors also provide theoretical results which bound the ATE and variance, and convergence to the oracle strategy in the limit.\n\nSTRENGTHS:\n* This is a novel task formulation for an important problem. The authors do a nice job of motivating the problem as well, and provide a clear description of the desired characteristics of the solution. \n\n* The proposed algorithm is simple and efficient. The authors do a nice job of laying out both problems A and B and clearly describing the update procedure. \n\n* Provided theory is nice in being able to reason about the behavior of the proposed procedure. The bound on the ATE and variance is very reasonable, with the proposed procedure taking only a modest penalty in terms of convergence rate. Asymptotic normality and convergence to the optimal strategy in the limit are also nice results.\n\nWEAKNESSES:\nWhile the strategy converges to the optimal strategy in the limit, it's not clear to me how much of an effect these parameters have on the procedure's behavior in finite samples. This is especially important in the design setting, where the main motivation of adaptivity is improving finite sample performance. It would be nice to see some simulated/empirical evidence that examines the robustness of the proposed procedure to misspecification. There is also a natural tension between the welfare and fairness constraints, it would be helpful if the authors provide a discussion of this. It also would have been useful if the authors provided a comparison to a non-adaptive (but balanced) procedure a as an additional baseline (e.g., rerandomizatoin, Gram-Schmidt walk). In this setting, treatment probabilities would remain fixed but variance would still be minimized by accounting for group status with an obvious tradeoff on the welfare constraint.\n\nQUESTIONS:\n* How does this procedure compare with adding regularization to a simple adaptive procedure like UCB? \n* Little guidance is given for selecting the constraints. In practice how much of an effect do these have on relative performance? Can the authors provide a sense of worst case behavior here?\n\nLIMITATIONS:\nThe authors overall do a nice job of motivating the problem of fairness in treatment allocation. However, I do think that the introduction of some of these constraints, while well motivated, still leaves a subjective decision in the hands of the experimenter. It seems given this that more attention/discussion should be given to this choice in light of the example used in the paper which has a significant impact on the lives of people who are subjected to the experiment.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-07"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper considers randomized experiments that aim to assess the effectiveness of some treatment or policy. The data and sample efficiency of such randomized experiments has been shown to improve by making them adaptive — i.e. by updating the treatment assignment probabilities on the fly, based on the data collected in the study. This paper argues that this can lead to fairness concerns, however. The goal of this paper is to strike a balance between maximizing information gain through randomized experiments and respecting fairness considerations. The fairness aspects considered in this paper include maximizing overall welfare of the participants (through treatments) and ensuring all participant groups in the experiment receive a “fair” exposure to the treatment.\nTowards achieving this, the paper proposes a “fair adaptive experimental design strategy” that integrates fairness and welfare considerations along with optimizing information gain. The approach is to build a non-parametric algorithm that calculates the mean and variance of potential outcomes at the group level, each time step, to determine future allocations. The paper presents theoretical results that show that this approach yields strongly consistent estimates. Numerical results show that this approach improves upon the fairness considerations.\n\nSTRENGTHS:\n– I think the key contribution/strength of the paper lies in recognizing and attempting to fill the fairness gap that can arise in adaptive experiments.\n \n– The problem formulation and contribution is cleanly spelled out and well-presented. Everything is easy to understand and follow. \n\n– The paper presents nice theoretical results backing the soundness of the proposed approach.\n\nWEAKNESSES:\n– The experimental section appears a little weak. The DGPs used are completely synthetic and seem to be hand-crafted to demonstrate the utility of this approach. It may be more convincing to have more real/realistic datasets or even synthetic data generated from some real DGP (somewhat similar to the case study, but having one more domain might make it stronger). \n\n– The abstract is either too dense or introduces too many technical terms that are discussed only later in the paper, making it very hard to understand. Perhaps cramming less info in the abstract might help. \n\n– THe paper doesn’t discuss/ quantify the trade-off involved in achieving fairness: how much information gain is given up as compared to other SOTA adaptive experimentation approaches. Having some insights/ theoretical guarantees here may be very useful.\n\nQUESTIONS:\n– In section 2.1, is $n_t$ the number of *new* participants added to the experiment at time $t$ or simply the total number of participants at time $t$? If it is the former, then why is $D_{it}$ defined only for $i = 1, …, n_t$? If it is the latter, then N= \\sum n_t doesn’t seem to make sense (because some participants may get over counted)? Could you please clarify this?\n\n– Why is Assumption 1 mild / why is it reasonable or practicable to assume? \n\n– In the statement of theorem 1, what is $n_1$ and do you need N tending to infinity? I thought that is already captured by the math? Or am I missing something?\n\n– Could you elaborate on the tradeoff involved (data efficiency vs fairness) as mentioned under the last bullet under wekanesses?\n\nLIMITATIONS:\nNo negative societal impacts discussed (but I don’t see any concerns). \n\nPerhaps the technical limitation is that the approach may give up some amount of data efficiency. This could be discussed more in the paper.", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 3, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-11"}