SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Fortunately, online optimization is a well-studied area with a rich source of techniques from which we may draw inspiration. However, to the best of our knowledge, existing regret minimization algorithms are not well-suited to minimizing the Neyman regret. For example, the multi-arm bandit literature typically defines regret in terms of a finite number of actions that can be taken [Lattimore and Szeptsvári, 2020] while Adaptive Neyman Allocation consists of a continuum of actions as  $P_{t} \in [0,1]$ . This means that algorithms like UCB [Auer et al., 2002a] and EXP3 [Auer et al., 2002b] are not appropriate for Adaptive Neyman Allocation. Our cost objectives  $f_{t}$  and action space  $[0,1]$  are both convex, so the problem of Adaptive Neyman Allocation is an instance of Online Convex Optimization (OCO) [Hazan, 2016]. Even so, the problem of minimizing Neyman regret is not immediately amenable to existing algorithms, which typically requires assumptions on the cost functions such as bounded gradients or known Lipschitz parameters. In this setting, the cost functions have gradients which blow up at the boundary and Lipschitz parameters cannot be guaranteed as they rely on the unknown heterogeneous potential outcomes. For these reasons, we must design a new algorithm specifically tailored to Adaptive Neyman Allocation.
4.2 CLIP-OGD: A Variant of Online Stochastic Projected Gradient Descent
We present CLIP-OGD, which aims to minimize the Neyman regret and thus recover the Neyman variance in large samples. The algorithm is based on the online stochastic projected gradient descent principle, but with a twist: the projection set continuously grows over the rounds. At each round  $t$ , a new treatment probability  $P_{t}$  is chosen by updating the previous sampling probability  $P_{t - 1}$  in the negative (estimated) gradient direction of the previous cost, and then projecting to an interval  $[\delta_t, 1 - \delta_t]$ . Initially, this projection interval contains only the point  $1/2$  and it grows as the rounds increase, allowing for larger amounts of exploitation in later rounds.
The gradient estimator  $G_{t}$  is obtained in the following way: the gradient of  $f_{t}$  at  $P_{t}$  is given as  $f^{\prime}(P_{t}) = -\frac{y_{t}(1)^{2}}{P_{t}^{2}} +\frac{y_{t}(0)^{2}}{(1 - P_{t})^{2}}$ . Only one outcome is observed, so we used the adaptive Horvitz-
Thompson principle using the conditional probability  $P_{t}$  to unbiasedly estimate the outcomes. CLIP-OGD is formally presented below as Algorithm 1, where the projection operator is defined as  $\mathcal{P}_c(x) = \max \{c,\min \{x,1 - c\}\}$ .
Algorithm 1: CLIP-OGD
Input: Step size  $\eta$  and decay parameter  $\alpha$
Initialize  $P_0\gets 1 / 2$  and  $G_{0}\leftarrow 0$
for  $t = 1\ldots T$  do
Set projection parameter  $\delta_t = (1/2) \cdot t^{-1/\alpha}$
Compute new treatment probability  $P_{t}\gets \mathcal{P}_{\delta_{t}}(P_{t - 1} - \eta \cdot G_{t - 1})$
Sample treatment assignment  $Z_{t}$  as 1 with probability  $P_{t}$  and 0 with probability  $1 - P_{t}$
Observe outcome  $Y_{t} = \mathbf{1}[Z_{t} = 1]y_{t}(1) + \mathbf{1}[Z_{t} = 0]y_{t}(0)$
Construct gradient estimator  $G_{t} = Y_{t}^{2}\left(-\frac{\mathbf{1}[Z_{t} = 1]}{P_{t}^{3}} + \frac{\mathbf{1}[Z_{t} = 0]}{(1 - P_{t})^{3}}\right)$
Unlike the two-stage design of [Hahn et al., 2011, Blackwell et al., 2022], CLIP-OGD does not feature explicit explore-exploit stages, but rather performs both of these simultaneously. The trade-off is implicitly controlled through parameters  $\eta$  and  $\alpha$ : smaller values of  $\eta$  limit the amount of that sampling probabilities can update and, likewise, larger values of  $\alpha$  prevent extreme probabilities in earlier stages. Because the gradient of the cost functions are inversely proportional to the treatment probabilities, limiting the extremeness of the treatment probabilities in this way ensures that the gradient estimates do not increase at a fast rate. By appropriately setting input parameters, CLIP-OGD achieves  $\widetilde{\mathcal{O}}(\sqrt{T})$  expected Neyman regret, where the  $\widetilde{\mathcal{O}}(\cdot)$  notation hides sub-polynomial factors.
Theorem 4.2. Under Assumption 1 the parameter values  $\eta = \sqrt{1 / T}$  and  $\alpha = \sqrt{5\log(T)}$  ensure the expected Neyman regret of CLIP-OGD is asymptotically bounded:  $\mathbb{E}[\mathcal{R}_T]\leqslant \tilde{\mathcal{O}} (\sqrt{T})$
Theorem 4.2 answers, in the affirmative, that it is possible to construct an adaptive experimental design whose variance recovers that of the Neyman variance, in large samples. Note that the amount of exploration (as given by the parameters  $\eta$  and  $\alpha$ ) should be increasing with  $T$  in order to recover these regret bounds. In Appendix C, we show that CLIP-OGD is somewhat robust to different values of the decay parameter, i.e. for any value  $\alpha > 5$ , the expected regret will be sublinear. We also show that if the experimenter presumes to have correctly specified bounds  $C$  and  $c$  appearing in Assumption 1, then the step size can be modified to improve the constant factors in the Neyman regret bound, which may lead to improved performance in moderate sample sizes. We conjecture that the minimax rate for expected Neyman regret is  $\mathcal{O}(\sqrt{T})$ , but proving this is beyond the scope of the current paper—we only remark that we do not know it to immediately follow from any existing regret lower bounds for OCO.
5 Inference in Large Samples
The proposed CLIP-OGD was constructed to ensure that the variance of the adaptive Horvitz-Thompson estimator quickly approaches the Neyman variance. In this section, we provide confidence intervals for the average treatment effect which also enjoy reduced width compared to non-adaptive counterparts.
A necessary condition for variance estimation is that the variance itself cannot be going to zero too quickly. In design-based inference, it is common to directly posit a so-called "non-superefficient" assumption that  $\mathrm{Var}(\hat{\tau}) = \Omega(1/T)$  [Aronow and Samii, 2017, Leung, 2022, Harshaw et al., 2022]. The non-superefficiency assumption may be seen as an additional regularity assumption on the outcomes, e.g. preventing  $y_{t}(1) = y_{t}(0) = 0$  for all  $t \in [T]$ . In this work, a similar lower bound on the rate of the adaptive variance is obtained through a different, perhaps more transparent, assumption on the expected Neyman regret.
Assumption 2. The outcome sequence is not overly-fit to CLIP-OGD:  $-\mathbb{E}[\mathcal{R}_T] = o(T)$
While we have shown that  $\mathbb{E}[\mathcal{R}_T] \leqslant \tilde{\mathcal{O}}(\sqrt{T})$ , the Neyman regret could in principle be negative if the adaptive design achieves variance which is strictly smaller than the best Bernoulli design. While this
seems unlikely to happen for "typical" outcomes, it is not impossible. Assumption 2 rules out these edge-case settings. We suspect that Assumption 2 would not be necessary in an i.i.d. setting, but proving this seems beyond the scope of the current paper. As shown in the appendix, Assumptions 1 and 2 imply that the adaptive variance achieves the parametric rate:  $\mathrm{Var}(\hat{\tau}) = \Theta (1 / T)$ .
5.1 Variance Estimation
In this section, we provide a variance estimator and show its stability in large samples. Rather than estimating the adaptive variance (which has no simple closed form), our approach is to estimate the Neyman variance directly. For an adaptive design achieving sublinear expected Neyman regret, these two quantities are asymptotically equivalent. In this way, our variance estimator may be appropriate not only for CLIP-OGD, but for any adaptive design achieving sublinear expected Neyman regret.
Recall that the Neyman variance is given by  $T \cdot V_{\mathrm{N}} = 2(1 + \rho)S_1S_0$ , where  $\rho$  is the outcome correlation,  $S_{1}$  is the second moment of treatment outcomes and  $S_{0}$  is the second moment of control outcomes. Unfortunately, the outcome correlation term is generally not estimable without strong assumptions in a design-based framework. Indeed, the difficulty is that terms like  $y_{t}(1)y_{t}(0)$  are unobservable due to the fundamental problem of causal inference [Imbens and Rubin, 2015]. A common solution to the problem is to opt for a conservative estimate of the variance, which will ensure validity of resulting confidence intervals.
We propose estimating the following upper bound on the variance:  $T \cdot \mathrm{VB} = 4S_0S_1$ . This upper bound on the Neyman variance is tight (i.e.  $\mathrm{VB} = V_{\mathrm{N}}$ ) when outcome correlation satisfies  $\rho = 1$ . For example, this occurs when all individual treatment effects are zero, i.e.  $y_{t}(1) = y_{t}(0)$  for all  $t \in [T]$ . Conversely, the upper bound will become looser for smaller values of the outcome correlation. In this sense, our bound resembles both the Neyman bound and the Aronow-Samii bound [Neyman, 1923, Aronow and Samii, 2013]. It may be possible to use the recent insights of Harshaw et al. [2021] in order to construct variance bounds which are tight in other scenarios, but that is beyond the scope of the current paper. Our variance estimator is defined as
$$
T \cdot \widehat {\mathbf {V B}} \triangleq 4 \sqrt {\left(\frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} ^ {2} \frac {\mathbf {1} [ z _ {t} = 1 ]}{p _ {t}}\right) \cdot \left(\frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} ^ {2} \frac {\mathbf {1} [ z _ {t} = 0 ]}{1 - p _ {t}}\right)} ,
$$
which is essentially a plug-in Horvitz-Thompson estimator for the second moments. Theorem 5.1 shows the error of the normalized variance estimator converges at a parametric rate.
Theorem 5.1. Under Assumptions 1 and 2, and the parameters stated in Theorem 4.2, the error of the normalized variance estimator under CLIP-OGD is  $T \cdot \widehat{VB} - T \cdot VB = \widetilde{\mathcal{O}}_p(T^{-1/2})$ .
5.2 Confidence Intervals
The variance estimator may be used to construct confidence intervals for the average treatment effect. This offers experimenters standard uncertainty quantification techniques when running CLIP-OGD. The following corollary shows that the resulting Chebyshev-type intervals are asymptotically valid.
Corollary 5.1. Under Assumptions 1 and 2, and parameters stated in Theorem 4.2, Chebyshev-type intervals are asymptotically valid: for all  $\alpha \in (0,1]$ ,  $\lim \inf_{T\to \infty}\operatorname *{Pr}(\tau \in \hat{\tau}\pm \alpha^{-1 / 2}\sqrt{\widehat{VB}})\geqslant 1 - \alpha$
While these confidence intervals are asymptotically valid under our regularity assumptions, they may be overly conservative in general. In particular, they will over cover when the Chebyshev tail bound is loose. We conjecture that the adaptive Horvitz-Thompson estimator under CLIP-OGD satisfies a Central Limit Theorem, which would imply asymptotic validity of the narrower Wald-type intervals where  $\alpha^{-1/2}$  scaling is replaced with the corresponding normal quantile,  $\Phi^{-1}(1 - \alpha/2)$ . As discussed in Section 7, the adaptive estimator appears approximately normal in simulations. Until this is formally shown, we recommend experimenters exhibit caution when using Wald-type confidence intervals for the adaptive Horvitz-Thompson estimator under CLIP-OGD.
6 Considering Alternative Designs
Explore-Then-Commit Two-stage adaptive designs have been proposed for the purpose of variance reduction [Hahn et al., 2011, Blackwell et al., 2022]. Due to its similarities to algorithms in the bandits
literature, we call these types of designs Explore-Then-Commit (ETC) [Lattimore and Szepesvári, 2020]. At a high level, an Explore-then-Commit design runs the Bernoulli design with  $p = 1/2$  for  $T_0 \leqslant T$  iterations, uses the collected data to estimate  $p^*$  by  $\hat{p}^*$ , and then runs the Bernoulli design with  $p = \hat{p}^*$  for the remaining  $T_1 = T - T_0$  iterations. These ETC designs are conceptually simpler than CLIP-OGD, and may be reasonable to apply in more restricted settings where changing the treatment probabilities is difficult or costly. However, we provide the following negative result which shows that they can suffer linear Neyman regret.
Proposition 6.1. For all explore phase lengths  $T_0$  satisfying  $T_0 = \Omega (T^\epsilon)$  for some  $\epsilon > 0$ , there exist a class of potential outcomes sequences satisfying Assumption 1 such that the Neyman regret under Explore-then-Commit is linear:  $\mathcal{R}_T = \Omega_p(T)$ .
The specific class of potential outcomes referenced in Propposition 6.1 is constructed explicitly in Appendix E.1. ETC designs suffer larger variance when the estimated  $\hat{p}^*$  may be far from the true optimal probability  $p^*$ . In a design-based setting, this happens when the units in the explore phase are not representative of the entire sequence. Formulating conditions under which Explore-then-Commit designs achieve low Neyman regret is beyond the scope of this paper, but the proof of Proposition 6.1 shows that additional regularity conditions on the order of the units will be required.
Multi Arm Bandit Algorithms Multi Arm Bandit (MAB) algorithms are often used for adaptive decision making settings, from online advertising to product development. The goal of MAB algorithms is to minimize the outcome regret, which measures the contrast between the overall value obtained from the actions relative to the value of the best action. The outcome regret is conventionally defined as  $\mathcal{R}_T^{\mathrm{outcome}} = \max_{k\in \{0,1\}}\sum_{t = 1}^{T}y_t(k) - \sum_{t = 1}^{T}Y_t$ . In certain contexts, minimizing outcome regret may be a more desirable goal than estimating a treatment effect to high precision. However, the following proposition illustrates that these two objectives are generally incompatible.
Proposition 6.2. Let  $\mathcal{A}$  be an adaptive treatment algorithm achieving sublinear outcome regret, i.e. there exists  $q\in (0,1)$  such that  $\mathbb{E}[\mathcal{R}_T^{outcme}] \leqslant O(T^q)$  for all outcome sequences satisfying Assumption 1. Then, there exists a class of outcome sequences satisfying Assumption 1 on which  $\mathcal{A}$  suffers super-linear Neyman regret, i.e.  $\mathbb{E}[\mathcal{R}_T] \geqslant \Omega (T^{2 - q})$
Proposition 6.2 demonstrates that the outcome regret and the Neyman regret cannot generally be simultaneously minimized. In particular, sublinear outcome regret implies that the variance of the estimator must converge slower than the  $\Theta(1/T)$  parametric rate. This result contributes to a growing body of work which highlights trade-offs between various possible objectives in sequential decision making [Burtini et al., 2015]. It is beyond the scope of the current paper to determine how such trade-offs ought to be resolved, though Appendix F discusses ethical considerations.
7 Numerical Simulations
We evaluate the performance of CLIP-OGD and Explore-then-Commit (ETC) for the purpose of Adaptive Neyman Allocation on the field experiment of Groh and McKenzie [2016], which investigates the effect of macro-insurance on micro-enterprises in post-revolution Egypt<sup>2</sup>. The experimental units are 2,961 clients of Egypt's largest microfinance organization and the treatment was a novel insurance product. Several outcomes were recorded including whether the clients took on loans, introduced a new product or service, and the amount invested in machinery or equipment following treatment. To allocate treatment, Groh and McKenzie [2016] use a non-adaptive matched pair experimental design. Our goal here is not to provide a new analysis of this study, but rather to construct a plausible experimental setting under which to evaluate adaptive experimental designs.
In our simulations, we focus on the numerical outcome "invested in machinery or equipment". The experimental data contains only observed outcomes, so we must impute the missing potential outcomes in order to simulate the experiment. We impute outcomes using the model  $y_{t}(1) - y_{t}(0) = \tau + \gamma_{t}$ , where  $\tau = 90,000$  and  $\gamma_{1} \ldots \gamma_{T} \sim \mathcal{N}(0, \sigma^{2})$  are independent with  $\sigma = 5,000$ . This randomness is invoked only to impute potential outcomes, i.e. not re-sampled during each run of the experiment. In order to increase the sample size, we create a larger population by repeating this processes 5 times, which yields a total of 14,445 units after those with missing entries are removed. Units are shuffled to appear in an arbitrary order and outcomes are normalized to be in the range[0, 1].
2A repository for reproducing simulations is: https://github.com/crharshaw/Clip-OGD-sims
(a) Original dataset
(b) First 100 units have flipped potential outcomes
Figure 1: Normalized Variance of Adaptive Estimator under Experimental Designs
Figure 1 presents two plots illustrating how the variance of the adaptive HT estimator varies with different designs. The  $x$  axis contains the number of rounds  $T$  and the  $y$  axis contains the normalized variance  $T \cdot \mathrm{Var}(\hat{\tau})$  under the designs. For each value of  $T$ , we take the population to be the first  $T$  units in the sequence. CLIP-OGD is run with the parameters recommended in Theorem 4.2 and ETC is run with  $T_0 = T^{1/3}$  so that the exploration phase grows with  $T$ . The variance under CLIP-OGD and ETC is estimated empirically from 50,000 runs of the experiment, while the variance under the Bernoulli and Neyman designs is computed exactly.
In Figure 1a, we observe that CLIP-OGD requires about  $T = 4,000$  samples to achieve variance equal to Bernoulli, but eventually converges to the Neyman variance. As discussed in Section 4.2, it may be possible to improve the convergence rate by incorporating knowledge of the outcome moments in the design parameters. On the other hand, ETC remains comparable with Bernoulli even for small values of  $T$ , but remains far away from the Neyman design for large samples. In Figure 1b, a similar simulation is run, except that the potential outcomes of the first 100 units are swapped, so that the first units have negative individual treatment effects. While this produces little effect on the performance of CLIP-OGD, it substantially worsens the performance of ETC, which relies on the early outcomes to estimate an optimal treatment probability. In particular, ETC performs worse than Bernoulli under this minor modification—even in large samples—corroborating Proposition 6.1.
In the appendix, we evaluate the proposed confidence intervals, showing that CLIP-OGD enjoys intervals of reduced width. We show that normal based intervals cover at the nominal level and provide further evidence that the estimator is asymptotically normal under CLIP-OGD. We run additional simulations to investigate the sensitivity of the step size, and to demonstrate that additional baselines which were not designed for Neyman allocation indeed perform poorly.
In this paper, we have proposed the Neyman ratio and Neyman regret as a performance measure of experimental designs for the Adaptive Neyman Allocation problem. To this end, we proposed CLIP-OGD which achieves  $\tilde{\mathcal{O}} (\sqrt{T})$  expected Neyman regret under mild regularity conditions on the outcomes. This formally establishes—for the first time—the existence of adaptive experimental designs under which the variance of the effect estimator quickly approaches the Neyman variance. Finally, we have provided a variance estimator which provides experimenters with uncertainty quantification methods when using CLIP-OGD. The main drawback of our analysis is that it is most relevant for moderate and large sample sizes; in particular, our work does not properly address whether adaptive designs are always beneficial in small samples.
There are several research directions which can improve relevance of this methodology to practice. First, establishing conditions under which a central limit theorem holds for CLIP-OGD will yield smaller and thus more desirable Wald-type confidence intervals. Second, investigations into batched treatment allocations and delayed observations of outcomes would allow practitioners more flexibility in their designs. Finally, investigating variants of Adaptive Neyman Allocation in the presence of interference [Aronow and Samii, 2017, Harshaw et al., 2022] would allow for more realistic inference in complex settings, e.g. social network experiments and marketplace experiments.
We thank P.M. Aronow, Molly Offer-Westort, Alexander Rakhlin, Benjamin Recht, Fredrik Sävje, and Daniel Spielman for insightful discussions which helped shaped this work. Part of this work was done while Christopher Harshaw was visiting the Simons Institute for the Theory of Computing. Christopher Harshaw gratefully acknowledges support from Foundations of Data Science Institute (FODSI) NSF grant DMS2023505.
P.M. Aronow and Cyrus Samii. Conservative variance estimation for sampling designs with zero pairwise inclusion probabilities. Survey Methodology, 39(1):231-241, 2013.
P.M. Aronow and Cyrus Samii. Estimating average causal effects under general interference. Annals of Applied Statistics, 11(4):1912-1947, 2017. doi: 10.1214/16-aoas1005.
Peter Auer, Nicolò Cesa-Bianchi, and Paul Fischer. Finite-time analysis of the multiarmed bandit problem. Machine Learning, 47(2-3):235-256, may 2002a. ISSN 0885-6125. doi: 10.1023/A:1013689704352.
Peter Auer, Nicolò Cesa-Bianchi, Yoav Freund, and Robert E. Schapire. The nonstochastic multiarmed bandit problem. SIAM Journal on Computing, 32(1):48-77, 2002b. doi: 10.1137/S0097539701398375.
AD Barker, CC Sigman, GJ Kelloff, NM Hylton, DA Berry, and LJ Esserman. I-spy 2: An adaptive breast cancer trial design in the setting of neoadjuvant chemotherapy. *Clinical Pharmacology & Therapeutics*, 86(1):97-100, 2009. doi: 10.1038/clpt.2009.68.
Matthew Blackwell, Nicole E. Pashley, and Dominic Valentino. Batch adaptive designs to improve efficiency in social science experiments. Working paper, Harvard University, 2022. URL https://www.mattblackwell.org/files/papers/batch_adaptive.pdf.
Jack Bowden and Lorenzo Trippa. Unbiased estimation for response adaptive clinical trials. Statistical methods in medical research, 26, 08 2015.
Giuseppe Burtini, Jason Loepppky, and Ramon Lawrence. A survey of online experiment design with the stochastic multi-armed bandit. arXiv preprint arXiv:1510.00757, 2015.
Yong Cai and Ahnaf Rafi. On the performance of the neyman allocation with small pilots. arXiv:2206.04643, 2022.
Jeffrey R. Eisele. The doubly adaptive biased coin design for sequential clinical trials. Journal of Statistical Planning and Inference, 38(2):249-261, 1994. ISSN 0378-3758.
National Commission for the Protection of Human Subjects of Biomedical and Behavioral Research. The Belmont Report: Ethical principles and guidelines for the commission for the protection of human subjects of biomedical and behavioral research. Technical report, US Department of Health, Education, and Welfare, 1978.
David A. Freedman. On regression adjustments to experimental data. Advances in Applied Mathematics, 40:180-193, 2008. doi: 10.1016/j.aam.2006.12.003.
Matthew Groh and David McKenzie. Macroinsurance for microenterprises: A randomized experiment in post-revolution Egypt. Journal of Development Economics, 118:13-25, 2016. doi: 10.1016/j.jdeveco.2015.08.003.
Vitor Hadad, David A. Hirshberg, Ruohan Zhan, Stefan Wager, and Susan Athey. Confidence intervals for policy evaluation in adaptive experiments. *PNAS*, 118(15), 2021. doi: 10.1073/pnas.2014602118.
Jinyong Hahn, Keisuke Hirano, and Dean Karlan. Adaptive experimental design using the propensity score. Journal of Business & Economic Statistics, 29(1):96-108, 2011.