SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Eric M. Schwartz, Eric T. Bradlow, and Peter S. Fader. Customer acquisition via display advertising using multi-armed bandit experiments. Marketing Science, 36(4):500-522, 2017.
H. Solomon and S. Zacks. Optimal design of sampling from finite populations: A critical review and indication of new research areas. Journal of the American Statistical Association, 65(330): 653-677, 1970.
A. Wald. Sequential Tests of Statistical Hypotheses. The Annals of Mathematical Statistics, 16(2): 117-186, 1945.
Yanxun Xu, Lorenzo Trippa, Peter Müller, and Yuan Ji. Subgroup-based adaptive (suba) designs for multi-arm biomarker trials. Statistics in Biosciences, 8, 02 2014.
Kelly Zhang, Lucas Janson, and Susan Murphy. Inference for batched bandits. In Advances in Neural Information Processing Systems, volume 33, pages 9818-9829. Curran Associates, Inc., 2020.
Kelly Zhang, Lucas Janson, and Susan Murphy. Statistical inference with m-estimators on adaptively collected data. In Advances in Neural Information Processing Systems, volume 34, pages 7460=7471. Curran Associates, Inc., 2021.
Lanju Zhang and William F. Rosenberger. Response-adaptive randomization for clinical trials with continuous outcomes. Biometrics, 62(2):562-569, 2006.
A Additional Simulation Results 15
A.1 Confidence Intervals 15
A.2 Sensitivity to Step Size 15
A.3 Alternative Designs 16
B General Analysis of Adaptive Neyman Allocation 17
B.1 Analysis of Adaptive Horvitz-Thompson Estimator (Propositions 2.1 and 2.2) 17
B.2 Derivation of the Neyman Design (Proposition 3.1) 18
B.3 Equivalence of Neyman Ratio and Neyman Regret (Theorem 4.1) 19
C Analysis of Neyman Regret for CLIP-OGD 19
C.1 Proof of Theorem 4.2 19
C.2 Selecting Parameters When Moment Bounds are Known 24
D Analysis for Inference in Large Samples 25
D.1 Conservative Variance Estimator (Theorem 5.1) 25
D.2 Valid Confidence Intervals (Corollary 5.1) 30
E Analysis of Alternative Designs 30
E.1 Analysis of Explore-then-Commit (Proposition 6.1) 30
E.2 Analysis of Designs for Outcome Regret (Proposition 6.2) 33
F Ethical Considerations 34
A Additional Simulation Results
In this section, we present additional simulation results on the Groh and McKenzie [2016] data. We refer to Section 7 for a review of the experimental set-up. In this section, we focus on the full dataset where  $T = 14,445$ . Simulations were run on a 2019 MacBook Pro with 2.4 GHz Quad-Core Intel Core i5 and 16 GB LPDDR3 RAM.
A.1 Confidence Intervals
Chebyshev Width	Chebyshev Coverage	Normal Width	Normal Coverage
Bernoulli (p=1/2)	0.0541	100%	0.0237	95.21%
CLIP-OGD	0.0507	99.99%	0.0222	95.22%
Table 1:  ${95}\%$  Confidence Intervals for Bernoulli and CLIP-OGD
Table 1 presents the Chebyshev-based and Normal-based intervals for the Bernoulli design  $(p = 1 / 2)$  and CLIP-OGD. We see that while Chebyshev over-covers, the normal-based confidence intervals cover at the nominal level with reduced width for both designs. The relative Neyman efficiency on this dataset is somewhat close to 1, so that the reduction of the width of the confidence intervals afforded by CLIP-OGD is present, though modest. The coverage of the normal-based confidence intervals provides further evidence supporting our conjecture that the adaptive Horvitz-Thompson estimator is asymptotically normal under CLIP-OGD.
Figure 2: Histogram of Studentized Adaptive Horvitz-Thompson estimator under CLIP-OGD ( $T = 14, 445$ )
Figure 2 plots the histogram of the studentized adaptive Horvitz-Thompson estimator under CLIP-OGD. By studentized, we mean that the histogram is plotting the draws of the random variable
$$
Z = \frac {\tau - \hat {\tau}}{\sqrt {\mathrm {V a r} (\hat {\tau})}}.
$$
We estimate the standard deviation empirically from 50,000 runs of the experiment. The estimator is said to be asymptotically normal if  $Z \xrightarrow{d} \mathcal{N}(0,1)$ . Figure 2 provides evidence that asymptotic normality is likely to hold in this setting. Formally establishing asymptotic normality is beyond the scope of the current paper as it would involve very different analytic techniques than those used to establish sublinear Neyman regret.
A.2 Sensitivity to Step Size
In this section, we explore through simulations how the performance of CLIP-OGD depends on the step size.
Figure 3: Comparing Step Sizes:
In Figure 3, we re-create Figure 1a in the main paper but we have added instances of CLIP-OGD with different step sizes of the forms  $\eta = c / \sqrt{T}$  for  $c\in \{0.25,0.5,1.0,2.0,4.0\}$ . We find that smaller step sizes improve convergence rates, effectively removing the "overhead of adaptivity" in this example. However, because the randomized experiment can only be run once, experimenters will typically not be able to try many step sizes. While it remains an open question about how to select a step size which best mitigates the "overhead of adaptivity", our recommendation of  $1 / \sqrt{T}$  still maintains good convergence properties.
A.3 Alternative Designs
In this section, we conduct additional experiments to compare the results of CLIP-OGD to alternative experimental designs which are not made for Neyman allocation. Indeed, we find that the alternative designs incur a high variance, relative to CLIP-OGD and the Neyman variance.
(a) DBCD only
Figure 4: Comparison of (Unnormalized) Variances:
(b) DBCD and EXP3
In Figure 4, we plot the variance of the adaptive designs on an unnormalized scale. We include "Doubly Biased Coin Design" proposed by Eisele [1994] as DBCD in Fig 4a and both DBCD and EXP3 in Fig 4b. We find that both DBCD and EXP3 suffer from higher variance. This is because they are not designed for Adaptive Neyman Allocation as defined in this paper: DBCD targets a different allocation rule and EXP3 minimizes outcome regret so that essentially only one arm is pulled. Both of these algorithms let the sampling probabilities  $P_{t}$  get too close to the boundary of  $[0,1]$ , resulting in excessively large variance.
B General Analysis of Adaptive Neyman Allocation
In this section, we provide general analysis relevant for the problem of Adaptive Neyman Allocation. In Section B.1, we analyze the adaptive Horvitz-Thompson estimator. In Section B.2, we derive the optimal non-adaptive Neyman design in terms of the potential outcomes. In Section B.3, we show the equivalence of Neyman ratio and expected Neyman regret. For completeness, all propositions re-appear in the appendix.
B.1 Analysis of Adaptive Horvitz-Thompson Estimator (Propositions 2.1 and 2.2)
Throughout these proofs, we break up the adaptive Horvitz-Thompson estimator into the sum of individual estimators. For each  $t \in [T]$ , define
$$
\hat {\tau} _ {t} = Y _ {t} \Big (\frac {\mathbf {1} [ Z _ {t} = 1 ]}{P _ {t}} - \frac {\mathbf {1} [ Z _ {t} = 0 ]}{1 - P _ {t}} \Big)
$$
so that the sequential Horvitz-Thompson estimator is equal to  $\hat{\tau} = (1 / T)\sum_{t = 1}^{T}\hat{\tau}_{t}$ . This mirrors how the average treatment effect is the average of individual treatment effects, i.e.  $\tau = (1 / T)\sum_{t = 1}^{T}\tau_{t}$ .
We begin by proving Proposition 2.1, which establishes the unbiasedness of the adaptive Horvitz-Thompson estimator, subject to a positivity condition.
Proof. Observe that by linearity of expectation, we can break the expectation of the adaptive Horvitz-Thompson estimator as
$$
\mathbb {E} [ \hat {\tau} ] = \frac {1}{T} \sum_ {t = 1} ^ {T} \mathbb {E} [ \hat {\tau} _ {t} ] .
$$
Thus, it suffices to show that the individual effect estimators are unbiased:  $\mathbb{E}[\hat{\tau}_t] = \tau_t$ . Observe that if the positivity condition holds, then we have that the conditional expectation may be computed as
$$
\begin{array}{l} \mathbb {E} [ \hat {\tau} _ {t} \mid \mathcal {F} _ {t} ] = \mathbb {E} \bigg [ Y _ {t} \bigg (\frac {\mathbf {1} [ Z _ {t} = 1 ]}{P _ {t}} - \frac {\mathbf {1} [ Z _ {t} = 0 ]}{1 - P _ {t}} \bigg) \mid \mathcal {F} _ {t} \bigg ] \\ = P _ {t} \cdot \left(\frac {y _ {t} (1)}{P _ {t}}\right) + (1 - P _ {t}) \cdot \left(\frac {y _ {t} (0)}{1 - P _ {t}}\right) \\ = y _ {t} (1) - y _ {t} (0) \\ = \tau_ {t}. \\ \end{array}
$$
The result follows by iterated expectation,  $\mathbb{E}[\hat{\tau}_t] = \mathbb{E}[\mathbb{E}[\hat{\tau}_t\mid \mathcal{F}_t]] = \tau_t$
Next, we prove Proposition 2.2, which derives the variance of the adaptive Horvitz-Thompson estimator.
Proof. We begin by decomposing the variance of the adaptive Horvitz-Thompson estimator as
$$
\mathrm {V a r} (\hat {\tau}) = \mathrm {V a r} \Big (\frac {1}{T} \sum_ {t = 1} ^ {T} \hat {\tau} _ {t} \Big) = \frac {1}{T ^ {2}} \sum_ {t = 1} ^ {T} \sum_ {s = 1} ^ {T} \mathrm {C o v} (\hat {\tau} _ {t}, \hat {\tau} _ {s}) .
$$
We now aim to compute each of these individual covariance terms. Before continuing, observe that, by construction, the individual effect estimators are conditionally unbiased  $\mathbb{E}[\hat{\tau}_t\mid \mathcal{F}_t] = \tau_t$ . It follows by iterated expectation that the individual effect estimators are unbiased (unconditionally), i.e.  $\mathbb{E}[\hat{\tau}_t] = \tau_t$ . Suppose that  $s > t$ . In this case, the covariance between the individual estimators is equal to zero as,
$$
\begin{array}{l} \operatorname {C o v} \left(\hat {\tau} _ {t}, \hat {\tau} _ {s}\right) = \mathbb {E} \left[ \hat {\tau} _ {t} \hat {\tau} _ {s} \right] - \mathbb {E} \left[ \hat {\tau} _ {t} \right] \mathbb {E} \left[ \hat {\tau} _ {s} \right] \\ = \mathbb {E} \big [ \hat {\tau} _ {t} \mathbb {E} \big [ \hat {\tau} _ {s} \mid \mathcal {F} _ {s} \big ] \big ] - \tau_ {t} \tau_ {s} \\ = \tau_ {s} \mathbb {E} \big [ \hat {\tau} _ {t} \big ] - \tau_ {t} \tau_ {s} \\ \end{array}
$$
$$
\begin{array}{l} = \tau_ {s} \tau_ {t} - \tau_ {t} \tau_ {s} \\ = 0. \\ \end{array}
$$
Now let us compute the variance of an individual effect estimator. Observe that the variance may be decomposed as
$$
\mathrm {V a r} \big (\hat {\tau} _ {t} \big) = \mathbb {E} \big [ \hat {\tau} _ {t} ^ {2} \big ] - \mathbb {E} \big [ \hat {\tau} _ {t} \big ] ^ {2}.
$$
Because the individual estimator is unbiased, we have that  $\mathbb{E}[\hat{\tau}_t]^2 = \tau_t^2$ . Let us now analyze the first term.
(iterated expectation)
$$
\begin{array}{l} \mathbb {E} \big [ \hat {\tau} _ {t} ^ {2} \big ] = \mathbb {E} \big [ \mathbb {E} \big [ \hat {\tau} _ {t} ^ {2} \mid \mathcal {F} _ {t} \big ] \big ] \\ = \mathbb {E} \left[ \mathbb {E} \left[ Y _ {t} ^ {2} \left(\frac {\mathbf {1} [ Z _ {t} = 1 ]}{P _ {t} ^ {2}} - \frac {\mathbf {1} [ Z _ {t} = 0 ]}{(1 - P _ {t}) ^ {2}}\right) \mid \mathcal {F} _ {t} \right] \right] \\ = \mathbb {E} \left[ y _ {t} (1) ^ {2} \frac {1}{P _ {t}} + y _ {t} (0) ^ {2} \frac {1}{1 - P _ {t}} \right] \\ = y _ {t} (1) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{P _ {t}} \right] + y _ {t} (0) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{1 - P _ {t}} \right]. \\ \end{array}
$$
Thus, this establishes that the variance of an individual estimator is equal to
$$
\operatorname {V a r} \left(\hat {\tau} _ {t}\right) = y _ {t} (1) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{P _ {t}} \right] + y _ {t} (0) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{1 - P _ {t}} \right] - \tau_ {t} ^ {2}.
$$
Combining terms, we have that the variance of the adaptive Horvitz-Thompson estimator is
$$
\begin{array}{l} T \cdot \operatorname {V a r} (\hat {\tau}) = \frac {1}{T} \sum_ {t = 1} ^ {T} \sum_ {s = 1} ^ {T} \operatorname {C o v} (\hat {\tau} _ {t}, \hat {\tau} _ {s}) \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} \operatorname {V a r} \left(\hat {\tau} _ {t}\right) \\ = \frac {1}{T} \sum_ {t = 1} ^ {T} \left(y _ {t} (1) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{P _ {t}} \right] + y _ {t} (0) ^ {2} \cdot \mathbb {E} \left[ \frac {1}{1 - P _ {t}} \right]\right) - \frac {1}{T} \sum_ {t = 1} ^ {T} \tau_ {t} ^ {2} \\ \end{array}
$$
B.2 Derivation of the Neyman Design (Proposition 3.1)
In this section, we prove Proposition 3.1 which derives the (infeasible) non-adaptive Neyman design in terms of the Neyman probability  $p^*$  and corresponding Neyman variance  $V_{\mathrm{N}}$ . We also show that, under Assumption 1, the Neyman variance achieves the parametric rate.
Proof. Using Proposition 2.2, we have that the variance of the (non-adaptive) Bernoulli design with probability  $p \in (0,1)$  is equal to
Thus, the optimal Neyman design is obtained by the  $p^*$  which minimizes the above. The first order condition stipulates that
$$
\left. \frac {\partial}{\partial p} \Big [ T \cdot V _ {p} \Big ] \right| _ {p = p ^ {*}} = 0 \Leftrightarrow - S (1) ^ {2} \Big (\frac {1}{p ^ {*}} \Big) ^ {2} + S (0) ^ {2} \Big (\frac {1}{1 - p ^ {*}} \Big) ^ {2} = 0 ,
$$
which is solved by  $p^* = \left(1 + S(0) / S(1)\right)^{-1}$ . Substituting this  $p^*$  back into the variance yields the Neyman variance:
$$
\begin{array}{l} T \cdot V _ {p ^ {*}} = S (1) ^ {2} \left(\frac {1}{p ^ {*}} - 1\right) + S (0) ^ {2} \left(\frac {1}{1 - p ^ {*}} - 1\right) + 2 \rho S (1) S (0) \\ = S (1) ^ {2} \cdot \frac {S (0)}{S (1)} + S (0) ^ {2} \cdot \frac {S (1)}{S (0)} + 2 \rho S (1) S (0) \\ = 2 (1 + \rho) S (1) S (0). \\ \end{array}
$$
Next, we show that under Assumption 1, the Neyman variance achieves the parametric rate.
Proposition B.1. Under Assumption 1, the Neyman variance achieves the parametric rate:  $V_{N} = \Theta (1 / T)$ .
Proof. Proposition 3.1, derives the Neyman variance:  $T \cdot V_{\mathrm{N}} = 2(1 + \rho)S(1)S(0)$ .
We begin by showing that the Neyman variance is asymptotically bounded from below. Moreover, Assumption 1 stipulates that there exists a constant  $c > 0$  which lower bounds the second moments as  $S(1) \geqslant c$  and  $S(0) \geqslant c$  and the correlation as  $(1 + \rho) \geqslant c$ . Thus, the normalized Neyman variance is bounded below as  $T \cdot V_{\mathrm{N}} \geqslant 2c^3$ .
Next, we show that the Neyman variance is asymptotically bounded from above at the same rate. Assumption 1 stipulates that there exists a constant  $C > 0$  which upper bounds the second moments as  $S(1) \leqslant C$  and  $S(0) \leqslant C$ . The correlation is bounded between  $\rho \in [-1, 1]$  so that  $(1 + \rho) \leqslant 2$ . These bounds together yield that the normalized Neyman variance is bounded above as  $T \cdot V_{\mathrm{N}} \leqslant 4C^2$ .
Together, these bounds establish that, under Assumption 1, we have that  $V_{\mathrm{N}} = \Theta (1 / T)$ .
B.3 Equivalence of Neyman Ratio and Neyman Regret (Theorem 4.1)
In this section, we prove Theorem 4.1, which demonstrates the equivalence between the Neyman Ratio and the expected Neyman regret.
Theorem 4.1. Under Assumption 1, the Neyman ratio is within a constant factor of the  $1 / T$ -scaled expected Neyman regret:  $\kappa_{T} = \Theta \left(\frac{1}{T}\mathbb{E}[\mathcal{R}_{T}]\right)$ .
Proof. Recall that the Neyman ratio is defined as
$$
\kappa_ {T} = \frac {V - V _ {\mathrm {N}}}{V _ {\mathrm {N}}} = \frac {T \cdot V - T \cdot V _ {\mathrm {N}}}{T \cdot V _ {\mathrm {N}}} ,
$$
where the second equality follows by multiplying the numerator and the denominator by  $T$ . Observe that by Proposition 2.2, the numerator is given by
$$
\begin{array}{l} T \cdot V - T \cdot V _ {\mathrm {N}} = \frac {1}{T} \sum_ {t = 1} ^ {T} \left(y _ {t} (1) ^ {2} \cdot \mathbb {E} \Big [ \frac {1}{P _ {t}} \Big ] + y _ {t} (0) ^ {2} \cdot \mathbb {E} \Big [ \frac {1}{1 - P _ {t}} \Big ]\right) \\ - \min  _ {p ^ {*} \in [ 0, 1 ]} \frac {1}{T} \sum_ {t = 1} ^ {T} \left(y _ {t} (1) ^ {2} \cdot \frac {1}{p ^ {*}} + y _ {t} (0) ^ {2} \cdot \frac {1}{1 - p ^ {*}}\right) \\ = \mathbb {E} \left[ \frac {1}{T} \sum_ {t = 1} ^ {T} f _ {t} \left(P _ {t}\right) \right] - \min  _ {p ^ {*} \in [ 0, 1 ]} \frac {1}{T} \sum_ {t = 1} ^ {T} f _ {t} \left(p ^ {*}\right) \\ = \frac {1}{T} \mathbb {E} \left[ \sum_ {t = 1} ^ {T} f _ {t} \left(P _ {t}\right) - \min  _ {p ^ {*} \in [ 0, 1 ]} \sum_ {t = 1} ^ {T} f _ {t} \left(p ^ {*}\right) \right] \\ = \frac {1}{T} \mathbb {E} \left[ \mathcal {R} _ {T} \right]. \\ \end{array}
$$
Proposition B.1 shows that under Assumption 1,  $T \cdot V_{\mathrm{N}} = \Theta(1)$  so that the denominator is asymptotically constant. Thus, we have that  $\kappa_T = \Theta(\frac{1}{T} \mathbb{E}[\mathcal{R}_T])$ .
C Analysis of Neyman Regret for CLIP-OGD
In this section, we will prove Theorem 4.2, which establish that CLIP-OGD achieves  $\mathcal{O}(\sqrt{T\log T})$  expected Neyman regret under our assumptions on the potential outcomes. While the main paper used capital letters  $P_{t}$  and  $G_{t}$  to signify that the treatment probability and gradient estimator were random variables, we use lower case letters  $p_t$  and  $g_{t}$  in the appendix for the purposes of more aesthetically appealing proofs. Throughout the analysis, we define  $\Delta_t = [\delta_t,1 - \delta_t]$  and  $a = 1 + C / c$  for notational convenience.
C.1 Proof of Theorem 4.2
The first lemma is a bound on the distance of treatment probability  $p_{t + 1}$  to the optimal  $p^*$  in terms of the previous treatment probability, gradient estimate, and whether the projection interval contains the optimal  $p^*$ .
Lemma C.1. For each iteration  $t \in [T]$ ,
$$
\left| p _ {t + 1} - p ^ {*} \right| \leqslant \left| \left(p _ {t} - \eta g _ {t}\right) - p ^ {*} \right| + \delta_ {t} \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right].
$$
Proof. If  $p^{\star} \in \Delta_{t}$ , then the statement holds by Pythagorean theorem. Otherwise, note that the most that the projection operation onto  $\Delta_{t}$  can move a point is exactly  $\delta_{t}$ . Therefore, the distance between  $\mathcal{P}_{\delta_t}(p_t - \eta g_t)$  and  $p^\star$  is at most  $\delta_{t}$  larger than that between  $p_t - \eta g_t$  and  $p^\star$ .
Next, we show that Assumption 1 implies that the optimal Neyman probability lies within an interval bounded away from zero.
Lemma C.2. Under Assumption 1,  $p^* \in \left[\frac{1}{a}, 1 - \frac{1}{a}\right]$ , where  $a = 1 + C / c \geqslant 2$ .
Proof. As shown previously, the optimal Neyman probability is equal to
$$
p ^ {*} = \left(1 + \sqrt {\frac {\sum_ {t = 1} ^ {T} y _ {t} (0) ^ {2}}{\sum_ {t = 1} ^ {T} y _ {t} (1) ^ {2}}}\right) ^ {- 1}
$$
Recall that Assumption 1 places the following moment conditions on the potential outcomes:
$$
c \leqslant \left(\frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (1) ^ {2}\right) ^ {1 / 2} \leqslant C \quad \text {a n d} \quad c \leqslant \left(\frac {1}{T} \sum_ {t = 1} ^ {T} y _ {t} (1) ^ {2}\right) ^ {1 / 2} \leqslant C.
$$
This bounds  $p^*$  by
$$
\left(1 + \frac {C}{c}\right) ^ {- 1} \leqslant p ^ {*} \leqslant \left(1 + \frac {c}{C}\right) ^ {- 1}.
$$
The result follows by using the definition of  $a = 1 + C / c$  to deduce that  $(1 / a) = (1 + C / c)^{-1}$  and  $(1 - 1 / a) = (1 + c / C)^{-1}$ .
The next lemma guarantees that after a fixed number of iterations, the projection interval will contain the Neyman optimal  $p^*$ .
Lemma C.3. We have that  $p^* \in \Delta_t$  for all  $t \geqslant (a/2)^\alpha$ .
Proof. Lemma C.2 guarantees that  $p^* \in [1/a, 1 - 1/a]$ , where  $a = 1 + C/c$ . Thus,  $p^* \in \Delta_t$  if  $\delta_t \leqslant 1/a$ . Using the definition of  $\delta_t$  and rearranging terms, we have that
$$
\delta_ {t} \leqslant 1 / a \Leftrightarrow (1 / 2) t ^ {- 1 / \alpha} \leqslant 1 / a \Leftrightarrow t \geqslant (a / 2) ^ {\alpha}.
$$
The next lemma bounds the expected difference between the cost objective  $f_{t}$  evaluated at  $p_t$  and the cost objective  $f_{t}$  evaluated at the Neyman optimal probability  $p^*$ .
Lemma C.4. For each iteration  $t \in [T]$ , we have the bound
$$
2 \mathbb {E} \Big [ f _ {t} (p _ {t}) - f _ {t} (p ^ {*}) \Big ] \leqslant \frac {1}{\eta} \Big [ \mathbb {E} \Big [ (p _ {t} - p ^ {*}) ^ {2} \Big ] - \mathbb {E} \Big [ (p _ {t + 1} - p ^ {*}) ^ {2} \Big ] \Big ] + \eta \mathbb {E} \big [ g _ {t} ^ {2} \big ] + 4 \mathbf {1} \Big [ t <   \left(\frac {a}{2}\right) ^ {\alpha} \Big ] \Big (\frac {\delta_ {t}}{\eta} + \frac {\delta_ {t}}{2} \mathbb {E} \Big [ | g _ {t} | \Big ] \Big) .
$$
Proof. Fix an iteration  $t \in [T]$ . By Lemma C.1, and using the triangle inequality, we have that
$$
\begin{array}{l} \left. \left(p _ {t + 1} - p ^ {*}\right) ^ {2} \leqslant \left(\left| \left(p _ {t} - \eta g _ {t}\right) - p ^ {*} \right| + \delta_ {t} \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right]\right) ^ {2} \right. \\ = \left(\left(p _ {t} - \eta g _ {t}\right) - p ^ {*}\right) ^ {2} + \delta_ {t} ^ {2} \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right] ^ {2} + 2 | \left(p _ {t} - \eta g _ {t}\right) - p ^ {*} | \delta_ {t} \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right] \\ = \left(\left(p _ {t} - p ^ {*}\right) - \eta g _ {t}\right) ^ {2} + \mathbf {1} [ p ^ {*} \notin \Delta_ {t} ] \Big (\delta_ {t} ^ {2} + 2 \delta_ {t} \cdot | (p _ {t} - p ^ {*}) - \eta g _ {t} | \Big) \\ \leqslant \left(\left(p _ {t} - p ^ {*}\right) - \eta g _ {t}\right) ^ {2} + 2 \delta_ {t} \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right] \left(\delta_ {t} + \left| p _ {t} - p ^ {*} \right| + \eta \left| g _ {t} \right|\right) \\ \end{array}
$$
Because  $p_t \in [\delta_t, 1 - \delta_t]$ , we have that  $|p_t - p^*| \leqslant 1 - \delta_t$  so that
$$
\leqslant \left(\left(p _ {t} - p ^ {*}\right) - \eta g _ {t}\right) ^ {2} + 2 \delta_ {t} \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right] \left(1 + \eta | g _ {t} |\right)
$$
$$
= \leqslant (p _ {t} - p ^ {*}) ^ {2} + \eta^ {2} g _ {t} ^ {2} - 2 \eta g _ {t} (p _ {t} - p ^ {*}) + 4 \eta \cdot {\bf 1} [ p ^ {*} \notin \Delta_ {t} ] \Big (\frac {\delta_ {t}}{\eta} + \frac {\delta_ {t}}{2} | g _ {t} | \Big)
$$
Rearranging terms yields that
$$
2 \eta g _ {t} \left(p _ {t} - p ^ {*}\right) \leqslant \left[ \left(p _ {t} - p ^ {*}\right) ^ {2} - \left(p _ {t + 1} - p ^ {*}\right) ^ {2} \right] + \eta^ {2} g _ {t} ^ {2} + 4 \eta \cdot \mathbf {1} \left[ p ^ {*} \notin \Delta_ {t} \right] \left(\frac {\delta_ {t}}{\eta} + \frac {\delta_ {t}}{2} | g _ {t} |\right).
$$
Dividing both sides by the step size  $2\eta$  yields
$$
g _ {t} \left(p _ {t} - p ^ {*}\right) \leqslant \frac {1}{2 \eta} \left[ \left(p _ {t} - p ^ {*}\right) ^ {2} - \left(p _ {t + 1} - p ^ {*}\right) ^ {2} \right] + \frac {\eta}{2} g _ {t} ^ {2} + 2 \mathbf {1} [ p ^ {*} \notin \Delta_ {t} ] \left(\frac {\delta_ {t}}{\eta} + \frac {\delta_ {t}}{2} | g _ {t} |\right).
$$
Using convexity of  $f_{t}$ , adding and subtracting terms, and using the above, we have that
$$
\begin{array}{l} f _ {t} \left(p _ {t}\right) - f _ {t} \left(p ^ {*}\right) \leqslant \left\langle \nabla f _ {t} \left(p _ {t}\right), p _ {t} - p ^ {*} \right\rangle (convexity) \\ = \left\langle g _ {t}, p _ {t} - p ^ {*} \right\rangle + \left\langle \nabla f _ {t} \left(p _ {t}\right) - g _ {t}, p _ {t} - p ^ {*} \right\rangle \quad (a d d i n g, s u b t r a c t i n g) \\ \leqslant \frac {1}{2 \eta} \left[ \left(p _ {t} - p ^ {*}\right) ^ {2} - \left(p _ {t + 1} - p ^ {*}\right) ^ {2} \right] + \frac {\eta}{2} g _ {t} ^ {2} + 2 \mathbf {1} [ p ^ {*} \notin \Delta_ {t} ] \left(\frac {\delta_ {t}}{\eta} + \frac {\delta_ {t}}{2} | g _ {t} |\right) (above) \\ + \left\langle \nabla f _ {t} \left(p _ {t}\right) - g _ {t}, p _ {t} - p ^ {*} \right\rangle \\ \end{array}
$$
By construction, we have that the gradient estimator is unbiased conditioned on  $D_{t}$ , i.e.  $\mathbb{E}[g_t\mid D_t] = \nabla f_t(p_t)$ . Thus, by iterated expectation we have that the gradient estimator is unbiased, i.e.
$$
\mathbb {E} \left[ \nabla f _ {t} \left(p _ {t}\right) - g _ {t} \right] = \mathbb {E} \left[ \mathbb {E} \left[ \nabla f _ {t} \left(p _ {t}\right) - g _ {t} \mid D _ {t} \right] \right] = 0.
$$
Thus, taking expectations of both sides and applying Lemma C.3 yields