SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Mehrdad Ghadiri
mehrdadg@mit.edu
David Arbour
Adobe Research
arbour@adobe.com
Tung Mai
tumai@adobe.com
Cameron Musco
cmusco@cs.umass.edu
Anup B. Rao
anuprao@adobe.com
The design and analysis of randomized experiments is fundamental to many areas, from the physical and social sciences to industrial settings. Regression adjustment is a popular technique to reduce the variance of estimates obtained from experiments, by utilizing information contained in auxiliary covariates. While there is a large literature within the statistics community studying various approaches to regression adjustment and their asymptotic properties, little focus has been given to approaches in the finite population setting with non-asymptotic accuracy bounds. Further, prior work typically assumes that an entire population is exposed to an experiment, whereas practitioners often seek to minimize the number of subjects exposed to an experiment, for ethical and pragmatic reasons. In this work, we study the problems of estimating the sample mean, individual treatment effects, and average treatment effect with regression adjustment. We propose approaches that use techniques from randomized numerical linear algebra to sample a subset of the population on which to perform an experiment. We give non-asymptotic accuracy bounds for our methods and demonstrate that they compare favorably with prior approaches.
Randomized experiments play a crucial role in estimating counterfactual outcomes. The strength of randomization lies in its ability to ensure independence between the treatment group and pretreatment covariates [14, 21]. This independence, in turn, results in unbiased estimation of treatment effects through the comparison of sample averages. However, in finite sample settings, the use of simple randomization techniques result in high variance of the estimated quantities. There have been two broad approaches to address this issue by reducing the variance of the estimated effects when subject level covariates are available: (1) design-based approaches, which focus on the mechanism for assigning individuals to treatment and control groups in order to minimize imbalance (c.f. [21, 18, 27, 34, 5, 15]), and (2) regression adjustment based approaches, which correct for imbalances post hoc by incorporating a regression of the covariates on the outcome (c.f. [9, 24, 7, 13, 29]).
Much of the prior work examining variance reduction for experimentation has focused on asymptotic properties of approaches [23, 24, 13, 21, 23]. Recently, there has been work focusing on design-based variance reduction, which ties the treatment assignment problem to discrepancy minimization and gives finite sample error bounds for estimating causal estimands such as the average treatment effect (ATE) [18, 1, 5] and individual treatment effect (ITE) [1]. In particular, Harshaw et al. propose a
design based on the Gram-Schmidt random walk (GSW) and give finite sample guarantees on the mean squared error for average treatment effect estimation when using this design with the classic Horvitz-Thompson estimator. Addanki, et al. [1] use GSW and leverage score sampling to provide algorithms and finite sample guarantees for both average and individual treatment effect estimation in the partial observation setting where only a subset of subjects are included in the experiment.
However, little work has examined the finite sample properties for experimental designs based on regression adjustment. Mou, et al. [28] provide non-asymptotic guarantees, but the analysis is in infinite population settings. The main contribution of this work is to analyze the finite population behavior of regression adjustment for variance reduction in treatment effect estimation, for ATE and ITE estimation in both the full and partial (sub-population) settings.
1.1 Problem Statement: Finite Population Treatment Effect Estimation
In the finite population treatment effect estimation problem, we have  $n$  units (or individuals) associated with covariates  $\mathbf{x}_1,\ldots ,\mathbf{x}_n\in \mathbb{R}^d$  which are the rows of a covariate matrix  $\mathbf{X}\in \mathbb{R}^{n\times d}$ . Each unit  $i$  is associated with two real numbers  $y_{i}^{(1)},y_{i}^{(0)}$  called the potential outcomes under the treatment and control, respectively. We denote the corresponding potential outcome vectors with  $\mathbf{y}^{(1)},\mathbf{y}^{(0)}\in \mathbb{R}^n$ . The Individual Treatment Effect (ITE) vector  $\mathbf{t}\in \mathbb{R}^n$  is defined as  $\mathbf{t}\coloneqq \mathbf{y}^{(1)} - \mathbf{y}^{(0)}$  and the Average Treatment Effect (ATE)  $\tau$  is defined as the mean of this vector, i.e.,  $\tau \coloneqq \frac{1}{n}\sum_{i = 1}^{n}t_{i}$ .
In treatment effect estimation, we seek to estimate either the ITE or ATE, under the constraint that at most one of the potential outcomes, either  $y_{i}^{(1)}$  or  $y_{i}^{(0)}$ , can be observed for each unit  $i$ . E.g., that unit may be assigned to a treatment group in a controlled trial, and thus  $y_{i}^{(1)}$  is observed while  $y_{i}^{(0)}$  is not.
In full observation treatment effect estimation, we observe one potential outcome for each unit in the population [9, 13, 24, 18]. In partial observation treatment effect estimation, we further restrict ourselves to observing outcomes for just a small subsample of the population. This setting is important when the goal is to minimize experimental costs by e.g., limiting the size of a controlled trial [30, 20, 32, 1].
To reduce variance, many approaches to ATE and ITE estimation, including our own, attempt to leverage the population covariates  $\mathbf{X}$  to make inferences about the treatment outcomes. In particular, throughout this work, we will target error bounds that depend on the best linear fit of the treatment outcomes to our covariates. Such linear effects models are common in the literature on treatment effect estimation [33, 18, 1] and provide a useful baseline for establishing theoretical bounds.
We give new approaches to both ITE and ATE estimation in the full and partial observation settings that combine the classic statistical technique of regression adjustment with techniques from randomized numerical linear algebra. Our algorithms give natural, non-asymptotic error bounds depending on the best linear fit for either the ITE vector  $\mathbf{t} \coloneqq \mathbf{y}^{(1)} - \mathbf{y}^{(0)}$  or the sum of the potential outcome vectors  $\pmb{\mu} \coloneqq \mathbf{y}^{(1)} + \mathbf{y}^{(0)}$  to our covariates.  $\pmb{\mu}$  is a natural quantity that arises e.g., in variance bounds for classic approaches to ATE estimation, such as the Hovitz-Thompson estimator [19].
1.2.1 Individual Treatment Effect Estimation
Our first result is for individual treatment effect estimation in the full and partial observation settings.
Theorem 1 (ITE Estimation). For any  $\epsilon, \delta, \alpha \in (0,1]$ , there exists a randomized algorithm (Algorithm 3) that observes a potential outcome for  $O(d\log(d/\delta)/\epsilon^2 + \alpha \cdot n)$  units and outputs a vector  $\widehat{\mathbf{t}}$ . Moreover there is an event  $\mathcal{E}$  such that  $\operatorname*{Pr}(\mathcal{E}) \geq 1 - \delta$  (over the randomness of the algorithm) and
$$
\mathbb {E} \left[ \| \widehat {\mathbf {t}} - \mathbf {t} \| _ {2} ^ {2} | \mathcal {E} \right] \leq (1 + \epsilon) \cdot \min  _ {\mathbf {b} \in \mathbb {R} ^ {d}} \| \mathbf {X b} - \mathbf {t} \| _ {2} ^ {2} + \frac {d}{\alpha} \cdot (1 + \epsilon) \cdot \left\| \mathbf {y} ^ {(1)} + \mathbf {y} ^ {(0)} \right\| _ {\infty} ^ {2}.
$$
Theorem 1 shows that with high probability (at least  $1 - \delta$ ), our algorithm achieves an expected error bound that nearly matches the best linear fit of the ITE vector to our covariates (i.e.,  $\min_{\mathbf{b} \in \mathbb{R}^d} \left\| \mathbf{X}\mathbf{b} - \mathbf{t} \right\|_2^2$ ), up to an additive term depending on the maximum magnitude of the sum of potential outcomes for any individual. In the full observation setting, when  $\alpha = 1$ , this additive
term is  $O(d)$ , assuming that the maximum magnitude does not scale with the population size  $n$ . In contrast, we expect the best fit error  $\left(\min_{\mathbf{b} \in \mathbb{R}^d} \|\mathbf{X}\mathbf{b} - \mathbf{t}\|_2^2\right)$  to grow as  $\Theta(n)$ , and thus the additive term is lower order. In the partial observation setting, as long as we set  $\alpha = \omega(1/n)$ , we still expect this additive term to be lower order. Thus, the case when  $d \ll n$ , we are able to nearly match the best fit error while only observing a very small subset of the full population.
We sketch the proof of Theorem 1 in Section 2. Our algorithm is based on pseudo-outcome regression, which is studied in the asymptotic setting by Kennedy et al. for conditional ATE estimation [22]. Roughly, in the full observation setting, we construct a vector  $\mathbf{v}$  where  $v_{i} = 2y_{i}^{(1)}$  with probability  $1 / 2$  and  $v_{i} = -2y_{i}^{(0)}$  with probability  $1 / 2$ . We can see that  $\mathbb{E}[\mathbf{v}] = \mathbf{t}$  (i.e.,  $\mathbf{v}$  is equal to the ITE vector in expectation), and importantly, that constructing  $\mathbf{v}$  only requires observing one potential outcome for each individual. By regressing  $\mathbf{v}$  onto our covariates, we obtain our estimate  $\hat{\mathbf{t}}$  for  $\mathbf{t}$ .
In the partial observation setting, we further subsample individuals according to the leverage scores of the covariate matrix  $\mathbf{X}$ . This is a standard technique in randomized linear algebra [26, 10, 35], which allows us to approximately solve the above regression problem while only observing a subset of  $\widetilde{O}(d/\epsilon^2)$  entries of  $\mathbf{v}$  (and thus only observing outcomes for a subset of the population). Addanki et al. [1] similarly propose an algorithm for the partial observation setting based on leverage score sampling. However, instead of pseudo-outcome regression, they learn separate approximations to both outcome vectors  $\mathbf{y}^{(1)}$  and  $\mathbf{y}^{(0)}$ . Thus, their final error depends on the best linear fit error for both these vectors, rather than the more natural best linear fit error for the ITE vector  $\mathbf{t}$ .
1.2.2 Average Treatment Effect Estimation
We can extend our general approach to give bounds for average treatment effect estimation. In the full observation setting we show:
Theorem 2 (ATE Estimation - Full Observation). For any  $\epsilon, \delta \in (0,1]$ , there exists a randomized algorithm (Algorithm 1) that computes an unbiased estimate  $\widehat{\tau}$  of the ATE  $\tau$ , and for which there is an event  $\mathcal{E}$  with  $\operatorname*{Pr}(\mathcal{E}) \geq 1 - \delta$  (over the randomness of the algorithm) such that
$$
\mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} \right] \leq \frac {8 (1 + \epsilon)}{n ^ {2}} \min _ {\mathbf {b} \in \mathbb {R} ^ {d}} \left(\| \mathbf {X b} - \boldsymbol {\mu} \| _ {2} ^ {2} + 1 0 0 \log (n / \delta) \cdot \zeta^ {2} \cdot \| \mathbf {b} \| _ {2} ^ {2}\right) + \frac {3 2 d}{n ^ {2}} \cdot \| \mathbf {y} ^ {(1)} - \mathbf {y} ^ {(0)} \| _ {\infty} ^ {2},
$$
where  $\zeta \coloneqq \max_{i\in [n]}\| \mathbf{x}_i\| _2$  and  $\pmb {\mu}\coloneqq \mathbf{y}^{(0)} + \mathbf{y}^{(1)}$  is the total outcome vector.
As with our result in ITE estimate, the error bound of Theorem 2 matches the error of the best linear fit of the total outcome vector  $\pmb{\mu}$  to our covariates up to a constant factor plus additive terms depending on 1) the maximum magnitude of the sum of potential outcomes for any individual, and 2) the norm of the coefficient vector used to approximately reconstruct  $\pmb{\mu}$  from the covariates. Again, we expect these additive terms to scale as  $O(1 / n^2)$  while we expect the term depending on the best fit error to scale as  $O(1 / n)$ . Thus, we generally expect the additive error to be lower order.
Harshaw et al. [18] proposed an algorithm based on the Gram-Schmidt walk (GSW) design (see Algorithm 5 in the appendix) for balancing assignments to treatment and control groups that achieves an unbiased estimator for ATE with variance of  $\frac{1}{n^2}\min_{\mathbf{b}}\left[\frac{1}{\phi}\|\pmb{\mu} - \mathbf{X}\mathbf{b}\|_2^2 + \frac{\zeta^2}{1 - \phi}\|\mathbf{b}\|_2^2\right]$ , where  $\phi \in (0,1)$  is chosen by the experiment designer. This guarantee is comparable to but stronger than ours, e.g., when one sets  $\phi = 1/2$ . If we ignore the additive error terms and focus just on the best linear fit error  $\|\mathbf{X}\mathbf{b} - \pmb{\mu}\|_2^2$ , then GSW is better than our guarantees by about a factor of 8. But the GSW design is computationally more expensive and, since it is based on balancing the covariates, requires availability of all covariates before running the experiment and hence cannot be applied in an online setting [4]. In contrast, our approach can be applied directly in the online setting since we place each unit independently in the treatment or control groups with equal probability. Moreover, as we show in our experiments, the empirical performance of our algorithm is much closer to GSW than the theoretical bounds we are able to show.
We sketch the ideas behind Theorem 2 in Section 2. Our algorithm is based on the classic Horvitz-Thompson estimator [19]. Roughly, this estimator randomly assigns individuals to control and treatment groups, and estimates ATE as the difference in the average outcome between these groups, appropriately weighted by the assignment probabilities. It is well known that the variance of this estimator is bounded by  $\frac{1}{n^2}\|\pmb{\mu}\|_2^2$ . To reduce this variance, we introduce full-observation regression
adjusted Horvitz-Thompson (RAHT) estimator, which estimates ATE as the following. We partition the units into two groups  $S$  and  $\overline{S}$  (this is a different partitioning than the partitioning into control and treatment), and we regress  $\mu_{S}$  and  $\mu_{\overline{S}}$  onto their corresponding covariates. As in the ITE case, since we cannot directly form  $\pmb{\mu}$ , we instead perform the above regressions using a random vector  $\mathbf{u}$  with  $\mathbb{E}[\mathbf{u}] = \pmb{\mu}$ , and where  $\mathbf{u}$  can be formed by only observing one potential outcome per individual. We then use the solution vector of group  $S$  to adjust the outcomes of the group  $\overline{S}$  and vice versa, and we apply the HT estimator to the adjusted outcomes. This gives the bound in Theorem 2.
Again, following a similar approach as in the ITE case, we can apply leverage score based subsampling to tackle ATE estimation in the partial observation setting as well, obtaining:
Theorem 3 (ATE Estimation - Partial Observation). For any  $\epsilon, \delta \in (0,1]$ ,  $\phi \in (0,1)$ , and  $m \in [n]$ , there exists a randomized algorithm (Algorithm 4) that observes a potential outcome for  $O(d\log(d/\delta)/\epsilon^2 + m)$  units and outputs an unbiased estimate  $\hat{\tau}$  of the ATE  $\tau$ , for which there is an event  $\mathcal{E}$  with  $\operatorname{Pr}(\mathcal{E}) \geq 1 - \delta$  (over the randomness of the algorithm), such that
$$
\begin{array}{l} \mathbb {E} \left[ (\widehat {\tau} - \tau) ^ {2} | \mathcal {E} \right] \leq \frac {1}{m n} \frac {1}{\phi} \| \mathbf {X b} ^ {*} - \boldsymbol {\mu} \| _ {2} ^ {2} + \frac {1}{m ^ {2}} \frac {\zeta^ {2}}{(1 - \phi)} \| \mathbf {b} ^ {*} \| _ {2} ^ {2} + \frac {1 0 0 d \cdot \log (d / \delta)}{n ^ {2} \epsilon^ {2}} \| \boldsymbol {\mu} \| _ {\infty} ^ {2} \\ + (1 + \epsilon) \cdot \left(\frac {1}{m} \| \boldsymbol {\mu} \| _ {\infty} ^ {2} + \frac {1}{m n} \| (\mathbf {X} - \overline {{\mathbf {X}}}) \widehat {\mathbf {b}} - (\mathbf {t} - \overline {{\mathbf {t}}}) \| _ {2} ^ {2} + \frac {\lambda}{m n} \| \widehat {\mathbf {b}} \| _ {2} ^ {2}\right), \\ \end{array}
$$
where  $\zeta := \max_{i \in [n]} \| \mathbf{x}_i \|_2$ ,  $\pmb{\mu} := \mathbf{y}^{(0)} + \mathbf{y}^{(1)}$ ,  $\lambda = \frac{6 \cdot \log(d / \delta)}{\epsilon^2} \cdot \zeta^2$ ,  $\overline{\mathbf{t}} \in \mathbb{R}^n$  is a vector where all the entries are equal to  $\bar{t} = \frac{1}{n} \sum_{i=1}^{n} t_i$ , and
$$
\mathbf {b} ^ {*} = \arg \min _ {b \in \mathbb {R} ^ {d}} \left[ \frac {m}{n} \| \mathbf {X b} - \boldsymbol {\mu} \| _ {2} ^ {2} + \zeta^ {2} \| \mathbf {b} \| _ {2} ^ {2} \right], \widehat {\mathbf {b}} = \underset {\mathbf {b} \in \mathbb {R} ^ {d}} {\arg \min } \left[ \| (\mathbf {X} - \overline {{\mathbf {X}}}) \mathbf {b} - (\mathbf {t} - \bar {\mathbf {t}}) \| _ {2} ^ {2} + \lambda \| \mathbf {b} \| _ {2} ^ {2} \right].
$$
The error bound of Theorem 3 depends on 1) the error of best linear fits of the covariates onto  $\pmb{\mu}$  and  $\mathbf{t} - \overline{\mathbf{t}}$ , 2) the norm of the coefficient vector used to approximately reconstruct  $\pmb{\mu}$  and  $\mathbf{t} - \overline{\mathbf{t}}$  from the covariates, and 3) the largest component of the total outcome vector  $\pmb{\mu}$ . We expect both the best linear fit and  $\| \pmb{\mu}\|_{\infty}$  terms to scale as  $O\left(\frac{1}{m}\right)$ . All other terms are lower order. Namely, The terms depending on the norms of the coefficient vectors scale as  $O\left(\frac{1}{mn}\right)$  and  $O\left(\frac{1}{m^2}\right)$ . For the case where  $md \ll n$ , by increasing the number of samples to  $O(d\log (d / \delta) / \epsilon^2 + md)$ , we can make the  $\| \mu \|_{\infty}$  term to scale as  $O(1 / m^2)$  as well.
We deploy a different regression adjustment technique for achieving Theorem 3, which we call partial-observation regression adjusted Horvitz-Thompson (RAHT) estimator. In the partial observation setting, a simple approach is just to apply the full observation algorithm to a uniformly selected subset of  $m$  units and then report the ATE estimate for these  $m$  units. However, this leads to a variance bound depending on  $\| \mathbf{t} - \bar{\mathbf{t}} \|_2^2$ . To reduce this variance, the partial observation RAHT adjusts the estimate obtained from the  $m$  units using regression adjustment techniques. We give the details behind the full algorithm in Section 2.3 and provide the full analysis in the appendix.
1.2.3 Experimental Evaluation
We compliment the above theoretical bounds with experiments using both synthetic and real data, focusing on the full observation ATE and ITE estimation problems. For ATE, we compare our method with classical regression adjustment [24], the GSW method [18], and the algorithm of [28]. We observe that our estimator (which is unbiased) performs as well or better than all other estimators except the classic regression adjustment, which is a biased estimator. For ITE, we compare our approach with the method of [1], and observe superior performance. A discussion about the running time of our algorithm and comparison to GSW design is included in the appendix.
1.3 Roadmap
We present the high-level ideas and techniques for ITE and ATE estimation for the full and partial observation settings in Section 2. We then present notation and preliminaries required for a more in-depth discussion of our result in Section 3. In Section 4, we give a detailed non-asymptotic analysis of the random vector (i.e., pseudo-outcome) regression technique, which forms the basis for all of our algorithms. We present our algorithm for the full observation ATE estimation and sketch the proof of its accuracy (Theorem 2). Finally, we present our experimental results in Section 5.
We relegate the details of other algorithms and proofs to the appendix, along with additional details and results for the experimental results. In the appendix, we also include a warm-up result for the mean estimation problem in the partial observation setting (where there is only one outcome  $\mathbf{y}$  and we seek to estimate  $\frac{1}{n}\sum_{i = 1}^{n}y_{i}$  from a small number of samples). This result illustrates the key ideas behind leverage score sampling, which is used for our results on both ITE and ATE estimation. In addition we discuss how to remove the  $1 - \delta$  probability from our results in Section C.1.
In this section, we sketch the key ideas behind our main results on ITE estimation (Theorem 1) and ATE estimation (Theorems 2 and 3).
2.1 ITE Estimation
We first consider ITE estimation in the full observation setting. The key challenge here is that we must infer the full treatment effect vector  $\mathbf{t} = \mathbf{y}^{(1)} - \mathbf{y}^{(0)}$ , while only observing one outcome,  $y_{i}^{(1)}$  or  $y_{i}^{(0)}$  for any unit  $i$ . To do so, we use the idea of pseudo-outcome regression in [22]. We construct a random vector  $\mathbf{v} \in \mathbb{R}^n$  that is equal to  $\mathbf{t}$  in expectation, i.e.,  $\mathbb{E}[v] = t$  but only requires observing one outcome per unit. Specifically, we independently set  $v_{i} = 2y_{i}^{(1)}$  with probability 0.5 and  $v_{i} = -2y_{i}^{(0)}$  with probability 0.5. We then regress  $\mathbf{v}$  onto our covariates, and use the result to estimate  $\mathbf{t}$ . In particular, we set  $\widehat{\mathbf{b}} := \arg \min_{\mathbf{b} \in \mathbb{R}^d} \| \mathbf{X}\mathbf{b} - \mathbf{v} \|_2^2$  and construct our ITE estimate as  $\widehat{\mathbf{t}} = \mathbf{X}\widehat{\mathbf{b}}$ .
To give a non-asymptotic error bound for this approach, we observe that we can write  $\mathbf{v} = \mathbf{t} + \mathbf{z} \odot \boldsymbol{\mu}$  where  $\mathbf{z} \in \{1, -1\}^n$  is a random sign vector,  $\boldsymbol{\mu} = \mathbf{y}^{(1)} + \mathbf{y}^{(0)}$  is the sum of potential outcome vectors, and  $\odot$  denotes the Hadamard (entrywise) product. Thus, letting  $\pi_{\mathbf{X}} \in \mathbb{R}^{n \times n}$  be the projection matrix onto the column span of  $\mathbf{X}$ , we can write  $\widehat{\mathbf{b}} = \pi_{\mathbf{X}}\mathbf{v} = \pi_{\mathbf{X}}\mathbf{t} + \pi_{\mathbf{X}}(\mathbf{z} \odot \boldsymbol{\mu})$ . In turn, we have
$$
\mathbb {E} \left[ \| \widehat {\mathbf {t}} - \mathbf {t} \| _ {2} ^ {2} \right] = \mathbb {E} \left[ \| \mathbf {X} \widehat {\mathbf {b}} - \mathbf {t} \| _ {2} ^ {2} \right] = \mathbb {E} \left[ \| \pi_ {\mathbf {X}} \mathbf {t} - \mathbf {t} \| _ {2} ^ {2} + \| \pi_ {\mathbf {X}} (\mathbf {z} \odot \boldsymbol {\mu}) \| _ {2} ^ {2} + 2 \mathbf {z} ^ {T} \mathbf {X} ^ {T} \mathbf {X} (\mathbf {z} \odot \boldsymbol {\mu}) \right].
$$
The first term on the righthand side is simply the best linear fit error for  $\mathbf{t}$ ,  $\| \pi_{\mathbf{X}}\mathbf{t} - \mathbf{t}\| _2^2 = \min_{\mathbf{b}\in \mathbb{R}^d}\| \mathbf{X}\mathbf{b} - \mathbf{t}\| _2^2$ . The third term is 0 in expectation since  $\mathbb{E}[\mathbf{z}\odot \boldsymbol {\mu}] = 0$ . Finally, since the entries of  $\mathbf{z}\circ \boldsymbol{\mu}$  are independent and mean 0, the expectation of the second term can be bounded by  $\mathrm{tr}(\pi_{\mathbf{X}})\cdot \| \boldsymbol {\mu}\|_{\infty}^{2} = d\| \boldsymbol {\mu}\|_{\infty}^{2}$ . Putting these all together, we have
$$
\mathbb {E} \left[ \| \widehat {\mathbf {t}} - \mathbf {t} \| _ {2} ^ {2} \right] \leq \min  _ {\mathbf {b} \in \mathbb {R} ^ {d}} \| \mathbf {X b} - \mathbf {t} \| _ {2} ^ {2} + d \| \boldsymbol {\mu} \| _ {\infty} ^ {2}, \tag {1}
$$
which gives the bound of Theorem 1 in the full observation setting. We note that this expected error bound can also be turned into a high probability bound using concentration inequalities, i.e., Hanson-wright inequality [31].