SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Anish Agarwal*
Department of IEOR
New York, NY 10027
aa5194@columbia.edu
Keegan Harris
School of Computer Science
Pittsburgh, PA 15213
keeganh@cs.cmu.edu
Justin Whitehouse
jwhiteho@cs.cmu.edu
Zhiwei Steven Wu
zhiweiw@cs.cmu.edu
Principal component regression (PCR) is a popular technique for fixed-design error-in-variables regression, a generalization of the linear regression setting in which the observed covariates are corrupted with random noise. We provide the first time-uniform finite sample guarantees for online (regularized) PCR whenever data is collected adaptively. Since the proof techniques for analyzing PCR in the fixed design setting do not readily extend to the online setting, our results rely on adapting tools from modern martingale concentration to the error-in-variables setting. As an application of our bounds, we provide a framework for experiment design in panel data settings when interventions are assigned adaptively. Our framework may be thought of as a generalization of the synthetic control and synthetic interventions frameworks, where data is collected via an adaptive intervention assignment policy.
An omnipresent task in machine learning, statistics, and econometrics is that of making predictions about outcomes of interest given an action and conditioned on observable covariates. An often overlooked aspect of the prediction task is that in many settings the learner only has access to imperfect observations of the covariates, due to e.g. measurement error or inherent randomness in the problem domain. Such settings are sometimes formulated as error-in-variables regression: a learner is given access to a collection of data  $(Z_{n},a_{n},Y_{n})_{n\geq 1}$ , where  $Z_{n}\in \mathbb{R}^{d}$  are the observed covariates,  $a_{n}\in \{1,\ldots ,A\}$  is the action taken, and  $Y_{n}\in \mathbb{R}$  is the outcome for each observation  $n$ . Typically, the outcomes are assumed to be generated by a linear model  $Y_{n}\coloneqq \langle \theta (a_{n}),X_{n}\rangle +\xi_{n}$  and  $Z_{n}\coloneqq X_{n} + \epsilon_{n}$  where  $X_{n}\in \mathbb{R}^{d}$  are the true covariates,  $\epsilon_n\in \mathbb{R}^d$  is the covariate noise,  $\theta (a_n)\in \mathbb{R}^d$  is an unknown slope vector associated with action  $a_{n}$ , and  $\xi_{n}\in \mathbb{R}$  is the response noise. Note that the learner does not get to see the true covariates  $X_{n}$ . Observe that when  $\epsilon_{n} = 0$  we recover the traditional linear regression setting. Such an error-in-variables model can encompass many forms of data corruption including measurement error, missing values, discretization, and differential privacy—see [10, 5] for details.
Our point of departure from previous work is that we allow the sequence of data  $(Z_{n},a_{n},Y_{n})_{n\geq 1}$  to be chosen adaptively. In other words, we provide bounds for learning in the error-in-variables regres
*For part of this work, Anish was a postdoc at Amazon, Core AI.
sion setting when the data seen at the current round  $n$  is allowed to depend on the previously-seen data  $(Z_{m},a_{m},Y_{m})_{1\leq m < n}$ . Adaptive data collection occurs when the choices of future observations can depend on the inference from previous data, which is common in learning paradigms such as multi-armed bandits [58, 78], active learning [74], and time-series analysis [76, 36]. Similar to prior work on adaptive data collection that shows that valid statistical inference can be done when the true covariates are observed [34, 67, 43, 91, 92], our work provides the first time-uniform finite sample guarantees for error-in-variables regression using adaptively collected data.
Concretely, we focus on analyzing principal component regression (PCR) [54], a method that has been shown to be effective for learning from noisy covariates [10, 7] and a central tool for learning from panel data [8, 5, 7]. At a high level, PCR "de-noises" the sequence of observed covariates  $(Z_{n})_{n\geq 1}$  as  $(\widehat{Z}_n)_{n\geq 1}$  by performing hard singular value thresholding, after which a linear model is learned using the observed outcomes  $(Y_{n})_{n\geq 1}$  and the denoised covariates  $(\widehat{Z}_n)_{n\geq 1}$ . See Section 3.2 for further technical background on PCR.
1. We derive novel time-uniform bounds for an online variant of regularized PCR when the sequence of covariates is chosen adaptively. The techniques used to derive bounds for PCR in the fixed-sample regime [7] do not extend to the setting in which data is collected adaptively; thus, we require new tools and ideas to obtain our results. Specifically, our results rely on applying recent advances in martingale concentration [50, 51], as well as more classical results on self-normalized concentration [3, 32, 33] which are commonly applied to online regression problems, to the error-in-variables setting. As an example of the bounds we obtain, consider the task of estimating the underlying relationship  $\theta(a)$  between true (i.e. noiseless) covariates and observations, given access to  $n$  adaptively-chosen noisy covariates and their corresponding actions and observations. The  $\ell_2$  estimation error of the adaptive PCR estimator  $\widehat{\theta}_n(a)$  can be bounded as
$$
\left\| \widehat {\theta} _ {n} (a) - \theta (a) \right\| _ {2} ^ {2} = \widetilde {O} \left(\frac {1}{\operatorname {s n r} _ {n} (a) ^ {2}} \kappa (\mathbf {X} _ {n} (a)) ^ {2}\right)
$$
with high probability, where  $\mathrm{snr}_n(a)$  is the signal-to-noise ratio associated with action  $a$  at round  $n$  (Definition 3.5), a measure of how well the true covariates stand out from the noise.  $\kappa (\mathbf{X}_n(a))$  is the condition number of the true covariates. Intuitively, if  $\mathrm{snr}_n(a)$  is high the true covariates can be well-separated from the noise, and therefore PCR accurately estimates  $\theta (a)$  as long as the true covariates are well-conditioned.
Despite the harder setting we consider, our PCR bounds for adaptively-collected data largely match the bounds currently known for the fixed-sample regime, and even improve upon them in two important ways: (1) Our bounds are computable, i.e. they depend on known constants and quantities available to the algorithm. (2) Unlike Agarwal et al. [7], our bounds do not depend on the  $\ell_1$ -norm of  $\theta(a)$ , i.e., we do not require approximate sparsity of  $\theta(a)$  for the bounds to imply consistency. This is important because PCR is a rotationally-invariant algorithm, and so its performance guarantees should not depend on the orientation of the basis representation of the space to be learned. The price we pay for adaptivity is that  $\mathrm{snr}_n(a)$  is defined with respect to a bound on the total amount of noise seen by the algorithm so far, instead of just the noise associated with the rounds that  $a$  is taken. As a result, our bound for  $\widehat{\theta}_n(a)$  may not be tight if  $a$  is seldomly selected.
2. We apply our PCR results to the problem of online experiment design with panel data. In panel data settings, the learner observes repeated, noisy measurements of units (e.g. medical patients, subpopulations, geographic locations) under different interventions (e.g. medical treatments, discounts, socioeconomic policies) over time. This is an ubiquitous method of data collection, and as a result, learning from panel data has been the subject of significant interest in the econometrics and statistics communities (see Section 2). A popular framework for learning from panel data is synthetic control (SC) [1, 2], which uses historical panel data to estimate counterfactual unit measurements under control. Synthetic interventions (SI) [8] is a recent generalization of the SC framework which allows for counterfactual estimation under treatment, in addition to control. By leveraging online PCR, we can perform counterfactual estimation of unit-specific treatment effects under both
treatment and control, as in the SI framework. However, unlike the traditional SI framework, we are the first to establish statistical rates for counterfactual unit outcome estimates under different interventions while allowing for both units and interventions to be chosen adaptively. Such adaptivity may naturally occur when treatments are prescribed to new units based on the outcomes of previous units. For example, this is the case when the intervention chosen for each unit is the one which appears to be best based on observations in the past.
Error-in-variables regression There is a rich literature on error-in-variables regression (e.g. [42, 57, 28, 44, 84, 47, 41]), with research focusing on topics such as high-dimensional [62, 56, 31, 72] and Bayesian settings [70, 81, 40]. Principal component regression (PCR) [54, 22, 10, 7] is a popular method for error-in-variables regression. The results of Agarwal et al. [7] are of particular relevance to us, as they provide finite sample guarantees for the fixed design (i.e. non-adaptive) version of the setting we consider.
Self-normalized concentration There has been a recent uptick in the application of self-normalized, martingale concentration to online learning problems. In short, self-normalized concentration aims to control the growth of processes that have been normalized by a random, or empirical, measure of accumulated variance [32, 33, 50, 51, 88]. Self-normalized concentration has led to breakthroughs in wide-ranging areas of machine learning such as differential privacy [85, 86], PAC-Bayesian learning [30], convex divergence estimation [63], and online learning [87, 29, 3]. Of particular importance for our work are the results of Abbasi-Yadkori et al. [3], which leverage self-normalized concentration results for vector-valued processes [32, 33] to construct confidence ellipsoids for online regression tasks. We take inspiration from these results when constructing our estimation error bounds for PCR in the sequel.
Learning in panel data settings Our application to panel data builds off of the SI framework [8, 9], which itself is a generalization of the canonical SC framework for learning from panel data [1, 2, 52, 35, 17, 60, 89, 13, 14, 59, 16, 20, 23, 27, 39]. In both frameworks, a latent factor model is often used to encode structure between units and time-steps [25, 61, 15, 18, 19, 68, 64, 65]. Specifically, it is assumed that unit outcomes are the product of unit- and time/intervention-specific latent factors, which capture the heterogeneity across time-steps, units, and interventions, and allows for the estimation of unit-specific counterfactuals under treatment and control. Other extensions of the SI framework include applications in biology [79], network effects [11], combinatorially-many interventions [4], and intervening under incentives [45, 66]. Finally, there is a growing line of work at the intersection of online learning and panel data. Chen [26] views the problem of SC as an instance of online linear regression, which allows them to apply the regret guarantees of the online learning algorithm follow-the-leader [55] to show that the predictions of SC are comparable to those of the best-in-hindsight weighted average of control unit outcomes. Farias et al. [38] build on the SC framework to estimate treatment effects in adaptive experimental design, while minimizing the regret associated with experimentation. The results of Farias et al. [38] are part of a growing line of work on counterfactual estimation and experimental design using multi-armed bandits [69, 77, 90, 24].
3 Setting and background
Notation We use boldface symbols to represent matrices. For  $N \in \mathbb{N}$ , we use the shorthand  $[N] := \{1, \ldots, N\}$ . Unless specified otherwise,  $\|v\|$  denotes the  $\ell_2$ -norm of a vector  $v$ , and  $\|\mathbf{A}\|_{op}$  the operator norm of matrix  $\mathbf{A}$ . We use  $\mathrm{diag}(a_1, \ldots, a_k)$  to represent a  $k \times k$  diagonal matrix with entries  $a_1, \ldots, a_k$ . For two numbers  $a, b \in \mathbb{R}$ , we use  $a \wedge b$  as shorthand for  $\min\{a, b\}$ , and  $a \vee b$  to mean  $\max\{a, b\}$ . Finally,  $\mathbb{S}^{d-1}$  denotes the  $d$ -dimensional unit sphere.
3.1 Problem setup
We now describe our error-in-variables setting. We consider a learner who interacts with an environment over a sequence of rounds. At the start of each round  $n \geq 1$ , the environment generates covariates  $X_{n} \in W^{*} \subset \mathbb{R}^{d}$ , where  $W^{*}$  is a low-dimensional linear subspace of dimension  $\dim(W^{*}) = r < d$ .
We assume that  $r$  (but not  $W^{*}$ ) is known to the learner. Such "low rank" assumptions are reasonable whenever, e.g. data is generated according to a latent factor model, a popular assumption in high-dimensional statistical settings [53, 12, 48]. As we will see in Section 5, analogous assumptions are often also made in panel data settings. The learner then observes noisy covariates  $Z_{n} \coloneqq X_{n} + \epsilon_{n}$ , where  $\epsilon_{n} \in \mathbb{R}^{d}$  is a random noise vector. Given  $Z_{n}$ , the learner selects an action  $a_{n} \in [A]$  and observes  $Y_{n} \coloneqq \langle \theta(a_{n}), X_{n} \rangle + \xi_{n}$ , where  $\xi_{n}$  is random noise and  $\theta(a)$  for  $a \in [A]$  are unknown slope vectors in  $W^{*}$  that parameterize action choices such that  $\| \theta(a) \|_{2} \leq L$  for some  $L \in \mathbb{R}$ . We require that the covariate noise is "well-behaved" according to one of the two the following assumptions:
Assumption 3.1 (SubGaussian Covariate Noise). For any  $n \geq 1$ , the noise variable  $\epsilon_{n}$  satisfies (a)  $\epsilon_{n}$  is  $\sigma$ -subGaussian, (b)  $\mathbb{E}\epsilon_{n} = 0$ , and (c)  $\| \mathbb{E}\epsilon_{n}\epsilon_{n}^{\top}\|_{op} \leq \gamma$ , for some constant  $\gamma > 0$ .
Assumption 3.2 (Bounded Covariate Noise). For any  $n \geq 1$ , the noise variable  $\epsilon_{n}$  satisfies (a)  $\| \epsilon_{n} \| \leq \sqrt{Cd}$ , (b)  $\mathbb{E}\epsilon_{n} = 0$ , and (c)  $\mathbb{E}\epsilon_{n}\epsilon_{n}^{\top} = \Sigma$ , for some positive-definite matrix  $\Sigma$  satisfying  $\| \Sigma \|_{op} \leq \gamma$ , for some constant  $\gamma > 0$ .
Note that Assumption 3.2 is a special case of Assumption 3.1, which allows us to get stronger results in some settings. We also impose the following constraint on the noise in the outcomes.
Assumption 3.3 (SubGaussian Outcome Noise). For any  $n \geq 1$ , the noise variable  $\xi_{n}$  satisfies (a)  $\mathbb{E}\xi_{n} = 0$ , (b)  $\xi_{n}$  is  $\eta$ -subGaussian, and (c)  $\mathbb{E}\xi_{n}^{2} \leq \alpha$ , for some constant  $\alpha$ .
Under this setting, the goal of the learner is to estimate  $\theta(a)$  for  $a \in [A]$  given an (possibly adaptively-chosen) observed sequence  $(Z_{n}, a_{n}, Y_{n})_{n \geq 1}$ . For  $n \geq 1$ , we define the matrix  $\mathbf{Z}_{n} \in \mathbb{R}^{n \times d}$  to be the matrix of observed (i.e. noisy) covariates, with  $Z_{1}, \ldots, Z_{n}$  as its rows. Similarly,  $\mathbf{X}_{n} = (X_{1}, \ldots, X_{n})^{T} \in \mathbb{R}^{n \times d}$  is the matrix of noiseless covariates (which are unobserved), and  $\mathcal{E}_{n} = (\epsilon_{1}, \ldots, \epsilon_{n})^{T} \in R^{n \times d}$ ,  $\mathbf{Y}_{n} = (Y_{1}, \ldots, Y_{n})^{T} \in \mathbb{R}^{n \times 1}$ , and  $\Xi_{n} = (\xi_{1}, \ldots, \xi_{n})^{T} \in \mathbb{R}^{n \times 1}$  are defined analogously. For any action  $a \in [A]$ , let  $N_{n}(a) := \{s \leq n : a_{s} = n\}$  be the set of rounds up to and including round  $n$  on which action  $a$  was chosen. Likewise, let  $c_{n}(a) := |N_{n}(a)|$  denote the number of rounds by round  $n$  on which action  $a$  was chosen. For  $a \in [A]$ , we enumerate  $N_{n}(a)$  in increasing order as  $i_{1} \leq \dots \leq i_{c_{n}(a)}$ . Finally, we define  $\mathbf{Z}_{n}(a) \in \mathbb{R}^{c_{n}(a) \times d}$  to be  $\mathbf{Z}(a) = (Z_{i_{1}}, \ldots, Z_{i_{c_{n}(a)}})^{T}$ , and define  $\mathbf{X}_{n}(a), \mathcal{E}_{n}(a), \mathbf{Y}_{n}(a)$ , and  $\Xi_{n}(a)$  analogously.
3.2 Principal component regression
Background on singular value decomposition Any matrix  $\mathbf{A} \in \mathbb{R}^{n \times d}$  may be written in terms of its singular value decomposition  $\mathbf{A} = \mathbf{U} \Sigma \mathbf{V}^T$ , where  $\mathbf{U} \in \mathbb{R}^{n \times d \wedge n}$  and  $\mathbf{V} \in \mathbb{R}^{d \times d \wedge n}$  are matrices with orthonormal columns, and  $\Sigma = \mathrm{diag}(\sigma_1(\mathbf{A}), \ldots, \sigma_{d \wedge n}(\mathbf{A})) \in \mathbb{R}^{(d \wedge n) \times (d \wedge n)}$  is a diagonal matrix containing the singular values of  $\mathbf{A}$ , where we assume  $\sigma_1(\mathbf{A}) \geq \dots \geq \sigma_{d \wedge n}(\mathbf{A}) \geq 0$ . Given a truncation level  $k$ , we define the truncation of  $\mathbf{A}$  onto its top  $k$  principal components as  $\mathbf{A}_k := \mathbf{U}_k \mathrm{diag}(\sigma_1(\mathbf{A}), \ldots, \sigma_{k \wedge d \wedge n}(\mathbf{A})) \mathbf{V}_k^T$ , where  $\mathbf{U}_k \in \mathbb{R}^{n \times k \wedge d \wedge n}$  is the matrix with the first  $k \wedge d \wedge n$  columns of  $\mathbf{U}$ , and  $\mathbf{V}_k \in \mathbb{R}^{n \times k \wedge d \wedge n}$  is defined analogously. Given such a singular value decomposition, we can define the projection matrix onto the subspace spanned by the top  $k$  right singular vectors as  $\mathbf{P}_k \in \mathbb{R}^{d \times d}$  given by  $\mathbf{P}_k := \mathbf{V}_k \mathbf{V}_k^T$ .
For  $n \geq 1$ ,  $a \in [A]$ , and  $\mathbf{Z}_n(a)$  as defined above, we write the  $k$ -truncated singular value decomposition of  $\mathbf{Z}_n(a)$  as  $\mathbf{Z}_{n,k}(a) = \widehat{\mathbf{U}}_{n,k}(a)\mathrm{diag}(\sigma_1(\mathbf{Z}_n(a)),\ldots,\sigma_{k\wedge n\wedge d}(\mathbf{Z}_n(a)))\widehat{\mathbf{V}}_{n,k}^T(a)$ , and the corresponding projection onto the top  $k$  right singular vectors of  $\mathbf{Z}_n(a)$  as  $\widehat{\mathbf{P}}_{n,k}(a)$ . When  $k = r$ , we leverage the simplified notation  $\widehat{\mathbf{P}}_n(a) \coloneqq \widehat{\mathbf{P}}_{n,r}(a)$ . (Recall  $r = \dim(W^*)$ .) By  $\mathbf{P}$ , we denote the projection matrix onto the true, underlying subspace  $W^*$ . While  $\mathbf{P}$  is never known, our results leverage the fact that  $\widehat{\mathbf{P}}_n(a)$  converges to  $\mathbf{P}$  nicely over time. We define the projected noisy covariate matrix matrix to be  $\widehat{\mathbf{Z}}_n(a) \coloneqq \mathbf{Z}_n(a)\widehat{\mathbf{P}}_n(a)$ , and define  $\widehat{\mathbf{X}}_n(a),\widehat{\mathcal{E}}_n(a)$  similarly. Any quantity with a “ $\asymp$ ” is defined equivalently to quantities with “ $\widehat{\cdot}$ ”, except with  $\mathbf{P}$  in place of  $\widehat{\mathbf{P}}_n(a)$ . We are now ready to introduce our procedure for estimating  $\theta(a)$  for  $a \in [A]$ , called adaptive (or online) principal component regression.
Definition 3.4 (Adaptive Principal Component Regression). Given regularization parameter  $\rho \geq 0$  and truncation level  $k\in \mathbb{N}$ , for  $a\in [A]$  and  $n\geq 1$  let  $\widehat{\mathbf{Z}}_n(a)\coloneqq \mathbf{Z}_n(a)\widehat{\mathbf{P}}_{n,k}(a)$  and  $\widehat{\mathcal{V}}_n(a)\coloneqq \widehat{\mathbf{Z}}_n(a)^T\widehat{\mathbf{Z}}_n(a) + \rho \widehat{\mathbf{P}}_{n,k}(a)$ . Regularized principal component regression estimates  $\theta (a)$  as
$$
\widehat {\theta} _ {n} (a) := \widehat {\mathcal {V}} _ {n} (a) ^ {- 1} \widehat {\mathbf {Z}} _ {n} (a) \mathbf {Y} _ {n} (a).
$$
Setting  $\rho = 0$  recovers the version of PCR used in Agarwal et al. [7]. In words, (unregularized) PCR "denoises" the observed covariates by projecting them onto the subspace given by their  $k$ -truncation, before estimating  $\theta(a)$  via linear regression using the projected covariates. We choose to regularize since it is known that regularization increases the stability of regression-style algorithms. This added stability from regularization allows us to exploit the online regression bounds of Abbasi-Yadkori et al. [3] to measure the performance of our estimates. Throughout the sequel, we only consider adaptive PCR with truncation level  $k = r$ .
3.3 Signal to noise ratio
We now introduce the concept of signal to noise ratio, which will be integral in stating and proving our results. The signal to noise ratio provides a measure of how strongly the true covariates (this is the "signal" of the problem, measured through  $\sigma_r(\mathbf{X}_n(a))$ ) stand out sequentially with respect to the "noise" induced by  $\mathcal{E}_n$  (which we will measure through the relevant high probability bounds on  $\| \mathcal{E}_n\|_{op}$ ).
Definition 3.5 (Signal to Noise Ratio). We define the signal to noise ratio associated with an action  $a \in [A]$  at round  $n$  as
$$
\operatorname {s n r} _ {n} (a) := \frac {\sigma_ {r} (\mathbf {X} _ {n} (a))}{U _ {n}},
$$
where  $(U_{n})_{n\geq 1}$  is a noise-dependent sequence growing as  $U_{n} = O\left(\sqrt{n} +\sqrt{d} +\sqrt{\log\left(\frac{1}{\delta}\log(n)\right)}\right)$ .
The price we pay for adaptivity is encoded directly into the definition of the signal to noise ratio,  $\mathrm{snr}_n(a)$ . While one may imagine defining  $\mathrm{snr}_n(a)$  as the ratio between  $\sigma_r(\mathbf{X}_n(a))$  and  $\| \mathcal{E}_n(a)\|_{op}$ , bounding  $\| \mathcal{E}_n(a)\|_{op}$  is a nontrivial task as the rows of  $\mathcal{E}_n(a)$  may be strongly correlated. To circumvent this, we apply the trivial bound  $\| \mathcal{E}_n(a)\|_{op} \leq \| \mathcal{E}_n\|_{op}$ . Thus, the price of adaptivity in our setting is that the signal from covariates associated with an action  $a$  must stand out with respect to the total covariate noise by time  $n$ . The growth condition on  $U_n$  presented in Definition 3.5 is motivated as follows: w.h.p  $\mathbb{E}\|\mathcal{E}_n\|_{op} \approx \sqrt{d} + \sqrt{n}$ , and the extra additive  $\sqrt{\log\left(\frac{1}{\delta}\log(n)\right)}$  factor is the price we pay for having high probability control of  $\| \mathcal{E}_n\|_{op}$  uniformly over rounds. Below we provide an exact definition for  $U_n$ , as this choice leads to bounds with known constants and simple conditions on  $\mathrm{snr}_n(a)$  for validity.
We consider the following two sequences  $(U_n)_{n\geq 1}$  in defining signal to noise ratio, which both satisfy  $\| \mathcal{E}_n\|_{op}\leq U_n,\forall n\geq 1$  with probability at least  $1 - \delta$ , per Lemma B.3.
$$
U _ {n} ^ {2} := \left\{ \begin{array}{l} \beta \left(3 \sqrt {n \ell_ {\delta / 2 \mathcal {N}} (n)} + 5 \ell_ {\delta / 2 \mathcal {N}} (n)\right) + n \gamma \quad \text {w h e n A s s u m p t i o n 3 . 1 h o l d s} \\ \frac {3}{2} \sqrt {n C d \gamma \ell_ {\delta} (n)} + \frac {7}{3} C d \ell_ {\delta} (n) + n \gamma \quad \text {w h e n A s s u m p t i o n 3 . 2 h o l d s}. \end{array} \right.
$$
In the above,  $\delta \in (0,1)$ ,  $\ell_{\delta}(n) := 2\log \log(2n) + \log\left(\frac{d\pi^2}{12\delta}\right)$ ,  $\beta = 32\sigma^2 e^2$ , and  $\mathcal{N} = 17^d$  is an upper bound on the  $1/8$ -covering number of  $\mathbb{S}^{d-1}$ . While the exact forms of the above sequences  $(U_n)_{n\geq 1}$  may appear complicated, it is helpful to realize that, under either Assumption 3.1 or 3.2, we have  $U_n = O\left(\sqrt{n} + \sqrt{d} + \sqrt{\log\left(\frac{1}{\delta}\log(n)\right)}\right)$ , i.e., the growth condition on  $U_n$  presented in Definition 3.5 is satisfied.
We can likewise define the empirical signal to noise ratio associated with action  $a$  as  $\widehat{\mathrm{snr}}_n(a) := \frac{\sigma_r(\mathbf{Z}_n(a))}{U_n}$ . Note that unlike the (true) signal to noise ratio  $\mathrm{snr}_n(a)$ , the empirical version  $\widehat{\mathrm{snr}}_n(a)$  is computable by the learner. Thus, it will be integral in stating our empirical-style bounds in the section that follows.