SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Licong Lin
Department of Statistics  
University of California, Berkeley  
liconglin@berkeley.edu
Department of Statistics
Rutgers University - New Brunswick
my426@scarletmail.rutgers.edu
Suvrojit Ghosh
Department of Statistics Rutgers University - New Brunswick sg1565@scarletmail.rutgers.edu
Koulik Khamaru  
Department of Statistics  
Rutgers University - New Brunswick  
kk1241@stat.rutgers.edu
Department of Statistics
Rutgers University - New Brunswick
czhang@stat.rutgers.edu
Estimation and inference in statistics pose significant challenges when data are collected adaptively. Even in linear models, the Ordinary Least Squares (OLS) estimator may fail to exhibit asymptotic normality for single coordinate estimation and have inflated error. This issue is highlighted by a recent minimax lower bound, which shows that the error of estimating a single coordinate can be enlarged by a multiple of  $\sqrt{d}$  when data are allowed to be arbitrarily adaptive, compared with the case when they are i.i.d. Our work explores this striking difference in estimation performance between utilizing i.i.d. and adaptive data. We investigate how the degree of adaptivity in data collection impacts the performance of estimating a low-dimensional parameter component in high-dimensional linear models. We identify conditions on the data collection mechanism under which the estimation error for a low-dimensional parameter component matches its counterpart in the i.i.d. setting, up to a factor that depends on the degree of adaptivity. We show that OLS or OLS on centered data can achieve this matching error. In addition, we propose a novel estimator for single coordinate inference via solving a Two-stage Adaptive Linear Estimating equation (TALE). Under a weaker form of adaptivity in data collection, we establish an asymptotic normality property of the proposed estimator.
Estimating a low-dimensional parameter component in a high-dimensional model is a fundamental problem in statistics and machine learning that has been widely studied in e.g., semiparametric statistics [35, 8], causal inference [16, 15] and bandit algorithms [1, 27]. When data are independently and identically distributed (i.i.d.), it is often possible to derive estimators that are asymptotically normal with a rate of convergence of  $\sqrt{n}$ , and that achieve the semi-parametric variance lower bound
that is independent of the dimension. There is now a rich body of literature that studies this problem under various scenarios [5, 6, 33, 36, 4, 8, 41].
In this work we are interested in the same estimation and inference problem but under the setting where the i.i.d. data assumption fails. Specifically, we consider an adaptive collection framework where the data collected at time  $i$  is allowed to be dependent on the historical data collected up to time  $i - 1$ . This adaptive framework incorporates datasets originated from applications in many fields, including sequential experimental design [15], bandit algorithm [27], time series modeling [7], adaptive stochastic approximation schemes [13, 24].
1.1 An interesting lower bound
To see the intrinsic difference between the i.i.d. and the adaptive data collection settings, we consider the canonical example of linear model  $y = \pmb{x}^{\top}\pmb{\theta}^{*} + \varepsilon$ , where the parameter  $\pmb{\theta}^{*} = (\theta_{1}^{*},\theta_{2}^{*})\in \mathbb{R}^{1}\times \mathbb{R}^{d - 1},\varepsilon \stackrel {iid}{\sim}\mathcal{N}(0,1)$ . Clearly, when the covariates  $\{\pmb{x}_i\}_{i\leq n}$  are deterministic, a straightforward calculation yields
$$
\widehat {\theta} _ {\mathrm {o l s}, 1} - \theta_ {1} ^ {*} \stackrel {{d}} {{=}} \mathcal {N} \left(0, \left(\mathbf {S} _ {n} ^ {- 1}\right) _ {1 1}\right), \quad \text {a n d} \quad \mathbb {E} \left[ \left(\mathbf {S} _ {n} ^ {- 1}\right) _ {1 1} ^ {- 1} \cdot \left(\widehat {\theta} _ {\mathrm {o l s}, 1} - \theta_ {1} ^ {*}\right) ^ {2} \right] = 1, \tag {1}
$$
where  $\widehat{\theta}_{\mathrm{ols}}$  is the OLS estimator and  $\mathbf{S}_n \coloneqq \sum_{t=1}^n \boldsymbol{x}_i \boldsymbol{x}_i^\top$  is the sample covariance matrix.
However, somewhat surprisingly, when the covariates  $\{\pmb{x}_i\}_{i\leq n}$  are allowed to be collected in an arbitrary adaptive manner, in a recent work [20] the authors proved the following (informal) counterintuitive minimax lower bound on the scaled-MSE (defined in Definition 2.2)
$$
\min  _ {\widehat {\theta}} \max  _ {\boldsymbol {\theta} ^ {*}} \mathbb {E} \left[ \left(\mathbf {S} _ {n} ^ {- 1}\right) _ {1 1} ^ {- 1} \cdot \left(\widehat {\theta} - \theta_ {1} ^ {*}\right) ^ {2} \right] \geq c d \cdot \log (n), \tag {2}
$$
where the extra  $d$ -factor enters the estimation of a single coordinate. This lower bound indicates that a dimension independent single coordinate estimation is infeasible when the data are collected arbitrarily adaptively. This is undesirable especially in the high dimensional scenario where  $d \to \infty$  since a  $\sqrt{n}$ -consistent estimation is unattainable. Motivated by the contrast between i.i.d. and adaptive data collection, we pose the following question in this work:
Can we bridge the gap between iid and adaptive data collection, and obtain an estimator for a low-dimensional parameter component in linear models, such that its performance depends on the degree of adaptivity?
In this work, we initiate the study of how the adaptivity of the collected data affects low-dimensional estimation in a high-dimensional linear model. We explore the previously posed question and provide an affirmative answer.
We begin by introducing a general data collection assumption, which we term  $(k,d)$ -adaptivity. Broadly speaking,  $(k,d)$ -adaptivity implies that the data pairs  $\{(x_i,y_i)\}_{i=1}^n \in \mathbb{R}^d \times \mathbb{R}$  are collected in a way that the first  $k$  coordinates of  $\boldsymbol{x}_i$  (denoted by  $\boldsymbol{x}_i^{\mathrm{ad}}$ ) are chosen adaptively based on the historical data, while the remaining  $d - k$  coordinates of  $\boldsymbol{x}_i$  (denoted by  $\boldsymbol{x}_i^{\mathrm{nad}}$ ) are i.i.d. across time  $i \in [n]$ .
Assume the collected data are  $(k,d)$ -adaptive from a linear model  $y = \pmb{x}^{\top}\pmb{\theta}^{*} + \varepsilon$ . We analyze the lower-dimensional estimation problem under the scenarios where the i.i.d. non-adaptive components  $\pmb{x}_i^{\mathrm{rad}}$  are either zero-mean or nonzero-mean. In the zero mean case, we show that the ordinary least squares estimator (OLS) for the first  $k$ -coordinate yields a scaled mean squared error (scaled-MSE) of  $k\log(n)$  (Theorem 3.1). For the nonzero-mean case, a similar result is achieved using the OLS estimator on centered data (Theorem 3.2). Consequently, we find that the degree of adaptivity significantly impacts the performance of single coordinate estimation, in the sense that the scaled-MSE is inflated by a factor of  $k$ , where  $k$  denotes the number of adaptive coordinates (see Corollary 3.3).
Although OLS for a single coordinate has a dimension independent scaled-MSE when the collected data are  $(1, d)$ -adaptive, it should be noted that OLS may exhibit non-normal asymptotic behavior [13, 20] when data are adaptively collected. Therefore, we propose a novel estimator by solving a Two-stage Adaptive Linear Estimating Equation (TALE). When the collected data are  $(1, d)$ -adaptive and the non-adaptive component is zero mean, we show that our new estimator is asymptotically normal and has a comparable scaled-MSE as the naive OLS estimator (see Theorem 3.4).
2 Problem set up
Consider a linear model
$$
y = \boldsymbol {x} ^ {\top} \boldsymbol {\theta} ^ {*} + \varepsilon , \tag {3}
$$
where the parameter  $\pmb{\theta}^{*}\in \mathbb{R}^{d}$ , and  $\varepsilon$  is a zero mean noise variable. Given access to a data set  $\{(x_i,y_i)\}_{i\leq n}$  from the model (3), we are interested in the estimation and inference problem of a low-dimensional parameter component  $\pmb{\theta}_{\mathrm{ad}}^{*}\in \mathbb{R}^{k}$ , where  $\pmb{\theta}^{*} = (\pmb{\theta}_{\mathrm{ad}}^{*\top},\pmb{\theta}_{\mathrm{nad}}^{*\top})^{\top}$ .
In this paper, we are interested in adaptive data collection regime. Concretely, we assume that the data are collected adaptively in the following way
Definition 2.1  $((k,d)$ -adaptivity) The collected samples  $\{(\pmb{x}_i,y_i)\}_{i\leq n}$  forms a filtration  $\{\mathcal{F}\}_{i = 0}^{\infty}$  with  $\mathcal{F}_0 = \emptyset$  and  $\mathcal{F}_i = \sigma (\pmb {x}_1,y_1,\dots ,\pmb {x}_i,y_i)$ . Let  $P$  be an unknown distribution on  $\mathbb{R}^{d - k}$ . We assume that at each stage,  $i\geq 1$
- The adaptive component  $\mathbf{x}_i^{\mathrm{ad}} = \mathbf{x}_{i,1:k}$  is collected from some unknown distribution that could depend on  $\mathcal{F}_{i - 1}$ .
- The non-adaptive component  $\mathbf{x}_i^{\mathrm{nad}} = \mathbf{x}_{i,k + 1:d}$  is a sample from  $P$  and independent of  $(\mathbf{x}_i^{\mathrm{ad}},\mathcal{F}_{i - 1})$ .
When  $k = 0$ , Definition 2.1 reduces to an i.i.d. data collection strategy; when  $k = d$ , it corresponds to the case where the data are allowed to be collected arbitrarily adaptively. Consequently,  $(k,d)$ -adaptivity connects two extreme scenarios, and the degree of adaptivity increases as  $k$  increases.
Example 2.1 (Treatment assignment) As a concrete example, consider the problem of treatment assignment to patients. At round  $i$ , we observe the health profile of the patient  $i$ , which we denote by  $\pmb{x}_i \in \mathbb{R}^{d-1}$ . Our job to assign a treatment  $A_i \in \{0,1\}$  based on the patient's health profile  $\pmb{x}_i$  and also our prior knowledge of effectiveness of the treatments. It is natural to capture our prior knowledge using  $\mathcal{F}_i = \sigma(A_1, \pmb{x}_1, y_1, \dots, A_{i-1}, \pmb{x}_{i-1}, y_{i-1})$  — the sigma field generated by previous data-points. As already pointed out in (2), in the adaptive regime the estimator error for treatment effect scales as  $\sqrt{d/n}$ ; in words, we have to pay for a dimension factor  $\sqrt{d}$  even if we are only interested in estimating a one-dimensional component. While for our treatment assignment example, the dimension  $d-1$  of the covariate vector  $\pmb{x}_i$  is large in practice, it is natural to assume that the treatment assignment is dependent on  $k-1 \ll d-1$ , a few (unknown) components. Under this assumption, it is easy to see that this treatment assignment problem is  $(k,d)$ -adaptive. We show that the treatment effect can be estimated at a rate  $\sqrt{k/n} \ll \sqrt{d/n}$ .
2.1 Statistical limits
Before we discuss how to obtain estimators for a low-dimensional parameter component of  $\theta^{\star}$ , we establish some baselines by recalling existing lower bounds. Throughout this section, we assume the noise  $\epsilon_i \stackrel{iid}{\sim} \mathcal{N}(0, \sigma^2)$ . We start with defining the metric for comparison.
Definition 2.2 (scaled mean squared error (scaled-MSE)) Given a subset  $\mathcal{I} \subseteq [d]$ . We define the scaled-MSE of an estimator  $\widehat{\pmb{\theta}}_{\mathcal{I}}$  for  $\pmb{\theta}_{\mathcal{I}}^{*} \in \mathbb{R}^{| \mathcal{I}|}$  to be  $\mathbb{E}[(\widehat{\pmb{\theta}}_{\mathcal{I}} - \widehat{\pmb{\theta}}_{\mathcal{I}})^{\top}[(\mathbf{S}_n^{-1})_{\mathcal{I}\mathcal{I}}]^{-1}(\widehat{\pmb{\theta}}_{\mathcal{I}} - \widehat{\pmb{\theta}}_{\mathcal{I}})]$ , where  $\mathbf{S}_n = \sum_{i=1}^{n} \pmb{x}_i \pmb{x}_i^\top$  is the sample Gram matrix.
Roughly speaking, when the covariates  $\boldsymbol{x}_i$  are all fixed, the scaled-MSE compares the performance of  $\widehat{\theta}_{\mathcal{I}}$  against the estimator with minimal variance (OLS). Moreover, we have the following result:
Proposition 2.2 (A simplified version of Theorem 2 in Khamaru et al. [20])
(a). Given a set  $\mathcal{I} \subseteq [d]$ . Suppose the data  $\{(x_i, y_i)\}_{i=1^n}$  are i.i.d.  $((0, d)$ -adaptive) from model (3). Then the scaled-MSE satisfies
$$
\inf  _ {\widehat {\boldsymbol {\theta}}} \sup  _ {\boldsymbol {\theta} ^ {*} \in \mathbb {R} ^ {d}} \mathbb {E} \left\| \widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*} \right\| _ {[ (\mathbf {S} _ {n} ^ {- 1}) _ {\mathcal {I I}} ] ^ {- 1}} ^ {2} \geq \sigma^ {2} | \mathcal {I} |. \tag {4}
$$
Furthermore, the equality holds when choosing  $\widehat{\theta}_{\mathcal{I}}$  to be the OLS estimator for  $\theta_{\mathcal{I}}^*$ .
(b). Suppose the data points  $\{(x_i, y_i)\}_{i=1}^n$  are allowed to be arbitrarily adaptive  $((d, d)$ -adaptive). For any  $(n, d)$  with  $d \geq 2$  and  $n \geq c \cdot d^3$ , and any non-empty set  $\mathcal{I} \in [d]$ , there exists a data collection algorithm such that
$$
\inf  _ {\widehat {\boldsymbol {\theta}}} \sup  _ {\boldsymbol {\theta} ^ {*} \in \mathbb {R} ^ {d}} \mathbb {E} \left\| \widehat {\boldsymbol {\theta}} _ {\mathcal {I}} - \boldsymbol {\theta} _ {\mathcal {I}} ^ {*} \right\| _ {[ (\mathbf {S} _ {n} ^ {- 1}) _ {\mathcal {I I}} ] ^ {- 1}} ^ {2} \geq c ^ {\prime} \cdot d \sigma^ {2} \log (n), \tag {5}
$$
where  $c, c' > 0$  are some universal constants.
Proposition 2.2 exhibits the striking difference between two extreme data collection mechanisms. While the scaled-MSE scales as  $\tilde{O}(|\mathcal{I}|)$  when data are i.i.d., the scaled-MSE for even a single coordinate (e.g., setting  $\mathcal{I} = \{1\}$ ) can be of the order  $O(d)$  if the data are allowed to be collected arbitrarily adaptively.
Let  $\mathcal{I}^c = [d] \setminus \mathcal{I}$ . By the matrix inverse formula, we have
$$
\left[ \left(\mathbf {S} _ {n} ^ {- 1}\right) _ {\mathcal {I I}} \right] ^ {- 1} = \left(\mathbf {S} _ {n}\right) _ {\mathcal {I I}} - \left(\mathbf {S} _ {n}\right) _ {\mathcal {I I} ^ {c}} \left[ \left(\mathbf {S} _ {n} ^ {- 1}\right) _ {\mathcal {I} ^ {c} \mathcal {I} ^ {c}} \right] ^ {- 1} \left(\mathbf {S} _ {n}\right) _ {\mathcal {I} ^ {c} \mathcal {I}} = \mathbf {X} _ {\mathcal {I}} ^ {\top} \left(\mathbf {I} _ {n} - \mathbf {P} _ {\mathbf {X} _ {\mathcal {I} ^ {c}}}\right) \mathbf {X} _ {\mathcal {I}},
$$
where  $\mathbf{P}_{\mathbf{X}_{\mathcal{I}^c}}$  denotes the projection onto the column space of  $\mathbf{X}_{\mathcal{I}^c}\in \mathbb{R}^{n\times |\mathcal{I}|}$ . In this work, we are often interested in the special cases where  $\mathcal{I} = [k]$  (or  $\{\ell \}$  for  $\ell \in [k]$ ), which denote (a single coordinate of) the adaptive component.
Adaptive linear model In the early works by Lai et al. [24, 23], the authors studied regression models when the data are adaptively collected. They established the asymptotic normality of OLS under a stability assumption on the covariate matrix. However, the stability assumption might be violated under various settings, including data collected from online bandit algorithms such as UCB [27, 2, 31, 34, 42], forecasting and autoregressive models [14, 38, 24]. Recent works [13, 20] addressed this issue and proposed debiasing estimators with inferential guarantee in linear models with fixed dimension. While allowing for arbitrarily adaptively collected data, their results impose an additional  $\sqrt{d}$  factor in the error bound for single coordinate estimation, limiting their applicability in linear models with increasing dimensions [19, 28].
Parameter estimation in bandit algorithms Though with the primary goal being achieving a low regret, the problem of parameter estimation under adaptive data are also studied when designing online bandit algorithms [1, 26, 29, 27]. Many online bandit algorithms are built based on the estimation or construction of adaptive confidence sets for the reward function [2, 9, 17], which can be viewed as finite sample estimation and inference of the unknown parameter of a model. Most related to our paper, in linear bandits, the works [1, 26] derived non-asymptotic upper bound on scaled-MSE of OLS for the whole parameter vector, as well as for a single coordinate. However, the upper bound on scaled-MSE for estimating a single coordinate is inflated by a factor of  $d$  compared with the i.i.d. case, as suggested by the lower bound in [20].
Inference using adaptively collected data The problem of estimation and inference using adaptively collected data has also been studied under other settings. The work by Hadad et al. [15] and Zhan et al. [40] proposed a weighted augmented inverse propensity weighted (AIPW, [32]) estimator for treatment effect estimation that is asymptotic normal. Zhang et al. [43] analyzed a weighted  $M$ -estimator for contextual bandit problems. Lin et al. [30] proposed a weighted  $Z$ -estimator for statistical inference in semi-parametric models. While investigating more intricate models, these works are built on the strong assumption that the adaptive data collection mechanism is known. In contrast, our  $(k,d)$ -adaptivity assumption allows the adaptive component to be collected in an arbitrary adaptive way.
Semi-parametric statistics A central problem in semi-parametric statistics is to derive  $\sqrt{n}$ -consistent and asymptotic normal estimators of a low-dimensional parameter component in high-dimensional or semi-parametric models [5, 6, 33, 36, 4, 8, 41]. Most works in this literature assume i.i.d. data collection and aim to obtain estimators that achieve the optimal asymptotic variance (i.e., semi-parametric efficient [35, 16]). On the other hand, our work focuses on a complementary perspective, with the goal of understanding how the data collection assumption affects the statistical limit of low-dimensional parameter estimation.
In the paper, we use the bold font to denote vectors and matrices (e.g.,  $\mathbf{x}$ ,  $\mathbf{x}$ ,  $\mathbf{X}$ ,  $\theta$ ,  $\varepsilon$ ), and the regular font to denote scalars (e.g.,  $x, \theta, \varepsilon$ ). Given data  $\left\{ (\boldsymbol{x}_i, y_i) \right\}_{i=1}^n$  from the linear model (3) that are  $(k, d)$ -adaptive, we use  $\boldsymbol{x}_i^{\mathrm{ad}} \in \mathbb{R}^k$ ,  $\boldsymbol{x}_i^{\mathrm{nad}} \in \mathbb{R}^{d-k}$  to denote the adaptive and non-adaptive covariates. We also write  $\boldsymbol{\theta}^* = (\boldsymbol{\theta}_{\mathrm{ad}}^{*\top}, \boldsymbol{\theta}_{\mathrm{nad}}^{*\top})^\top$  to denote the components that correspond to the adaptive and non-adaptive covariates. Let  $\mathbf{X} = (\boldsymbol{x}_1^\top, \ldots, \boldsymbol{x}_n^\top)^\top \in \mathbb{R}^{n\times d}$  be the covariate matrix, with  $\mathbf{X}_{\mathrm{ad}}$  (or  $\mathbf{X}_{\mathrm{nad}}$ ) representing the submatrices consisting of the adaptive (or non-adaptive) columns. We use  $\mathbf{x}_j$  to denote the  $j$ -th column of the covariate matrix.
For a matrix  $\mathbf{M}$  with  $n$  rows, let  $\mathbf{M}_{-j}$  be the matrix obtained by deleting the  $j$ -th column of  $\mathbf{M}$ . We define the projection operator  $\mathbf{P}_{\mathbf{M}} := \mathbf{M}(\mathbf{M}^{\top}\mathbf{M})^{-1}\mathbf{M}^{\top}$  and the (columnwise) centered matrix  $\widetilde{\mathbf{M}} := (\mathbf{I}_n - \mathbf{P}_{\mathbf{1}_n})\mathbf{M}$ , where  $\mathbf{I} \in \mathbb{R}^{n \times n}$  is the identity matrix and  $\mathbf{1}_n \in \mathbb{R}^n$  is the all-one vector. For a symmetric  $\mathbf{M} \succeq 0$ , we define  $\| \boldsymbol{x} \|_{\mathbf{M}} := \sqrt{\boldsymbol{x}^{\top}\mathbf{M}\boldsymbol{x}}$ . Lastly, we use  $c, c', c'' > 0$  to denote universal constants and  $C, C', C'' > 0$  to denote constants that may depend on the problem specific parameters but not on  $k, d, n$ . We allow the values of the constants to vary from place to place.
This section is devoted to our main results on low-dimensional estimation and inference. In Section 3.1 and 3.2 we discuss the problem of estimating a low-dimensional component of  $\theta^{\star}$ , and Section 3.3 is devoted to inference of low-dimensional components.
3.1 Low-dimensional estimation
Suppose the collected data  $\{(\pmb{x}_i, y_i)\}_{i=1}^n$  are  $(k, d)$ -adaptive. In this section, we are interested in estimating the adaptive parameter component  $\pmb{\theta}_{\mathrm{ad}}^* \in \mathbb{R}^k$ .
In addition to  $(k,d)$ -adaptivity, we introduce the following assumptions on the collected data  $\{(x_i,y_i)\}_{i = 1}^n$ .
Assumption A
(A1) There exists a constant  $\mathrm{U}_{\mathrm{x}} > 0$  such that
$$
1 \leq \sigma_ {\min } \left(\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}\right) \leq \sigma_ {\max } \left(\mathbf {X} _ {\mathrm {a d}} ^ {\top} \mathbf {X} _ {\mathrm {a d}}\right) \leq n U _ {x}.
$$
(A2) The non-adaptive components  $\{\pmb{x}_i^{\mathrm{nad}}\}_{i = 1}^n$  are i.i.d. sub-Gaussian vectors with parameter  $\nu >0$ , that is, for any unit direction  $\pmb {u}\in \mathbb{S}^{d - 1}$
$$
\mathbb {E} [ \exp \{\lambda \langle \boldsymbol {u}, \boldsymbol {x} _ {i} ^ {\mathrm {n a d}} - \mathbb {E} [ \boldsymbol {x} _ {i} ^ {\mathrm {n a d}} ] \rangle \} ] \leq e ^ {\lambda^ {2} \nu^ {2} / 2} \quad \forall \lambda \in \mathbb {R}.
$$
(A3) There exist some constants  $0 \leq \sigma_{\min} \leq \sigma_{\max}$  such that the covariance matrix of the non-adaptive component  $\boldsymbol{\Sigma} \coloneqq \operatorname{Cov}[\boldsymbol{x}_i^{\mathrm{nad}}]$  satisfies,
$$
0 <   \sigma_ {\min } \leq \sigma_ {\min } (\boldsymbol {\Sigma}) \leq \sigma_ {\max } (\boldsymbol {\Sigma}) \leq \sigma_ {\max }.
$$
(A4) Conditioned on  $(\pmb{x}_i, \mathcal{F}_{i-1})$ , the noise variable  $\varepsilon_i$  in (3) is zero mean sub-Gaussian with parameter  $v > 0$ , i.e.,
$$
\mathbb {E} \left[ \varepsilon_ {i} \mid \boldsymbol {x} _ {i} ^ {\text {a d}}, \mathcal {F} _ {i - 1} \right] = 0, \text {a n d} \mathbb {E} \left[ e ^ {\lambda \varepsilon_ {i}} \mid \boldsymbol {x} _ {i} ^ {\text {a d}}, \mathcal {F} _ {i - 1} \right] \leq e ^ {\lambda^ {2} v ^ {2} / 2} \quad \forall \lambda \in \mathbb {R},
$$
and has conditional variance  $\sigma^2 = \mathbb{E}[\varepsilon_i^2 | \boldsymbol{x}_i, \mathcal{F}_{i-1}]$  for all  $i \in [n]$ .
Let us clarify the meaning of the above assumptions. Assumption (A1) is the regularity assumption on the adaptive component. Roughly speaking, we allow the adaptive component to be arbitrarily adaptive as long as  $\mathbf{X}_{\mathrm{ad}}^{\top}\mathbf{X}_{\mathrm{ad}}$  is not close to be singular and  $x_{i}^{\mathrm{ad}}$  has bounded  $\ell_2$ -norm. This is weaker than the assumptions made in [15, 43, 30], which assume that the conditional distribution of  $\mathbf{X}_{\mathrm{ad}}$  is known. Assumption (A2), (A3) on the non-adaptive component, assume its distribution is non-singular and light-tailed. Assumption (A4) is a standard assumption that characterizes the tail behavior of the zero-mean noise variable. We remark that the equal conditional variance assumption in Assumption (A4) is mainly required in Theorem 3.4, while it is sufficient to assume  $\sigma^2$  being a uniform upper bound of the conditional variance in Theorem 3.1 and 3.2.
3.1.1 Warm up: zero-mean non-adaptive component
We start with discussing a special case where the non-adaptive component  $\boldsymbol{x}_i^{\mathrm{rad}}$  is zero-mean. In this case, we prove that the Ordinary Least Squares (OLS) estimator on  $(\mathbf{X},\mathbf{y})$  for  $\boldsymbol{\theta}_{\mathrm{ad}}^{*}$  is near-optimal; see the discussion in Section 2.1. Denote the OLS estimator by  $\widehat{\pmb{\theta}} = (\widehat{\pmb{\theta}}_{\mathrm{ad}}^{\top},\widehat{\pmb{\theta}}_{\mathrm{nad}}^{\top})^{\top}$ . Throughout, we assume that the sample size  $n$  and dimension  $d$  satisfies the relation
$$
\frac {n}{\log^ {2} (n / \delta)} \geq C d ^ {2}, \tag {6}
$$
where  $C$  is an independent of  $(n, d)$  but may depend on other problem specific parameters. With this set up, our first theorem states
Theorem 3.1 Given data points  $\{(x_i, y_i)\}_{i=1}^n$  from a  $(k, d)$ -adaptive model, and tolerance level  $\delta \in (0, 1/2)$ . Let, assumption (A1)-(A4) and the bound (6) in force, and the non-adaptive component  $x_i^{\mathrm{nad}}$  is drawn from a zero-mean distribution. Then, we have