SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Multi-Task Learning with Summary Statistics
Parker Knight
pknight@g.harvard.edu
Rui Duan
rduan@hsph.harvard.edu
Multi-task learning has emerged as a powerful machine learning paradigm for integrating data from multiple sources, leveraging similarities between tasks to improve overall model performance. However, the application of multi-task learning to real-world settings is hindered by data-sharing constraints, especially in healthcare settings. To address this challenge, we propose a flexible multi-task learning framework utilizing summary statistics from various sources. Additionally, we present an adaptive parameter selection approach based on a variant of Lepski's method, allowing for data-driven tuning parameter selection when only summary statistics are available. Our systematic non-asymptotic analysis characterizes the performance of the proposed methods under various regimes of the sample complexity and overlap. We demonstrate our theoretical findings and the performance of the method through extensive simulations. This work offers a more flexible tool for training related models across various domains, with practical implications in genetic risk prediction and many other fields.
The growing availability of extensive and intricate datasets presents an opportunity to integrate data from multiple sources. Multi-task learning has emerged as a promising machine learning approach that enables the simultaneous learning of multiple related models, leveraging shared structure between tasks to enhance the performance on each task individually [36, 35]. In healthcare and biomedical research, the practical application of multi-task learning is often hindered by data-sharing constraints, which stem from concerns about the ownership and privacy of individual-level data [32, 8]. Patient data in these domains is typically sensitive and less likely to be publicly available or shared across study sites, limiting researchers' access to individual-level data from different domains.
To overcome this limitation, researchers have increasingly integrated summary statistics into analysis pipelines as a substitute for individual-level data [22, 13, 12]. Summary statistics are straightforward, interpretable measures derived from raw data that can offer insights into data distribution, variability, and relationships among variables. Furthermore, they can be aggregated across studies to facilitate data integration and reused in various research projects. Recently, the use of summary statistics has garnered interest in healthcare and biomedical research. For example, many genetic risk prediction methods rely on summary-level statistics such as associations from Genome-wide Association Studies (GWAS), Linkage Disequilibrium estimations (LD), and minor allele frequencies (MAFs) [6]. These summary statistics can help predict an individual's likelihood of developing specific diseases based on their genetic profile.
Inspired by a potential use case in genetic risk prediction, we propose a multi-task learning framework that enables simultaneous learning of multiple genetic risk prediction models using only publicly available summary statistics. Our proposed framework can be used in the context of predicting genetic risks for multiple traits leveraging potentially shared genetic pathways, and can also be used
to develop trans-ethnic genetic risk prediction models that account for potential heterogeneity across populations, improving generalizability and real-world applicability. Beyond genetic risk prediction, the ability to learn from summary statistics offers a versatile tool for developing models across a wide range of domains, including healthcare, finance, and marketing.
To summarize, the contributions of this work are threefold: First, we propose a flexible multi-task learning framework which allows training multiple models simultaneously using basic summary statistics characterizing marginal relationship between outcomes and features, which are often publicly available. We allow summary statistics corresponding to each task to be generated from distinct or potentially overlapping samples. Secondly, we conducted a systematic non-asymptotic analysis which characterizes how the performance of the proposed methods are influenced by the characteristics of summary statistics. In particular, we show that there are multiple regimes of performance depending on the sample complexity of the source datasets and their overlap. The theoretical results are supported with extensive simulations. Lastly, We propose an adaptive scheme for tuning parameter selection based on the variant of Lepski's method [14] given in [5]. This allows us to select a data-driven tuning parameter when only summary statistics are available and cross-validation is not feasible. We prove that tuning parameters chosen by this method satisfy an oracle inequality with high probability, and demonstrate the effectiveness of the method via simulations.
The use of summary statistics for regression modeling has been considered in the statistical genetics literature [6]. The lassosum method for polygenic risk prediction was introduced by [22], which considered fitting a  $L_{1}$  penalized linear regression with summary statistics, and its theoretical properties were studied in depth by [15]. In [4], the authors extend these ideas to polygenic risk prediction with binary traits. The summary statistics used in these methods include the marginal associations between genetic variants and phenotypes, and statistics summarizing the covariance structures among all genetic variants oftentimes derived from a reference genotype dataset. Empirical studies have demonstrated that the efficacy of such models is significantly influenced by the choices of the GWAS summary statistics and the reference dataset [30]. However, there's still limited theoretical understanding regarding how the overlap of samples and the inherent heterogeneity between datasets impact the model performance. Moreover, most current approaches devise models for a single trait within a single ancestral population. Considering shared genetic architectures could potentially enhance performance by employing a multi-task learning strategy [23]. The authors of [24] take this approach, and describe a multi-task estimator for multi-ancestry pQTL analysis. However, they do not consider the setting when only summary statistics are available for each task.
Our methods build upon classical multi-task learning techniques, and enable fitting models only using basic summary statistics which are often made publicly available. The sparse regularized estimator extends the group-sparse estimators studied in [19, 18], while the nuclear norm estimator expands on the low-rank regression model described in [25]. The nuclear norm approach is closely related to the linear representation learning problem [7, 31], which constrains the regression coefficients to a shared low-dimensional subspace.
Another recent line of work studies the multi-task learning problem under data-sharing constraints. In [17], the authors describe a federated multi-task learning linear regression model for privacy-preserving data analysis. Similarly, [2] presents a computational framework for multi-task learning under DataSHIELD [11] constraints. The formulation of these methods is conceptually similar to ours, but they do not provide theoretical guarantees for their estimators, and we consider a more flexible setting where the summary statistics can be derived from different sources.
Finally, our methods are closely related to the one-shot federated learning paradigm, in which only one round of communication is permitted between the primary local research site and additional sites. [9] presents an algorithm for fitting logistic regression models using summary statistics from different research sites. The works of [20, 34] extend these ideas to linear mixed effects models and generalized mixed effects models, respectively. [21] presents a federated algorithm for fitting the Cox proportional hazards model, and [16] studies federated transfer learning methods for fitting generalized linear models. Nevertheless, the summary statistics addressed in our research are frequently reported in existing studies and can be employed across various models. This is in contrast to the one-shot federated algorithm, where the summary statistics are model specific and the implementation relies on the infrastructure of a collaborative environment.
2 Problem setup and methods
Consider the setting where we are interested in learning a total of  $Q$  tasks simultaneously. For each  $q \in [Q]$ , we posit the linear model
$$
\mathbf {Y} ^ {(q)} = \mathbf {X} ^ {(q)} \boldsymbol {\beta} ^ {(q)} + \varepsilon^ {(q)}
$$
where  $\mathbf{Y}^{(q)}\in \mathbb{R}^{n_q}$ ,  $\mathbf{X}^{(q)}\in \mathbb{R}^{n_q\times p}$ , and  $\varepsilon^{(q)}$  is mean-zero random noise. Each index  $q$  corresponds to the  $q_{th}$  task. The dataset  $\mathcal{D}^{(q)} = (\mathbf{Y}^{(q)},\mathbf{X}^{(q)})$  contains the individual-level observations of the outcome and features respectively for the  $q_{th}$  task. We consider the generic setting where the features  $\mathcal{D}^{(q)}$  might be collected from either overlapping or non-overlapping samples across tasks. Our estimation of interest is the matrix  $\mathbf{B}^* = [\beta^{(1)},\dots,\beta^{(Q)}]\in \mathbb{R}^{p\times Q}$ , where the  $q_{th}$  column of  $\mathbf{B}^*$  is  $\beta^{(q)}$ . Furthermore, let  $e_i$  denote the  $i_{th}$  standard basis vector, so that  $\beta^{(q)} = \mathbf{B}^* e_q$ .
If all the individual-level observations  $\mathcal{D}^{(q)}$  are available, a natural estimator of  $\mathbf{B}^*$  is the regularized multi-task least-squares estimator
$$
\widehat {\mathbf {B}} = \underset {\mathbf {B}} {\arg \min } \left\{\sum_ {q \in [ Q ]} \frac {1}{2 n _ {q}} \left\| \mathbf {Y} ^ {(q)} - \mathbf {X} ^ {(q)} \mathbf {B} e _ {q} \right\| _ {2} ^ {2} + \lambda \mathcal {P} (\mathbf {B}) \right\} \tag {1}
$$
where  $\mathcal{P}$  is a suitable penalty, chosen to enforce similarity structure between tasks, with tuning parameter  $\lambda > 0$ . However, in many applications, we are less likely to observe  $\mathcal{D}^{(q)}$ . Rather, summary statistics which contains information of the feature-outcome and feature-feature relationships may be more likely to be made publicly available. Motivated by the use case in genetic risk prediction, we assume that only summary statistics  $\widehat{\mathbf{S}}^{(q)}$  and  $\widetilde{\mathbf{S}}^{(q)}$  are observable, where  $\widehat{\mathbf{S}}^{(q)} = \frac{1}{n_q}\mathbf{X}^{(q)^T}\mathbf{Y}^{(q)}$  are derived from  $\{\mathbf{X}^{(q)},\mathbf{Y}^{(q)}\}$ , which we termed as the discovery data, and  $\widetilde{\Sigma}^{(q)} = \frac{1}{\tilde{n}_q}\widetilde{\mathbf{X}}^{(q)T}\widetilde{\mathbf{X}}^{(q)}$  is a sample covariance matrix computed from the proxy data  $\widetilde{\mathbf{X}}^{(q)} \in \mathbb{R}^{\tilde{n}_q \times p}$ , which may or may not have overlap with  $\mathbf{X}^{(q)}$ .
Our goal is to estimate  $\mathbf{B}^*$  using two sets of summary statistics  $\widehat{\mathbf{S}}^{(q)}$  and  $\widetilde{\Sigma}^{(q)}$ . We note that  $\widetilde{\mathbf{X}}^{(q)}$  is not necessarily equal to  $\mathbf{X}^{(q)}$ . In practice, the studies which report  $\widehat{\mathbf{S}}^{(q)}$  may not be the same as the ones reporting  $\widetilde{\Sigma}^{(q)}$ . Intuitively, we hope that  $\widetilde{\mathbf{X}}^{(q)}$  is generated from a similar population as  $\mathbf{X}^{(q)}$ , but this may not hold in general. In Section 3, our theoretical analysis reveals how the overlap between  $\widetilde{\mathbf{X}}^{(q)}$  and  $\mathbf{X}^{(q)}$  and their distributional shift can influence the accuracy of multi-task learning.
To construct an estimator that uses only the information provided by  $\widetilde{D}^{(q)}$ , we notice that the least-squares loss can be written as
$$
\mathcal {L} (\beta) = \| \mathbf {Y} - \mathbf {X} \beta \| _ {2} ^ {2} = \mathbf {Y} ^ {T} \mathbf {Y} - 2 \langle \boldsymbol {\beta}, \mathbf {X} ^ {T} \mathbf {Y} \rangle + \boldsymbol {\beta} ^ {T} \mathbf {X} ^ {T} \mathbf {X} \boldsymbol {\beta}
$$
By dropping the constant term, we arrive at a loss function that can be computed using only summary-level information, namely the matrices  $\mathbf{X}^T\mathbf{Y}$  and  $\mathbf{X}^T\mathbf{X}$ . This motivates our general strategy for constructing an estimator only using summary statistics: we substitute  $\widehat{\mathbf{S}}^{(q)}$  and  $\widetilde{\Sigma}^{(q)}$  where appropriate in each least-square loss function in Equation 1 and arrive at the following optimization problem.
$$
\widehat {\mathbf {B}} = \arg \min  _ {\mathbf {B}} \left\{\sum_ {q \in [ Q ]} \frac {1}{2} \left\| \widetilde {\mathbf {S}} ^ {(q) 1 / 2} \mathbf {B} e _ {q} \right\| _ {2} ^ {2} - \langle \widehat {\mathbf {S}} ^ {(q)}, \mathbf {B} e _ {q} \rangle + \lambda \mathcal {P} (\mathbf {B}) \right\} \tag {2}
$$
There are many possible choices of  $\mathcal{P}$  for enforcing structure similarities across tasks. For instance, the recent works of [29] and [12] study low-rank and angle-based penalties for enforcing a shared orientation among the task-specific parameters. In this work, we study two estimators obtained under the  $\ell_{2,1}$  norm penalty, denoted  $\| .\|_{2,1}$ , and the nuclear norm penalty, denoted  $\| .\|_{*}$ . These penalties are chosen for their intuitive interpretation: the  $\ell_{2,1}$  penalty is more likely to be effective if a common set of variables are active across the tasks. If the task-specific parameters tend to be "correlated",
in the sense that they lie in a low-dimensional subspace, the nuclear norm penalty is preferred. In practice, certain domain knowledge can be incorporated to determine the penalty structure, or it can be chosen in a data-driven way in the existence of a validation dataset.
The corresponding estimators are expressed as follows:
$$
\widehat {\mathbf {B}} ^ {(s p)} = \arg \min  _ {\mathbf {B}} \left\{\sum_ {q \in [ Q ]} \frac {1}{2} \left\| \widetilde {\mathbf {S}} ^ {(q) 1 / 2} \mathbf {B} e _ {q} \right\| _ {2} ^ {2} - \langle \widehat {\mathbf {S}} ^ {(q)}, \mathbf {B} e _ {q} \rangle + \lambda \| \mathbf {B} \| _ {2, 1} \right\} \tag {3}
$$
$$
\widehat {\mathbf {B}} ^ {(l r)} = \arg \min  _ {\mathbf {B}} \left\{\sum_ {q \in [ Q ]} \frac {1}{2} \left\| \widetilde {\mathbf {S}} ^ {(q) 1 / 2} \mathbf {B} e _ {q} \right\| _ {2} ^ {2} - \langle \widehat {\mathbf {S}} ^ {(q)}, \mathbf {B} e _ {q} \rangle + \lambda \| \mathbf {B} \| _ {*} \right\} \tag {4}
$$
The superscripts  $(sp)$  and  $(lr)$  stand for "sparse" and "low-rank" respectively.
Before presenting our theoretical results, we first introduce the relevant notation. Let  $N$  denote the total size of discovery observations and proxy observation across all  $Q$  tasks. Formally,
$$
N = \sum_ {q = 1} ^ {Q} (n _ {q} + \tilde {n} _ {q})
$$
We note that  $N$  may double-count individuals who are part of both the proxy data and the discovery data. Define the subset  $\mathcal{I}_q\subset [N]$  as the index set for the discovery data points in the  $q_{th}$  task; in other words  $i\in \mathcal{I}_q$  implies  $X_{i}\in \mathbb{R}^{p}$  is a row of  $\mathbf{X}^{(q)}$ . We define  $\widetilde{\mathcal{I}}_q$  analogously for the proxy data;  $i\in \widetilde{\mathcal{I}}_q$  implies  $X_{i}$  is a row of  $\widetilde{\mathbf{X}}^{(q)}$ . Let  $\tilde{\rho}_{q} = |\mathcal{I}_{q}\cap \widetilde{\mathcal{I}}_{q}| / \tilde{n}_{q}$  denote the proportion of proxy samples which are also in the discovery dataset for the  $q_{th}$  task. In the results that follow, let
$$
\gamma_ {q} = 1 + \left\| \beta^ {(q)} \right\| _ {2} ^ {2} \left(\frac {n _ {q}}{\tilde {n} _ {q}} + 1 - 2 \tilde {\rho} _ {q}\right)
$$
and take  $\gamma = \max_q\gamma_q$ . Additionally, let  $\Xi \in \mathbb{R}^{p\times Q}$  be the matrix with its  $q_{th}$  column equal to  $(\Sigma_1^{(q)} - \Sigma_2^{(q)})\beta^{(q)}$ , where  $\Sigma_1^{(q)}$  and  $\Sigma_2^{(q)}$  are the population-level covariance matrices of  $\mathbf{X}^{(q)}$  and  $\widetilde{\mathbf{X}}^{(q)}$  respectively. The quantities  $\gamma$  and  $\Xi$  play important roles in our results that follow. In particular,  $\gamma$  is a multiplicative factor in our bounds that represents the cost of using proxy data rather than individual-level data. Similarly,  $\Xi$  will represent the cost of using a proxy dataset with a distributional shift from the discovery data. Finally, we will let  $n_{\mathrm{min}}$  and  $\tilde{n}_{\mathrm{min}}$  denote the smallest sample size of discovery and proxy data, respectively. All proofs are given in the supplement.
3.1 Guarantees for  $\ell_{2,1}$ -norm estimator
In this section, we formally state our assumptions and results for the  $\widehat{\mathbf{B}}^{(sp)}$  estimator. The assumptions are standard for high-dimensional regularized estimators, see [26] for a deeper discussion of these conditions.
Assumption 3.1 (Sub-gaussian design and noise). The following holds for each  $q \in [Q]$ : The rows of  $\mathbf{X}^{(q)}$  are independent and identically distributed according to a sub-Gaussian distribution with covariance matrix  $\boldsymbol{\Sigma}_1^{(q)} \in \mathbb{R}^{p \times p}$ . Similarly, the rows of  $\widetilde{\mathbf{X}}^{(q)}$  are independent and identically distributed according to a sub-Gaussian distribution with covariance  $\boldsymbol{\Sigma}_2^{(q)} \in \mathbb{R}^{p \times p}$ . The matrices  $\boldsymbol{\Sigma}_1^{(q)}$  and  $\boldsymbol{\Sigma}_2^{(q)}$  have bounded eigenvalues. The entries of  $\varepsilon^{(q)}$  are independent and identically distributed according to a sub-Gaussian distribution with parameter  $\sigma^2$ . The  $\mathbf{X}^{(q)}$  and  $\varepsilon^{(q)}$  are independent of one another.
Assumption 3.2 (Shared support). There exists a subset  $S^{*} \subset [p]$  such that  $\operatorname{supp}(\beta^{(q)}) = S^{*}$  for each  $q$ .
Definition 3.1 (Sparse cone). For any  $S \subset [p]$ , let
$$
\mathcal {C} _ {\alpha} (S) = \left\{\Delta \in \mathbb {R} ^ {p \times Q}: \| \Delta_ {S ^ {c}} \| _ {2, 1} \leq \alpha \| \Delta_ {S} \| _ {2, 1} \right\}
$$
Assumption 3.3 (Restricted strong convexity). There exists a constant  $\kappa >0$  and a sequence  $a_{N}\to 0$  as  $N\rightarrow \infty$  such that the following inequality holds for each  $\Delta \in \mathcal{C}_3(S^*)$  with probability at least  $1 - a_N$ :
$$
\sum_ {q = 1} ^ {Q} \left\| \widetilde {\boldsymbol {\Sigma}} ^ {(q) 1 / 2} \Delta e _ {q} \right\| _ {2} ^ {2} \geq \frac {1}{\kappa} \| \Delta \| _ {F} ^ {2}
$$
Theorem 3.1. Under assumptions 3.1, 3.2, and 3.3, there exist constants  $c_{1}$  and  $c_{2}$  depending only on the  $\sigma^2$  and the eigenvalues of  $\pmb{\Sigma}_{1}^{(q)}$  and  $\pmb{\Sigma}_{2}^{(q)}$  such that if  $n_{\mathrm{min}}\wedge \tilde{n}_{\mathrm{min}}\geq c_1\| \mathbf{B}^*\|_{\infty ,\infty}(Q + \log p)$  and  $\lambda = O(\sqrt{\gamma(Q + \log p) / n_{\mathrm{min}}} +\| \Xi \|_{2,\infty})$ , the following inequality holds with probability at least  $1 - e^{-\log p} - a_N$ :
$$
\left\| \widehat {\mathbf {B}} ^ {(s p)} - \mathbf {B} ^ {*} \right\| _ {F} \leq c _ {2} \left(\sqrt {\frac {\gamma s (Q + \log p)}{n _ {\operatorname* {m i n}}}} + \sqrt {s} \| \Xi \| _ {2, \infty}\right)
$$
In the subsequent discussion, we take  $q^{*} = \arg \max_{q\in [Q]}\gamma_{q}$  and  $(n,n,\tilde{\rho}) = (n_{q^*},\tilde{n}_{q^*},\tilde{\rho}_{q^*})$  so that the triplet  $(n,\tilde{n},\tilde{\rho})$  corresponds to the same sizes and overlap factor used to compute  $\gamma$ .
There are three main quantities in this upper bound that are of novel interest: the ratio of discovery data size to proxy data size  $n / \tilde{n}$ , the proportion of overlap between the discovery and proxy data  $\tilde{\rho}$ , and the error in specifying the proxy data distribution  $\Xi = (\Sigma^{(1)} - \Sigma^{(2)})\beta$ . The first two of these are captured by the factor  $\gamma$ . Our results show that with fixed  $n$  and  $\tilde{n}$ , the larger proportion of overlap leads to better estimation accuracy. When the proxy data and discovery data are precisely the same, meaning that  $\mathcal{I}_q = \tilde{\mathcal{I}}_q$  for all  $q$ , we recover the minimax rate of estimation for the  $\ell_{2,1}$  penalized multi-task learning problem established by Theorem 6.1 of [18]. If the proxy data and the discovery data are disjoint, meaning that  $\mathcal{I}_q \cap \tilde{\mathcal{I}}_q = \emptyset$  for all  $q$ , the error is increased relative to the minimax rate by a factor of  $(1 + n / \tilde{n})\| \beta \| _2^2$ . This recovers the result of Theorem 2.1 in [15] up to a constant factor, assuming that  $\Xi = 0$ . The novelty of Theorem 3.1 is that we are able to characterize the convergence rate of  $\widehat{\mathbf{B}}^{(sp)}$  for any values of  $n / \tilde{n}$ ,  $\tilde{\rho}$ , and  $\Xi$ . Additionally, we emphasize that the form of the  $\gamma$  term implies that a price is paid anytime when  $\mathbf{X}^{(q)}$  is not fully contained in  $\widetilde{\mathbf{X}}^{(q)}$ . Indeed, if  $\tilde{\rho} < 1 / 2$  and we take  $\tilde{n}\rightarrow \infty$  we still have that  $\gamma >1$  as long as the signal is nonzero. Counter-intuitively, this indicates that an oracle model which has full access to the population-level covariance matrix of the covariates will perform worse in terms of estimation error than an estimator which has access to individual-level data. Furthermore, if  $\tilde{\rho} >1 / 2$ , our theorem predicts that the estimator will out-perform the oracle estimator that uses the population covariance matrix. These phenomena are validated in our simulation studies in Section 5.
3.2 Guarantees for the nuclear norm estimator
Now we state our results for the low-rank proxy data estimator, when the penalty is taken to be the nuclear norm. Once again, these assumptions are standard for high-dimensional regression problems with the nuclear norm [26, 33].
Assumption 3.4 (Low rank). The matrix  $\mathbf{B}^*$  has rank  $r < p \wedge Q$ . Let  $\mathcal{U}^*$  and  $\mathcal{V}^*$  denote the column space and row space of  $\mathbf{B}^*$  respectively. Note that  $\mathcal{U}^*$  and  $\mathcal{V}^*$  each have dimension  $r$ .
Definition 3.2 (Subspaces). Let  $\mathcal{U}$  denote a dimension  $k\leq p\land Q$  subspace of  $\mathbb{R}^p$ , and let  $\mathcal{V}$  denote a dimension  $k\leq p\land Q$  subspace of  $\mathbb{R}^Q$ . Define
$$
\mathbb {M} = \mathbb {M} (\mathcal {U}, \mathcal {V}) := \left\{\Delta \in \mathbb {R} ^ {p \times Q}: \operatorname {r o w} (\Delta) = \mathcal {V}, \operatorname {c o l} (\Delta) = \mathcal {U} \right\}
$$
$$
\mathbb {M} ^ {\perp} = \mathbb {M} ^ {\perp} (\mathcal {U}, \mathcal {V}) = \left\{\Delta \in \mathbb {R} ^ {p \times Q}: \operatorname {r o w} (\Delta) \perp \mathcal {V}, \operatorname {c o l} (\Delta) \perp \mathcal {U} \right\}
$$
Furthermore, for any subspace  $\Omega$  of  $\mathbb{R}^{p\times Q}$ , let  $\Delta_{\Omega}$  denote the projection of  $\Delta$  onto  $\Omega$ .
We will denote  $\mathbb{M}^* = \mathbb{M}(\mathcal{U}^*,\mathcal{V}^*)$
Definition 3.3 (Low rank cone). For any set  $\mathbb{M}$  as defined above, let
$$
\mathcal {C} _ {\alpha} (\mathbb {M}) = \left\{\Delta \in \mathbb {R} ^ {p \times Q}: \| \Delta_ {\mathbb {M} ^ {\perp}} \| _ {*} \leq \alpha \| \Delta_ {\mathbb {M}} \| _ {*} \right\}
$$
Assumption 3.5 (Restricted strong convexity). There exists a constant  $\kappa >0$  and a sequence  $b_{N}\to 0$  as  $N\to \infty$  such that the following inequality holds for each  $\Delta \in \mathcal{C}_3(\mathbb{M}^*)$  with probability at least  $1 - b_{N}$ :
Theorem 3.2. Under assumptions 3.1, 3.4, and 3.5, there exist constants  $c_{1}$  and  $c_{2}$  depending only on  $\sigma^2$  and the eigenvalues of  $\boldsymbol{\Sigma}_1^{(q)}$  and  $\boldsymbol{\Sigma}_2^{(q)}$  such that if  $n_{\mathrm{min}} \wedge \tilde{n}_{\mathrm{min}} \geq c_1 \| \mathbf{B}^* \|_{\infty, \infty}(Q + p)$  and  $\lambda = O(\sqrt{\gamma(Q + p) / n_{\mathrm{min}}} + \| \Xi \|_{\mathrm{op}})$ , the following inequality holds with probability at least  $1 - e^{-p} - b_N$ :