SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {\theta} _ {\mathrm {S L}} \leftarrow \mathcal {A} _ {\mathrm {S L}} (\mathcal {D} _ {l})
$$
$$
\hat {\theta} _ {\mathrm {U L} +} \leftarrow \mathcal {A} _ {\mathrm {U L} +} \left(\mathcal {D} _ {u}, \mathcal {D} _ {l}\right)
$$
$$
\mathbf {i f} s \leq \min  \left\{\sqrt {\frac {d}{n _ {l}}}, \left(\frac {d}{n _ {u}}\right) ^ {1 / 4} \right\}
$$
$$
\begin{array}{c} \left. \begin{array}{l} \hline \hat {\boldsymbol {\theta}} _ {\mathrm {S S L - S}} = 0 \end{array} \right. \end{array}
$$
$$
\text {e l s e i f} \left\{\sqrt {\frac {d}{n _ {l}}}, \left(\frac {d}{n _ {u}}\right) ^ {1 / 4} \right\} <   s \leq \sqrt {\frac {n _ {l}}{n _ {u}}}
$$
$$
\left\lfloor \hat {\boldsymbol {\theta}} _ {\mathrm {S S L - S}} = \hat {\boldsymbol {\theta}} _ {\mathrm {S L}} \right.
$$
$$
\left. \begin{array}{l} \hline \hat {\boldsymbol {\theta}} _ {\mathrm {S L S - S}} = \hat {\boldsymbol {\theta}} _ {\mathrm {U L +}} \end{array} \right.
$$
return  $\theta_{\mathrm{SSL - S}}$
that the SSL-S algorithm chooses to output the low-SNR regime, the trivial estimator  $\hat{\theta} = 0$ .  $\Phi (0)\simeq e^{-s^2 /2}s$  where  $\Phi$  is the CDF of the
Moreover, the technical condition  $s \in (0,1]$  is not overly restrictive as this range for the SNR is already sufficient for seeing all the relevant regimes, e.g. SSL-S switches from using SL to UL+ within this range. Finally, while the rates of either SL or UL+ cannot be improved further using SSL algorithms, it is nonetheless possible to improve the error by a constant factor, independent of  $n_l$  and  $n_u$ . To see this, in Section 4 we describe an algorithm that uses both  $\mathcal{D}_l$  and  $\mathcal{D}_u$  effectively and can hence achieve a provable improvement in error over both SL and UL+.
3.2 Comparison of SSL with UL+ and SL in different regimes
To understand whether an SSL algorithm is using the labeled and unlabeled data effectively, we compare the error rate of SSL algorithms to the minimax rates for SL and UL+ algorithms.
A gap in these rates for a certain SSL algorithm would indicate that the algorithm can obtain lower error than both SL and  $\mathrm{UL + }$  when presented with the same amount of (large enough) data. We study the improvement rates defined below, which capture the error ratio for the worst-case distributions between the minimax rate for SSL and the minimax rate for SL or  $\mathrm{UL + }$ .
Definition 1 (SSL improvement rates). For a set of parameters  $\Theta \subseteq \mathbb{R}^d$ , we define the improvement rates of SSL over SL and  $UL+$  as  $h_l$  and  $h_u$ , respectively, where
$$
h _ {l} \left(n _ {l}, n _ {u}, \Theta\right) := \frac {\inf  _ {\mathcal {A} _ {S S L}} \sup  _ {\boldsymbol {\theta} ^ {*} \in \Theta} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {S S L} \left(\mathcal {D} _ {l} , \mathcal {D} _ {u}\right) , \boldsymbol {\theta} ^ {*}\right) \right]}{\inf  _ {\mathcal {A} _ {S L}} \sup  _ {\boldsymbol {\theta} ^ {*} \in \Theta} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {S L} \left(\mathcal {D} _ {l} , \emptyset\right) , \boldsymbol {\theta} ^ {*}\right) \right]}, \tag {6}
$$
$$
h _ {u} \left(n _ {l}, n _ {u}, \Theta\right) := \frac {\inf  _ {\mathcal {A} _ {S S L}} \sup  _ {\boldsymbol {\theta} ^ {*} \in \Theta} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {S S L} \left(\mathcal {D} _ {l} , \mathcal {D} _ {u}\right) , \boldsymbol {\theta} ^ {*}\right) \right]}{\inf  _ {\mathcal {A} _ {U L +}} \sup  _ {\boldsymbol {\theta} ^ {*} \in \Theta} \mathbb {E} \left[ \mathcal {E} \left(\mathcal {A} _ {U L +} \left(\mathcal {D} _ {l} , \mathcal {D} _ {u}\right) , \boldsymbol {\theta} ^ {*}\right) \right]}, \tag {7}
$$
where the expectations are over  $\mathcal{D}_l\sim (P_{XY}^*)^{n_l}$  and  $\mathcal{D}_u\sim (P_X^\theta)^{n_u}$
To simplify notation, we denote the improvement rates of SL and  $\mathrm{UL + }$  over  $\Theta^{(s)}$  as  $h_l(n_l,n_u,s)$  and  $h_u(n_l,n_u,s)$ , respectively. A straightforward upper bound for these rates is  $h_l,h_u\leq 1$ , achieved when utilizing an SL and  $\mathrm{UL + }$  algorithm, respectively. SSL demonstrates an enhanced error rate over SL and  $\mathrm{UL + }$ , if both  $\lim_{n_l,n_u\to \infty}h_l(n_l,n_u,\Theta) = 0$  and  $\lim_{n_l,n_u\to \infty}h_u(n_l,n_u,\Theta) = 0$ . If  $h_l$  or  $h_u$  lies in (0,1) without converging to zero as  $n_l,n_u\to \infty$ , then SSL surpasses SL or  $\mathrm{UL + }$ , respectively, only by a constant factor.
SNR Regime	Rate of growth of nu vs nl	hl(nl, nu, s)	hu(nl, nu, s)
s = o(√1/nu)	Any	cSL	0
fixed s > 0	nu = o(nl)	cSL	0
nu = ω(nl)	0	cUL
limnl, nu → ∞ n_u / n_l = c	(1/1+cs^2) cSL	(s^2c/1+s^2c) cUL
Table 2: SSL improvement rates over SL and UL+ for different regimes of  $s, n_l$  and  $n_u$ , where  $h_l$  and  $h_u$  are evaluated for  $\lim_{n_l, n_u \to \infty}$ .
Given Theorem 1, we can directly derive the improvement rates of SSL in the following corollary.
Corollary 1. Assuming the setting of Theorem 1, the improvement rates of SSL can be written as:
$$
\text {I m p r o v e m e n t} h _ {l} \left(n _ {l}, n _ {u}, s\right) \asymp \frac {n _ {l}}{n _ {l} + s ^ {2} n _ {u}}.
$$
$$
\text {I m p r o v e m e n t} \quad h _ {u} \left(n _ {l}, n _ {u}, s\right) \asymp \frac {s ^ {2} n _ {u}}{n _ {l} + s ^ {2} n _ {u}}.
$$
Based on this corollary, we now argue how no SSL algorithm can simultaneously improve the rates of both SL and  $\mathrm{UL + }$  for any regime of  $s,n_{l}$  and  $n_u$  , as summarized in Table 2.
1. SSL rate is not faster than SL, but is faster than UL+. For extremely low values of SNR (i.e.  $s \to 0$  faster than  $\sqrt{n_u} \to \infty$ ), we have that  $\lim_{n_l, n_u \to \infty} h_l(n_l, n_u, s) = c_{SL} > 0$  and hence SSL fails to improve the SL rates even with infinite unlabeled data. This setting has been the focus of previous worst-case analyses for SSL [6, 36, 20]. Alternatively, the SSL rate is also not better than the SL rate for a fixed SNR, when the labeled dataset is significantly larger than the unlabeled data, i.e.  $n_u = o(n_l)$ .
2. SSL rate is faster than SL, but not faster than UL+. As mentioned in Gopfert et al. [20], in order for SSL to lead to a rate improvement compared to SL, it is necessary that the unlabeled set size is at least a superlinear function of the labeled set size, i.e.  $n_u = \omega(n_l)$ . Corollary 1 shows that this condition is, in fact, sufficient for 2-GMM distributions: for  $s > 0$ , as long as  $n_u$  grows superlinearly with respect to  $n_l$ ,  $h_l(n_l, n_u, s) \to 0$ , and hence, SSL can achieve faster rates than SL. Despite the improvement over SL, for this setting, the asymptotic error ratio between SSL and UL+ does not vanish, i.e.  $\lim_{n_l, n_u \to \infty} h_u(n_l, n_u, s) = c_{\mathrm{UL}} > 0$ .
3. SSL rate is not faster than either SL or UL+. Finally, in the regime where the unlabeled dataset size depends linearly on the size of the labeled set i.e.  $\lim_{n_l,n_u\to \infty}\frac{n_u}{n_l}\to c$  for some constant  $c > 0$ , neither of the improvement rates vanishes for  $n_l,n_u\to \infty$ .
As explained in Section 2.2, prior  $\mathrm{UL}+$  algorithms that work well in practice do not use the labeled data for the unsupervised learning step. Interestingly, for the case when both the labeled and unlabeled data are used for the unsupervised step, the trends in Table 2 remain the same, with only one exception: the improvement rates in the last line of the table (i.e.  $\frac{n_u}{n_l}\rightarrow c\in (0,\infty)$ ) would only change by a small constant. More importantly, the main takeaway from Table 2 remains the same: SSL cannot achieve better rates than both  $\mathrm{UL}+$  and SL at the same time since there is no regime for which  $h_l$  and  $h_u$  are simultaneously 0.
4 Empirical improvements over the rate-optimal SSL-S algorithm
Section 3 shows that the SSL minimax rates can be achieved with a simple algorithm that switches between a minimax optimal SL and a minimax optimal UL+ algorithm. Despite being optimal in terms of statistical rates, this SSL Switching algorithm does not make the most effective use of the available data: SL algorithms solely uses the labeled data whereas UL+ learns the decision boundary using just the unlabeled data and the labeled data is used to only label the different prediction regions. Alternatively, one could use both labeled and unlabeled data to learn the decision boundary. In this section, we investigate the following question: Is it possible to improve over the error of SSL-S even though the rate would be, at best, as good as the 'wasteful' SSL-S algorithm as per Section 3?
In this section, we present experiments to show that, in fact, a remarkably simple algorithm (i.e. a weighted ensemble of the SL and  $\mathrm{UL}+$  classifiers) can outperform the minimax optimal SSL-S algorithm. We show that algorithms such as self-training, which have been shown to excel in practice [40, 34, 22], can also improve over SSL-S. Consequently, it remains an exciting avenue for future work to derive tight analyses that characterize the improvement of such algorithms over  $\mathrm{SL} / \mathrm{UL}+$ .
A simple algorithm more effective than SSL-S A natural approach to improve upon the naive SSL-S algorithm is to construct a weighted ensemble of an SL and a  $\mathrm{UL + }$  estimator, trained on  $\mathcal{D}_l$  and  $\mathcal{D}_u$ , respectively, with a controllable weighting hyperparameter  $t$ . We call this the SSL Weighted algorithm (SSL-W) shown in Algorithm 3. With an appropriate choice of  $t$ , it is possible to show that the SSL-W algorithm performs better (up to sign permutation) than SSL-S. The formal statement of this re
Algorithm 3: SSL-W algorithm
Input:  $\mathcal{D}_l, \mathcal{D}_u, t$
Result:  $\hat{\theta}_{\mathrm{SSL - W}}$
$$
\begin{array}{l} \boldsymbol {\theta} _ {\mathrm {S L}} \leftarrow \mathcal {A} _ {\mathrm {S L}} \left(\mathcal {D} _ {l}\right) \\ \hat {\theta} _ {\mathrm {U L} +} \leftarrow \mathcal {A} _ {\mathrm {U L} +} \left(\mathcal {D} _ {l}, \mathcal {D} _ {u}\right) \\ \hat {\boldsymbol {\theta}} _ {\mathrm {S S L - W}} (t) = t \hat {\boldsymbol {\theta}} _ {\mathrm {S L}} + (1 - t) \hat {\boldsymbol {\theta}} _ {\mathrm {U L} +} \\ \text {r e t u r n} \theta_ {\mathrm {S S L - W}} (t) \\ \end{array}
$$
sult together with the proof are deferred to Appendix D. In practice, one can fix the sign permutation of the  $\hat{\theta}_{\mathrm{SSL - W}}$  estimator using a small amount of labeled data. The intuition for this improvement is that the ensemble estimator  $\hat{\theta}_{\mathrm{SSL - W}}$  achieves lower error than the constituent estimators of the ensemble (i.e.  $\hat{\theta}_{\mathrm{SL}}$  and  $\hat{\theta}_{\mathrm{UL + }}$ ), which, in turn, determine the error of the SSL-S algorithm.
4.1 Empirical improvements over the minimax optimal SSL-S algorithm
In this section we present linear classification experiments on synthetic and real-world data to show that there indeed exist SSL algorithms that can improve over the error of the (minimax optimal) SSL Switching Algorithm. For both synthetic and real-world data, we use  $\hat{\theta}_{\mathrm{SL}} = \frac{1}{n_l}\sum_{i = 1}^{n_l}Y_iX_i$  as the SL estimator and an Expectation-Maximization (EM) algorithm for the UL algorithm (see Appendix E for details). The optimal switching point for SSL-S and the optimal weight for SSL-W, as well as the optimal  $\ell_2$  penalty for logistic regression are chosen using a validation set.
Synthetic data. We consider data drawn from a symmetric and isotropic 2-GMM distribution  $P_{XY}^{\theta^{*}}$  over  $\mathbb{R}^2$ . The labeled and unlabeled set sizes are set to 20 and 2000, respectively. Figure 1a shows the gap between SSL-W and SL or UL+ as a function of the SNR  $s$  (Figure 3 in Appendix F shows the dependence of the error gap on  $n_l$ ). There are two main takeaways. First, for varying SNR values  $s$ , SSL-W always outperforms SL and UL+, and hence, also SSL-S. Second, as argued in Section 3.2, SSL-W improves more over UL+ for small values of the SNR  $s$ , and it improves more over SL for large values of the SNR.
Real-world data. We consider 10 binary classification real-world datasets: five from the OpenML repository [37] and five 2-class subsets of the MNIST dataset [13]. For the MNIST subsets, we choose class pairs that have a linear Bayes error varying between  $0.1\%$  and  $2.5\%$ .<sup>4</sup> We choose from OpenML datasets that have a large enough number of samples compared to dimensionality (see Appendix E for details on how we choose the datasets). The OpenML datasets span a range of Bayes errors that varies between  $3\%$  and  $34\%$ .
In the absence of the exact data-generating process, we quantify the difficulty of SSL on real-world datasets using a notion of compatibility, reminiscent of Balcan and Blum [4]. Specifically, we consider the compatibility given by  $\rho^{-1}$  with  $\rho := \frac{1}{2\sqrt{d}}\left(\Delta(\pmb{\theta}_{UL+}^{*}, \pmb{\theta}_{Bayes}^{*}) + \mathcal{R}_{pred}(\pmb{\theta}_{Bayes}^{*})\right)$ , where  $\Delta(\pmb{\theta}_{UL+}^{*}, \pmb{\theta}_{Bayes}^{*}) := \frac{\mathcal{R}_{pred}(\pmb{\theta}_{UL+}^{*}) - \mathcal{R}_{pred}(\pmb{\theta}_{Bayes}^{*})}{\mathcal{R}_{pred}(\pmb{\theta}_{Bayes}^{*})}$ ,  $d$  is the dimension of the data,  $\pmb{\theta}_{Bayes}^{*}$  is obtained via SL on the entire dataset and  $\pmb{\theta}_{UL+}^{*}$  determines the predictor with optimal sign obtained via UL on the entire dataset. Intuitively, this notion of compatibility captures both the Bayes error of a dataset, as well as how compatible the 2-GMM parametric assumption actually is for the given data.
In addition to SSL-S (Algorithm 2) and SSL-W (Algorithm 3) we also evaluate the performance of self-training, using a procedure similar to the one analyzed in Frei et al. [18]. We use a logistic regression estimator for the pseudolabeling, and train logistic regression with a ridge penalty in the second stage of the self-training procedure. Note that an  $\ell_2$  penalty corresponds to input consistency regularization [38] with respect to  $\ell_2$  perturbations.
4We estimate the Bayes error of a dataset by training a linear classifier on the entire labeled dataset.
(a) 2-GMM synthetic data.
(b) Real-world data.
Figure 1: Error gap between SL or UL+ and SSL-W for varying task difficulties. We assess problem difficulty using the SNR (for 2-GMMs) or the compatibility  $\rho^{-1}$  (for real-world data). We see the same trends for both synthetic and real-world data. Moreover, self-training also exhibits the same trend as  $\hat{\theta}_{\mathrm{SSL - W}}$ .
Figure 1b shows the improvement in classification error of SSL algorithms (i.e. SSL-W and self-training) compared to SL and  $\mathrm{UL}+$ . The positive values of the error gap indicate that SSL-W outperforms both SL and  $\mathrm{UL}+$  even on real-world datasets. This finding suggests that the intuition presented in this section carries over to more generic distributions beyond just 2-GMMs. Finally, Figure 4 in Appendix F shows that SSL-W improves over the error of the minimax optimal SSL-S algorithm, for varying values of  $n_l$ . These findings provide some proof-of-concept evidence that existing SSL algorithms may already be obtaining lower error than the minimax optimal SSL-S algorithm. We hope this observation will encourage future research into characterizing this error gap for practically relevant algorithms such as self-training.
Other theoretical analyses of SSL algorithms. Beyond the theoretical studies highlighted in Section 2, there are a few others pertinent to our research. Specifically, Azizyan et al. [1], Singh et al. [33] present upper bounds for semi-supervised regression, which are contingent on the degree to which the marginal  $P_{X}$  informs the labeling function. This is akin to the results we derive in this work. However, obtaining a minimax lower bound for semi-supervised regression remains an exciting direction for future work. We refer to [27] for an overview of prior theoretical results for SSL.
Balcan and Blum [4] introduced a compatibility score, denoted as  $\chi(f, P_X) \in [0, 1]$ , which connects the space of marginal distributions to the space of labeling functions. While their findings hint that SSL may surpass the SL minimax rates, they offer no comparisons with UL/UL+. Moreover, the paper does not discuss minimax optimality of the proposed SSL algorithms.
On another note, even though SSL does not enhance the rates of UL, Sula and Zheng [35] demonstrate that labeled samples can bolster the convergence speed of Expectation-Maximization within the context of our study.
To conclude, Scholkopf et al. [32] leveraged a causality framework to pinpoint scenarios where SSL does not offer any advantage over SL. In essence, when the covariates, represented by  $X$ , act as causal ancestors to the labels  $Y$ , the independent causal mechanism assumption dictates that the marginal  $P_{X}$  offers no insights about the labeling function.
Minimax rates for SL and UL. The proofs in this work rely on techniques used to derive minimax rates for SL and UL algorithms. Most of these prior results consider the same distributional assumptions as our paper. Wu and Zhou [39] show a tight minimax lower bound for estimation error for spherical 2-GMMs from  $\mathcal{P}_{2\text{-GMM}}$ . Moreover, Azizyan et al. [2], Li et al. [24] derive minimax rates over  $\mathcal{P}_{2\text{-GMM}}$  for classification and clustering (up to permutation).
In addition to the SL and UL algorithms considered in Section 3, Expectation-Maximization (EM) is another family of algorithms that is commonly analyzed for the same distributional setting considered in our paper. For instance, Wu and Zhou [39] rely on techniques from several previous seminal papers [12, 3, 14-16] to obtain upper bounds for EM-style algorithms.
6 Conclusions and limitations
In this paper, we establish a tight lower bound for semi-supervised classification within the class of 2-GMM distributions. Our findings demonstrate that SSL cannot simultaneously improve the error rates of both SL and UL across all signal-to-noise ratios. However, empirical evidence suggests that SSL can improve upon the error of minimax optimal SL or UL algorithms. This observation calls for careful analyses of the error of SSL algorithms that also track constant factors, not only rates.
Our theoretical analysis focuses exclusively on isotropic and symmetric GMMs due to limitations in the technical tools employed for the proofs. Similar constraints can be observed in recent analyses of SL or UL algorithms [24, 39]. However, it is worth noting that whenever the bounds for SL and UL can be extended to more general distributions in the future, these results can be seamlessly used to also extend Theorem 1 to these settings.
AT was supported by a PhD fellowship from the Swiss Data Science Center. We also thank the anonymous reviewers for their helpful comments.
[37] Joaquin Vanschoren, Jan N. van Rijn, Bernd Bischl, and Luis Torgo. OpenML: networked science in machine learning. SIGKDD Explorations, 2013.
[38] Colin Wei, Kendrick Shen, Yining Chen, and Tengyu Ma. Theoretical analysis of self-training with deep networks on unlabeled data. In International Conference on Learning Representations, 2021.
[39] Yihong Wu and Harrison H. Zhou. Randomly initialized EM algorithm for two-component gaussian mixture achieves near optimality in  $O(\sqrt{n})$  iterations. Mathematical Statistics and Learning, 2021.
[40] Qizhe Xie, Minh-Thang Luong, Eduard Hovy, and Quoc V. Le. Self-training with noisy student improves ImageNet classification. June 2020.
[41] David Yarowsky. Unsupervised word sense disambiguation rivaling supervised methods. In 33rd Annual Meeting on Association for Computational Linguistics, 1995.
[42] Barret Zoph, Golnaz Ghiasi, Tsung-Yi Lin, Yin Cui, Hanxiao Liu, Ekin Dogus Cubuk, and Quoc Le. Rethinking pre-training and self-training. In Advances in Neural Information Processing Systems 33, 2020.
A Proof of Proposition 1
In this section we provide the formal statement of Proposition 1 and its proof.
Proposition 2 (Fixing the sign of  $\hat{\theta}_{\mathrm{UL}}$ ). There exist universal constants  $c_{0}, C_{1}, C_{2}, C_{3}, C_{4} > 0$  such that for  $n_u \geq (160 / s)^2 d$ ,  $d \geq 2$  and  $s \in (0, 1]$ , it holds that:
$$
\begin{array}{l} \mathbb {E} \left[ \| \hat {\pmb {\theta}} _ {U L +} - \pmb {\theta} ^ {*} \| \right] \leq C _ {1} \sqrt {\frac {d}{s ^ {2} n _ {u}}} + C _ {2} s e ^ {- \frac {1}{2} n _ {l} s ^ {2} (1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}) ^ {2}}, \quad a n d \\ \mathbb {E} \left[ \mathcal {E} (\hat {\pmb {\theta}} _ {U L +}) \right] \leq C _ {3} e ^ {- \frac {1}{2} s ^ {2}} \frac {d \log (d n _ {u})}{s ^ {3} n _ {u}} + C _ {4} e ^ {- \frac {1}{2} s ^ {2} n _ {l} \left(1 - c _ {0} \sqrt {\frac {d \log (n _ {u})}{s ^ {2} n _ {u}}}\right) ^ {2}}. \\ \end{array}
$$
Proof. Recall that we consider the UL+ estimator  $\hat{\pmb{\theta}}_{\mathrm{UL + }} = \mathrm{sign}\left(\hat{\pmb{\theta}}_{\mathrm{SL}}^{\top}\hat{\pmb{\theta}}_{\mathrm{UL}}\right)\hat{\pmb{\theta}}_{\mathrm{UL}}$  and denote  $\hat{\beta} := \mathrm{sign}\left(\hat{\pmb{\theta}}_{\mathrm{SL}}^{\top}\hat{\pmb{\theta}}_{\mathrm{UL}}\right)$ . Now let  $\beta := \mathrm{sign}(\pmb{\theta}^{*}^{\top}\hat{\pmb{\theta}}_{\mathrm{UL}}) = \arg \min_{\tilde{\beta} \in \{-1, + 1\}}\|\tilde{\beta}\hat{\pmb{\theta}}_{\mathrm{UL}} - \pmb{\theta}^{*}\|^{2}$ .
Note that we can write the expected estimation error of  $\hat{\theta}_{\mathrm{UL + }}$  as
$$
\begin{array}{l} \mathbb {E} \left[ \left\| \hat {\boldsymbol {\theta}} _ {\mathrm {U L} +} - \boldsymbol {\theta} ^ {*} \right\| \right] = \mathbb {E} \left[ \left\| \hat {\beta} \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} - \boldsymbol {\theta} ^ {*} \right\| \right] \\ = \mathbb {E} \left[ \mathbb {1} _ {\{\hat {\beta} = \beta \}} \| \beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} - \boldsymbol {\theta} ^ {*} \| + \mathbb {1} _ {\{\hat {\beta} \neq \beta \}} \| \beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} + \boldsymbol {\theta} ^ {*} \| \right] \\ \leq \mathbb {E} \left[ \mathbb {1} _ {\{\hat {\beta} = \beta \}} \| \beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} - \boldsymbol {\theta} ^ {*} \| \right] + \mathbb {E} \left[ \mathbb {1} _ {\{\hat {\beta} \neq \beta \}} (\| \beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} - \boldsymbol {\theta} ^ {*} \| + 2 \| \boldsymbol {\theta} ^ {*} \|) \right] \\ \leq \mathbb {E} \left[ \| \beta \hat {\boldsymbol {\theta}} _ {\mathrm {U L}} - \boldsymbol {\theta} ^ {*} \| \right] + 2 s \mathbb {P} (\hat {\beta} \neq \beta). \tag {8} \\ \end{array}
$$
Next, we recall a result by Wu and Zhou [39] established for this particular UL estimator, namely that  $\mathbb{E}\left[\| \beta \hat{\pmb{\theta}}_{\mathrm{UL}} - \pmb{\theta}^{*}\| \right]\lesssim \sqrt{\frac{d}{s^{2}n_{u}}}$ . Moreover, the probability of incorrectly estimating the sign (permutation) can be written as