SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\mathbb {P} \left(\left| \Phi_ {\mathrm {P O}} (\boldsymbol {G}) - \lambda \right| > t\right) \leq 2 \mathbb {P} \left(\left| \Phi_ {\mathrm {A O}} (\boldsymbol {g}, \boldsymbol {h}) - \lambda \right| > t\right).
$$
An immediate corollary of this result (by choosing  $\lambda = \mathbb{E}[\Phi_{\mathrm{AO}}(\pmb {g},\pmb {h})]$ ) is that if the optimal cost of AO problem concentrates in probability, then the optimal cost of the corresponding PO problem also concentrates, in probability, around the same value. In addition, as shown in part (iii) of [30](Theorem 3), concentration of the optimal solution of the AO problem implies concentration of the optimal solution of the PO around the same value. Therefore, the two optimization are intimately connected and by analyzing the AO problem, which is substantially simpler, one can derive corresponding properties of the PO problem.
The CGMT framework has been used to infer statistical properties of estimators in certain high-dimensional asymptotic regime. The intermediate steps in the CGMT framework can be summarized as follows: First form an PO problem in the form of (A.1) and construct the corresponding AO problem. Second, derive the point-wise limit of the AO objective in terms of a convex-concave optimization problem, over only few scalar variables. This step is called 'scalarization'. Next, it is possible to establish uniform convergence of the scalarized AO to the (deterministic) min-max optimization problem using convexity conditions. Finally, by analyzing the latter deterministic problem, one can derive the desired asymptotic characterizations.
Of course implementing the above steps involved problem-specific intricate calculations. Our proofs of Theorems 3.1, 3.2, 3.3 in the supplementary follow this general strategy.
B Proof of theorems and technical lemmas
B.1 Proof of Lemma 2.1
By substituting for  $y$  from (2.1) in the definition of risk we obtain
$$
\begin{array}{l} \operatorname {R i s k} (\boldsymbol {\theta}) = \mathbb {E} \left[ (y - \boldsymbol {x} ^ {\top} \boldsymbol {\theta}) ^ {2} \right] \\ = \mathbb {E} \left[ \left(\boldsymbol {x} ^ {\top} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {\theta}\right)\right) ^ {2} \right] + \mathbb {E} \left[ \varepsilon^ {2} \right] \\ \stackrel {(a)} {=} \sum_ {\ell \in [ k ]} \pi_ {\ell} \mathbb {E} \left[ \left(\left(\boldsymbol {\mu} _ {\ell} + \boldsymbol {z}\right) ^ {\top} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {\theta}\right)\right) ^ {2} \right] + \mathbb {E} \left[ \varepsilon^ {2} \right] \\ = \sum_ {\ell \in [ k ]} \pi_ {\ell} \mathbb {E} \left[ \left(\boldsymbol {\mu} _ {\ell} ^ {\mathrm {T}} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {\theta}\right)\right) ^ {2} \right] + \sum_ {\ell \in [ k ]} \pi_ {\ell} \| \boldsymbol {\theta} - \boldsymbol {\theta} _ {0} \| _ {\ell_ {2}} ^ {2} + \sigma^ {2} \\ \stackrel {(b)} {=} \left(\boldsymbol {\theta} - \boldsymbol {\theta} _ {0}\right) ^ {\top} M \operatorname {d i a g} (\boldsymbol {\pi}) M ^ {\top} \left(\boldsymbol {\theta} _ {0} - \boldsymbol {\theta}\right) + \left\| \boldsymbol {\theta} _ {0} - \boldsymbol {\theta} \right\| _ {\ell_ {2}} + \sigma^ {2}, \\ \end{array}
$$
where  $(a)$  follows from the Gaussian-Mixture model (2.2) and  $(b)$  holds since  $\sum_{\ell \in [k]} \pi_{\ell} = 1$ .
B.2 Proof of Theorem 3.1 and Theorem 3.2
Recall that the look-alike estimator is defined as the min-norm estimator over the feature matrix  $X_{L}$  where the look-alike representations are used instead of individual sensitive features; see (3.2).
To analyze risk of  $\widehat{\theta}_L$ , we consider the ridge regression estimator given by
$$
\widehat {\boldsymbol {\theta}} _ {\lambda} = \arg \min  _ {\boldsymbol {\theta}} \frac {1}{2 n} \left\| \boldsymbol {y} - \boldsymbol {X} _ {L} ^ {\top} \boldsymbol {\theta} \right\| _ {\ell_ {2}} ^ {2} + \lambda \left\| \boldsymbol {\theta} \right\| _ {\ell_ {2}} ^ {2}.
$$
The minimum-norm estimator is given by  $\widehat{\pmb{\theta}}_L = \lim_{\lambda \to 0^+}\widehat{\pmb{\theta}}_\lambda$
We follow the CGMT framework explained in Section A. Recall that
$$
\boldsymbol {X} _ {L} = \left[ \begin{array}{c} M _ {\mathrm {s}} \boldsymbol {\Lambda} \\ M _ {\mathrm {n s}} \boldsymbol {\Lambda} + \boldsymbol {Z} _ {\mathrm {n s}} \end{array} \right],
$$
and therefore by substituting for  $y, X$ , and  $X_{L}$ , we get
$$
\begin{array}{l} \frac {1}{2 n} \left\| \boldsymbol {y} - \boldsymbol {X} _ {L} ^ {\top} \boldsymbol {\theta} \right\| _ {\ell_ {2}} ^ {2} = \frac {1}{2 n} \left\| \boldsymbol {\varepsilon} + \boldsymbol {X} ^ {\top} \boldsymbol {\theta} _ {0} - \boldsymbol {X} _ {L} ^ {\top} \boldsymbol {\theta} \right\| _ {\ell_ {2}} ^ {2} \\ = \frac {1}{2 n} \left\| \boldsymbol {\varepsilon} + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) + \boldsymbol {Z} _ {\mathrm {s}} ^ {\mathsf {T}} \boldsymbol {\theta} _ {0, \mathrm {s}} + \left(\boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {n s}} ^ {\mathsf {T}} + \boldsymbol {Z} _ {n s} ^ {\mathsf {T}}\right) \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) \right\| _ {\ell_ {2}} ^ {2}. \\ \end{array}
$$
We define the primary optimization loss as follows:
$$
\mathcal {L} _ {P O} (\boldsymbol {\theta} _ {\mathrm {s}}, \boldsymbol {\theta} _ {\mathrm {n s}}) := \frac {1}{2 n} \left\| \boldsymbol {\varepsilon} + \boldsymbol {\Lambda} ^ {\top} \boldsymbol {M} _ {\mathrm {s}} ^ {\top} (\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}) + \boldsymbol {Z} _ {\mathrm {s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {s}} + (\boldsymbol {\Lambda} ^ {\top} \boldsymbol {M} _ {\mathrm {n s}} ^ {\top} + \boldsymbol {Z} _ {n s} ^ {\top}) (\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}) \right\| _ {\ell_ {2}} ^ {2} + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} ^ {2}
$$
We continue by deriving the auxiliary optimization (AO) problem. By duality, we have
$$
\begin{array}{l} \mathcal {L} _ {P O} \left(\boldsymbol {\theta} _ {\mathrm {s}}, \boldsymbol {\theta} _ {\mathrm {n s}}\right) = \max  _ {\boldsymbol {v}} \frac {1}{n} \left(\boldsymbol {v} ^ {\intercal} \boldsymbol {\varepsilon} + \boldsymbol {v} ^ {\intercal} \boldsymbol {\Lambda} ^ {\intercal} \boldsymbol {M} _ {\mathrm {s}} ^ {\intercal} \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) + \boldsymbol {v} ^ {\intercal} \boldsymbol {Z} _ {\mathrm {s}} ^ {\intercal} \boldsymbol {\theta} _ {0, \mathrm {s}} + \boldsymbol {v} ^ {\intercal} \left(\boldsymbol {\Lambda} ^ {\intercal} \boldsymbol {M} _ {\mathrm {n s}} ^ {\intercal} + \boldsymbol {Z} _ {n s} ^ {\intercal}\right) \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) - \frac {\| \boldsymbol {v} \| _ {\ell_ {2}} ^ {2}}{2}\right) \\ + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} ^ {2} \\ \end{array}
$$
Note that the above is jointly convex in  $(\theta_{\mathrm{s}},\theta_{\mathrm{ns}})$  and concave in  $\pmb{v}$ , and the Gaussian matrix  $Z$  is independent of everything else. Therefore, the AO problem reads:
$$
\begin{array}{l} \mathcal {L} _ {A O} \left(\boldsymbol {\theta} _ {\mathrm {s}}, \boldsymbol {\theta} _ {\mathrm {n s}}\right) = \max  _ {\boldsymbol {v}} \frac {1}{n} \left(\boldsymbol {v} ^ {\top} \boldsymbol {\varepsilon} + \boldsymbol {v} ^ {\top} \boldsymbol {\Lambda} ^ {\top} \boldsymbol {M} _ {\mathrm {s}} ^ {\top} \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) \right. \\ + \left\| \boldsymbol {\theta} _ {0, \mathrm {s}} \right\| _ {\ell_ {2}} \boldsymbol {g} _ {\mathrm {s}} ^ {\top} \boldsymbol {v} + \left\| \boldsymbol {v} \right\| _ {\ell_ {2}} \boldsymbol {h} _ {\mathrm {s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {s}} \\ + \left\| \boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} \boldsymbol {g} _ {\mathrm {n s}} ^ {\top} \boldsymbol {v} + \left\| \boldsymbol {v} \right\| _ {\ell_ {2}} \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) \\ + \pmb {v} ^ {\mathsf {T}} \pmb {\Lambda} ^ {\mathsf {T}} \pmb {M} _ {\mathrm {n s}} ^ {\mathsf {T}} (\pmb {\theta} _ {0, \mathrm {n s}} - \pmb {\theta} _ {\mathrm {n s}}) - \frac {\| \pmb {v} \| _ {\ell_ {2}} ^ {2}}{2}) + \lambda \| \pmb {\theta} _ {\mathrm {s}} \| _ {\ell_ {2}} ^ {2} + \lambda \| \pmb {\theta} _ {\mathrm {n s}} \| _ {\ell_ {2}} ^ {2}, \\ \end{array}
$$
where  $g_{\mathrm{s}}, g_{\mathrm{ns}} \in \mathbb{R}^n$  and  $h_{\mathrm{s}} \in \mathbb{R}^{p}$ ,  $h_{\mathrm{ns}} \in \mathbb{R}^{d - p}$  are independent Gaussian random vectors with i.i.d.  $\mathrm{N}(0,1)$  entries.
We next fix norm of  $\| \pmb{v}\|_{\ell_2} = \beta$  , and maximize over its direction to obtain
$$
\begin{array}{l} \mathcal {L} _ {A O} \left(\boldsymbol {\theta} _ {\mathrm {s}}, \boldsymbol {\theta} _ {\mathrm {n s}}\right) = \max  _ {\beta \geq 0} \frac {1}{n} \Big (\beta \left\| \boldsymbol {\varepsilon} + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) + \left\| \boldsymbol {\theta} _ {0, \mathrm {s}} \right\| _ {\ell_ {2}} \boldsymbol {g} _ {\mathrm {s}} + \left\| \boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} \boldsymbol {g} _ {\mathrm {n s}} + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {n s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) \right\| _ {\ell_ {2}} \\ + \beta \boldsymbol {h} _ {\mathrm {s}} ^ {\mathsf {T}} \boldsymbol {\theta} _ {0, \mathrm {s}} + \beta \boldsymbol {h} _ {\mathrm {n s}} ^ {\mathsf {T}} (\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}) - \frac {\beta^ {2}}{2}) + \lambda \| \boldsymbol {\theta} _ {\mathrm {s}} \| _ {\ell_ {2}} ^ {2} + \lambda \| \boldsymbol {\theta} _ {\mathrm {n s}} \| _ {\ell_ {2}} ^ {2} \\ = \max  _ {\beta \geq 0} \frac {1}{n} \Big (\beta \left\| \boldsymbol {\varepsilon} + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {n s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) + \sqrt {\left\| \boldsymbol {\theta} _ {0 , \mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \left\| \boldsymbol {\theta} _ {0 , \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} ^ {2}} \boldsymbol {g} \right\| _ {\ell_ {2}} \\ + \beta \boldsymbol {h} _ {\mathrm {s}} ^ {\mathsf {T}} \boldsymbol {\theta} _ {0, \mathrm {s}} + \beta \boldsymbol {h} _ {\mathrm {n s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) - \frac {\beta^ {2}}{2}) + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} ^ {2}, \\ \end{array}
$$
where we used that  $g_{\mathrm{s}}, g_{\mathrm{ns}} \in \mathbb{R}^n$  have independent Gaussian entries. Here,  $g \in \mathbb{R}^n$  has i.i.d entries from  $\mathsf{N}(0,1)$ . Next, note that the above optimization over  $\beta$  has a closed form. Using the identity  $\max_{\beta \geq 0} (\beta x - \beta^2 / 2) = x_+^2 / 2$ , with  $x_+ = \max(x, 0)$ , we get
$$
\begin{array}{l} \mathcal {L} _ {A O} \big (\boldsymbol {\theta} _ {\mathrm {s}}, \boldsymbol {\theta} _ {\mathrm {n s}} \big) = \frac {1}{2 n} \Big (\left\| \boldsymbol {\varepsilon} + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {s}} ^ {\mathsf {T}} \big (\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}} \big) + \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {n s}} ^ {\mathsf {T}} \big (\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}} \big) + \sqrt {\left\| \boldsymbol {\theta} _ {0 , \mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \left\| \boldsymbol {\theta} _ {0 , \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} ^ {2}} \boldsymbol {g} \right\| _ {\ell_ {2}} \\ + \boldsymbol {h} _ {\mathrm {s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {s}} + \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) \bigg) _ {+} ^ {2} + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \lambda \left\| \boldsymbol {\theta} _ {\mathrm {n s}} \right\| _ {\ell_ {2}} ^ {2}. \tag {B.1} \\ \end{array}
$$
Scalarization of the auxiliary optimization (AO) problem. We next proceed to scalarize the AO problem. Consider the singular value decomposition
$$
\boldsymbol {M} _ {\mathrm {s}} = \boldsymbol {U} _ {\mathrm {s}} \boldsymbol {\Sigma} _ {\mathrm {s}} \boldsymbol {V} _ {\mathrm {s}} ^ {\top},
$$
with  $\pmb{U}_{\mathrm{s}} \in \mathbb{R}^{p \times r}$ ,  $\pmb{\Sigma}_{\mathrm{s}} \in \mathbb{R}^{r \times r}$ ,  $\pmb{V}_{\mathrm{s}} \in \mathbb{R}^{k \times r}$ , where  $r = \mathrm{rank}(M_{\mathrm{s}}) \leq k$ . Decompose  $\pmb{q}_{\mathrm{s}} := \pmb{\theta}_{0,\mathrm{s}} - \pmb{\theta}_{\mathrm{s}}$  in its projections onto the space spanned by the columns  $\pmb{u}_{1,\mathrm{s}}, \dots, \pmb{u}_{r,\mathrm{s}}$  of  $U_{\mathrm{s}}$ , and the orthogonal component:
$$
\boldsymbol {q} _ {\mathrm {s}} = \sum_ {i = 1} ^ {r} \alpha_ {i} \boldsymbol {u} _ {i, \mathrm {s}} + \alpha_ {0} \boldsymbol {q} _ {\mathrm {s}} ^ {\perp},
$$
where  $\| q_{\mathrm{s}}^{\perp}\|_{\ell_2} = 1$ ,  $\alpha_0 \geq 0$ , and  $\boldsymbol{U}_{\mathrm{s}}^{\mathsf{T}}\boldsymbol{q}_{\mathrm{s}}^{\perp} = \mathbf{0}$ . Using the shorthand  $\alpha = (\alpha_{1},\dots,\alpha_{r})$ , we write
$$
\boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) = \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {V} _ {\mathrm {s}} \boldsymbol {\Sigma} _ {\mathrm {s}} \boldsymbol {U} _ {\mathrm {s}} ^ {\mathsf {T}} \boldsymbol {q} _ {\mathrm {s}} = \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {V} _ {\mathrm {s}} \boldsymbol {\Sigma} _ {\mathrm {s}} \boldsymbol {\alpha}.
$$
$$
\begin{array}{l} \left\| \boldsymbol {\theta} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} = \left\| \boldsymbol {\theta} _ {0, \mathrm {s}} - \left(\boldsymbol {\theta} _ {0, \mathrm {s}} - \boldsymbol {\theta} _ {\mathrm {s}}\right) \right\| _ {\ell_ {2}} ^ {2} \\ = \left\| \boldsymbol {\theta} _ {0, \mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \left\| \boldsymbol {q} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} - 2 \left\langle \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {q} _ {\mathrm {s}} \right\rangle \\ = \left\| \boldsymbol {\theta} _ {0, \mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \left\| \mathbf {q} _ {\mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} - 2 \left\langle \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {U} _ {\mathrm {s}} \boldsymbol {\alpha} \right\rangle - 2 \alpha_ {0} \left\langle \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {q} _ {\mathrm {s}} ^ {\perp} \right\rangle \\ = \left\| \boldsymbol {\theta} _ {0, s} \right\| _ {\ell_ {2}} ^ {2} + \left\| \boldsymbol {q} _ {s} \right\| _ {\ell_ {2}} ^ {2} - 2 \left\langle \boldsymbol {U} _ {s} ^ {\top} \boldsymbol {\theta} _ {0, s}, \boldsymbol {\alpha} \right\rangle - 2 \alpha_ {0} \left\langle \boldsymbol {\theta} _ {0, s}, \boldsymbol {q} _ {s} ^ {\perp} \right\rangle \\ = \left\| \boldsymbol {\theta} _ {0, \mathrm {s}} \right\| _ {\ell_ {2}} ^ {2} + \left(\alpha_ {0} ^ {2} + \left\| \boldsymbol {\alpha} \right\| _ {\ell_ {2}} ^ {2}\right) - 2 \left\langle \boldsymbol {U} _ {\mathrm {s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {\alpha} \right\rangle - 2 \alpha_ {0} \left\langle \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {q} _ {\mathrm {s}} ^ {\perp} \right\rangle . \tag {B.2} \\ \end{array}
$$
Similarly, we define  $\pmb{q}_{\mathrm{ns}} = \pmb{\theta}_{0,\mathrm{ns}} - \pmb{\theta}_{\mathrm{ns}}$  and consider the singular value decomposition
$$
\boldsymbol {M} _ {\mathrm {n s}} = \boldsymbol {U} _ {\mathrm {n s}} \boldsymbol {\Sigma} _ {\mathrm {n s}} \boldsymbol {V} _ {\mathrm {n s}} ^ {\mathsf {T}},
$$
with  $U_{\mathrm{ns}} \in \mathbb{R}^{(d - p) \times t}$ ,  $\Sigma_{\mathrm{ns}} \in \mathbb{R}^{t \times t}$ ,  $V_{\mathrm{ns}} \in \mathbb{R}^{k \times t}$ , where  $t = \operatorname{rank}(M_{\mathrm{ns}}) \leq k$ . Decomposing  $q_{\mathrm{ns}}$  in its projections on the orthogonal columns  $u_{1,\mathrm{ns}}, \ldots, u_{r,\mathrm{ns}}$  of  $U_{\mathrm{ns}}$ , and the orthogonal component we write
$$
\boldsymbol {q} _ {\mathrm {n s}} = \sum_ {i = 1} ^ {t} \gamma_ {i} \boldsymbol {u} _ {i, \mathrm {n s}} + \gamma_ {0} \boldsymbol {q} _ {\mathrm {n s}} ^ {\perp},
$$
with  $\| q_{\mathrm{ns}}^{\perp}\|_{\ell_2} = 1, \gamma_0 \geq 0$ , and  $U_{\mathrm{ns}}^{\mathsf{T}}q_{\mathrm{ns}}^{\perp} = 0$ . Define  $\gamma = (\gamma_1, \dots, \gamma_t)$ . In this notation, we have
$$
\boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {M} _ {\mathrm {n s}} ^ {\mathsf {T}} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) = \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {V} _ {\mathrm {n s}} \boldsymbol {\Sigma} _ {\mathrm {n s}} \boldsymbol {U} _ {\mathrm {n s}} ^ {\mathsf {T}} \boldsymbol {q} _ {\mathrm {n s}} = \boldsymbol {\Lambda} ^ {\mathsf {T}} \boldsymbol {V} _ {\mathrm {n s}} \boldsymbol {\Sigma} _ {\mathrm {n s}} \boldsymbol {\gamma}.
$$
Also,  $\| \pmb{\theta}_{0,\mathrm{ns}} - \pmb{\theta}_{\mathrm{ns}}\|_{\ell_2} = \| \pmb{q}_{\mathrm{ns}}\|_{\ell_2} = \sqrt{\gamma_0^2 + \|\pmb{\gamma}\|_{\ell_2}^2}$ . In addition,
$$
\boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \left(\boldsymbol {\theta} _ {0, \mathrm {n s}} - \boldsymbol {\theta} _ {\mathrm {n s}}\right) = \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \boldsymbol {q} _ {\mathrm {n s}} = \sum_ {i = 1} ^ {t} \gamma_ {i} \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \boldsymbol {u} _ {i, \mathrm {n s}} + \gamma_ {0} \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \boldsymbol {q} _ {\mathrm {n s}} ^ {\perp}.
$$
Using the above identities in (B.1), we have
$$
\begin{array}{l} \mathcal {L} _ {A O} (\pmb {\theta} _ {\mathrm {s}}, \pmb {\theta} _ {\mathrm {n s}}) = \frac {1}{2 n} \Bigg (\left\| \pmb {\varepsilon} + \pmb {\Lambda} ^ {\mathsf {T}} \pmb {V} _ {\mathrm {s}} \pmb {\Sigma} _ {\mathrm {s}} \pmb {\alpha} + \pmb {\Lambda} ^ {\mathsf {T}} \pmb {V} _ {\mathrm {n s}} \pmb {\Sigma} _ {\mathrm {n s}} \pmb {\gamma} + \sqrt {\| \pmb {\theta} _ {0 , \mathrm {s}} \| _ {\ell_ {2}} ^ {2} + \gamma_ {0} ^ {2} + \| \pmb {\gamma} \| _ {\ell_ {2}} ^ {2}} \pmb {g} \right\| _ {\ell_ {2}} \\ + \boldsymbol {h} _ {\mathrm {s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {s}} + \sum_ {i = 1} ^ {t} \gamma_ {i} \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \boldsymbol {u} _ {i, \mathrm {n s}} + \gamma_ {0} \boldsymbol {h} _ {\mathrm {n s}} ^ {\top} \boldsymbol {q} _ {\mathrm {n s}} ^ {\perp}) _ {+} ^ {2} \\ + \lambda \| \boldsymbol {\theta} _ {0, \mathrm {s}} \| _ {\ell_ {2}} ^ {2} + \lambda \left(\alpha_ {0} ^ {2} + \| \boldsymbol {\alpha} \| _ {\ell_ {2}} ^ {2}\right) - 2 \lambda \left\langle \boldsymbol {U} _ {\mathrm {s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {\alpha} \right\rangle - 2 \lambda \alpha_ {0} \left\langle \boldsymbol {\theta} _ {0, \mathrm {s}}, \boldsymbol {q} _ {\mathrm {s}} ^ {\perp} \right\rangle \\ + \lambda \| \boldsymbol {\theta} _ {0, \mathrm {n s}} \| _ {\ell_ {2}} ^ {2} + \lambda \left(\gamma_ {0} ^ {2} + \| \boldsymbol {\gamma} \| _ {\ell_ {2}} ^ {2}\right) - 2 \lambda \left\langle \boldsymbol {U} _ {\mathrm {n s}} ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {n s}}, \boldsymbol {\gamma} \right\rangle - 2 \lambda \gamma_ {0} \left\langle \boldsymbol {\theta} _ {0, \mathrm {n s}}, \boldsymbol {q} _ {\mathrm {n s}} ^ {\perp} \right\rangle . \tag {B.3} \\ \end{array}
$$
By the above characterization, minimization over  $\theta_{\mathrm{s}}$  and  $\theta_{\mathrm{ns}}$  reduces to minimization over  $\alpha_0,\gamma_0,\alpha$ $\gamma ,q_{\mathrm{s}}^{\perp}$  and  $q_{\mathrm{ns}}^{\perp}$ . Further, these variables are free from each other and can be optimized over separately. For  $q_{\mathrm{s}}^{\perp}$ , there is only one term involving this variable and therefore, minimization over it reduces to
$$
\min _ {\pmb {q} _ {\mathrm {s}} ^ {\perp}, \| \pmb {q} _ {\mathrm {s}} ^ {\perp} \| _ {\ell_ {2}} = 1} - \langle \pmb {\theta} _ {0, \mathrm {s}}, \pmb {q} _ {\mathrm {s}} ^ {\perp} \rangle = \min _ {\pmb {q} _ {\mathrm {s}} ^ {\perp}, \| \pmb {q} _ {\mathrm {s}} ^ {\perp} \| _ {\ell_ {2}} = 1} - \langle \pmb {U} _ {\mathrm {s}} ^ {\perp} (\pmb {U} _ {\mathrm {s}} ^ {\perp}) ^ {\top} \pmb {\theta} _ {0, \mathrm {s}}, \pmb {q} _ {\mathrm {s}} ^ {\perp} \rangle = - \left\| (\pmb {U} _ {\mathrm {s}} ^ {\perp}) ^ {\top} \pmb {\theta} _ {0, \mathrm {s}} \right\| _ {\ell_ {2}}.
$$
For  $q_{\mathrm{ns}}^{\perp}$ , we note that there are two terms involving this variable, namely  $\langle \frac{h_{\mathrm{ns}}}{\sqrt{n}}, q_{\mathrm{ns}}^{\perp} \rangle$  and  $\langle (U_{\mathrm{ns}}^{\perp})^{\mathsf{T}} \theta_{0,\mathrm{ns}}, q_{\mathrm{ns}}^{\perp} \rangle$ . Since  $\| q_{\mathrm{ns}}^{\perp} \|_{\ell_2} = 1$ , it is easy to see that the optimal  $q_{\mathrm{ns}}^{\perp}$  should be in the span of  $h_{\mathrm{ns}}^{\perp}$  and  $(U_{\mathrm{ns}}^{\perp})^{\mathsf{T}} \theta_{0,\mathrm{ns}}$ . In addition,
$$
\langle \frac {h _ {\mathrm {n s}} ^ {\perp}}{\sqrt {n}}, (U _ {\mathrm {n s}} ^ {\perp}) ^ {\top} \boldsymbol {\theta} _ {0, \mathrm {n s}} \rangle \stackrel {(p)} {\rightarrow} 0,
$$
by the law of large numbers. In words, these two vectors are asymptotically orthogonal. Hence, we can consider the following decomposition of the optimal  $\pmb{q}_{\mathrm{ns}}^{\perp}$ :
$$
\boldsymbol {q} _ {\mathrm {n s}} ^ {\perp} = - \xi \frac {\boldsymbol {h} _ {\mathrm {n s}} ^ {\perp}}{\| \boldsymbol {h} _ {\mathrm {n s}} ^ {\perp} \| _ {\ell_ {2}}} + \sqrt {1 - \xi^ {2}} \frac {\boldsymbol {U} _ {\mathrm {n s}} ^ {\perp} (\boldsymbol {U} _ {\mathrm {n s}} ^ {\perp}) ^ {\mathsf {T}} \boldsymbol {\theta} _ {0 , \mathrm {n s}}}{\| (\boldsymbol {U} _ {\mathrm {n s}} ^ {\perp}) ^ {\mathsf {T}} \boldsymbol {\theta} _ {0 , \mathrm {n s}} \| _ {\ell_ {2}}},
$$
where  $\xi \geq 0$  and  $h_{\mathrm{ns}}^{\perp}$  denotes the projection of  $h_\mathrm{ns}$  onto the (left) null space of  $U_{\mathrm{ns}}$ . This brings us to