SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\boldsymbol {Z} - \kappa \nabla_ {\boldsymbol {Z}} R ^ {c} (\boldsymbol {Z} \mid \boldsymbol {U} _ {[ K ]}) \approx (1 - \kappa \gamma) \boldsymbol {Z} + \kappa \gamma^ {2} \left[ \boldsymbol {U} _ {1}, \dots , \boldsymbol {U} _ {K} \right] \left[ \begin{array}{c} \operatorname {S S A} (\boldsymbol {Z} | \boldsymbol {U} _ {1}) \\ \vdots \\ \operatorname {S S A} (\boldsymbol {Z} | \boldsymbol {U} _ {K}) \end{array} \right]. \tag {76}
$$
A.3 Companion to Section 2.4
We again wish to re-iterate the core contribution of our approach in Section 2.4.
- Within the framework of alternating minimization of the sparse rate reduction objective, we show that the second alternating step — gradient descent on the overall coding rate plus a sparse regularization term — has heuristic connections to a particular LASSO optimization.
- We show that the unrolling of the proximal gradient step to solve this LASSO optimization resembles the MLP which immediately follows the self-attention layer within transformer blocks.
In the main text, our connection between the second step of the alternating minimization and the LASSO optimization was high-level and heuristic. In some sense, the choice to pose the minimization step as a LASSO was a simple, reliable, and interpretable choice which works well in practice, but is nonetheless not backed up by rigorous theoretical justification. In the following subsection, we provide a mathematical justification for a reformulation of the minimization step using a majorization-minimization framework. We further show that the associated unrolled optimization step bears a strong resemblance to the ISTA step. This confirms our earlier discussion — we took the simplest possible choice in designing CRATE, but by more rigorous derivation we can uncover alternative operators which nonetheless have the same conceptual function and may perform better in practice.
Assumptions. In this section, we present a rigorous optimization analysis of an incremental minimization approach to the objective (13). We will show that under two simplifying assumptions, namely
1. The columns of  $Z^{\ell + 1/2}$  are normalized, in the sense that  $\mathrm{diag}((Z^{\ell + 1/2})^* Z^{\ell + 1/2}) = 1$ ;<sup>10</sup>
2. We have  $d \geq N,^{11}$  and the columns of  $Z^{\ell + 1/2}$  are orthogonal, so that  $(Z^{\ell + 1/2})^* Z^{\ell + 1/2} = I$ .<sup>12</sup>
the approach leads to an update iteration that is equal to a slightly simplified version of the ISTA block (17). We see this as a justification for our derivation in Section 2.4, which obtained the ISTA block by introducing an additional simplifying assumption on the distribution of the data at layer  $\ell$ .
Analysis. Following (16), we will consider the natural relaxation of the  $\ell_0$  "norm" to the  $\ell^1$  norm, and incorporate a nonnegativity constraint. Consider the objective
$$
\varphi (\boldsymbol {Z}) = \lambda \| \boldsymbol {Z} \| _ {1} + \chi_ {\{\boldsymbol {Z} \geq \mathbf {0} \}} (\boldsymbol {Z}) - \underbrace {\frac {1}{2} \log \det  \left(\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}\right)} _ {R (\boldsymbol {Z})}, \tag {77}
$$
where  $\mathbf{Z} \in \mathbb{R}^{d \times N}$  and  $\alpha = d / N\varepsilon^2$ , and  $\chi_{\{\mathbf{Z} \geq 0\}}$  denotes the characteristic function for the set of elementwise-nonnegative matrices  $\mathbf{Z}$ . As in Appendix A.2, we calculate
$$
\nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z}) = \alpha \boldsymbol {Z} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}\right) ^ {- 1}. \tag {78}
$$
This is a natural assumption in transformer-type architectures such as CRATE due to the use of LayerNorm blocks—although these blocks (indeed, as we use them in CRATE) include trainable mean and scale offsets as well as an additional mean subtraction operation [64], they are initialized to have zero mean and unit norm, hence this assumption corresponds to an analysis of the network at its initialization.
This assumption is without loss of generality, as we will see in the analysis below. The reason is that  $Z^{*}Z$  and  $Z^{*}Z$  have the same nonzero eigenvalues regardless of the shape of  $Z$ , which implies that  $\log \det (I + \alpha Z^{*}Z) = \log \det (I + \alpha ZZ^{*})$ . In particular, interpreting the norms appropriately (with a slight abuse of notation), we have  $\varphi (Z) = \varphi (Z^{*})$ , so for the purposes of analysis we can always proceed as though  $Z$  is a tall matrix (as long as we do not use any special properties of  $\alpha$  in our derivation).
<sup>12</sup>This assumption is strictly stronger than the previous one, and strictly stronger than an assumption of incoherence on the columns. It corresponds to the representation  $Z^{\ell + 1/2}$  being non-collapsed, which we expect to hold at initialization due to the projections  $U_{[K]}$  being random.
We consider an incremental optimization scheme for the highly nonlinear and nonconvex objective  $\varphi$ . Following Section 2.3, we optimize locally at a "post-compression" iterate  $Z^{\ell +1 / 2}$ . We follow the standard proximal majorize-minimize framework [70] for incremental/local optimization: this begins with the second-order Taylor expansion for the smooth part of  $\varphi$  in a neighborhood of the current iterate  $Z^{\ell +1 / 2}$ :
$$
\begin{array}{l} R (\boldsymbol {Z}) = R (\boldsymbol {Z} ^ {\ell + 1 / 2}) + \left\langle \nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z} ^ {\ell + 1 / 2}), \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2} \right\rangle \\ + \int_ {0} ^ {1} (1 - t) \left\langle \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2}, \nabla^ {2} R (\boldsymbol {Z} _ {t}) \left(\boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2}\right) \right\rangle \mathrm {d} t, \tag {79} \\ \end{array}
$$
where for any  $\mathbf{Z} \in \mathbb{R}^{d \times N}$ ,  $\mathbf{Z}_t = t\mathbf{Z}^{\ell + 1/2} + (1 - t)\mathbf{Z}$ . The proximal majorization-minimization approach alternates two steps to minimize  $\varphi$ :
1. First, use assumptions on  $Z^{\ell + 1/2}$  to derive an upper bound on the operator norm of the Hessian  $\nabla^2 R(Z)$  over the effective domain of the optimization problem. We will write  $L$  for this (uniform) upper bound. This yields a quadratic upper bound for the smooth part of the objective  $\varphi$ .
2. Then, alternately minimize the smooth part of the quadratic upper bound as a function of  $Z$ , and take a proximal step on the nonsmooth part. It can be shown [70] that corresponds to the iteration
$$
\boldsymbol {Z} ^ {+} = \operatorname {p r o x} _ {\frac {\lambda}{L} \left(\| \cdot \| _ {1} + \chi_ {\{\boldsymbol {z} \geq 0 \}}\right)} \left(\boldsymbol {Z} + \frac {1}{L} \nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z})\right) \tag {80}
$$
In the alternating minimization setting of this paper for optimizing (1), we only take one such step, starting at  $Z^{\ell + 1/2}$ .
We will instantiate this program below, showing quantitative error bounds related to our assumptions above as necessary. Rather than directly applying the iteration (80), we will derive it below under our aforementioned assumptions.
Starting at (79), our first task is to upper bound the quadratic residual. This corresponds to estimating
$$
\begin{array}{l} \left\langle \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2}, \nabla^ {2} R \left(\boldsymbol {Z} _ {t}\right) \left(\boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2}\right) \right\rangle (81) \\ \leq \sup  _ {t \in [ 0, 1 ]} \left\| \nabla^ {2} R \left(\boldsymbol {Z} _ {t}\right)\right\| _ {\ell^ {2} \rightarrow \ell^ {2}} \left\| \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2} \right\| _ {\mathrm {F}} ^ {2} (82) \\ \end{array}
$$
with Cauchy-Schwarz. Using Lemma 5, we can estimate the operator norm term in the previous bound in terms of properties of  $Z^{\ell +1 / 2}$ . We need to bound
$$
\left. \alpha \sup  _ {\| \boldsymbol {\Delta} \| _ {F} \leq 1} \left\| (\boldsymbol {\Delta} - \alpha \boldsymbol {Z} _ {t} (\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}) ^ {- 1} (\boldsymbol {Z} _ {t} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \boldsymbol {Z} _ {t})) (\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}) ^ {- 1} \right\| _ {F}, \right. \tag {83}
$$
and Lemma 6 gives that this term is no larger than  $9\alpha /4$  for any  $\pmb{Z}$  and any  $t$ . With this estimate and (79), we have a quadratic upper bound for  $-R(Z)$ :
$$
- R (\boldsymbol {Z}) \leq - R \left(\boldsymbol {Z} ^ {\ell + 1 / 2}\right) + \left\langle - \nabla_ {\boldsymbol {Z}} R \left(\boldsymbol {Z} ^ {\ell + 1 / 2}\right), \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2} \right\rangle + \frac {9 \alpha}{8} \left\| \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2} \right\| _ {\mathrm {F}} ^ {2}. \tag {84}
$$
Meanwhile, by our assumptions above, we have
$$
- \nabla_ {\boldsymbol {Z}} R \left(\boldsymbol {Z} ^ {\ell + 1 / 2}\right) = - \alpha \boldsymbol {Z} ^ {\ell + 1 / 2} (\boldsymbol {I} + \alpha \boldsymbol {I}) ^ {- 1} = - \frac {\alpha}{1 + \alpha} \boldsymbol {Z} ^ {\ell + 1 / 2}. \tag {85}
$$
We now minimize the preceding quadratic upper bound as a function of  $Z$ . Differentiating, the minimizer  $Z_{\mathrm{opt}}$  is calculated as
$$
\boldsymbol {Z} _ {\text {o p t}} = \left(1 + \frac {4}{9 (1 + \alpha)}\right) \boldsymbol {Z} ^ {\ell + 1 / 2}, \tag {86}
$$
and it is well-known that the proximal operator of the sum of  $\chi_{\{\mathbf{Z}\geq \mathbf{0}\}}$  and  $\lambda \| \cdot \| _1$  is simply the one-sided soft-thresholding operator [70]
$$
\operatorname {p r o x} _ {\chi \left\{\mathbf {Z} \geq \mathbf {0} \right\}} + \lambda \| \cdot \| _ {1} (\mathbf {Z}) = \max  \left\{\mathbf {Z} - \lambda \mathbf {1}, \mathbf {0} \right\}, \tag {87}
$$
where the maximum is applied elementwise. As in Section 2.4, we may write this elementwise maximum simply as ReLU. Thus, one step of proximal majorization-minimization under our simplifying assumptions takes the form
$$
\boldsymbol {Z} ^ {\ell + 1} = \operatorname {R e L U} \left(\left(1 + \frac {4}{9 (1 + \alpha)}\right) \boldsymbol {Z} ^ {\ell + 1 / 2} - \frac {4 \lambda}{9 \alpha} \mathbf {1}\right). \tag {88}
$$
Finally, we point out one additional elaboration which introduces the dictionary  $D$  that appears in the ISTA block in Section 2.4. Notice that for any orthogonal  $D$ , one has  $R(DZ) = R(Z)$  for every  $Z$ . This symmetry implies equivariance properties of  $\nabla_Z R(Z)$  and  $\nabla_Z^2 R(Z)$ : for every  $Z$  and every  $\Delta$  and every orthogonal  $D$ ,
$$
\boldsymbol {D} \nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z}) = \nabla_ {\boldsymbol {Z}} R (\boldsymbol {D} \boldsymbol {Z}), \tag {89}
$$
$$
\langle \boldsymbol {D} \boldsymbol {\Delta}, \nabla_ {\boldsymbol {Z}} ^ {2} R (\boldsymbol {Z}) (\boldsymbol {D} \boldsymbol {\Delta}) \rangle = \langle \boldsymbol {\Delta}, \nabla_ {\boldsymbol {Z}} ^ {2} R (\boldsymbol {D} \boldsymbol {Z}) (\boldsymbol {\Delta}) \rangle . \tag {90}
$$
Hence the quadratic Taylor expansion (79) can be written equivalently as
$$
\begin{array}{l} R (\boldsymbol {Z}) = R \left(\boldsymbol {D} ^ {*} \boldsymbol {Z} ^ {\ell + 1 / 2}\right) + \left\langle \nabla_ {\boldsymbol {Z}} R \left(\boldsymbol {D} ^ {*} \boldsymbol {Z} ^ {\ell + 1 / 2}\right), \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2} \right\rangle \\ + \int_ {0} ^ {1} (1 - t) \left\langle \boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2}, \nabla^ {2} R \left(\boldsymbol {D} ^ {*} \boldsymbol {Z} _ {t}\right) \left(\boldsymbol {Z} - \boldsymbol {Z} ^ {\ell + 1 / 2}\right) \right\rangle \mathrm {d} t, \tag {91} \\ \end{array}
$$
for any orthogonal  $D$ . The significance of this is that we have obtained an expression equivalent to (79), but with  $Z^{\ell +1 / 2}$  replaced by  $D^{*}Z^{\ell +1 / 2}$ ; moreover, because our approximation arguments above are not affected by left-multiplication of  $Z^{\ell +1 / 2}$  by an orthogonal matrix (this operation does not change the norms of the columns of  $Z^{\ell +1 / 2}$ , or their correlations, and hence the matrix's incoherence), we can apply exactly the same line of reasoning above to obtain that an equivalent proximal majorization-minimization iteration is given by
$$
\boldsymbol {Z} ^ {\ell + 1} = \operatorname {R e L U} \left(\left(1 + \frac {4}{9 (1 + \alpha)}\right) \boldsymbol {D} ^ {*} \boldsymbol {Z} ^ {\ell + 1 / 2} - \frac {4 \lambda}{9 \alpha} \mathbf {1}\right), \tag {92}
$$
for any orthogonal dictionary  $D$ . This gives an update quite similar to the ISTA block (17) in the case where the dictionary used in Section 2.4 is orthogonal, but without a skip connection.
We thus obtain a natural white-box version of this part of the architecture, along with the natural interpretation that its purpose is to sparsify the compressed tokens  $Z^{\ell +1 / 2}$  in a (learnable) dictionary, which accords with recent empirical studies [79].
Other architectures? As we mentioned at the start of this section, the preceding derivation is performed in the most elementary possible setting in order to demonstrate the majorization-minimization approach for layer design. More precise approximations or assumptions may lead to superior layer designs that better optimize the target objective (1) (and in particular (13)). We mention two here:
1. Beyond exactly-incoherent features: our derivations above assumed that the incoming representations  $Z^{\ell + 1/2}$  were already maximal for the expansion term  $R$  in (13). It is desirable to obtain a 'perturbative' derivation, which applies in cases where  $Z^{\ell + 1/2}$  is not fully orthogonal, but instead near-orthogonal, in particular incoherent [70]. The derivations above can be adapted to this setting; the perturbation bounds become slightly more delicate, and the ultimate layer (92) changes to involve additional normalization.
2. Beyond orthogonal dictionaries: The symmetries of the expansion term  $R$  in (13) may be followed to lead to a pair of dictionaries  $D$  and  $D'$  and an objective that sparsifies  $DZD'$ . This type of transformation is suggestive of popular architectures that mix over tokens [54, 67], however we consider the simpler form  $DZ$  in this work. In addition, we have focused for simplicity on orthogonal dictionaries  $D$ ; as in the previous bullet, one may consider in a similar way dictionaries  $D$  which are complete and near-orthogonal. Adapting the derivation to overcomplete dictionaries is an interesting future direction that we expect to improve the scalability of CRATE; one avenue to achieve this could be increasing the number of projections  $U_{[K]}$  and their embedding dimensions.
A.3.1 Auxiliary Lemmas
Lemma 5. Consider the function
$$
R (\boldsymbol {Z}) = \frac {1}{2} \log \det  \left(\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}\right), \tag {93}
$$
where  $\alpha > 0$  is a constant. Then we have
$$
\nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z}) = \alpha \boldsymbol {Z} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}\right) ^ {- 1}, \tag {94}
$$
and the Hessian operator  $\nabla_{\mathbf{Z}}^{2}R(\mathbf{Z})\colon \mathbb{R}^{d\times N}\to \mathbb{R}^{d\times N}$  satisfies that for any  $\pmb {\Delta}\in \mathbb{R}^{d\times N}$
$$
\begin{array}{l} \nabla_ {\mathbf {Z}} ^ {2} R (\mathbf {Z}) (\Delta) (95) \\ = \alpha \boldsymbol {\Delta} (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} - \alpha^ {2} \boldsymbol {Z} (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} (\boldsymbol {Z} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \boldsymbol {Z}) (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1}. (96) \\ \end{array}
$$
Proof. The gradient calculation follows from [46], for example. For the Hessian, we use the usual approach to calculating derivatives: if  $\pmb{\Delta}$  is any matrix with the same shape as  $\mathbf{Z}$  and  $t > 0$ ,
$$
\nabla_ {\boldsymbol {Z}} ^ {2} R (\boldsymbol {Z}) (\boldsymbol {\Delta}) = \left. \frac {\partial}{\partial t} \right| _ {t = 0} [ t \mapsto \nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z} + t \boldsymbol {\Delta}) ], \tag {97}
$$
valid since  $R$  is smooth. We have
$$
\begin{array}{l} \nabla_ {\boldsymbol {Z}} R (\boldsymbol {Z} + t \boldsymbol {\Delta}) \\ = \alpha (\boldsymbol {Z} + t \Delta) (\boldsymbol {I} + \alpha (\boldsymbol {Z} + t \Delta) ^ {*} (\boldsymbol {Z} + t \Delta)) ^ {- 1} \\ = \alpha (Z + t \Delta) (I + \alpha Z ^ {*} Z + \alpha t [ Z ^ {*} \Delta + \Delta^ {*} Z + t \Delta^ {*} \Delta ]) ^ {- 1} \\ = \alpha (\boldsymbol {Z} + t \Delta) \left(\boldsymbol {I} + \alpha t (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} [ \boldsymbol {Z} ^ {*} \Delta + \Delta^ {*} \boldsymbol {Z} + t \Delta^ {*} \Delta ]\right) ^ {- 1} (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} \\ = \alpha (\boldsymbol {Z} + t \Delta) \left(\sum_ {k = 0} ^ {\infty} (- \alpha t) ^ {k} \left((\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} [ \boldsymbol {Z} ^ {*} \Delta + \Delta^ {*} \boldsymbol {Z} + t \Delta^ {*} \Delta ]\right) ^ {k}\right) (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1}, \\ \end{array}
$$
where in the fourth line we require that  $t$  is sufficiently close to 0 in order to invoke the Neumann series. First, notice that the term involving  $\Delta^{*}\Delta$  does not play a role in the final expression: after we differentiate with respect to  $t$  and take a limit  $t\to 0$ , terms arising due to differentiation of  $t\mapsto t\Delta^{*}\Delta$  go to zero, because whenever the summation index  $k > 0$  we have a term  $(-\alpha t)^k$  that goes to zero as  $t\rightarrow 0$ . We thus obtain with the product rule
$$
\begin{array}{l} \left. \frac {\partial}{\partial t} \right| _ {t = 0} [ t \mapsto \nabla_ {\mathbf {Z}} R (\mathbf {Z} + t \Delta) ] (98) \\ = \alpha \boldsymbol {\Delta} (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} - \alpha^ {2} \boldsymbol {Z} (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1} (\boldsymbol {Z} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \boldsymbol {Z}) (\boldsymbol {I} + \alpha \boldsymbol {Z} ^ {*} \boldsymbol {Z}) ^ {- 1}. (99) \\ \end{array}
$$
Lemma 6. One has
$$
\sup  _ {\| \boldsymbol {\Delta} \| _ {F} \leq 1} \left\| \left(\boldsymbol {\Delta} - \alpha \boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \left(\boldsymbol {Z} _ {t} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \boldsymbol {Z} _ {t}\right)\right) \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {F} \leq \frac {9}{4}. \tag {100}
$$
Proof. Fix  $\pmb{\Delta}$  satisfying  $\| \pmb {\Delta}\|_{\mathrm{F}}\leq 1$  . By the triangle inequality,
$$
\begin{array}{l} \left\| \left(\boldsymbol {\Delta} - \alpha \mathbf {Z} _ {t} \left(\boldsymbol {I} + \alpha \mathbf {Z} _ {t} ^ {*} \mathbf {Z} _ {t}\right) ^ {- 1} \left(\mathbf {Z} _ {t} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \mathbf {Z} _ {t}\right)\right) \left(\boldsymbol {I} + \alpha \mathbf {Z} _ {t} ^ {*} \mathbf {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} (101) \\ \leq \left\| \boldsymbol {\Delta} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} + \alpha \left\| \boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \left(\boldsymbol {Z} _ {t} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \boldsymbol {Z} _ {t}\right) \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}}. (102) \\ \end{array}
$$
For the first term, we note that
$$
\left\| \boldsymbol {\Delta} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} = \left\| \left(\left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \otimes \boldsymbol {I}\right) \operatorname {v e c} (\boldsymbol {\Delta}) \right\| _ {\mathrm {F}}, \tag {103}
$$
and since  $(\pmb{I} + \alpha \pmb{Z}_t^*\pmb{Z}_t)^{-1}\preceq \pmb{I}$ , we obtain from Cauchy-Schwarz<sup>13</sup>
$$
\left\| \boldsymbol {\Delta} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} \leq \| \boldsymbol {\Delta} \| _ {\mathrm {F}}. \tag {104}
$$
<sup>13</sup>Recall that the eigenvalues of a Kronecker product of symmetric matrices are the tensor product of the eigenvalues (with multiplicity).
We can use a similar idea to control the second term. We have from the triangle inequality
$$
\begin{array}{l} \left\| \boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \left(\boldsymbol {Z} _ {t} ^ {*} \boldsymbol {\Delta} + \boldsymbol {\Delta} ^ {*} \boldsymbol {Z} _ {t}\right) \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} (105) \\ \leq \left\| \boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {\Delta} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} (106) \\ + \left\| \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*} \Delta \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*} \right\| _ {\mathrm {F}}. (107) \\ \end{array}
$$
$$
\begin{array}{l} \left\| \boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*} \Delta \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \right\| _ {\mathrm {F}} (108) \\ = \left\| \left(\left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \otimes \boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*}\right) \operatorname {v e c} (\boldsymbol {\Delta}) \right\| _ {\mathrm {F}} (109) \\ \leq \sigma_ {\max } \left(\left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1}\right) \sigma_ {\max } \left(\boldsymbol {Z} _ {t} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*}\right) \| \boldsymbol {\Delta} \| _ {\mathrm {F}} (110) \\ \leq \frac {1}{\alpha} \| \boldsymbol {\Delta} \| _ {\mathrm {F}}. (111) \\ \end{array}
$$
The last estimate follows from a computation using the SVD of  $Z_{t}$ . Meanwhile, we have for the second term by a similar argument (using the fact that the singular values of  $A$  and  $A^{*}$  are identical for any matrix  $A$ )
$$
\begin{array}{l} \left\| \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {\Delta} \left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*} \right\| _ {\mathrm {F}} \leq \sigma_ {\max } \left(\left(\boldsymbol {I} + \alpha \boldsymbol {Z} _ {t} ^ {*} \boldsymbol {Z} _ {t}\right) ^ {- 1} \boldsymbol {Z} _ {t} ^ {*}\right) ^ {2} \| \boldsymbol {\Delta} \| _ {\mathrm {F}} (112) \\ \leq \frac {1}{4 \alpha} \| \boldsymbol {\Delta} \| _ {\mathrm {F}}, (113) \\ \end{array}
$$
where once again the estimate follows from a computation involving the SVD of  $Z_{t}$  (together with the fact that the function  $\sigma \mapsto \sigma / (1 + \alpha \sigma^{2})$  is bounded on  $\sigma \geq 0$  by  $1 / (2\sqrt{\alpha})$ ). Putting it together, we have obtained