SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Khashayar Gatmiry
gatmiry@mit.edu
Ching-Yao Chuang
cychuang@mit.edu
Sashank Reddi
sashank@google.com
TU Munich & MIT
Recent works on over-parameterized neural networks have shown that the stochasticity in optimizers has the implicit regularization effect of minimizing the sharpness of the loss function (in particular, the trace of its Hessian) over the family zero-loss solutions. More explicit forms of flatness regularization also empirically improve the generalization performance. However, it remains unclear why and when flatness regularization leads to better generalization. This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in an important setting: learning deep linear networks from linear measurements, also known as deep matrix factorization. We show that for all depth greater than one, with the standard Restricted Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters (i.e., the product of all layer matrices), which in turn leads to better generalization. We empirically verify our theoretical findings on synthetic datasets.
Modern deep neural networks are typically over-parametrized and equipped with huge model capacity, but surprisingly, they generalize well when trained using stochastic gradient descent (SGD) or its variants [51]. A recent line of research suggested the implicit bias of SGD as a possible explanation to this mysterious ability. In particular, Damian et al. [10], Li et al. [29], Arora et al. [4], Lyu et al. [33], Wen et al. [48], Liu et al. [31] have shown that SGD can implicitly minimize the sharpness of the training loss, in particular, the trace of the Hessian of the training loss, to obtain the final model. However, despite the strong empirical evidence on the correlation between various notions of sharpness and generalization [25, 23, 38, 24] and the effectiveness of using sharpness regularization on improving generalization [16, 49, 53, 39], the connection between penalization of the sharpness of training loss and better generalization still remains majorly unclear [13, 2] and has only been proved in the context of two-layer linear models [29, 37, 12]. To further understand this connection beyond the two layer case, we study the inductive bias of penalizing the trace of the Hessian of training loss and its effect on the generalization in an important theoretical deep learning setting: deep linear networks (or equivalently, deep matrix factorization [3]). We start by briefly describing the problem setup.
Deep Matrix Factorization. Consider an  $L$ -layer deep network where  $L \in \mathbb{N}^+$ ,  $L \geq 2$  is the depth of the model. Let  $W_i \in \mathbb{R}^{d_i \times d_{i-1}}$  and  $d_i$  denote the layer weight matrix and width of the  $i^{\text{th}}$  ( $i \in [L]$ )
layer respectively. We use  $\mathbf{W}$  to denote the concatenation of all the parameters  $(W_{1},\ldots ,W_{L})$  and define the end-to-end matrix of  $\mathbf{W}$  as
$$
E (\mathbf {W}) \triangleq W _ {L} W _ {L - 1} \dots W _ {1}. \tag {1}
$$
In this paper, we focus on models that are linear in the space of the end-to-end matrix  $E(W)$ . Suppose  $M^{*} \in \mathbb{R}^{d_{L} \times d_{0}}$  is the target end-to-end matrix, and we observe  $n$  linear measurements (matrices)  $A_{i} \in \mathbb{R}^{d_{L} \times d_{0}}$  and the corresponding labels  $b_{i} = \langle A_{i}, M^{*} \rangle$ . The training loss of  $\mathbf{W}$  is the mean-squared error (MSE) between the prediction  $\langle A_{i}, W_{L}W_{L-1} \cdots W_{1} \rangle$  and the observation  $b_{i}$ :
$$
\mathcal {L} (\mathbf {W}) \triangleq \frac {1}{n} \sum_ {i = 1} ^ {n} \left(\left\langle A _ {i}, W _ {L} W _ {L - 1} \dots W _ {1} \right\rangle - b _ {i}\right) ^ {2}. \tag {2}
$$
Throughout this paper, we assume that  $d_{i}\geq \min (d_{0},d_{L})$  for each  $i\in [L]$  and, thus, the image of the function  $E(\cdot)$  is the entire  $\mathbb{R}^{d_L\times d_0}$ . In particular, this ensures that the deep models are sufficiently expressive in the sense that  $\min_{\boldsymbol{W}}\mathcal{L}(\boldsymbol {W}) = 0$ . For this setting, we aim to understand the structure of the trace of the Hessian minimization, as described below. The trace of Hessian is the sum of the eigenvalues of Hessian, which is an indicator of sharpness and it is known that variants of SGD, such as label noise SGD or 1-SAM, are biased toward models with a smaller trace of Hessian [29, 48].
Min Trace of Hessian Interpolating Solution. Our primary object of study is the interpolating solution with the minimum trace of Hessian, defined as:
$$
\boldsymbol {W} ^ {*} \in \underset {\boldsymbol {W}: \mathcal {L} (\boldsymbol {W}) = 0} {\arg \min } \operatorname {t r} \left[ \nabla^ {2} \mathcal {L} (\boldsymbol {W}) \right]. \tag {3}
$$
As we shall see shortly, the solution to the above optimization problem is not unique. We are interested in understanding the underlying structure of any minimizer  $\mathbf{W}^*$ . This will, in turn, inform us about the generalization nature of these solutions.
Before delving into the technical details, we state our main results in this section. This also serves the purpose of highlighting the primary technical contributions of the paper. First, since the generalization of  $\mathbf{W}$  only depends on its end-to-end matrix  $E(\mathbf{W})$ , it is informative to derive the properties of  $E(\mathbf{W}^{*})$  for any min trace of the Hessian interpolating solution  $\mathbf{W}^{*}$  defined in (3). Indeed, penalizing the trace of Hessian in the  $W$  space induces an equivalent penalization in the space of the end-to-end parameters. More concretely, given an end-to-end parameter  $M$ , let the induced regularizer  $F(M)$  denote the minimum trace of Hessian of the training loss at  $\mathbf{W}$  among all  $\mathbf{W}$ 's that instantiate the end-to-end matrix  $M$  i.e.,  $E(\mathbf{W}) = M$ .
Definition 1 (Induced Regularizer). Suppose  $M \in \mathbb{R}^{d_L \times d_0}$  is an end-to-end parameter that fits the training data perfectly (that is,  $\langle A_i, M \rangle = b_i$ ,  $\forall i \in [n]$ ). We define the induced regularizer as
$$
F (M) \triangleq \min  _ {\boldsymbol {W}: E (\boldsymbol {W}) = M} \operatorname {t r} [ \nabla^ {2} \mathcal {L} (\boldsymbol {W}) ] \tag {4}
$$
Since the image of  $E(\cdot)$  is the entire  $\mathbb{R}^{d_L\times d_0}$  by our assumption that  $d_{i}\geq \min (d_{0},d_{L})$ , function  $F$  is well-defined for all  $M\in \mathbb{R}^{d_L\times d_0}$ . It is easy to see that minimizing the trace of the Hessian in the original parameter space (see (3)) is equivalent to penalizing  $F(M)$  in the end-to-end parameter. Indeed, the minimizers of the implicit regularizer in the end-to-end space are related to the minimizers of the implicit regularizer in the  $W$  space, i.e.,
$$
\operatorname *{arg  min}_{M:\mathcal{L}^{\prime}(M) = 0}F(M) = \left\{E(\boldsymbol{W}^{*})\mid \boldsymbol{W}^{*}\in \operatorname *{arg  min}_{\boldsymbol {W}:\mathcal{L}(\boldsymbol {W}) = 0}\operatorname {tr}[\nabla^{2}\mathcal{L}(\boldsymbol {W})]\right\} ,
$$
where for any  $M \in \mathbb{R}^{d_L \times d_0}$ , we define  $\mathcal{L}'(M) \triangleq \frac{1}{n} \sum_{i=1} (\langle A_i, M \rangle - b_i)^2$  and thus  $\mathcal{L}(\pmb{W}) = \mathcal{L}'(E(\pmb{W}))$ . This directly follows from the definition of  $F$  in (4). Our main result characterizes the induced regularizer  $F(M)$  when the data satisfies the RIP property.
Theorem 1 (Induced regularizer under RIP). Suppose the linear measurements  $\{A_i\}_{i=1}^n$  satisfy the  $(1, \delta)$ -RIP condition.
Settings	Induced Regularizer F(M)/L	Theorem
(1,δ)-RIP	(1 ± O(δ))(d0dL)1/L||M||*2-2/L	Theorem 1
L=2	||(1/nAiAiT)1/2M (1/nAiT Ai)1/2||*	Theorem 5 ([12])
n=1	||(ATM)L-1AT||2/LS2/L	Theorem 7
Table 1: Summary of properties of the induced regularizer in the end-to-end matrix space. Here  $\left\| \cdot \right\|_{S_p}$  denotes the Schatten  $p$ -norm for  $p \in [1, \infty]$  and Schatten  $p$ -quasinorm for  $p \in (0, 1)$  (see Definition 2).  $\left\| \cdot \right\|_*$  denotes the Schatten 1-norm, also known as the nuclear norm.
1. For any  $M \in \mathbb{R}^{d_L \times d_0}$  such that  $\langle A_i, M \rangle = b_i$ ,  $\forall i \in [n]$ , it holds that
$$
(1 - \delta) L \left(d _ {0} d _ {L}\right) ^ {1 / L} \| M \| _ {*} ^ {2 (L - 1) / L} \leq F (M) \leq (1 + \delta) L \left(d _ {0} d _ {L}\right) ^ {1 / L} \| M \| _ {*} ^ {2 (L - 1) / L}. \tag {5}
$$
2. Let  $\mathbf{W}^{*} \in \arg \min_{\mathbf{W}: \mathcal{L}(\mathbf{W}) = 0} \operatorname{tr}[\nabla^{2}\mathcal{L}(\mathbf{W})]$  be an interpolating solution with minimal trace of Hessian. Then  $E(\mathbf{W}^{*})$  roughly minimizes the nuclear norm among all interpolating solutions of  $\mathcal{L}'$ . That is,
$$
\| E \left(\boldsymbol {W} ^ {*}\right) \| _ {*} \leq \frac {1 + \delta}{1 - \delta} \min  _ {\mathcal {L} ^ {\prime} (M) = 0} \| M \| _ {*}.
$$
However, for more general cases, it is challenging to compute the closed-form expression of  $F$ . In this work, we derive closed-form expressions for  $F$  in the following two cases: (1) depth  $L$  is equal to 2 and (2) there is only one measurement, i.e.,  $n = 1$  (see Table 1). Leveraging the above characterization of induced regularizer, we obtain the following result on the generalization bounds:
Theorem 2 (Recovery of the ground truth under RIP). Suppose the linear measurements  $\{(A_i)\}_{i=1}^n$  satisfy the  $(2, \delta(n))$ -RIP (Definition 3). Then for any  $\mathbf{W}^* \in \operatorname{arg min}_{\mathbf{W}: \mathcal{L}(\mathbf{W}) = 0} \operatorname{tr}[\nabla^2 \mathcal{L}(\mathbf{W})]$ , we have
$$
\left\| E \left(\boldsymbol {W} ^ {*}\right) - M ^ {*} \right\| _ {F} ^ {2} \leq \frac {8 \delta (n)}{(1 - \delta (n)) ^ {2}} \| M ^ {*} \| _ {*} ^ {2}. \tag {6}
$$
where  $\delta (n)$  depends on the number of measurements  $n$  and the distribution of the measurements.
If we further suppose  $\{A_i\}_{i=1}^n$  are independently sampled from some distribution over  $\mathbb{R}^{d_L \times d_0}$  satisfying that  $\mathbb{E}_A\langle A, M \rangle^2 = \|M\|_F^2$ , e.g., the standard multivariate Gaussian distribution, denoted by  $\mathcal{G}_{d_L \times d_0}$ , we know  $\delta(n) = O\left(\sqrt{\frac{d_L + d_0}{n}}\right)$  from Candes and Plan [7] (see Section 5.1 for more examples).
Theorem 3. For  $n \geq \Omega(r(d_0 + d_L))$ , with probability at least  $1 - \exp(\Omega(d_0 + d_L))$  over the randomly sampled  $\{A_i\}_{i=1}^n$  from multivariate Gaussian distribution  $\mathcal{G}$ , for any minimum trace of Hessian interpolating solution  $\boldsymbol{W}^* \in \operatorname{arg min}_{\boldsymbol{W}: \mathcal{L}(\boldsymbol{W}) = 0} \operatorname{tr}[\nabla^2 \mathcal{L}(\boldsymbol{W})]$ , the population loss  $\overline{\mathcal{L}}(\boldsymbol{W}^*) \triangleq \mathbb{E}_{A \sim \mathcal{G}}(\langle A, E(\boldsymbol{W}^*) \rangle - \langle A, M^* \rangle)^2$  satisfies that
$$
\overline {{\mathcal {L}}} (\boldsymbol {W} ^ {*}) = \| E (\boldsymbol {W} ^ {*}) - M ^ {*} \| _ {F} ^ {2} \leq O \left(\frac {d _ {0} + d _ {L}}{n} \| M ^ {*} \| _ {*} ^ {2} \log^ {3} n\right).
$$
Next, we state a lower bound for the conventional estimator for overparameterized models that minimizes the norm. The lower bound states that, to achieve a small error, the number of samples should be as large as the product of the dimensions of the end-to-end matrix  $d_0d_L$  as opposed to  $d_0 + d_L$  in case of the min trace of Hessian minimizer. It is proved in Appendix F.
Theorem 4 (Lower bound for  $\ell_2$  regression). Suppose  $\{A_i\}_{i=1}^n$  are randomly sampled from multivariate Gaussian distribution  $\mathcal{G}$ , let  $\tilde{\boldsymbol{W}} = \arg \min_{\boldsymbol{W}:\mathcal{L}(\boldsymbol{W}) = 0} \|E(\boldsymbol{W})\|_F$  to be the minimum Frobenius norm interpolating solution, then the expected population loss is
$$
\mathbb {E} \overline {{\mathcal {L}}} (\tilde {\boldsymbol {W}}) = \left(1 - \frac {\min  \left\{n , d _ {0} d _ {L} \right\}}{d _ {0} d _ {L}}\right) \| M ^ {*} \| _ {F} ^ {2}.
$$
The lower bound in Theorem 4 shows in order to obtain an  $O(1)$ -relatively accurate estimates of the ground truth in expectation, namely to guarantee  $\mathbb{E}\overline{\mathcal{L}} (\tilde{W})\leq O(1)\| M^{*}\|_{F}^{2}$ , the minimum Frobenius norm interpolating solution needs at least  $\Omega (d_0d_L)$  samples. In contrast, the minimizer of trace of Hessian in the same problem only requires  $O((d_0 + d_L)\| M^*\| _*^2 /\| M^*\| _F^2)$  samples, which is at least  $\tilde{O} (\frac{\min\{d_0,d_L\}}{r})$  times smaller. We further illustrate experimentally the superior generalization ability of sharpness minimization algorithms like label noise SGD [6, 10, 29] compared to vanilla mini-bactch SGD Figure 1. Due to the space limits, we defer the full setting for experiments into Appendix A.
Connection Between Sharpness and Generalization. Research on the connection between generalization and sharpness dates back to Hochreiter and Schmidhuber [21]. Keskar et al. [25] famously observe that when increasing the batch size of SGD, the test error and the sharpness of the learned solution both increase. Jastrzebski et al. [23] extend this observation and found that there is a positive correlation between sharpness and the ratio between learning rate and batch size. Jiang et al. [24] perform a large-scale empirical study on various notions of generalization measures and show that sharpness-based measures correlate with generalization best. Liu et al. [31] find that among language models with the same validation pretraining loss, those that have smaller sharpness can have better downstream performance. On the other hand, Dinh et al. [13] argue that for networks with scaling invariance, there always exist models with good generalization but with arbitrarily large sharpness. We note this does not contradict our main result here, which only asserts the interpolation solution with a minimal trace of Hessian generalizes well, but not vice versa. Empirically, sharpness minimization is also a popular and effective regularization method for overparametrized models [39, 17, 53, 49, 26, 32, 54, 52, 1].
Implicit Bias of Sharpness Minimization. Recent theoretical works [6, 10, 29, 31] show that SGD with label noise is implicitly biased toward local minimizers with a smaller trace of Hessian under the assumption that the minimizers locally connect as a manifold. Such a manifold setting is empirically verified by Draxler et al. [14], Garipov et al. [18] in the sense that the set of minimizers of the training loss is path-connected. It is the same situation for the deep matrix factorization problem studied in this paper, although we do not study the optimization trajectory. Instead, we directly study properties of the minimum trace of Hessian interpolation solution.
Sharpness-reduction implicit bias can also happen for deterministic GD. Arora et al. [4] show that normalized GD implicitly penalizes the largest eigenvalue of the Hessian. Ma et al. [35] argues that such sharpness reduction phenomena can also be caused by a multi-scale loss landscape. Lyu et al. [33] show that GD with weight decay on a scale-invariant loss function implicitly decreases the spherical sharpness, i.e., the largest eigenvalue of the Hessian evaluated at the normalized parameter. Another line of work focuses on the sharpness minimization effect of a large learning rate in GD, assuming that it converges at the end of training. This has been studied mainly through linear stability analysis [50, 8, 34, 9]. Recent theoretical analysis [11, 30] showed that the sharpness minimization effect of a large learning rate in GD does not necessarily rely on convergence and linear stability, through a four-phase characterization of the dynamics at the Edge of Stability regime [8].
Notation. We use  $[n]$  to denote  $\{1,2,\dots,n\}$  for every  $n\in \mathbb{N}$ . We use  $\| M\| _F$ ,  $\| M\| _*$ ,  $\| M\| _2$  and  $\operatorname {tr}(M)$  to denote the Frobenius norm, nuclear norm, spectral norm and trace of matrix  $M$  respectively. For any function  $f$  defined over set  $S$  such that  $\min_{x\in S}f(x)$  exists, we use  $\arg \min_S f$  to denote the set  $\{y\in S\mid f(y) = \min_{x\in S}f(x)\}$ . Given a matrix  $M$ , we use  $h_M$  to denote the linear map  $A\mapsto \langle A,M\rangle$ . We use  $\mathcal{H}_r$  to denote the set  $\mathcal{H}_r\triangleq \{h_M\mid \| M\|_*\leq r\}$ .  $M_{i}$ : and  $M_{:j}$  are used to denote the  $i$ th row and  $j$ th column of the matrix  $M$ .
The following definitions will be important to the technical discussion in the paper.
Rademacher Complexity. Given  $n$  data points  $\{A_i\}_{i=1}^n$ , the empirical Rademacher complexity of function class  $\mathcal{H}$  is defined as
$$
\mathcal {R} _ {n} (\mathcal {H}) = \frac {1}{n} \mathbb {E} _ {\epsilon \sim \{\pm 1 \} ^ {n}} \sup _ {h \in \mathcal {H}} \sum_ {i = 1} ^ {n} \epsilon_ {i} h (A _ {i}).
$$
Given a distribution  $P$ , the population Rademacher complexity is defined as follows:  $\overline{\mathcal{R}}_n(\mathcal{H}) = \mathbb{E}_{A_i \sim P}^{iid} \mathcal{R}_n(\mathcal{H})$ . This is mainly used to upper bound the generalization gap of SGD.
Definition 2 (Schatten  $p$ -quasi)norm). Given any  $d, d' \in \mathbb{N}^+$ ,  $p \in (0, \infty)$  a matrix  $M \in \mathbb{R}^{d \times d'}$  with singular values  $\sigma_1(M), \ldots, \sigma_{\min(d, d')} (M)$ , we define the Schatten  $p$ -(semi)norm as
$$
\| M \| _ {S _ {p}} = \left(\sum_ {i = 1} ^ {\min (d, d ^ {\prime})} \sigma_ {i} ^ {p} (M)\right) ^ {1 / p}.
$$
Note that in this definition  $\| \cdot \|_{S_p}$  is a norm only when  $p\geq 1$ . When  $p\in (0,1)$ , the triangle inequality does not hold. Note that when  $p\in (0,1)$ ,  $\| A + B\|_{S_p}\leq 2^{1 / p - 1}(\| A\|_{S_p} + \| B\|_{S_p})$  for any matrices  $A$  and  $B$ , however,  $2^{1 / p - 1} > 1$ .
We use  $L$  to denote the depth of the linear model and  $\pmb{W} = (W_{1},\dots ,W_{L})$  to denote the parameters, where  $W_{i}\in \mathbb{R}^{d_{i}\times d_{i - 1}}$ . We assume that  $d_{i}\geq \min (d_{0},d_{L})$  for each  $i\in [L - 1]$  and, thus, the image of  $E(\pmb {W})$  is the entire  $\mathbb{R}^{d_L\times d_0}$ . Following is a simple relationship between nuclear norm and Frobenius norm that is used frequently in the paper.
Lemma 1. For any matrices  $A$  and  $B$ , it holds that  $\| AB \|_{*} \leq \| A \|_{F} \| B \|_{F}$ .
4 Exact Formulation of Induced Regularizer by Trace of Hessian
In this section, we derive the exact formulation of trace of Hessian for  $\ell_2$  loss over deep matrix factorization models with linear measurements as a minimization problem over  $W$ . We shall later approximate this formula by a different function in Section 5, which allows us to calculate the implicit bias in closed-form in the space of end-to-end matrices.
We first introduce the following simple lemma showing that the trace of the Hessian of the loss is equal to the sum of squares of norms of the gradients of the neural network output.
Lemma 2. For any twice-differentiable function  $\{f_i(\mathbf{W})\}_{i=1}^n$ , real-valued labels  $\{b_i\}_{i=1}^n$ , loss function  $\mathcal{L}(\mathbf{W}) = \frac{1}{n}\sum_{i=1}^{n}(f_i(\mathbf{W}) - b_i)^2$ , and any  $\mathbf{W}$  satisfying  $\mathcal{L}(\mathbf{W}) = 0$ , it holds that
$$
\operatorname {t r} \left(\nabla^ {2} \mathcal {L} (\mathbf {W})\right) = \frac {2}{n} \sum_ {i = 1} ^ {n} \| \nabla f _ {i} (\mathbf {W}) \| ^ {2}.
$$
Using Lemma 2, we calculate the trace of Hessian for the particular loss defined in (2). To do this, we consider  $\mathbf{W}$  in Lemma 2 to be the concatenation of matrices  $(W_{1},\ldots ,W_{L})$  and we set  $f_{i}(\mathbf{W})$  to be the linear measurement  $\langle A_i,E(\mathbf{W})\rangle$ , where  $E(\mathbf{W}) = W_L\dots W_1$  (see (1)). To calculate the trace of Hessian, according to Lemma 2, we need to calculate the gradient of  $\mathcal{L}(\mathbf{W})$  in (2). To this end, for a fixed  $i$ , we compute the gradient of  $\langle A_i,E(\mathbf{W})\rangle$  with respect to one of the weight matrices  $W_{j}$ .
$$
\begin{array}{l} \nabla_ {W _ {j}} \left\langle A _ {i}, E (\boldsymbol {W}) \right\rangle = \nabla_ {W _ {j}} \operatorname {t r} \left(A _ {i} ^ {\top} W _ {L} \dots W _ {1}\right) \\ = \nabla_ {W _ {j}} \operatorname {t r} \left(\left(W _ {j - 1} \dots W _ {1} A _ {i} ^ {\top} W _ {L} \dots W _ {j + 1}\right) W _ {j}\right) \\ = \left(W _ {j - 1} \dots W _ {1} A _ {i} ^ {\top} W _ {L} \dots W _ {j + 1}\right) ^ {\top}. \\ \end{array}
$$
According to Lemma 2, trace of Hessian is given by
$$
\operatorname {t r} \left(\nabla^ {2} L\right) (\mathbf {W}) = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {L} \left\| \nabla_ {W _ {j}} \left\langle A _ {i}, E (\mathbf {W}) \right\rangle \right\| _ {F} ^ {2} = \frac {1}{n} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {L} \left\| W _ {j - 1} \dots W _ {1} A _ {i} ^ {\top} W _ {L} \dots W _ {j + 1} \right\| _ {F} ^ {2}.
$$
As mentioned earlier, our approach is to characterize the minimizer of the trace of Hessian among all interpolating solutions by its induced regularizer in the end-to-end matrix space. The above
calculation provides the following more tractable characterization of induced regularizer  $F$  in (12):
$$
F (M) = \min  _ {E (\boldsymbol {W}) = M} \sum_ {i = 1} ^ {n} \sum_ {j = 1} ^ {L} \| W _ {j - 1} \dots W _ {1} A _ {i} ^ {\top} W _ {L} \dots W _ {j + 1} \| _ {F} ^ {2}. \tag {7}
$$
In general, we cannot solve  $F$  in closed form for general linear measurements  $\{A_i\}_{i=1}^n$ ; however, interestingly, we show that it can be solved approximately under reasonable assumption on the measurements. In particular, we show that the induced regularizer, as defined in (7), will be approximately proportional to a power of the nuclear norm of  $E(\mathbf{W})$  given that the measurements  $\{A_i\}_{i=1}^n$  satisfy a natural norm-preserving property known as the Restricted Isometry Property (RIP) [7, 42].
Before diving into the proof of the general result for RIP, we first illustrate the connection between nuclear norm and the induced regularizer for the depth-two case. In this case, fortunately, we can compute the closed form of the induced regularizer. This result was first proved by Ding et al. [12]. For self-completeness, we also provide a short proof.
Theorem 5 (Ding et al. [12]). For any  $M \in \mathbb{R}^{d_L \times d_0}$ , it holds that
$$
F (M) \triangleq \min  _ {W _ {2} W _ {1} = M} \operatorname {t r} [ \nabla^ {2} \mathcal {L} ] (\boldsymbol {W}) = 2 \left\| \left(\frac {1}{n} \sum_ {i} A _ {i} A _ {i} ^ {\top}\right) ^ {1 / 2} M \left(\frac {1}{n} \sum_ {i} A _ {i} ^ {\top} A _ {i}\right) ^ {1 / 2} \right\| _ {*}. \tag {8}
$$
Proof of Theorem 5. We first define  $B_{1} = \left(\sum_{i=1}^{n} A_{i} A_{i}^{T}\right)^{\frac{1}{2}}$  and  $B_{2} = \left(\sum_{i=1}^{n} A_{i}^{T} A_{i}\right)^{\frac{1}{2}}$ . Therefore we have that
$$
\operatorname {t r} \left[ \nabla^ {2} \mathcal {L} \right] (\boldsymbol {W}) = \sum_ {i = 1} ^ {n} \left(\left\| A _ {i} ^ {T} W _ {2} \right\| _ {F} ^ {2} + \left\| W _ {1} A _ {i} ^ {T} \right\| _ {F} ^ {2}\right) = \left\| B _ {1} W _ {2} \right\| _ {F} ^ {2} + \left\| W _ {1} B _ {2} \right\| _ {F} ^ {2}.
$$
Further applying Lemma 1, we have that
$$
\begin{array}{l} F (M) = \min  _ {W _ {2} W _ {1} = M} \operatorname {t r} [ \nabla^ {2} \mathcal {L} ] (\boldsymbol {W}) = \min  _ {W _ {2} W _ {1} = M} \sum_ {i = 1} ^ {n} \left(\left\| A _ {i} ^ {T} W _ {2} \right\| _ {F} ^ {2} + \left\| W _ {1} A _ {i} ^ {T} \right\| _ {F} ^ {2}\right) \\ \geq \min _ {W _ {2} W _ {1} = M} 2 \| B _ {1} W _ {2} W _ {1} B _ {2} \| _ {*} ^ {2} = 2 \| B _ {1} M B _ {2} \| _ {*} ^ {2}. \\ \end{array}
$$