SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Next we show this lower bound of  $F(M)$  can be attained. Let  $U\Lambda V^T$  be the SVD of  $B_{1}MB_{2}$ . The equality condition happens for  $W_{2}^{*} = B_{1}^{\dagger}U\Lambda^{1 / 2},W_{1}^{*} = \Lambda^{1 / 2}V^{T}B_{2}^{\dagger}$ , where we have that  $\sum_{i = 1}^{n}\| A_{i}^{T}W_{2}^{*}\|_{F}^{2} + \| W_{1}^{*}A_{i}^{T}\|_{F}^{2} = 2\| \Lambda \|_{F}^{2} = 2\| B_{1}MB_{2}\|_{F}^{2}$ . This completes the proof.
The right-hand side in (8) will be very close to the nuclear norm of  $M$  if the two extra multiplicative terms are close to the identity matrix. It turns out that  $\{A_i\}_{i=1}^n$  satisfying the  $(1, \delta)$ -RIP exactly guarantees the two extra terms are  $O(\delta)$ -close to identity. However, the case for deep networks where depth is larger than two is fundamentally different from the two-layer case, where one can obtain a closed form for  $F$ . To the best of our knowledge, it is open whether one obtains a closed form for the induced-regularizer for the trace of Hessian when  $L > 2$ . Nonetheless, in Section 5.1, we show that under RIP, we can still approximate it with the nuclear norm.
5 Results for Measurements with Restricted Isometry Property (RIP)
In this section, we present our main results for the generalization benefit of flatness regularization in deep linear networks. We structure the analysis as follows:
1. In Section 5.1, we first recap some preliminaries on the RIP property.
2. In Section 5.2, we prove that the induced regularizer by trace of Hessian is approximately the power of nuclear norm for  $(1,\delta)$ -RIP measurements (Theorem 1).
3. In Section 5.3, we prove that the minimum trace of Hessian interpolating solution with  $(2,\delta)$ -RIP measurements can recover the ground truth  $M^{*}$  up to error  $\delta \| M^{*}\|_{*}^{2}$ . For  $\{A_i\}_{i = 1}^n$  sampled from Gaussian distributions, we know  $\delta = O(\sqrt{\frac{d_0 + d_L}{n}})$ .
4. In Section 5.4, we prove a generalization bound with faster rate of  $\frac{d_0 + d_L}{n} \| M^* \|_*^2$  using local Rademacher complexity based techniques from Srebro et al. [44].
Next, we discuss important distributions of measurements for which the RIP property holds.
5.1 Preliminaries for RIP
Definition 3 (Restricted Isometry Property (RIP)). A family of matrices  $\{A_i\}_{i=1}^n$  satisfies the  $(r, \delta)$ -RIP iff for any matrix  $X$  with the same dimension and rank at most  $r$ :
$$
(1 - \delta) \| X \| _ {F} ^ {2} \leq \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle A _ {i}, X \right\rangle^ {2} \leq (1 + \delta) \| X \| _ {F} ^ {2}. \tag {9}
$$
Next, we give two examples of distributions where  $\Omega (r(d_0 + d_L))$  samples guarantee  $(r,O(1))$ -RIP. The proofs follow from Theorem 2.3 in [7].
Example 1. Suppose for every  $i \in \{1, \ldots, n\}$ , each entry in the matrix  $A_i$  is an independent standard Gaussian random variable, i.e.,  $A_i \stackrel{i.i.d.}{\sim} \mathcal{G}_{d_L \times d_0}$ . For every constant  $\delta \in (0,1)$ , if  $n \geq \Omega(r(d_0 + d_L))$ , then with probability  $1 - e^{\Omega(n)}$ ,  $\{A_i\}_{i=1}^n$  satisfies  $(r,\delta)$ -RIP.
Example 2. If each entry of  $A_{i}$  is from a symmetric Bernoulli random variable with variance 1, i.e. for all  $i, k, \ell$ , entry  $[A_{i}]_{k,\ell}$  is either equal to 1 or -1 with equal probabilities, then for any  $r$  and  $\delta$ ,  $(r,\delta)$ -RIP holds with same probability as in Example 1 if the same condition there is satisfied.
5.2 Induced Regularizer of Trace of Hessian is Approximately Nuclear Norm
This section focuses primarily on the proof of Theorem 2. Our proof consists of two steps: (1) we show that the trace of Hessian of training loss at the minimizer  $\mathbf{W}$  is multiplicatively  $O(\delta)$ -close to the regularizer  $R(\mathbf{W})$  defined below (Lemma 3) and (2) we show that the induced regularizer of  $R$ ,  $F'(M)$ , is proportional to  $\| M\|_{*}^{2(L - 1) / L}$  (Lemma 4).
$$
R (\boldsymbol {W}) \triangleq \| W _ {L} \dots W _ {2} \| _ {F} ^ {2} d _ {0} + \sum_ {j = 2} ^ {L - 1} \| W _ {L} \dots W _ {j + 1} \| _ {F} ^ {2} \| W _ {j - 1} \dots W _ {1} \| _ {F} ^ {2} + \| W _ {L - 1} \dots W _ {1} \| _ {F} ^ {2} d _ {L}. \tag {10}
$$
Lemma 3. Suppose the linear measurement  $\{A_i\}_{i=1}^n$  satisfy  $(1, \delta)$ -RIP. Then, for any  $\mathbf{W}$  such that  $\mathcal{L}(\mathbf{W}) = 0$ , it holds that
$$
(1 - \delta) R (\boldsymbol {W}) \leq \operatorname {t r} (\nabla^ {2} L) (\boldsymbol {W}) \leq (1 + \delta) R (\boldsymbol {W}).
$$
Since  $\mathrm{tr}(\nabla^2\mathcal{L})(\pmb {W})$  closely approximates  $R(\pmb {W})$  , we can study  $R$  instead of  $\mathrm{tr}[\nabla^2\mathcal{L}]$  to understand the implicit bias up to a multiplicative factor  $(1 + \delta)$  . In particular, we want to solve the induced regularizer of  $R(W)$  on the space of end-to-end matrices,  $F^{\prime}(M)$  ..
$$
F ^ {\prime} (M) \triangleq \min  _ {\mathbf {W}: W _ {L} \dots W _ {1} = M} R (\mathbf {W}). \tag {11}
$$
Surprisingly, we can solve this problem in closed form.
Lemma 4. For any  $M \in \mathbb{R}^{d_L \times d_0}$ , it holds that
$$
F ^ {\prime} (M) \triangleq \underset {\boldsymbol {W}: W _ {L} \dots W _ {1} = M} {\min } R (\boldsymbol {W}) = L \left(d _ {0} d _ {L}\right) ^ {1 / L} \| M \| _ {*} ^ {2 (L - 1) / L}. \tag {12}
$$
Proof of Lemma 4. Applying the  $L$ -version of the AM-GM to Equation (10):
$$
\begin{array}{l} \left(R (\mathbf {W}) / L\right) ^ {L} \geq d _ {0} \| W _ {L} \dots W _ {2} \| _ {F} ^ {2} \cdot \| W _ {1} \| _ {F} ^ {2} \| W _ {L} \dots W _ {3} \| _ {F} ^ {2} \dots \| W _ {L - 1} \dots W _ {1} \| _ {F} ^ {2} d _ {L}. \tag {13} \\ = d _ {0} d _ {L} \prod_ {j = 1} ^ {L - 1} \left(\| W _ {L} \dots W _ {j + 1} \| _ {F} ^ {2} \| W _ {j} \dots W _ {1} \| _ {F} ^ {2}\right) \\ \end{array}
$$
Now using Lemma 1, we have for every  $1 \leq j \leq L - 1$ :
$$
\left\| W _ {L} \dots W _ {j + 1} \right\| _ {F} ^ {2} \left\| W _ {j} \dots W _ {1} \right\| _ {F} ^ {2} \geq \left\| W _ {L} \dots W _ {1} \right\| _ {*} ^ {2} = \left\| M \right\| _ {*} ^ {2}. \tag {14}
$$
Multiplying Equation (14) for all  $1 \leq j \leq L - 1$  and combining with Equation (13) implies
$$
\min  _ {\{W \mid W _ {L} \dots W _ {1} = M \}} R (\mathbf {W}) \geq L \left(d _ {0} d _ {L}\right) ^ {1 / L} \| M \| _ {*} ^ {2 (L - 1) / L}. \tag {15}
$$
Now we show that equality can indeed be attained. To construct an example in which the equality happens, consider the singular value decomposition of  $M$ :  $M = U\Lambda V^T$ , where  $\Lambda$  is a square matrix with dimension  $\mathrm{rank}(M)$ .
For  $1 \leq i \leq L$ , we pick  $Q_{i} \in \mathbb{R}^{d_{i} \times \mathrm{rank}(M)}$  to be any matrix with orthonormal columns. Note that  $\mathrm{rank}(M)$  is not larger than  $d_{i}$  for all  $1 \leq i \leq L$ , hence such orthonormal matrices  $Q_{i}$  exist. Then we define the following with  $\alpha, \alpha' > 0$  being constants to be determined:
$$
W _ {L} = \alpha^ {\prime} \alpha^ {- (L - 2) / 2} U \Lambda^ {1 / 2} Q _ {L - 1} ^ {T} \in \mathbb {R} ^ {d _ {L} \times d _ {L - 1}},
$$
$$
W _ {i} = \alpha Q _ {i} Q _ {i - 1} ^ {T} \in \mathbb {R} ^ {d _ {i} \times d _ {i - 1}}, \quad \forall 2 \leq i \leq L - 1,
$$
$$
W _ {1} = \alpha^ {\prime - 1} \alpha^ {- (L - 2) / 2} Q _ {1} \Lambda^ {1 / 2} V ^ {T} \in \mathbb {R} ^ {d _ {1} \times d _ {0}}.
$$
Note that  $\Lambda$  is a square matrix with dimension  $\mathrm{rank}(M)$ . First of all, note that the defined matrices satisfy
$$
W _ {L} W _ {L - 1} \dots W _ {1} = \alpha^ {L - 2} \alpha^ {- (L - 2)} U \Lambda^ {1 / 2} \Lambda^ {1 / 2} V ^ {T} = M.
$$
To gain some intuition, we check that the equality case for all the inequalities that we applied above. We set the value of  $\alpha$  in a way that these equality cases can hold simultaneously. Note that for the matrix holder inequality that we applied in Equation (14):
$$
\| W _ {L} \dots W _ {j + 1} \| _ {F} ^ {2} \| W _ {j} \dots W _ {1} \| _ {F} ^ {2} = \| W _ {L} \dots W _ {1} \| _ {*} ^ {2} = \| \Lambda^ {1 / 2} \| _ {F} ^ {2},
$$
independent of the choice of  $\alpha$ . It remains to check the equality case for the AM-GM inequality that we applied in Equation (13). We have for all  $2 \leq j \leq L - 1$ :
$$
\begin{array}{l} \left\| W _ {L} \dots W _ {j + 1} \right\| _ {F} \left\| W _ {j - 1} \dots W _ {1} \right\| _ {F} \\ = \alpha^ {j - 2} \alpha^ {- (L - 2) / 2} \alpha^ {L - j - 1} \alpha^ {- (L - 2) / 2} \| U \Lambda^ {1 / 2} \| _ {F} \| \Lambda^ {1 / 2} V ^ {T} \| _ {F} = \alpha^ {- 1} \| \Lambda^ {1 / 2} \| _ {F} ^ {2}, \tag {16} \\ \end{array}
$$
Hence, equality happens for all of them. Moreover, for cases  $j = 1$  and  $j = L$ , we have
$$
d _ {0} \| W _ {L} \dots W _ {2} \| = \| \Lambda^ {1 / 2} \| _ {F} d _ {0} \alpha^ {\prime} \alpha^ {L - 2} \alpha^ {- (L - 2) / 2} = \| \Lambda^ {1 / 2} \| _ {F} d _ {0} \alpha^ {\prime} \alpha^ {(L - 2) / 2}. \tag {17}
$$
$$
d _ {L} \| W _ {L - 1} \dots W _ {1} \| = \| \Lambda^ {1 / 2} \| _ {F} d _ {L} \alpha^ {\prime - 1} \alpha^ {L - 2} \alpha^ {- (L - 2) / 2} = \| \Lambda^ {1 / 2} \| _ {F} d _ {L} \alpha^ {\prime - 1} \alpha^ {(L - 2) / 2}. \tag {18}
$$
Thus it suffices to set  $\alpha' = \left(\frac{d_L}{d_0}\right)^{1/2}$  and  $\alpha = \left(\frac{\|\Lambda^{1/2}\|_F}{\sqrt{d_0 d_L}}\right)^{2/L} = \left(\frac{\|M\|_*}{d_0 d_L}\right)^{1/L}$  so that the left-hand sides of (16), (17), and (18) are equal, which implies that the lower bound in Equation (15) is actually an equality. The proof is complete.
Now we can prove Theorem 1 as an implication of Lemma 4.
Proof of Theorem 1. The first claim is a corollary of Lemma 3. We note that
$$
F(M) = \min_{W_{L}\dots W_{1} = M}\operatorname {tr}[\nabla^{2}\mathcal{L}](M)\leq (1 + \delta)\min_{W_{L}\dots W_{1} = M}R(\mathbf{W}) = (1 + \delta)F^{\prime}(M)
$$
$$
F (M) = \min _ {W _ {L} \ldots W _ {1} = M} \operatorname {t r} [ \nabla^ {2} \mathcal {L} ] (M) \geq (1 - \delta) \min _ {W _ {L} \ldots W _ {1} = M} R (\mathbf {W}) = (1 - \delta) F ^ {\prime} (M).
$$
For the second claim, pick  $\bar{\mathbf{W}}$  that minimizes  $R(\bar{\mathbf{W}})$  over all  $\mathbf{W}$ 's that satisfy the linear measurements, thus we have that
$$
R (\bar {\mathbf {W}}) = L \left(d _ {0} d _ {L}\right) ^ {1 / L} \| E (\bar {\mathbf {W}}) \| _ {*} ^ {2 (L - 1) / L} = L \left(d _ {0} d _ {L}\right) ^ {1 / L} \min  _ {\mathcal {L} ^ {\prime} (M) = 0} \| M \| _ {*} ^ {2 (L - 1) / L}. \tag {19}
$$
Now from the definition of  $E(\mathbf{W}^{*})$ ,
$$
\operatorname {t r} \left(\nabla^ {2} L\right) \left(\mathbf {W} ^ {*}\right) \leq \operatorname {t r} \left(\nabla^ {2} L\right) \left(\bar {\mathbf {W}}\right) \leq (1 + \delta) R (\bar {\mathbf {W}}), \tag {20}
$$
where the last inequality follows from the definition of  $W$ . On the other hand
$$
\left. \operatorname {t r} \left(\nabla^ {2} L\right) \left(\mathbf {W} ^ {*}\right) \geq (1 - \delta) R (\bar {\mathbf {W}}) \geq (1 - \delta) L \left(d _ {0} d _ {L}\right) ^ {1 / L} \| E \left(\mathbf {W} ^ {*}\right) \| _ {*} ^ {2 (L - 1) / L}. \right. \tag {21}
$$
Combining (19), (20) and (21),
$$
\| E (\mathbf {W} ^ {*}) \| _ {*} \leq \left(\frac {1 + \delta}{1 - \delta}\right) ^ {\frac {L}{2 (L - 1)}} \min  _ {\mathcal {L} ^ {\prime} (M) = 0} \| M \| _ {*}.
$$
The proof is completed by noting that  $\frac{L}{2(L - 1)} \leq 1$  for all  $L \geq 2$ .
Thus combining Example 1 and Theorem 1 with  $\delta = 1 / 2$ , we have the following corollary.
Corollary 1. Let  $\{A_i\}_{i=1}^n$  be sampled independently from Gaussian distribution  $\mathcal{G}_{d_L \times d_0}$  where  $n \geq \Omega((d_0 + d_L))$ , with probability at least  $1 - \exp(\Omega(n))$ , we have
$$
\| E (\boldsymbol {W} ^ {*}) \| _ {*} \leq 3 \min  _ {\mathcal {L} ^ {\prime} (M) = 0} \| M \| _ {*} \leq 3 \| E (\boldsymbol {W} ^ {*}) \| _ {*}.
$$
5.3 Recovering the Ground truth
In this section, we prove Theorem 2. The idea is to show that under RIP, the empirical loss  $\mathcal{L}(\pmb{W})$  is a good approximation for the Frobenius distance of  $E(\pmb{W})$  to the ground truth  $M^{*}$ . To this end, we first introduce a very useful Lemma 5 below, whose proof is deferred to Appendix E.
Lemma 5. Suppose the measurements  $\{A_i\}_{i=1}^n$  satisfy the  $(2, \delta)$ -RIP condition. Then for any matrix  $M \in \mathbb{R}^{d_L \times d_0}$ , we have that
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \langle A _ {i}, M \rangle^ {2} - \| M \| _ {F} ^ {2} \right| \leq 2 \delta \| M \| _ {*} ^ {2}.
$$
We note that if  $\{A_i\}_{i=1}^n$  are i.i.d. random matrices with each coordinate being independent, zero mean, and unit variance (like standard Gaussian distribution), then  $\|W - M^*\|_F^2$  is the population squared loss corresponding to  $W$ . Thus, Theorem 2 implies a generalization bound for this case. Now we are ready to prove Theorem 2.
Proof of Theorem 2. Note that from Theorem 1,
$$
\| E (\boldsymbol {W} ^ {*}) \| _ {*} \leq \frac {1 + \delta}{1 - \delta} \min  _ {\mathcal {L} ^ {\prime} (M) = 0} \| M \| _ {*} \leq \frac {1 + \delta}{1 - \delta} \| M ^ {*} \| _ {*},
$$
which implies the following by triangle inequality,
$$
\left\| E \left(\boldsymbol {W} ^ {*}\right) - M ^ {*} \right\| _ {*} \leq \left\| \tilde {E} \left(\boldsymbol {W} ^ {*}\right) \right\| _ {*} + \left\| M ^ {*} \right\| _ {*} \leq \frac {2}{1 - \delta} \| M ^ {*} \| _ {*}. \tag {22}
$$
Combining (22) with Lemma 5 (with  $M = E(\pmb{W}^{*}) - M^{*}$ ):
$$
\left| \frac {1}{n} \sum_ {i = 1} ^ {n} \left\langle A _ {i}, E \left(\boldsymbol {W} ^ {*}\right) - M ^ {*} \right\rangle^ {2} - \| E \left(\boldsymbol {W} ^ {*}\right) - M ^ {*} \| _ {F} ^ {2} \right| \leq \frac {8 \delta}{(1 - \delta) ^ {2}} \| M ^ {*} \| _ {*} ^ {2}.
$$
Since  $W^{*}$  satisfies the linear constraints  $\operatorname{tr}(A_i E(\mathbf{W}^*)) = b_i$ ,  $\frac{1}{n} \sum_{i=1}^{n} \langle A_i, E(\mathbf{W}^*) - M^* \rangle^2 = \frac{1}{n} \sum_{i=1}^{n} \left( \langle A_i, E(\mathbf{W}^*) \rangle - b_i \right)^2 = 0$ , which completes the proof.
5.4 Generalization Bound
In this section, we prove the generalization bound in Theorem 3, which yields a faster rate of  $O\left(\frac{d_0 + d_L}{n}\| M^*\| _*^2\right)$  compared to  $O\left(\sqrt{\frac{d_0 + d_L}{n}}\| M^*\| _*\right)$  in Theorem 2. The intuition for this is as follows: By Corollary 1, we know that with very high probability, the learned solution has a bounded nuclear norm for its end-to-end matrix, no larger than  $3\| M^{*}\|_{2}$ , where  $M^{*}$  is the ground truth. The key mathematical tool is Theorem 6, which provides an upper bound on the population error of the learned interpolation solution that is proportional to the square of the Rademacher complexity of the function class  $\mathcal{H}_{3\| M^{*}\|_{*}} = \{h_M\mid \| M\|_{*}\leq 3\| M^{*}\|_{*}\}$ .
Theorem 6 (Theorem 1, Srebro et al. [44]). Let  $\mathcal{H}$  be a class of real-valued functions and  $\ell : \mathbb{R} \times \mathbb{R} \to \mathbb{R}$  be a differentiable non-negative loss function satisfying that (1) for any fixed  $y \in \mathbb{R}$ , the partial derivative  $\ell(\cdot, y)$  with respect to its first coordinate is  $H$ -Lipschitz and (2)  $|\sup_{x, y} \ell(x, y)| \leq B$ , where  $H, B$  are some positive constants. Then for any  $p > 0$ , we have that with probability at least  $1 - p$  over a random sample of size  $n$ , for any  $h \in \mathcal{H}$  with zero training loss,
$$
\bar {\mathcal {L}} (h) \leq O \left(H \log^ {3} n \mathcal {R} _ {n} ^ {2} (\mathcal {H}) + \frac {B \log (1 / p)}{n}\right). \tag {23}
$$
One technical difficulty is that Theorem 6 only works for bounded loss functions, but the  $\ell_2$  loss on Gaussian data is unbounded. To circumvent this issue, we construct a smoothly truncated variant of  $\ell_2$  loss (41) and apply Theorem 6 on that. Finally, we show that with a carefully chosen threshold, this truncation happens very rarely and, thus, does not change the population loss significantly. The proof can be found in Appendix E.
Figure 1: Train and test loss. Label noise SGD leads to better generalization results due to the sharpness-minimization implicit biases (as shown in Figure 2), while mini-batch SGD without label noise finds solutions with much larger test loss.
6 Result for the Single Measurement Case
Quite surprisingly, even though in the general case we cannot compute the closed-form of the induced regularizer in (12), we can find its minimum as a quasinorm function of the  $E(\mathbf{W})$  which only depends on the singular values of  $E(\mathbf{W})$ . This yields the following result for multiple layers  $L$  (possibly  $L > 2$ ) with a single measurement.
Theorem 7. Suppose there is only a single measurement matrix  $A$ , i.e.,  $n = 1$ . For any  $M \in \mathbb{R}^{d_L \times d_0}$ , the following holds:
$$
F (M) = \min  _ {W _ {L} \dots W _ {1} = M} \operatorname {t r} [ \nabla^ {2} \mathcal {L} ] (\boldsymbol {W}) = L \left\| \left(A ^ {T} M\right) ^ {L - 1} A ^ {T} \right\| _ {S _ {2 / L}} ^ {2 / L}. \tag {24}
$$
To better illustrate the behavior of this induced regularizer, consider the case where the measurement matrix  $A$  is identity and  $M$  is symmetric with eigenvalues  $\{\sigma_i\}_{i=1}^d$ . Then, it is easy to see that  $F(M)$  in (24) is equal to  $F(M) = \sum_{i} \sigma_i^{2(L-1)/L}$ . Interestingly, we see that the value of  $F(M)$  converges to the Frobenius norm of  $M$  and not the nuclear norm as  $L$  becomes large, which behaves quite differently (e.g. in the context of sparse recovery). This means that beyond RIP, the induced regularizer can behave very differently, and perhaps the success of training deep networks with SGD is closely tied to the properties of the dataset.
In this paper, we study the inductive bias of the minimum trace of the Hessian solutions for learning deep linear networks from linear measurements. We show that trace of Hessian regularization of loss on the end-to-end matrix of deep linear networks roughly corresponds to nuclear norm regularization under restricted isometry property (RIP) and yields a way to recover the ground truth matrix. Furthermore, leveraging this connection with the nuclear norm regularization, we show a generalization bound which yields a faster rate than Frobenius (or  $\ell_2$  norm) regularizer for Gaussian distributions. Finally, going beyond RIP conditions, we obtain closed-form solutions for the case of a single measurement. Several avenues for future work remain open, e.g., more general characterization of trace of Hessian regularization beyond RIP settings and understanding it for neural networks with non-linear activations.
TM and ZL would like to thank the support from NSF IIS 2045685. KG and SJ acknowledge support by NSF award CCF-2112665 (TILOS AI Institute) and NSF award 2134108.