SlowGuess/NeurIPS_2023_OCR huggingface dataset:
Bochen Lyu  
DataCanvas Lab, DataCanvas  
Beijing, China  
bochen.lv@gmail.com
Zhanxing Zhu  
Changping National Lab & Peking University  
Beijing, China  
zhanxing.zhu@pku.edu.cn
Studying the implicit bias of gradient descent (GD) and stochastic gradient descent (SGD) is critical to unveil the underlying mechanism of deep learning. Unfortunately, even for standard linear networks in regression setting, a comprehensive characterization of the implicit bias is still an open problem. This paper proposes to investigate a new proxy model of standard linear network, rank-1 linear network, where each weight matrix is parameterized as a rank-1 form. For overparameterized regression problem, we precisely analyze the implicit bias of GD and SGD—by identifying a "potential" function such that GD converges to its minimizer constrained by zero training error (i.e., interpolation solution), and further characterizing the role of the noise introduced by SGD in perturbing the form of this potential. Our results explicitly connect the depth of the network and the initialization with the implicit bias of GD and SGD. Furthermore, we emphasize a new implicit bias of SGD jointly induced by stochasticity and over-parameterization, which can reduce the dependence of the SGD's solution on the initialization. Our findings regarding the implicit bias are different from that of a recently popular model, the diagonal linear network. We highlight that the induced bias of our rank-1 model is more consistent with standard linear network while the diagonal one is not. This suggests that the proposed rank-1 linear network might be a plausible proxy for standard linear net.
Gradient Descent (GD) and its stochastic variant, Stochastic Gradient Descent (SGD), are probably the most important optimization techniques in deep learning. To unveil the underlying mechanism of modern neural networks, it is highly fundamental to understand the thrilling and mysterious properties of GD and SGD. Some recent works [24, 18, 3, 22] have made significant efforts in this direction by exploring their implicit bias: among all global minimum, i.e., interpolation solutions, what particular ones will GD and SGD prefer without adding an explicit regularization? This question highlights the crucial roles of these algorithms in the generalization performance of the trained models.
Among the vast number of architectures in deep learning, the first object to investigate is the deep linear network, i.e., without any nonlinear activations,
$$
F (x; W) = W _ {L} \dots W _ {1} x \tag {1}
$$
where  $W_{k}$ 's are weight matrices and  $x$  is the input data. The linear network can be obviously treated as an over-parameterized model of standard linear model,  $\theta^T = W_L \cdots W_1$ . However, the introduction of the over-parameterization brings significant non-convexity, and thus complicates dynamics during learning. Until now, the direct analysis of implicit bias of GD and, particularly, SGD for standard deep linear networks with any depth and initialization on regression problems is still an open problem.
(b)
Figure 1: (a). Standard linear networks. (b). Rank-1 linear networks. One neuron in the middle hidden layer is active and fully connected. (c). Diagonal linear networks. Every neuron is active but not fully connected.
In order to study the implicit bias of linear networks in regression settings, a simplified version of standard linear network, diagonal linear network (i.e., each neuron is only connected with a single neuron of the next layer [26, 22]), was proposed as a proxy to underpin the bias of the standard one. With this simplified model, the solution selected by GD solves a constrained norm minimization problem that interpolates between the  $\ell_1$  and  $\ell_2$  norms up to the initialization scale [26]. For SGD, the solution is closer to that of the sparse regression when compared to GD with the same initialization [22].
However, in Section 3.2, our theoretical analysis shows that the induced implicit bias of diagonal linear networks is not consistent with standard linear network, at least for GD in regression settings. It is then natural to ask: are there any proxy architectures that could produce similar implicit bias to that of standard linear networks? If so, we still expect the new proxy model be amenable to tractable analysis and provide insights for investigating the implicit bias of SGD and a wider spectrum of architectures. This is the main goal of our work.
On the other hand, [11] showed that weight matrices of linear networks for linearly separable classification will become low-rank when trained with GD and logistic loss. [16] also observed such low-rank bias for deep matrix factorization. [8] showed that SGD and weight decay jointly induces a low-rank bias in the weight matrices when training a neural network. In fact, for linear networks with a single output trained with GD, all weight matrices will automatically become rank-1 and will maintain this property during training when the initialization is balanced [11].
Our contributions. Inspired by this low-rank bias and to avoid the complicated analysis of standard linear networks, we propose a novel model rank-1 deep linear network as a plausible proxy of standard linear networks, to reveal the implicit bias of GD and SGD on over-parameterized linear models for regression problems. A depth- $L$  rank-1 deep linear network  $f(x; u, v)$  is defined by
$$
f (x; u, v) := w _ {L} W _ {L - 1} \dots W _ {1} x = \theta^ {T} x, s. t. W _ {k} = u _ {k} v _ {k} ^ {T}, \forall k \in \{1, \dots , L - 1 \} \tag {2}
$$
where  $u_{k} \in \mathbb{R}^{d_{k + 1}}$  and  $v_{k} \in \mathbb{R}^{d_{k}}$  are vectors while  $u$  and  $v$  are denoted as collections of  $u_{k}$  and  $v_{k}$ , respectively. See Figure 1 for the difference between three types of linear networks. In this work we will show that the formulation above could offer us the possibility to understand the bias of GD and SGD for standard linear networks through the lens of results on rank-1 linear networks.
With the proposed model, this paper targets on precisely identifying the "potential" function  $V(\theta)$  such that GD converges to its minimizer constrained by zero training error (i.e., interpolation solution), and further characterizing the role of the noise induced by adding sampling stochasticity to GD, i.e., SGD, in perturbing the form of  $V(\theta)$ . For the convenience of theoretical analysis, we focus on the continuous versions of GD and SGD, i.e., we study gradient flow (GF) and stochastic gradient flow (SGF). In particular, this paper establishes the following findings:
- For GF, we show that the solution implicitly minimizes a potential function  $V(\theta)$  that depends on the initialization and depth subject to  $f(x; u, v)$  achieving zero training error (Theorem 1). The single layer case recovers the standard linear regression results, while a depth larger than one immediately changes the form of  $V(\theta)$ , which clearly connects the implicit bias of GF with the model architecture. We emphasize that our results explicitly reveal how depth and initialization jointly influence the implicit bias of GF.
More importantly, in Theorem 2, by showing the similarity of the implicit bias of GF for standard and rank-1 linear networks, we conclude that rank-1 linear networks are standard linear networks with special initialization when trained with GF, highlighting that our rank-1 linear network is a plausible proxy of standard linear networks. This offers us the possibility to explore the implicit bias of SGD for standard linear networks by analyzing the rank-1 linear nets. On the other hand, the diagonal linear networks exhibit drastically different implicit bias of GF when compared to standard linear networks, e.g., GF prefers  $\ell_2$  minimum norm solution for standard linear networks when the initialization is small while, on the contrary, it prefers such solution when the initialization is large for diagonal linear networks.
- For SGF, we show that the sampling noise brings an extra effect that depends on several hyper-parameters such as the learning rate, batch size and network depth compared to GF. When  $L > 1$ , this extra effect "alleviates" the influence of the initialization, i.e., the final solution reduces its dependence on the initialization. More intriguingly, when  $L = 1$ , i.e., without over-parameterization, this effect brought by SGF immediately disappears. Thus this implicit bias is jointly induced by both model depth and stochasticity from SGD. To the best of our knowledge, we are the first to show such an implicit bias jointly affected by the two factors through the lens of the rank-1 networks, which is also empirically verified on standard linear and nonlinear networks. Our findings on SGF are summarized in Theorem 3.
Organization. In Section 2, we summarize the notations and the setup of our work. Section 3 and Section 4 present our main results where Section 3 focuses on the implicit bias of GD while Section 4 is about SGD. Numerical experiments are presented in Section 5 and we conclude this paper in Section 6. Some technical details are deferred to Appendix.
The study of implicit bias of GD has been pioneered by [24] on linearly separable classification problem and was generalized to GD for deep neural networks and different training strategies [11, 18, 5, 21, 15, 17]. Recently extensive works have made progress in this direction by focusing on the diagonal linear network model. Assuming the existence of a perfect solution in the regression setting, [26] showed that the solution selected by GF interpolates between  $\ell_2$  minimum norm solutions and  $\ell_1$  ones up to the initialization scale. Besides the full-batch gradient descent, [22] then analyzed SGF for diagonal linear networks and concluded that the sampling noise brought by SGF reduces the effective initialization scale when compared with GF, leading its solution to be closer to a sparse one. Aside from the sampling noise induced by SGF, [9, 23] also investigated the influence of the label noise in the diagonal linear networks setting. For GD and SGD with moderate learning rate, [7] studied their implicit bias for diagonal linear networks and revealed the corresponding influence of the finite learning rate.
This paper considers the rank-1 deep linear network to study the implicit bias of GD and SGD. We briefly introduce differences between the settings considered here and those in previous works. For GD on standard linear networks, [3] did not restrict the initialization for the 2-layer case while [28] required the initialization for standard linear networks to be nearly-zero without restricting the number of layers. [25] considered GD for shallow ReLU nets. In this work, we consider linear networks without requirements either on the scale of the initialization or the number of layers. For GD and SGD on the diagonal linear networks [3, 26, 22], we point out that the results are different with that for standard linear networks. Note that this is not to downgrade diagonal linear network, and instead the point is to show that different architectures could induce different implicit bias. The rank-1 linear networks considered here are close to the standard linear networks since it can be seen as standard linear networks with special initialization. Furthermore, the classification problem [24, 18, 5] and linear regression problem [1] have been comprehensively investigated and we focus on the over-parameterized regression. Finally, [4, 10, 19, 27, 9] focused on the flatness of the loss landscape and model parameters while our analysis is for the overall parametrization  $\theta$ .
Notations. Given a dataset of  $n$  samples  $\{(x_i, y_i)\}_{i=1}^n$ ,  $x_i \in \mathbb{R}^d$  represents a  $d$ -dimensional data vector with scalar label  $y_i \in \mathbb{R}$ . We use  $X \in \mathbb{R}^{n \times d}$  to denote the data matrix and use  $y = (y_1, \ldots, y_n)$
to denote the collection of  $y_{i}$ .  $\| \cdot \|$  denotes the  $\ell_2$ -norm and  $\| \cdot \|_F$  is the Frobenius norm.  $\langle a, b \rangle$  is the inner product. For any vector  $u$  and matrix  $A$ , we use  $u(0)$  and  $A(0)$  to denote their initialization. We let  $\operatorname{tr}(\cdot)$  be the trace operator. For the weight matrix  $W_{k} \in \mathbb{R}^{d_{k} \times d_{k + 1}}$ ,  $W_{k;ij}$  is its  $i$ -th row  $j$ -th column element. For a parametric model  $f(x; \theta) = \theta^T x$  with parameters  $\theta$ , we use  $\mathcal{L}(\theta) = \frac{1}{n} \sum_{i}^{n} \ell_i(\theta)$  to represent the empirical loss where  $\ell_i(\theta)$  is the loss function for the sample  $(x_{i}, y_{i})$ .
Over-parameterized regression. We focus on the regression problem where  $n < d$  and assume the existence of solutions that perfectly fit the dataset, i.e., there exist interpolating parameters  $\theta^*$  such that  $\forall i \in \{1, \dots, n\} : \langle x_i, \theta^* \rangle = y_i$ . The quadratic loss  $\ell_i(\theta)$  is used in our setting. The empirical loss is then  $\mathcal{L}(\theta) = \frac{1}{n} \sum_{i=1}^{n} \ell_i(\theta) = \frac{1}{n} \sum_{i=1}^{n} (y_i - \langle x_i, \theta \rangle)^2$ .
Rank-1 deep linear networks. In this paper we consider the rank-1 deep linear network  $f(x; u, v)$  (see Eq. (2)). We use  $u$  and  $v$  to denote the collections of  $u_k$ 's and  $v_k$ 's, respectively. For convenience,
$$
\rho_ {k} = w _ {L} ^ {T} W _ {L - 1} \dots W _ {k + 2} u _ {k + 1}, \rho_ {L - 1} = 1, \text {a n d} \rho_ {- k} = v _ {k} ^ {T} W _ {k - 1} \dots W _ {2} u _ {1}, \rho_ {- 1} = 1 \tag {3}
$$
such that the network can be written as  $f(x;u,v) = \rho_k v_{k+1}^T u_k \rho_{-k} v_1^T x$  for  $k \in \{1, \dots, L-1\}$  when  $L \geq 2$ . Through this paper, we treat the depth  $L$  as a hyper-parameter of the network and try to precisely characterize its role in the implicit bias.
Definition 1 (Balanced initialization for rank-1 linear networks). Given an  $L$ -layer rank-1 linear network Eq. (2), for any  $k \in \{1, \dots, L - 1\}$ , the balanced initialization means that
$$
\frac {\left\langle v _ {k + 1} (0) , u _ {k} (0) \right\rangle^ {2}}{\| v _ {k + 1} (0) \| ^ {2} \| u _ {k} (0) \| ^ {2}} = 1, \| v _ {k + 1} (0) \| = \| u _ {k} (0) \| = \| v _ {1} (0) \|.
$$
We add more discussions on the initialization in Appendix C.1.
3 Equivalence Between Implicit Bias of GD for Standard and Rank-1 Nets
3.1 Implicit bias of GF for rank-1 linear networks
In this section, we characterize the implicit bias of the continuous version of GD for the rank-1 deep linear networks  $f(x; u, v)$  Eq. (2). Note that the model parameters are updated according to the gradient flow
$$
\frac {d u _ {k}}{d t} = - \nabla_ {u _ {k}} \mathcal {L} (\theta), \frac {d v _ {k + 1}}{d t} = - \nabla_ {v _ {k + 1}} \mathcal {L} (\theta) \tag {4}
$$
for  $k \in \{1, \dots, L - 1\}$  and  $dv_{1} / dt = -\nabla_{v_{1}}\mathcal{L}(\theta)$ , which clarifies that  $u_{k}$  and  $v_{k}$ , rather than  $W_{k}$ , are the model parameters.
Previous work [26] showed that the solution selected by GF interpolates between the  $\ell_1$  minimum norm solution and  $\ell_2$  one depending on the initialization scale for diagonal linear networks. In this section we examine whether this is the case for the rank-1 deep linear network with depth  $L$ . For convenience, we define
$$
\Omega_ {L} = \frac {2 L}{2 L - 1}, \lambda_ {L} = \frac {2 (L - 1)}{2 L - 1} \tag {5}
$$
where  $L$  is the number of layers. Recall that  $\theta(0)$  represents the initialization of  $\theta$ , we now state the main theorem regarding the implicit bias of GF for rank-1 linear networks below:
Theorem 1 (Implicit bias of GF for rank-1 linear networks). For a rank-1 linear network, if the initialization is balanced across layers (Definition 1) and if the gradient flow solution  $\theta (\infty)$  satisfies that  $X\theta (\infty) = y$ , then gradient flow converges to a minimizer of the potential function  $V(\theta)$ :
$$
\theta (\infty) = \underset {\theta} {\arg \min } V (\theta), \quad s. t. X \theta = y,
$$
$$
V (\theta) = \frac {1}{\Omega_ {L}} \| \theta \| ^ {\Omega_ {L}} - \theta^ {T} \frac {\theta (0)}{\| \theta (0) \| ^ {\lambda_ {L}}}. \tag {6}
$$
Figure 2: The plot of  $V(\theta)$  for different numbers of layers and initialization scales in a two-dimensional parameter space. The initialization in the first column is [0.08, 0.06] and is multiplied by different scale factors for the other two columns. In the first column where the initialization is nearly zero, GD has similar implicit bias for rank-1 linear networks of different layers. When we increase the initialization scale, the shape of the contour for linear regression potential function does not change, while for linear networks as shown in the last column, the contour of the potential function gradually presents a sharp angle as we increase the number of layers.
While previous works focused on 2-layer standard linear networks [3] or multiple-layer linear networks with nearly zero initialization [28], i.e.,  $\| \theta (0)\| \approx 0$ , Theorem 1 builds the potential function  $V(\theta)$  that depends on both the initialization  $\theta (0)$  and the depth of the network  $L$  explicitly. We also plot the contours of  $V(\theta)$  for different numbers of layers and scales of initialization in Fig. 2 in a 2-dimensional space. Theorem 1 clearly reveals how over-parameterization and the initialization of  $\theta$  guide GD to select different solutions.
Effects of depth. When  $L = 1$ ,  $V(\theta)$  becomes  $\frac{1}{2} \| \theta \|^2 - \langle \theta, \theta(0) \rangle$  which is the same as the potential of the least square  $\| \theta - \theta(0) \|^2$  except for a constant term. When  $L > 1$ ,  $V(\theta)$  is no longer an Euclidean distance. The form of  $V(\theta)$  depends on the depth and GD will favor different interpolating solutions. A particular interesting case is when  $L \to \infty$  where we have  $\Omega_L \to 1$  and  $\lambda_L \to 1$ . As a result,  $V(\theta)$  becomes  $V(\theta) \to \| \theta \| - \langle \theta, \theta(0) / \| \theta(0) \| \rangle$ , which reflects the difference between the norm of  $\theta$  and the norm of its projection on the direction of the initialization  $\theta(0)$ . Therefore, the direction of the initialization  $\theta(0)$  matters for the potential function  $V(\theta)$  and the final solution  $\theta(\infty)$  while they are rather less sensitive to the scale of  $\theta(0)$ . This may serve as a benefit of the over-parameterization in the sense that it makes the network more stable to different scales of the initialization. As a comparison, both direction and scale of  $\theta(0)$  are crucial for the least square potential function  $\| \theta - \theta(0) \|^2$ .
Effects of initialization. When  $L$  is finite, to inspect the effects of the initialization, we can rewrite  $V(\theta) = \| \theta \|_{\Omega_L}^{\Omega_L} / \Omega_L - \| \theta (0)\|^{1 / (2L - 1)}\langle \theta ,\theta (0) / \| \theta (0)\| \rangle$ . The second term will be more important when the initialization scale  $\| \theta (0)\|$  is getting larger thus the initialization affects the implicit bias of GD more significantly. On the other hand, as  $\| \theta (0)\| \to 0$ , the second term vanishes. Thus we have the following corollary:
Corollary 1.1. Under conditions of Theorem 1, if we further assume that the initialization is infinitesimal  $\| \theta (0)\| \to 0$  and the depth is finite, then the GF solution  $\theta (\infty)$  is an  $\ell_2$ -norm minimization solution:
$$
\theta (\infty) = \min  _ {\theta} \| \theta \| \quad s. t. X \theta = y.
$$
We now summarize the effects of initialization on the training regime as follows. (i). According to [26], for any  $D$ -homogeneous model ( $D$  is a positive integer), the lazy regime (or NTK regime) is reached for large initialization. Since the rank-1 linear network is a homogeneous model, the NTK regime is reached for large initialization and the implicit bias is given by the RKHS norm predictor accordingly; (ii). For vanishing initialization, according to Corollary 1.1, an  $\ell_2$ -norm minimization predictor which can not be captured by the NTK kernel is returned. Therefore, as the initialization becomes smaller, we "escape" from the lazy regime and falls into the "Anti-NTK" regime defined
Figure 3: GD maintains the shape of rank-1 initialization for the weight matrices (the upper panel), but destroys that of the diagonal initialization for standard linear networks (the lower panel), as formally described by Proposition 1.
by [3]. Furthermore, based on Corollary 1.1, it is now worth to mention that, assuming vanishing initialization, linear regression (i.e., 1-layer linear networks), 2-layer standard linear networks [3] and rank-1 linear networks with finite depth share a similar implicit bias if trained with GD. And the first column of Fig. 2 shows that rank-1 linear networks with different number of layers exhibit potential contours with a similar shape.
3.2 Comparison between different architectures
The aforementioned phenomenon that GD will return an  $\ell_2$ -norm minimization solution for both 2-layer standard linear networks and rank-1 linear networks with small initialization drives us to further investigate the comparison and similarities between different network architectures. For diagonal linear networks, GD prefers  $\ell_1$ -norm minimization solution with nearly-zero initialization, which is drastically different from the case for rank-1 and standard linear networks. On the contrary, it prefers  $\ell_2$ -norm minimization solution when the initialization is sufficiently large [26]. This inconsistency begs us to ask the question will standard linear networks and rank-1 linear networks share a similar implicit bias of GD when the initialization is large? To answer this question, in the following, we first analyze the implicit bias of GD for standard linear networks  $f(x;W) = w_L^T W_{L - 1}\dots W_1x$  with any depth  $L$ , which corresponds to the parameterization  $\theta^T = w_L^T W_{L - 1}\dots W_1$ , with balanced initialization (Definition 2).
Theorem 2 (Implicit bias of GF for standard linear networks with any initialization). For an  $L$ -layer standard linear network, if the initialization is balanced across layers, i.e.,  $W_{k+1}^{T}(0)W_{k+1}(0) = W_{k}(0)W_{k}^{T}(0)$  for all layers, and if the gradient flow solution  $\theta(\infty)$  satisfies that  $X\theta(\infty) = y$ , then gradient flow converges to a minimizer of the potential function  $V(\theta)$ :
$$
\theta (\infty) = \underset {\theta} {\arg \min } V _ {s t d} (\theta), \quad s. t. X \theta = y,
$$
$$
V _ {s t d} (\theta) = \frac {L}{L + 1} \| \theta \| ^ {\frac {L + 1}{L}} - \theta^ {T} \frac {\theta (0)}{\| \theta (0) \| ^ {\frac {L - 1}{L}}}, \tag {7}
$$
Although we state before that the parameters of a rank-1 deep linear network are  $u_{k}$ 's and  $v_{k}$ 's rather than  $W_{k}$ 's, the form of the potential for rank-1 linear networks Eq. (6) is very similar to that of standard linear networks Eq. (7):  $V(\theta)$  for an  $L$ -layer rank-1 linear network is the same as  $V_{std}(\theta)$  for a  $(2L - 1)$ -layer standard linear network. In the following, we explain the reason behind this fact and conclude that rank-1 linear network can be seen as a qualified proxy for the standard linear network when studying the implicit bias of GD, and potentially SGD.
Rank-1 linear networks are standard linear networks with special initialization. We first present a useful proposition regarding a special kind of initialization for standard linear networks.
Proposition 1 (Effects of diagonal and rank-1 initialization for standard linear networks). Given an  $L$ -layer standard linear network  $f(x;W) = w_{L}^{T}W_{L - 1}\dots W_{1}x$  where  $W_{k}\in \mathbb{R}^{d_{k + 1}\times d_{k}}$  for  $k\in \{1,\ldots ,L\}$ , if the weights are initialized such that only one column of  $W_{k}$  is non-zero when  $k$  is even and only one row of  $W_{k}$  is non-zero otherwise, i.e., for an integer  $p$