SlowGuess/NeurIPS_2023_OCR huggingface dataset:
$$
\begin{array}{l} \forall k = 2 p + 1 \in \{1, \dots , L \}: W _ {k; i j} (0) = 0 \text {i f} i \neq c _ {k}, \text {w h e r e} c _ {k} \in \{1, \dots d _ {k + 1} \} \\ \forall k = 2 p \in \{1, \dots , L \}: W _ {k; i j} (0) = 0 \text {i f} j \neq c _ {k}, \\ \end{array}
$$
then for any  $t > 0$ :
$$
\forall k = 2 p + 1 \in \{1, \dots , L \}: W _ {k; i j} (t) = 0 \text {i f} i \neq c _ {k};
$$
$$
\forall k = 2 p \in \{1, \dots , L \}: W _ {k; i j} (t) = 0 \text {i f} j \neq c _ {k}.
$$
Furthermore, if the weights are initialized as the diagonal shape, i.e.,  $\forall k\in \{1,\ldots ,L - 1\}$ ,  $W_{k;i j}(0) = 0$  if  $i\neq j$ , then  $GD$  does not maintain the diagonal shape of weight matrices.
Remark. A  $(2L - 1)$ -layer standard linear network initialized as in Proposition 1 with  $c_k = 1$  for all layers has the same formulation as our  $L$ -layer rank-1 architecture. According to Proposition 1, the special structure of the initialization of such standard linear network will be maintained if we run GD, which suggests that it will always have the same formulation as our  $L$ -layer rank-1 linear network, see the upper panel of Fig. 3. Therefore, an  $L$ -layer rank-1 linear network can be seen as a  $(2L - 1)$ -layer standard one with special initialization, and they exhibit similar implicit bias of GD. On the other hand, if the standard linear network is initialized as the diagonal shape as in the diagonal net, GD will not maintain this property (the lower panel of Fig. 3), i.e., the diagonal structure cannot be seen as a standard one. In this sense, the diagonal network exhibits special implicit bias particularly due to its structure, while the rank-1 linear network is a more qualified proxy of standard linear networks.
4 Implicit bias of SGD for Rank-1 Linear Networks
Recently, [23, 22] developed the characterization of the implicit bias of GD with label noise and SGD in diagonal linear networks. However, as mentioned earlier, its diagonal structure is special in the sense that the corresponding results can not be generalized to the case for standard linear networks. On the contrary, our results in Section 3 reveal that rank-1 linear networks can be seen as standard linear networks with special initialization. In this section, to take a step forward towards understanding the implicit bias of SGD for standard linear networks, we explore the continuous part of SGD, stochastic gradient flow (SGF), for the rank-1 linear networks in the over-parameterized regression setting. We begin with the introduction of the definition of SGD and our modelling techniques.
SGD. Unlike the full-batch GD where the parameters are updated according to Eq. (4), the SGD dynamics is
$$
u _ {k} (t + 1) = u _ {k} (t) - \frac {\eta}{b} \sum_ {i \in \mathcal {B} _ {t}} \nabla_ {u _ {k}} \ell_ {i} (\theta), v _ {k + 1} (t + 1) = v _ {k + 1} (t) - \frac {\eta}{b} \sum_ {i \in \mathcal {B} _ {t}} \nabla_ {v _ {k + 1}} \ell_ {i} (\theta) \tag {8}
$$
for  $k \in \{1, \dots, L - 1\}$  where  $t$  denotes the iteration step,  $\eta$  is the learning rate,  $b$  is the batch-size, and  $\mathcal{B}_t$  consists of  $b$  points randomly sampled from the uniform distribution  $\mathcal{U}[1, n]$ .
Continuous Modelling of SGD. The continuous modelling techniques for SGD have been widely applied in recent works [1, 9, 23, 22] to study the dynamics of SGD. In our setting, recalling the definition of  $\rho_{k}$  and  $\rho_{-k}$  in Eq. (3), the continuous counterpart of SGD, SGF, is given by the following set of stochastic differential equations (SDE):
$$
d u _ {k} = - \frac {2}{n} v _ {1} ^ {T} X ^ {T} r \rho_ {k} \rho_ {- k} v _ {k + 1} d t + 2 \sqrt {\frac {\eta \mathcal {L}}{n b}} \left(\rho_ {k} \rho_ {- k}\right) v _ {k + 1} v _ {1} ^ {T} X ^ {T} d \mathcal {W} _ {t} \tag {9}
$$
$$
d v _ {k + 1} = - \frac {2}{n} v _ {1} ^ {T} X ^ {T} r \rho_ {k} \rho_ {- k} u _ {k} d t + 2 \sqrt {\frac {\eta \mathcal {L}}{n b}} \left(\rho_ {k} \rho_ {- k}\right) u _ {k} v _ {1} ^ {T} X ^ {T} d \mathcal {W} _ {t} \tag {10}
$$
where  $r = (f(x_1;u,v) - y_1,\dots ,f(x_n;u,v) - y_n)^T\in \mathbb{R}^n$  is the residual, and  $\mathcal{W}_t$  is a standard Brownian motion in  $\mathbb{R}^n$ . For the parameterization of  $\theta$  (Eq. (2)), we now aim to characterize the implicit bias of SGD by showing the existence of a function  $V(\theta)$  such that the solution of the stochastic dynamics converges to its minimizer under the constraint of zero-training loss as follows.
Theorem 3 (Implicit bias of SGF for rank-1 linear networks). For the rank-1 linear network Eq. (2) that is trained with SGF (Eq. (9) and (10)) in the over-parameterization regression, if the initialization is balanced across layers, then the dynamics of  $\theta$  gives us
$$
d \nabla_ {\theta} V ^ {S} (\theta , t) = - \nabla_ {\theta} \mathcal {L} d t + 2 \sqrt {\frac {\eta | \xi | \mathcal {L}}{n b}} X ^ {T} d \mathcal {W} _ {t} \tag {11}
$$
(a) The implicit bias jointly induced by stochasticity and architectures "alleviates" the effects of initialization
Figure 4: (a) The blue trajectory is for gradient flow (GF) that converges to  $\theta_{\mathrm{GF}}^{*}$  (blue square). The green trajectory that converges to  $\theta_{\mathrm{SGF}}^{*}$  (green square) is for stochastic gradient flow (SGF).  $\theta(0)$  denotes the initialization of  $\theta$  and  $\theta_{\ell_2}^{*}$  (red square) is the  $\ell_2$ -norm minimum solution. Black dot marks the origin. Compared to  $\theta_{\mathrm{GF}}^{*}$ ,  $\theta_{\mathrm{SGF}}^{*}$  is closer to  $\theta_{\ell_2}^{*}$  since its dependence on the initialization is reduced. Note that when  $\theta(0)$  is nearly zero (the left figure), both  $\theta_{\mathrm{GF}}^{*}$  and  $\theta_{\mathrm{SGF}}^{*}$  are close to  $\theta_{\ell_2}^{*}$  (Corollary 1.1). (b) The red arrow denotes the direction of  $\alpha(\infty)$  while the black one denotes that of  $\theta(0)$ , which, as shown in the figure, is particularly important to the direction of  $\alpha(\infty)$ . See more experiments in Appendix A.3.
(b) The trajectory of  $\alpha$  starting from the origin converging to  $\alpha (\infty)$
$$
V ^ {\mathcal {S}} (\theta , t) = \frac {1}{\Omega_ {L}} \| \theta \| ^ {\Omega_ {L}} + \frac {\theta^ {T} \theta (0)}{\| \theta (0) \| ^ {\lambda_ {L}}} - \frac {2 \lambda_ {L} \eta \theta^ {T}}{n b} \int_ {0} ^ {t} \frac {\mathcal {L} (\theta (s)) \operatorname {t r} \left(P _ {\perp} (\theta (s)) X ^ {T} X\right)}{\| \theta (s) \| ^ {2 - \lambda_ {L}}} \theta (s) d s \tag {12}
$$
with  $P_{\perp}(\theta) = I - \theta \theta^{T} / \| \theta \|^{2}$  being the orthogonal projection operator of  $\theta$ . Furthermore, if the SGF solution  $\theta (\infty)$  satisfies that  $X\theta = y$ , then the model parameter  $\theta$  converges to a minimizer of  $V^{\mathcal{S}}(\theta ,\infty)$  as  $t\to \infty$ :
$$
\theta (\infty) = \underset {\theta} {\arg \min } V ^ {\mathcal {S}} (\theta , \infty), \quad s. t. X \theta = y.
$$
One can immediately notice that the R.H.S of Eq. (11) includes a noise term, which can be interpreted as that the model parameter  $\theta$  follows a mirror flow with a noise term. Furthermore, Eq. (12), which characterizes the optimization geometry, depends on time explicitly. This partly reflects its stochastic nature—the optimization trajectory is not deterministic. Furthermore, as in [7], it is also possible to generalize the current vanishing learning rate results to the moderate learning rate analysis by deriving a stochastic mirror descent recursion with time varying potentials.
Implicit bias jointly induced by stochasticity and architectures. The sampling noise of SGD introduces an extra term (the last term of Eq. (12)) that depends on several parameters such as the learning rate and the data matrix  $X$  when compared with that of GD (Eq. (6)). Interestingly, this extra term only exists when the model is simultaneously over-parameterized and trained with the existence of the sampling noise. If the model is not over-parameterized, i.e., standard linear regression when  $L = 1$ , then  $\lambda_{L} = 0$  and this term disappears. On the other hand, if there is no sampling noise this term also vanishes. Indeed, the  $L = 1$  case has been widely studied by recent works [1], and it turns out that GD and SGD share similar implicit bias when there is no over-parameterization. Theorem 3 confirms this phenomenon and, more importantly and intriguingly, reveals a new connection between the architecture-induced over-parameterization and the implicit bias of the optimization algorithms.
"Alleviating" the effects of initialization. When  $L \to \infty$ , the last term of Eq. (12) brought by the sampling noise can be seen as "alleviating" the influence of the initialization  $\theta(0)$  such that the training dynamics reduces its dependence on the initialization (Fig. 4(a)). Due to the difficulty of explicitly solving the stochastic integral in Eq. (12), we give here a qualitative interpretation of this alleviating effect. As  $L \to \infty$ , i.e., infinitely deep linear network, we have  $\lambda_L \to 1$ , therefore the integral in Eq. (12) becomes  $-\frac{2\eta}{nb} \int_0^\infty \mathcal{L}(\theta) \mathrm{tr}\left(P_\perp(\theta) X^T X\right) \frac{\theta}{\|\theta\|} ds$ . In a  $d$ -dimensional space, we let  $\alpha(t) := \int_0^t \mathcal{L}(\theta) \mathrm{tr}\left(P_\perp(\theta) X^T X\right) \theta / \|\theta\| ds$  be the position of a particle  $A$  that starts from the origin. Then the integral of Eq. (12) amounts to the final position  $\alpha(\infty)$  of  $A$ , which moves
(b)
Figure 5: For different  $\| \theta (0)\|$ : (a)  $D(\theta (\infty),\theta_{\ell_2}^*)$  for rank-1 linear networks. (b)  $D(\theta (\infty),\theta^{*})$  for rank-1 linear networks. (c)  $D(\theta (\infty),\theta_{\ell_2}^*)$  for standard linear networks. (d)  $D(\theta (\infty),\theta^{*})$  for standard linear networks.
along the direction of  $\theta(t) / \| \theta(t)\|$  with speed proportional to  $\mathcal{L}(\theta(t))$ $\operatorname{tr}(P_{\perp}(\theta(t))X^TX)$ . Since  $\| X\|_F^2 - \| X\|_2^2 \leq \operatorname{tr}(P_{\perp}(\theta(t))X^TX) \leq \| X\|_F^2$  where  $\| X\|_F^2$  is finite, it is highly likely that the direction of  $\alpha(\infty)$  heavily depends on  $\theta(0)$ . This is because the velocity magnitude, which is proportional to the value of empirical loss  $\mathcal{L}(\theta(t))$ , is large when  $t = 0$  and quickly shrinks along its trajectory. As a result, in Eq. (12), the effect coming from the initialization  $\theta^T\theta(0)/\|\theta(0)\|$  is alleviated by the extra term  $-\theta^T\alpha(\infty)$  induced by the sampling noise and over-parameterization. In this sense,  $V^{\mathcal{S}}(\theta,\infty)$  is closer to a simple  $\ell_2$ -norm  $\|\theta\|$  when compared with GD, i.e., the SGD solution is more likely to be an  $\ell_2$  minimum norm solution when compared to GD. We also present the trajectory of  $\alpha$  in Fig. 4(b). The trajectory is obtained by first training a rank-1 linear network using SGD for 5000 iterations and computing  $\alpha(t)$  at every step followed by a projection of the trajectory  $\{\alpha(t)\}_{t=1}^{5000}$  into a 2-dimensional space according to the method described in [13]. It can be seen that the direction of the final position  $\alpha(\infty)$  is close to that of  $\theta(0)$  ( $\langle \alpha(\infty), \theta(0) \rangle / (\|\alpha(\infty)\|\theta(0)) = 0.9314$ ). We also conduct additional experiments in Appendix A.3 to further verify our finding.
Comparison with diagonal linear networks. In [22], the authors showed that the extra effects of the sampling noise of SGD is equivalent to multiplying a shrinking coefficient which depends on the training dynamics to the initialization scale. This effect leads the solution of SGD to be closer to that of sparse regression when compared with GD. Our Theorem 3 does not show such bias that the solution interpolates between the  $\ell_2$ -norm and  $\ell_1$ -norm minimization solution. Therefore the conclusion of implicit bias of SGD for diagonal linear networks can not be directly applied to rank-1 linear networks and standard linear networks. This reveals an important finding that the implicit bias of GD or SGD is strongly tied with network architecture. On the other hand, similar to the case of diagonal linear networks, Theorem 3 also reveals an initialization cancellation effect induced by the SGD sampling noise. It is interesting for future work to explore whether this effect can be generalized to other architectures and can be seen as a special benefit brought by SGD.
In this section, we consider the over-parameterized regression problem with rank-1 linear, standard linear, and non-linear networks for different initialization scales to verify our theoretical claims. We define the distance  $D(\theta_1,\theta_2) = \| \theta_1 - \theta_2\| ^2 /\| \theta_2\| ^2$  to measure the relative difference between  $\theta_{1}$  and  $\theta_{2}$ . For a linear network and the parameterization  $\theta^T = w_L^T W_{L - 1}\dots W_1$ , we use  $\theta (\infty)$  to denote the solution returned by the optimization algorithms (GD or SGD) and  $\theta (0)$  to denote the corresponding initialization.  $\theta^{*}$  is the ground truth solution, and  $\theta_{\ell_2}^*$  is the  $\ell_{2}$  minimum norm solution (Corollary 1.1). Details of the experiments and more numerical experiments are deferred to Appendix A.1 and we focus on the results here.
Rank-1 linear networks. As shown in Fig. 5(a), for all different initialization scales  $\| \theta (0)\|$ $D(\theta (\infty),\theta_{\ell_2}^*)$  is smaller if  $\theta (\infty)$  is returned by SGD, i.e., the SGD solution is closer to the  $\ell_2$  minimum norm solution  $\theta_{\ell_2}^*$  when compared to the GD solution. Similarly, Fig. 5(b) shows a benefit of such alleviating dependence of initialization bias of SGD: its solution is closer to the ground truth solution  $\theta^{*}$  when compared to the GD solution for different  $\| \theta (0)\|$ . Furthermore, in Fig. 6(a), we show  $D(\theta (t),\theta_{\ell_2}^*)$  along training, which further reveals that the final solution returned by SGD is closer to  $\theta_{\ell_2}^*$ . Note that when the initialization scales are small, both GD solution and SGD solution
are close to  $\theta_{\ell_2^5}$ . These experiments well support our theoretical findings. Besides, we present additional experiments to compare diagonal and rank-1 linear networks in Appendix A.2.
(b)
Figure 6: Numbers in the bracket denote the scale of the initialization. (a).  $D(\theta(t), \theta_{\ell_2}^*)$  along training for rank-1 linear networks. (b).  $D(\theta(t), \theta_{\ell_2}^*)$  along training for standard linear networks. (c). Test error along training for non-linear networks.
Standard linear networks and non-linear networks. Proposition 1 states similarity between rank-1 and standard linear nets, therefore they have similar results. We conduct numerical experiments for standard linear networks as in the case of rank-1 linear networks and show in Fig. 5(c) and 5(d) that  $D(\theta(\infty), \theta_{\ell_2}^*)$  and  $D(\theta(\infty), \theta^*)$  are also smaller if we run SGD, which supports the generalization of conclusion of rank-1 linear network to standard linear networks.  $D(\theta(t), \theta_{\ell_2}^*)$  along training plotted in Fig. 6(b) further supports this phenomenon. For non-linear networks (Fig. 6(c)), we report the test error on a newly sampled test set for both GD and SGD along training, where SGD solutions have smaller test error when compared to GD solutions.
Our work proposes the rank-1 linear network that is a plausible proxy of standard linear networks. We analyze the implicit bias of GD and SGD for this new net and find that it approximates standard linear networks better than diagonal linear networks. We further reveal the joint role of over-parameterization and stochasticity in characterizing the implicit bias of SGD. Similar to the diagonal linear networks, our results also reveal an initialization alleviating effect of SGD sampling noise, suggesting a future direction that investigates whether such effect is general across different architectures. See Appendix B for more discussions.
**Limitation.** We do not generalize the analysis to non-linear networks due to its lack of the form of  $\theta$  as in our current approach. Furthermore, an exact characterization of the stochastic integral is absent.
B. Lyu thanks Di Wang for useful discussion. Z. Zhu is supported by Beijing Nova Program (No. 202072) from Beijing Municipal Science Technology Commission.
[1] Alnur Ali, Edgar Dobriban, and Ryan Tibshirani. The implicit regularization of stochastic gradient flow for least squares. In International Conference on Machine Learning, 2020.
[2] Sanjeev Arora, Nadav Cohen, and Elad Hazan. On the optimization of deep networks: Implicit acceleration by overparameterization. In arXiv: 1802.06509, 2018.
[3] Shahar Azulay, Edward Moroshko, Mor Shpigel Nacson, Blake Woodworth, Nathan Srebro, Amir Globerson, and Daniel Soudry. On the implicit bias of initialization shape: beyond infinitesimal mirror descent. In NeurIPS, 2021.
[4] Xiang Cheng, Dong Yin, Peter Bartlett, and Michael Jordan. Stochastic gradient and Langevin processes. In International Conference on Machine Learning, 2020.
[25] Francis Williams, Matthew Trager, Claudio Silva, Daniele Panozzo, Denis Zorin, and Joan Bruna. Gradient dynamics of shallow univariate relu networks. In NeurIPS, 2019.
[26] Blake Woodworth, Suriya Gunasekar, Jason D Lee, Edward Moroshko, Pedro Savarese, Itay Golan, Daniel Soudry, and Nathan Srebro. Kernel and rich regimes in overparametrized models. In Conference on Learning Theory, 2020.
[27] Lei Wu, Chao Ma, and Weinan E. How sgd selects the global minima in over-parameterized learning: A dynamical stability perspective. In NeurIPS, 2018.
[28] Chulhee Yun, Shankar Krishnan, and Hossein Mobahi. A unifying view on implicit bias in training linear neural networks. In International Conference on Learning Representations, 2021.
Organization of Appendix. In Appendix A we present details of the numerical experiments and additional numerical experiments. We give more discussions of our work in Appendix B. Appendix C provides missing technical details of Section 3 while Appendix D provides those of Section 4.
A Details of Numerical Experiments and Additional Experiments
In A.1, we present the details of the numerical experiments. We compare the rank-1 linear networks with the diagonal linear networks empirically in A.2. Finally, in A.3, we conduct additional experiments to further verify the "alleviting" effect of the SGD sampling noise mentioned in Theorem 3. In A.4, we conduct experiments when the balanced initialization condition is not satisfied.
Data. We conduct over-parameterized regression with different linear networks. For the dataset  $\{(x_i, y_i)\}_{i=1}^n$  where  $x_i \in \mathbb{R}^d$  and  $y_i \in \mathbb{R}$ , we set  $n = 40, d = 100$  and  $x_i \sim \mathcal{N}(0, I)$ . For  $i \in \{1, \dots, n\}$ ,  $y_i$  is generated by  $y_i = \theta^{*T} x_i$  where  $\theta^* \in \mathbb{R}^d$ , i.e.,  $\theta^*$  is the ground truth solution. We let 20 components of  $\theta^*$  be informative.
A.1 Details of Numerical Experiments in Section 5
Figure 7:  $D(\theta(t), \theta_{\ell_2^*})$  along training for rank-1 linear networks when the initialization is extremely large. SGD solution is closer to  $\theta_{\ell_2^*}$  when compared to the GD solution for rank-1 linear networks.
We now present the details of numerical experiments conducted in Section 5.
$\ell_2$  minimum norm solution. To get the  $\ell_2$  minimum norm solution  $\theta_{\ell_2}^*$ , we train a single layer linear network with zero initialization using GD for 20000 iterations, since GD will return an  $\ell_2$  minimum norm solution solution in this case according to Corollary 1.1 and [26].
Rank-1 linear networks. For the rank-1 linear network  $f(x; u, v) = w_{L}^{T} W_{L - 1} \cdots W_{1} x$  where  $W_{k} = u_{k} v_{k}^{T} \in \mathbb{R}^{d_{k} \times d_{k + 1}}$ , we let  $L = 3$  and  $\forall k \in \{1, \dots, L\} : d_{k} = 100$ . The learning rate is  $10^{-3}$  and the batch size is 4 if we run SGD. We construct different rank-1 linear networks as follows: for a randomly sampled  $\tilde{\theta} \in \mathbb{R}^{100}$ , we let the initialization  $\theta_{i}(0)$  of the  $i$ -th rank-1 networks have the same direction as  $\tilde{\theta}$  but with different scales. We then train each rank-1 linear network with GD and SGD, respectively, for 20000 iterations. In particular:
1. Fig. 5(a) presents the results of the distances between  $\theta_{\ell_2}^*$  and GD and SGD solutions, respectively, of each trained rank-1 networks with different initialization scales.
2. In Fig. 5(b), we measure the distances between  $\theta^{*}$  and GD and SGD solutions, respectively, for each trained rank-1 networks with different initialization scales.
3. Fig. 6(a) plots the distances between  $\theta_{\ell_2}^*$  and the model parameters  $\theta$  along training when the initialization scales are different for both GD and SGD. The numbers in the bracket denote  $\| \theta (0)\|$ .
4. Fig. 7 is about the distances between  $\theta_{\ell_2}^*$  and the model parameters  $\theta$  along training for both GD and SGD when  $\| \theta (0)\|$  is extremely large.
Standard linear networks. For the standard linear network  $f(x;W) = w_{L}^{T}W_{L - 1}\dots W_{1}x$  where  $W_{k}\in \mathbb{R}^{d_{k}\times d_{k + 1}}$ , we let  $L = 4$  and  $\forall k\in \{1,2,3\} :d_k = 100$ . The learning rate is  $10^{-3}$  and the batch size is 4 if we run SGD. Other settings are similar to that of rank-1 linear networks.
1. In Fig. 5(c), we plot the distances between  $\theta_{\ell_2}^*$  and GD and SGD solutions, respectively. Similar to the case of rank-1 linear networks, for all initialization scales,  $D(\theta(\infty), \theta_{\ell_2}^*)$  is smaller if the network is trained with SGD when compared to GD.
2. Fig. 5(d) presents the results of the distances between  $\theta^{*}$  and GD and SGD solutions.
3. Fig. 6(b) plots the distances between  $\theta_{\ell_2}^*$  and the model parameters  $\theta$  along training when the initialization scales  $\| \theta (0)\|$  are different for both GD and SGD. The numbers in the bracket denote  $\| \theta (0)\|$ .
Non-linear networks. For the non-linear network  $f(x;W) = w_L^T\sigma (W_{L - 1}\dots \sigma (W_1x))$  where  $W_{k}\in \mathbb{R}^{d_{k}\times d_{k + 1}}$ , we let  $L = 4$  and  $\forall k\in \{1,2,3\} :d_k = 100$ . The learning rate is  $10^{-3}$  and the batch size is 4 if we run SGD. We use the ReLU activation  $\sigma (x) = \mathrm{ReLU}(x)$ . We use the same dataset as in the experiments of rank-1 linear networks. Since the non-linear networks do not have the overall parameterization of  $\theta$  as in the linear networks case, to measure the initialization scale, we first straight all weight matrices to vectors and stack them to get a single vector, then we calculate the  $\ell_2$  norm of this vector as the scale of the initialization of a non-linear network, i.e., we use  $\sqrt{\left(\sum_{k}\|W_{k}(0)\|_{F}^{2}\right)}$  as the initialization scale where  $\| \cdot \| _F$  is the Frobenius norm. Due to the same reason, we can not measure quantities such as  $D(\theta ,\theta_{\ell_2}^*)$ , therefore, we report the test error of the model in a newly sampled test set instead. Fig. 6(c) plots the test error of the model along training when the initialization scales are different for both GD and SGD. The numbers in the bracket denote initialization scales.
A.2 Additional Experiments of Comparison with Diagonal Nets