Catastrophic Fisher Explosion: Early Phase Fisher Matrix Impacts Generalization
Authors: Stanislaw Kamil Jastrzebski
Devansh Arpit
Oliver Åstrand
Giancarlo Kerg
Huan Wang
Caiming Xiong
richard socher
Kyunghyun Cho
Krzysztof J. Geras
Venue:  (2021)
Keywords: early phase of training
implicit regularization
SGD
learning rate
batch size
Hessian
Fisher Information Matrix
curvature
gradient norm

Abstract: The early phase of training has been shown to be important in two ways for deep neural networks. First, the degree of regularization in this phase significantly impacts the final generalization. Second, it is accompanied by a rapid change in the local loss curvature influenced by regularization choices. Connecting these two findings, we show that stochastic gradient descent (SGD) implicitly penalizes the trace of the Fisher Information Matrix (FIM) from the beginning of training. We argue it is an implicit regularizer in SGD by showing that explicitly penalizing the trace of the FIM can significantly improve generalization. We further show that the early value of the trace of the FIM correlates strongly with the final generalization. We highlight that in the absence of implicit or explicit regularization, the trace of the FIM can increase to a large value early in training, to which we refer as catastrophic Fisher explosion. Finally, to gain insight into the regularization effect of penalizing the trace of the FIM, we show that it limits memorization by reducing the learning speed of examples with noisy labels more than that of the clean examples, and 2) trajectories with a low initial trace of the FIM end in flat minima, which are commonly associated with good generalization.

Under review as a conference paper at ICLR 2021
CATASTROPHIC FISHER EXPLOSION: EARLY PHASE FISHER MATRIX IMPACTS GENERALIZATION
Anonymous authors Paper under double-blind review
ABSTRACT
The early phase of training has been shown to be important in two ways for deep neural networks. First, the degree of regularization in this phase significantly impacts the final generalization. Second, it is accompanied by a rapid change in the local loss curvature influenced by regularization choices. Connecting these two findings, we show that stochastic gradient descent (SGD) implicitly penalizes the trace of the Fisher Information Matrix (FIM) from the beginning of training. We argue it is an implicit regularizer in SGD by showing that explicitly penalizing the trace of the FIM can significantly improve generalization. We further show that the early value of the trace of the FIM correlates strongly with the final generalization. We highlight that in the absence of implicit or explicit regularization, the trace of the FIM increase Fisher explosion. Finally, to gain insight into the regularization effect of penalizing the trace of the FIM, we show that limits memorization by reducing the learning speed of examples with noisy labels more than that of the clean examples, and 2) trajectories with a low initial trace of the FIM end in flat minima, which are commonly associated with good generalization.
large value early in training, to which we refer catastrophic
INTRODUCTION
Implicit regularization in gradient- based training of deep neural networks (DNNs) remains rel- atively poorly understood, despite being considered critical component in their empirical suc- cess (Neyshabur, 2017; Zhang et al., 2016 Jiang et al., 2020b). Recent work suggests that the early phase of training of DNNs might hold the key to understanding these implicit regularization effects. Golatkar et al. (2019); Keskar et al. (2017); Sagun et al. (2018); Achille et al. (2019) show that by introducing regularization later, a drop in performance due to lack of regularization in this phase is hard to recover from, while on the other hand, removing regularization after the early phase has relatively small effect on the final performance.
Other works show that the early phase of training also has dramatic effect the trajectory in terms of properties such as the local curvature of the loss surface or the gradient norm (Jastrzebski et al., 2020; Frankle et al., 2020). In partieular, Achille et al. (2019); Jastrzębski et al. (2019); Golatkar et al. (2019); Lewkowycz et al. (2020); Leclerc & Madry (2020) independently suggest that rapid changes in the local curvature of the loss surface in the early phase critically affects the final generalization. Closely related to our work, Lewkowycz et al. (2020); Jastrzębski et al. (2019) show that using a large learning rate has a dramatic effect on the early optimization trajectory in terms of the loss curvature. These observations lead to question: what is the mechanism by which regularization in the early phase impacts the optimization trajectory and generalization? We investigate this question mainly through the lens of the Fisher Information Matrix (FIM), matrix that can be seen as approximating the local curvature of the loss surface DNNs (Martens, 2020; Thomas et al., 2020).
Our main contribution is to show that the implicit regularization effect of using large learning rate or a small batch size can be modeled as an implicit penalization of the trace of the FIM (Tr(F)) from the very beginning of training. We demonstrate early in training correlates with the final generalization performance across settings with different learning rates batch sizes. We then show evidence that explicitly regularizing Tr(F) (which we call Fisher penalty) significantly improves generalization in training with a sub-optimal learning rate. On the other hand, growth of Tr(F) early in training, which may occur in practice when using
image classification tasks that the value of Tr(F)

---

Under review as a conference paper at ICLR 2021
150
0.0316
0.03316
Validation, Accuracy
0.0010
100
30-
20
50
50
10
71
0.0010
300
300
100
Epoch
Epoch
(a) Validation accuracy
(h) The of the FIM
(c) The of the gradient
The catastrophic Fisher explosion phenomenon demonstrated for Wide ResNet trained using stochastic gradient descent on the Tiny Imngeee dataset. Training is done with either a learning rate optimized using grid search (7) 0.0316, red), or small learning rate (722 = 0.001, blue). Training with 1/2 leads to large overfitting (left) and a sharp increase the trace the Fisher Information Matrix (FIM, middle). The trace of the FIM is closely related to the gradient norm (right).
Figure 1:
Test Accuracy
Test Accuracy
Test Accuracy
60.0
93
55
92
50
55.0
91
45
52.5
90
40
10000
20000
1000
1250
1000
2000
3000
4000
Tr(F,)
Tr(F,)
Tr(F))
(a) ImageNet (w/o augmentation)
(b) 10FAR-10 (w/ augmentation)
(c) CIFAR-100 (w/o augmentation)
Figure 2: Association between value of Tr(F) in the initial phase of training (Tr(F;)) and test accuracy ImageNet, CIFAR-10 and CIFAR- 100 datasets. Each point corresponds to multiple seeds and a specific value of learning rate. Tr(F;) is recorded during the early phase of training (2-7 epochs, the main text for details). The plots show that early Tr(F) is predictive of final generalization. Analogous results illustrating the influence batch size are shown in Appendix A.
relatively small learning rate, coincides with poor generalization. We call this phenomenon the catastrophic Fisher explosion. Figure illustrates this effect on the TinyImageNet dataset (Le & Yang, 2015).
Our second contribution an analysis of why implicitly or explicitly regularizing Tr(F) impacts generalization. We reveal two effects of implicit or explicit regularization of Tr(F): (1) penalizing Tr(F)) discourages memorizing noisy labels, (2) small Tr(F) in the early phase of training biases optimization towards flat minimum, characterized by the trace of the Hessian.
IMPLICIT AND EXPLICIT REGULARIZATION OF THE FIM
Fisher Information Matrix Consider probabilistic classification model (y(y) where O de- notes its parameters. Let (xx, 0) be the cross-entropy loss function calculated for input æ and label Let g(x, y; 0) (60 f(x, 0) denote the gradient computed for example (x, The central object that we study is the Fisher Information Matrix defined as
F(0) = Sp-pooolllgs- ()) 1)
(I)
where the expectation often approximated using the empirical distribution A induced by the training 2² set. We denote its trace by Tr(F). Later, we also look into the Hessian H(0) = aga {(x, y: 0). We
denote its trace by Tr(H).
The FIM can be seen as an approximation to the Hessian (Martens, 2020). In particular, as p(y] mm; 0) p())=), where p(y(a)) is the empirical label distribution, the FIM converges to the
2

---

Under review as a conference paper at ICLR 2021
Hessian. Thomas et al. (2020) showed on image classifications tasks that Tr(H) Tr(FF) along the optimization trajectory, which we also evidence in Appendix F.
Fisher Penalty Several studies have presented evidence that the early phase has a drastic effect on the trajectory in terms of the local curvature of the loss surface (Achille et al., 2019; Jastrzębski et al. 2019; Gur-Ari al., 2018; Lewkowycz al., 2020; Leclere & Madry, 2020). In particular, Lewkowycz et al. (2020); Jastrzebski et al. (2019) show that using a large learning rate in stochastic gradient descent biases training towards low curvature regions of the loss surface very early in training. For example, using a large learning rate in SGD was shown to result in rapid decay of Tr(H) along the optimization trajectory Jastrzębski al. (2019).
Our main contribution is to propose and investigate a specific mechanism by which using large learning rate or a small batch size implicitly influences final generalization. Our first insight is to shift the focus from studying the Hessian, to studying properties of the FIM. Concretely, we hypothesize that using a large learning rate or a small batch size improves generalization by implicitly penalizing Tr(F) from the very beginning of training.
The benefit of studying the FIM is that it can be directly and efficiently manipulated during training. In order to study the effect of implicit regularization of Tr(F), we introduce regularizer, which refer to as Fisher penalty, explicitly penalizing Tr(F), We derive this regularizer in the following
way. First, we note that Tr(F) can be written as (r(F) = Eg (o-o((l)) [Il ae ((x, 0)
To regularize Tr(F), we add the following term to the loss function:
B (a. yz; 0) +
B g(xi,
(Z B::: 0)
Σ B
- Σ B
Ii)
(2)
i=1
where (æ1.B. M1.B) is a mini-batch, Di is sampled from (o (z;), and a is a hyperparameter. We refer to this regularizer as Fisher penalty. The formulation is based on the empirical observation
that | Σi g(., it)) |2 and Tr(F) correlate well during training. Crucially, this allows us to reduce the added computational cost of Fisher penalty to that of single additional backpropagation call (Drucker & Le Cun, 1992). Finally, we compute the gradient of the second term only every 10 optimization steps, and in a given iteration use the most recently computed gradient. We discuss these approximations in detail in Appendix C.
Catastrophic Fisher Explosion To illustrate the concepts mentioned in this section, we train a Wide ResNet model (depth 44, width 3) (Zagoruyko & Komodakis, 2016) on the TinylmageNet dataset with SGD and two different learning rates. We illustrate in Figure that the small learning rate leads to dramatic overfitting, which coincides with a sharp increase in Tr(F) in the early phase of training. We also show in Appendix D that these effects cannot be explained by the difference in learning speed between runs with smaller and learning rates. We call this phenomenon the catastrophic Fisher explosion.
3 EARLY-PHASE Tr(F) CORRELATES WITH FINAL GENERALIZATION
Using a large learning rate (1)) or a small batch size (S) in SGD steers optimization to a lower curvature region of the loss surface. However, it remains hotly debated topic whether this explains their strong regularization effect (Dinh et al., 2017; He et al., 2019; Maddox et al., 2020; Tsuzuku et al., 2019; Yoshida & Miyato, 2017). We begin by studying the connection between Tr(F) and generalization in experiments across which we vary 7) or S in SGD.
Experimental setup We run our experiments in two settings: (1) ResNet-18 with Fixup He et al. (2015); Zhang et al. (2019) trained on the ImageNet dataset (Deng al., 2009), (2) ResNet-26 initialized with Arpit et al. (2019) trained on the CIFAR-10 10 and CIFAR- 100 datasets (Krizhevsky, 2009). We train each architecture using SGD, with various values of n, S, and random seed.
We define (r,F,) as (r(F) during the initial phase of training. The early- Tr(F) is measured when the training loss crosses task-specific threshold For ImageNet, we use learning rates 0.001,
3

---

Under review as a conference paper at ICLR 2021
Table I: Using a 10-30x smaller learning rate (Baseline) results in up to 9% degradation in test accuracy on popular image classification benchmarks (c.f. to optimal ry". Adding Fisher penalty (FP) substantially improves generalization and closes the gap to 17". We do not use data augmentation with CIFAR-10 and CIFAR-IOK to ensure that using a small learning rate does not lead to under-fitiing.
Setting
17*
Baseline
GPx
GP
FP
GP,
WResNevTinylmageNer (aug.)
54.67%
52.57%
52.79%
56.44%
56.73 %
55.41%
DenseNeu/C100 aug.)
66.09%
58.51%
62.12%
64.42%
66.41 %
66.39%
VGG11/C100 (w/o aug.)
45.86%
36.86%
45.26%
47.35%
49.87 %
48.26%
WResNer/C100 (w/o aug.)
53.96%
46.38%
58.68% %
57.68%
57.05%
58.15%
SimplecNNiCIO(w/o..n aug.)
76.94%
71.32%
75.68%
75.73%
79.66%
79.76%
0.01, 0.1, and E = 3.5. For CIFAR-10, we use learning rates 0.007, 0.01, 0.05, and e = 1.2. For CIFAR-100, we use learning rates 0.001, 0.005, 0.01, and = 3.5. In all cases, training loss reaches between and epochs across different byper-parameter settings. We repeat similar experiments for different batch sizes in Appendix A.1. The remaining training details can be found in Appendix G.1.
Results Figure shows the correlation between Tr(F;) and test accuracy across runs with different learning rates. We show results for CIFAR-I0 and CIFAR-100 when varying the batch size in Figure 7 in the Appendix. We find that Tr(F;) correlates well with the final generalization in our setting, which provides initial evidence for the importance of Tr(F). It also serves as stepping stone towards developing a more granular understanding of the role of implicit regularization of Tr(F) in the following sections.
4 FISHER PENALTY
To better understand the significance of the identified correlation between Tr(F;) and generalization, we now run experiments in which we directly penalize Tr(F). We focus our attention on the identified effect of high learning rate on Tr(F).
Experimental setting We use a similar setting as in the previous section, but we include larger models. We run experiments using Wide ResNet (Zagoruyko & Komodakis, 2016) (depth 44 and width 3, with or without BN layers), SimpleCNN (without BN layers), DenseNet (L=40, K=12) (Huang et al., 2017) and VGG-11 (Simonyan & Zisserman, 2015). We train these models on either the CIFAR-10 or the CIFAR-100 datasets. Due to larger computational cost, we replace ImageNet with the TinyImageNe dataset (Le & Yang, 2015) in these experiments.
To investigate the correlation of Tr(F;) and final generalization holds more generally, we apply Fisher penalty in two settings. First, we which both incur up to 9% degradation in test accuracy and results in large value of Tr(F)). ).. We also remove data augmentation from the CIFAR-10 10 and the CIFAR- 100 datasets to ensure that training with small learning rate does not result in underfitting. In the second setting, we add Fisher penalty in training with an optimized learning rate using grid search (1)") and train with data augmentation.
learning rate 10-30x smaller than the optimal one,
Fisher penalty penalizes the gradient norm computed using labels sampled from (o(y)x). We hypothesize that a similar, but weaker, effect can be introduced by other gradient norm regularizers. First, we compare FP to penalizing the input gradient norm ||g.|| = d ((x, y). which we denote by GP, (Varga et al., 2018; Rifai et al., 2011; Drucker & Le Cun, 1992). We also experiment with penalizing the vanilla mini-batch gradient Gulrajani et al. (2017), which we denote by GP. Finally, experiment with penalizing the mini-batch gradient computed with random labels ||g+ = where ÿ is sampled from uniform distribution the label set (GP.). We are not Be
& f(x, g)
of any prior work using GP or GP, in supervised training, with the exception of Alizadeh et al. (2020) where the authors penalized & norm of gradients to compress the network towards the end of training.
We tune the hyperparameters on the validation set. More specifically for a, we test 10 different values spaced uniformly between 10-1 x to 10a x e logarithmic scale with E R... For Ting
4

---

Under review as a conference paper at ICLR 2021
Table 2: Fisher penalty (FP) improves generalization in out of 5 settings when applied with the optimal learning rate 1) and trained using standard data augmentation. In 3 out of 5 settings the difference between FP and 1j" small (below 1%), which is expected given that FP is aimed at reproducing the regularization effect of large 1,, and we compare to training with the optimal 17".
Setting
7
FP
DenseNes/C100 (aug.)
74.41 %
74.19-00.51%
VGG11/C100 (aug.)
59.82++1 1.22-----23%
65.08+00.53%
WResNes/C110 (aug.)
69.48+00.30%
71.53++1.2299
SimpleCNN/C10 (aug.)
87.16=t0.166
87.52++0.50 %
WResNe/TinylmageNe (aug.)
54.70-10.04%
60.00.10.07% %
140
Velnation AccuUNt
1.00
00
0.2
0.2
20
0.0
Epach
Esoch
Epoch
Esoch
CP.
GP.
(a) Wide ResNet on CIFAR-I00 (w/o aug.)
(b) VGG-11 on CIFAR- 100 (w/o aug.)
Figure 3: Training with FP GPx improves generalization and limits early peak of Tr(F). Each subfigure shows validation accuracy (left) and Tr(F) (right) for training with 17" or small learning rate (blue) and for training with either GP, or FP (red). Curves were smoothed for clarity.
we test 5 alternatives instead. pick the optimal learning rate, we evaluate 5 values spaced equally on a logarithmic scale. We include the remaining experimental details in the Appendix G.2.
Fisher Penalty improves generalization Table 1 summarizes the results of the main experiment. First, we observe that a suboptimal learning rate (10-30x lower than the optimal) leads to dramatic overfitting. We observe degradation of up 9% in test accuracy, while achieving perfect training accuracy (see Table 6 in the Appendix).
Fisher penalty closes the gap test accuracy between the small and optimal learning rate, and even achieves better performance than the optimal learning rate. A similar performance was observed when minimizing [g- |]. We will come back this observation in the next section.
GP and GP, reduce the early value of Tr(F) (see Table 4 in the Appendix). They, however, generally perform worse than TF(F) or GP, and do not fully close the gap between small and optimal learning rate. We hypothesize they improve generalization by similar but less direct mechanism than Tr(F) and GP,.
In the second experimental setting, apply FP to network trained with the optimal learning rate 17" According to Table 2, Fisher Penalty improves generalization in out of settings. The gap between the baseline and FP is small in out of settings (below 1 1%), which is natural given that we already regularize training implicitly by using the optimal 17 and data augmentation.
Geometry and generalization in the early phase of training Here, we investigate the temporal aspect of Fisher Penalty on CIFAR-I0 10 and CIFAR-100. In particular, we study whether early penalization of Tr(F) matters for final generalization.
First, we observe that all gradient- norm regularizers reduce the early value of Tr(F) closer to Tr(F) achieved when trained with the optimal learning rate 17". We show this effect with Wide ResNet and VGG-11 on CIFAR- 100 in Figure 3, and for other experimental settings in the Appendix. We also tabulate the maximum achieved values of Tr(F) over the optimization trajectory in Appendix A.2.
5

---

Under review as a conference paper at ICLR 2021
Vallaadd Accuncc P50
40
40.0
375
50
100
33.0
20
03
150
FEder
(a) Wide ResNet CIFAR-
(b) VGG-11 CIFAR-100
(c) Simple CNN CIFAR-
(d) DenseNet CIFAR-100
100 (w/o aug.)
(w/o aug.)
10 (w/o aug.)
(w/o aug.)
Figure 4: Each subplot summarizes an experiment in which we apply Fisher Penalty starting from certain epoch (x axis) and measure the final test accuracy (y axis). Fisher Penalty has to be applied from the beginning of training to close the generalization gap to the optimal learning rate (c.f. the red horizontal line to the blue horizontal line).
To test the importance of explicitly penalizing Tr(F) early in training, start applying it after certain number of epoch E E 11,2,4,8, 16,32, 32, 64, 128). ) We use the hest hyperparameter set from the previous experiments. Figure summarizes the results. For both datasets, observe consistent pattern. When FP is applied starting from later epoch, final generalization is significantly worse, and the generalization gap arising from suboptimal learning rate is not closed.
4.1 FISHER PENALTY REDUCES MEMORIZATION
It is not self-evident how regularizing (r(F) influences generalization. In this section, provide evidence that regularizing Tr(F) slows down learning on data with noisy labels. To study this, we replace labels of the examples in the CIFAR- 1-000 dataset (25% 50% of the training set) with labels sampled uniformly. While label noise in real datasets is not uniform, methods that perform well with uniform label noise generally
robust label noise in real datasets (Jiang et al., 2020a). We also know that datasets such as CIFAR-IOK 100 contain many labeling errors (Song et al., 2020). As such, examining how (F) reduces memorization synthetic label noise provides an insight into how improves generalization in our prior experiments.
We expect FP to reduce memorization. When the predictive distribution po(y)æ) and the true label distribution p" (y/z)) are both uniform, Tr(F) of the specific example æ is equivalent to the squared loss gradient norm of the sample example. The proposed Fisher penalty thus minimizes the contribution of the loss gradient from the training examples whose labels were sampled uniformly. In other words, the Fisher penalty implicitly suppresses learning noisy examples, under the assumption that clean examples' label distributions not uniform.
To study whether the above happens in practice, we compare FP to GPx, GP,, and mixup (Zhang et al., 2018). While mixup is not the state-of-the-art approach to learning with noisy labels, it is competitive among approaches that do not require additional data nor multiple stages of training. In particular, it is a component in several lfte-of-the-aa approaches (Li et al., 2020; Song et al., 2020). For gradient norm based regularizers, we evaluate 6 different hyperparameter values spaced uniformly on logarithmic scale, and for mixup evaluate 3 E {0.2, 0.4, 0.8. 3.2, 6.4). We experiment with the Wide ResNet and VGG-11 models. We describe remaining experimental details in the Appendix G.3.
Results We begin by studying the learning dynamics on data with noisy labels through the lens of training accuracy and mini-batch gradient norm. We show the results for VGG-11 and ResNet-50 in Figure and Figure in the Appendix. We observe that FP limits the ability of the model to memorize data more strongly than it limits its ability to learn from clean data. We can further confirm our interpretation of the effect Tr(F) has on training by studying the gradient norms. As visible in Figure 5, the gradient norm examples with noisy labels is larger than clean examples, and the ratio is closer to when large regularization is applied.
We report test accuracy (at the best validation point) in Table 3. We observe that Tr(F) reduces memorization competitively to mixup. Furthermore, FP performs similarly to GP,, which agrees with our interpretation of why FP limits learning on examples with noisy labels.

---

Under review as a conference paper at ICLR 2021
1.0
low early regularization high early regularization
0.0
1500
2000
2500
2000
3000
4000
50
54
Tr(F,)
Tr(Hi)
Test. Accuracy
Figure 6: Small Tr(F) during the early phase of training is more likely to reach wider minima. Left: two ResNet-56 models are trained with two different levels of regularization for 20 epochs on CIFAR-100. Tr(F) at the end of 20 epochs (Tr(Fi)) ()) is shown. Middle: Each model is then continued trained using the low regularization configuration with different random seeds. histogram of Tr(H) at best test accuracy along the trajectory (Tr(Hp) is shown. Right: histogram of test accuracy.
Table 3: Fisher Penalty (FP) and GP, both reduce memorization competitively to mixup. We measure test accuracy the best validation point training with either 25% or 50% examples with noisy labels in the CIFAR-100 dataset.
Noise
Setting
Baseline
Mixup
GPx
FP
GP,
25%
VGG-11/C100
41.74%
52.31%
45,94%
60.18% %
58.46
ResNet-52/C100
53.30%
61.61 %
52.70%
58.31%
57.60%
50%
VGG-11/C100 30.05%
39.15%
34.26%
51.33% % 47.99%
50.33%
ResNet-52/C100
43.35%
51.71 %
42.99%
50.08%
Training Accuracy 180
100
Grad, Norm Rallo
15
50
30
100
Epoch
Epoch
Figure 5: Fisher penalty slows down training on data with noisy labels more strongly than it slows down training clean data for VGG-11 on 100. This likely happens because FP penalizes more strongly gradient norm on data with noisy labels. Left plot shows the training accuracy on examples with clean/noisy labels (solid/dashed line). Middle plot shows the gradient norm evaluated on examples with clean/noisy labels (solid/dashed). Right plot shows the ratio of gradient norm on clean to noisy data. Red blue color represents the regularization coefficient (from 10-2 to 10
EARLY Tr(F) INFLUENCES FINAL CURVATURE
To provide further insight why it is important regularize Tr(F) during the early phase of training, establish a connection between the early phase training and the wide minima hypothesis (Hochreiter Schmidhuber, 1997; Keskar et al., 2017) which states that flat minima typically correspond to better generalization. Here, we use TH as a measure of flatness.
Experimental setting We investigate how likely it is for an optimization trajectory to end up in wide minimum in two scenarios. 1) When optimization exhibits small Tr(F) early on. 2) When optimization exhibits large Tr(F) early on. Specifically, we train two separate ResNet-26 models for 20 epochs using high and low regularization configurations. At epoch 20 record Tr(F) for each model. We then use these two models as initialization for 8 separate models each, and continue training using the low regularization configuration with different random seeds. The motivation behind this experiment is to show that the degree of regularization in the early phase biases the model towards minima with certain flatness (Tr(H)) even though further high regularization

---

Under review as a conference paper at ICLR 2021
configurations are used during the rest of the training. For all these runs, record the best test accuracy along the optimization trajectory along with Tr(H) at the point corresponding to the best test accuracy. We describe the remaining experimental details in Appendix G.4.
Results We present the result in Figure 6 for the CIFAR-100 100 datasets, and for CIFAR-10 in Appendix A.4. A training run that shows lower Tr(F) during the early phase is more likely to end up in a wider minimum as opposed to one that reaches large Tr(F) during the early phase. This happens despite that the late phases of both sets of models use the low regularization configuration. The latter runs have a high variance in the best test accuracy and always end up in sharper minima. In Appendix G.4 we also show evolution of Tr (H) throughout training, which suggests that this behavior can be attributed to curvature stabilization happening early during training.
6 RELATED WORK
SGD's implicit regularization effect has been argued to a critical component of the empirical success of DNNs (Neyshabur, 2017; Zhang et al., 2016). Much of it is attributed to the choice of hyperparameters (Keskar et al., 2017; Smith & Le, 2018; Jastrzebski et al., 2017), the low complexity bias induced by gradient descent (Xu, 2018; Jacot et al., 2018; Hu et al., 2020) or the cross-entropy loss function (Poggio et al., 2018; Soudry et al., 2018). However, a more mechanistic understanding of how SGD implicitly regularizes DNNs remains largely unsolved problem.
Prior work on replicating SGD's s implicit regularization focused mainly on the loss curvature at the final minimum (Hochreiter & Schmidhuber, 1997). Chaudhari et al. (2019) propose Langevin dynamics based algorithm for finding update directions that point towards wide minima. Wen et al. (2018) propose to find wide minima by averaging gradients at the neighborhood of the current parameter state. In contrast, we shift the focus to the FIM and the early phase of training. This new perspective allows us to more directly test our theory by explicitly penalizing (r(
Penalizing Tr Tr(F) is related to regularizing the input gradient norm, which was shown to be an effective regularizer for deep neural networks (Drucker & Le Cun, 1992; Varga et al., 2018). Chatterjee (2020); Fort et al. (2020) show that SGD avoids memorization by extracting commonalities between examples due to following gradient descent directions shared between examples. Our work complementary. We argue that SGD implicitly penalizes Tr(F), which also reduces memorization. Concurrently, Barrett & Dherin (2020) show that SGD implicitly penalizes the gradient norm for large learning rates and propose GP as an explicit regularizer. Similarly, we found that SGD implicitly regularizes Tr(F), which is the squared gradient norm under labels sampled from pe (e(y))). In contrast to them, connected the implicit regularization effect of SGD to large curvature in the early phase. We also found GP to be a generally less effective regularizer than FP.
7 CONCLUSION
Inspired by recent findings of rapid changes to the local curvature of the loss surface that happen in the early phase of training (Achille et al., 2019; Jastrzębski et al., 2019; Lewkowycz et al., 2020), we investigated more closely the connection between the loss geometry in the early phase of training of neural networks and generalization.
We proposed and investigated a hypothesis that SGD influences generalization by implicitly penalizing the trace of the Fisher Information Matrix (Tr(F)) from the very beginning of training. We show that (1) the value of early Tr(F) correlates with final generalization, and (2) explicitly regularizing Tr(F) can substantially improve generalization.
To gain further insight into the mechanism by which penalizing Tr(F) improves generalization, we investigated training noisy data. We found that penalizing Tr(F) reduces memorization by penalizing examples with noisy labels more strongly than clean ones, which seems to happen because it penalizes more strongly their gradient norm. This sheds new light onto implicit regularization effects in SGD, and suggests the utility of penalizing Tr(F) as an explicit regularizer.
An interesting topic for the future is put findings in the context of transfer and continual learning. We hypothesize that catastrophic Fisher explosion (the initial growth of Tr(F) to a large value) can negatively impact not only generalization, but also transferability of the model.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Alessandro Achille, Matteo Rovere, and Stefano Soatto. Critical learning periods in deep networks. In International Conference on Learning Representations, 2019.
Milad Alizadeh, Arash Behboodi, Mart van Baalen, Christos Louizos, Tijmen Blankevoort, and Max Welling. Gradient & regularization for quantization robustness. In International Conference Learning Representations, 2020.
Anonymous. Gradient descent neural networks typically occurs at the edge of stability. In Submitted to International Conference on Learning Representations, 2021. URL https:// openreview net wooooumiid-jji jo- under review.
Devansh Arpit, Víctor Campos, and Yoshua Bengio. How initialize your network? robust initialization for weightnorm & resnets. In Advances in Neural Information Processing Systems, 2019.
David G. T. Barrett and Benoit Dherin. Implicit gradient regularization, 2020.
Satrajit Chatterjee. Coherent gradients: An approach to understanding generalization in gradient descent-based optimization. In International Conference on Learning Reprexentations, 2020.
Pratik Chaudhari, Anna Choromanska, Stefano Soatto, Yann LeCun, Carlo Baldassi, Christian Borgs, Jennifer Chayes, Levent Sagun, and Riccardo Zecchina. Entropy-sgd: Biasing gradient descent into wide valleys. Journal of Statistical Mechanics: Theory and Experiment, 2019.
François Chollet and others. Keras. GitHub, 2015.
Soham De and Samuel Smith, Batch normalization biases deep residual networks towards shallow paths, 2020.
J. Deng, W. Dong, R. Socher, L.-J. Li. K. Li, and L. Fei-Fei. ImageNet: A Large-Scale Hicrarchical Image Database. In CVPR09, 2009.
Laurent Dinh, Razvan Pascanu, Samy Bengio, and Yoshua Bengio. Sharp Minima Can Generalize For Deep Nets. In Proceedings of the 34th International Conference on Machine Learning, volume 70 of Proceedings of Machine Learning Research. PMLR, 2017.
H. Drucker and Y. Le Cun. Improving generalization performance using double backpropagation. IEEE Transactionsf on Neural Networks, 1992.
Stanislav Fort, Pawet Krzysztof Nowak, Stanislaw Jastrzebski, and Srini Narayanan. Stiffness: A new perspective on generalization in neural networks, 2020.
Jonathan Frankle, David J. Schwab, and Ari S. Morcos. The early phase of neural network training. In International Conference on Learning Representations, 2020.
Aditya Sharad Golatkar, Alessandro Achille, and Stefano Soatto. Time matters in regularizing deep networks: Weight decay and data augmentation affect early learning dynamics, matter little near convergence. In Advances in Neural Information Processing Systems 32, 2019.
Ishaan Gulrajani, Faruk Ahmed, Martin Arjovsky, Vincent Dumoulin, and Aaron C Courville. Improved training of wasserstein gans. Advances in Neurul Information Processing Systems 30. 2017.
Guy Gur-Ari, Daniel A. Roberts, and Ethan Dyer. Gradient descent happens in tiny subspace, 2018.
Haowei He, Gao Huang, and Yang Yuan. Asymmetric valleys: Beyond sharp and flat local minima. In Advances in Neural Information Processing Systems 32. 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 2016 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2015. 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Flat minima. Neural Computation, 1997.

---

Under review as a conference paper at ICLR 2021
Wei Hu, Lechao Xiao, Ben Adlam, and Jeffrey Pennington. The surprising simplicity of the early-time learning dynamics neural networks, 2020.
G. Huang, Z. Liu, L. Van Der Maaten, and K. Q. Weinberger. Densely connected convolutional networks. In 2017 IEEE Conference on Computer Vision and Pattern Recognition (CVPR), 2017.
Michael F Hutchinson. A stochastic estimator of the trace the influence matrix for laplacian smoothing splines. Communications in Sttistics-Simullttion and Computation, 1990.
Arthur Jacot, Franck Gabriel, and Clement Hongler. Neural tangent kernel: Convergence and generalization in neural networks. In S. Bengio, H. Wallach, H. Larochelle, K. Grauman, N. Cesa- Bianchi, and R. Garnett (eds.), Advances in Neural Information Processing Systems 31. Curran Associates, Inc., 2018.
Stanislaw Jastrzebski. Zachary Kenton, Devansh Arpit, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amos J. Storkey. Three Factors Influencing Minima in SGD. 2017.
Stanislaw Jastrzebski, Maciej Szymczak. Stanislav Fort, Devansh Arpit, Jacek Tabor, Kyunghyun Cho and Krzysztof Geras The break-even point on optimization trajectories of deep neural networks. In International Conference Learning Representations, 2020.
Stanistaw Jastrzebski. Zachary Kenton, Nicolas Ballas, Asja Fischer, Yoshua Bengio, and Amost Storkey. On the relation between the sharpest directions of DNN loss and the SGD step length. In International Conference on Learning Representations, 2019.
Lu Jiang, Di Huang, Mason Liu, and Weilong Yang. Beyond synthetic noise: Deep learning on controlled noisy labels. In ICML, 2020a.
Yiding Jiang, Behnam Neyshabur, Dilip Krishnan, Hossein Mobahi, and Samy Bengio. Fantastic Generalization Measures and Where to Find Them. In International Conference on Learning Representations, 2020b.
Nitish Shirish Keskar, Dheevatsa Mudigere, Jorge Nocedal, Mikhail Smelyanskiy, and Ping Tak Peter Tang. On Large-Batch Training for Deep Learning: Generalization Gap and Sharp Minima. In 5th International Conference on Learning Representations, ICLR, 2017.
Alex Krizhevsky. Learning multiple layers of features from tiny images. Technical report, 2009.
Y. Le and X. Yang. Tiny imagenet visual recognition challenge. 2015.
Guillaume Leclere and Aleksander Madry. The two regimes of deep network training, 2020.
Aitor Lewkowycz, Yasaman Bahri, Ethan Dyer, Jascha Sohl-Dickstein, and Guy Gur-Ari. The large learning rate phase of deep learning: the catapult mechanism. 2020.
Junnan Li, Richard Socher, and Steven C.H. Hoi. Dividemix: Learning with noisy labels as semi- supervised learning. In International Conference on Learning Representations, 2020.
Wesley Maddox. Gregory Benton, and Andrew Gordon Wilson. Rethinking parameter counting in deep models: Effective dimensionality revisited, 2020.
James Martens. New insights and perspectives the natural gradient method, 2020.
Behnam Neyshabur. Implicit regularization in deep learning. 2017.
Tomaso Poggio, Kenji Kawaguchi, Qianli Liao, Brando Miranda, Lorenzo Rosasco, Xavier Boix, Jack Hidary, and Hrushikesh Mhaskar. Theory of deep learning ii:: explaining the non-overfitting puzzle, 2018.
Salah Rifai, Pascal Vincent, Xavier Muller, Xavier Glorot, and Yoshua Bengio. Contractive auto- encoders: Explicit invariance during feature extraction. In ICMI,, 2011.
Levent Sagun, Utku Evci. V. Ugur Guney, Yann Dauphin, and Leon Bottou. Empirical analysis the hessian of over-parametrized neural networks, 2018.
10

---

Under review as a conference paper at ICLR 2021
Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale image recognition. In International Conference on Learning Representations, 2015.
Samuel L. Smith and Quoe V. Le. A bayesian perspective on generalization and stochastic gradient descent. In International Conference on Learning Representations, 2018.
Jiaming Song, Lunjia Hu, Michael Auli, Yann Dauphin, and Tengyu Ma. Robust and on-the-fly dataset denoising for image classification, 2020.
Daniel Soudry, Elad Hoffer, Mor Shpigel Nacson, Suriya Gunasekar, and Nathan Srebro. The implicit bias of gradient descent on separable data. The Journal of Machine Learning Research, 2018.
Valentin Thomas, Fabian Pedregosa, Bart Merriënboer, Pierre-Antoine Manzagol, Yoshua Bengio, and Nicolas Le Roux. On the interplay between noise and curvature and its effect on optimization and generalization. In International Conference on Artificial Intelligence and Statistics. PMLR, 2020.
Yusuke Tsuzuku, Issei Sato, and Masashi Sugiyama. Normalized flat minima: Exploring scale invariant definition of flat minima for neural networks using pac-bayesian analysis, 2019.
Dániel Varga, Adrián Csiszárik, and Zsolt Zombori. Gradient regularization improves accuracy of discriminative models, 2018
Wei Wen, Yandan Wang, Feng Yan, Cong Xu, Chunpeng Wu, Yiran Chen, and Hai Li. Smoothout: Smoothing out sharp minima to improve generalization in deep learning, 2018.
Zhiqin John Xu. Understanding training and generalization in deep learning by fourier analysis, 2018.
Yuichi Yoshida and Takeru Miyato. Spectral norm regularization for improving the generalizability of deep learning, 2017.
Sergey Zagoruyko and Nikos Komodakis. Wide residual networks. In Proceedings of the British Machine Vision Conference (BMVC), 2016.
Chiyuan Zhang, Samy Bengio, Moritz Hardt, Benjamin Recht, and Oriol Vinyals. Understand- ing deep learning requires rethinking generalization. In International Conference on Learning Representations, 2016.
Hongyi Zhang, Moustapha Cisse, Yann N. Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. In International Conference on Learning Representations, 2018.
Hongyi Zhang, Yann N. Dauphin, and Tengyu Ma. Residual learning without normalization via better initialization. In International Conference on Learning Representations, 2019.

---

Under review as a conference paper at ICLR 2021
APPPENDIX
A ADDITIONAL RESULTS
A.1 EARLY PHASE TF() CORRELATES WITH FINAL GENERALIZATION
In this section, we present the additional experimental results for Section 3. The experiments with varying batch size for CIFAR-IO0 100 and CIFAR-10 are shown in Figure 7, The conclusions are the same as discussed in the main text in Section 3.
Test Test Accuracy
Test Accuracy
94
55
92
400
600
800
1000
500
1000
Tr(F.)
Tr(F.)
(a) CIFAR-10 (w/ augmentation) (b) CIIAR-I00 100 (w/o augmentation)
Figure 7: Association between early phase values of TF(F) and generalization, holds on the CIFAR- 10 and the CIFAR-100 datasets. Each point corresponds to multiple runs with randomly chosen seeds and specific value of batch size. TrF; is recorded during early phase (2-7 epochs, see main text for details), while the test accuracy the maximum value along the entire optimization path (averaged across runs with the same batch size). The horizontal and vertical error bars show the standard deviation of values across runs. The plots show that early phase Tr(F) is predictive of final generalization.
A.2 FISHER PENALTY
We first show additional metries for experiments summarized in Table 1. In Table we show the final training accuracy. Table 4 confirms that generally all gradient norm regularizers reduce the maximum value of TrFF)) (we measure Tr(F) starting from after one epoch of training because Tr(F) explodes in networks with batch normalization layers initialization). Finally, Table 5 confirms that the regularizers incurred a relatively small additional computational cost.
Figure 8 is counterpart of Figure 3 for the other two models on the CIFAR-10 and the CIFAR-100 datasets.
3.6
Aecurary 190
0.6
T:0F
0.5
0.3
1000
200
1.00
2:00
300
1.00
200
300
100
200
300
Epoch
FP
GP.
GP.
0.3n"
(a) DenseNet on CIFAR-100 100 (w/o aug.)
(b) SimpleCNN on CIFAR-10 (w/o aug.)
Figure 8: Same as Figure 3, but for DenseNet on CIFAR-100, and SimpleCNN on CIFAR-IO, Curves smoothed for visual elarity.

---

Under review as a conference paper at ICLR 2021
Table 4: The maximum value of (r(F) along the optimization trajectory for experiments on CIFAR-10
or CIFAR- 1000 included in Table 1.
Setting
7
Baseline
GPx
GP
FP
GP,
DenseNeu/C100 (w/o aug.)
24.68
98.17
83.64
66.24
73.66
VGG11/C100 (wo aug.)
50.88
148.19
102.95
58.53
64.93
62.96
WResNeuC110 (w/o aug.)
26.21
91.39
41.43
40.94
56.53
39.31
SCNN/C10 (w/o aug.)
24.21
52.05
47.96
25.03
19.63
25.35
Table 5: Time per epoch (in seconds) for experiments in Table 1.
Setting
7
Baseline
GPx
GP
FP
GP;
WResNer/TinylaageNe (aug.)
214.45
142.69
233.14
143.78
208.62
371.74
DenseNes/C100 (w/o aug.)
78.88
57.40
77.89
78.66
97.25
75.96
VGG11/C100 (w/o aug.)
30.50
35.27
31.54
32.52
43.41
42,40
WResNet/C100 (w/o aug.)
49.64
47.99
71.33
61.36
76.93
53.25
SCNN/C10 (w/o aug.)
18.64
19.51
26.09
19.91
21.21
20.55
Table The final epoch training accuracy for experiments shown in Table Experiments with small learning rate reach no lower accuracy than experiments corresponding to large learning rate 17"
Setting
Bascline
GPx
GP
FP
GP:
WResNee/TinylmageNet (aug.)
99.84%
99.96%
99.97%
93.84%
81.05%
86.46%
DenseNeu/1100 (w/o aug)
99.98%
99.97%
99.96%
99.91%
99.91% %
99.39%
VGG11/C100 (w/o aug)
99. 99.98%
99.98%
99.85%
99.62%
97.73%
86.32%
WResNet/C100 (w/o aug)
99.98%
99.98%
99.97%
99.96%
99.99%
99.94%
SCNN/C10 (w/o aug)
100.00%
100.00%
97.79%
100.00%
93.80%
94.64%
A.3 FISHER PENALTY REDUCES MEMORIZATION
In this section, we describe additional experimental results for Section 4.1. Figure the same as Figure 5, but for R0.Net-50.
Training Accuracy (9))
15
0.8
10
100
200
300
Epoch
Epech
Figure 9: Same as Figure 5, but for ResNet-50.
A.4 EARLY Tr(F) INFLUENCES FINAL CURVATURE
this section, we present additional experimental results for Section 5. The experiment on CIFAR-10 shown in Figure 10. The conclusions are the same discussed in the main text in Section 5.
2

---

Under review as a conference paper at ICLR 2021
0.5
early regularization high early regula rizatien
500
1000
1250
750
1000
1250
1500
87.5
Tr(R)
Tr(Hi)
Test Accuracy
Figure 10: Small (F(F) during the early phase training is more likely to reach wider minima as measured by Tr [rH)). Left: 2 models are trained with different levels of regularization for 20 epochs on CIFAR-10. Tr(F)) at the end 20 epochs (denoted as ir(F;)) is shown. Middle: Each model is then used as initialization and trained until convergence using the low regularization configuration with different random seeds. A histogram of (r(H) at the point corresponding to the best test accuracy along the trajectory (denoted by [r(H))) is shown. Right: a histogram of the best test accuracy corresponding to middle figure is shown.
5-529 TA-90-72)
5-120 TA-ds
ned -0 TA-01
n-0 2e
n-d 8010 -4,54444112
4-0 0180
2e 74-35-0-500000
TA-50.76)
(wo 8012 (1-40 (A=52.89)
n-0 0120
20 TA-56.12)
107
100
1780
10'
50
100
15:0
200
30
100
130
200
Epochs
Epochs
(a) CIFAR-10 10 (w/ augmentation)
(b) CIFAR- -100 (w/o augmentation)
The value of (r(H) over the course training. Each point corresponds to runs with different seeds and specific value of learning rate Y and batch size S. l and TA respectively denote the minimum training loss and the maximum test accuracy along the entire trajectory for the corresponding runs (averaged across seeds). The plots show that flatter optimization trajectories become biased towards flatter minima early during training, at a coarse scale values (red vs blue).
Figure 11:
Next, understand why smaller (rFF) during early phase is more likely to end up in wider final minimum, we track Tr(H) during the entire coarse of training and show that it stabilizes early during training. In this experiment, For CIFAR-10, 10, we use batch size 5 E A U B, where A = (480, 500, 520} and B = (80, 100, 120)- For all batch size configurations, learning rate of 0.02 is used. Overloading the symbols A and B for CIFAR-100, we use learning rate E A B, where A = {0.0008, 0.001, 0.0012] and B (0.00,, 0.01, 0.012). For all learning rate configurations, batch size of 100 is used. In both cases, the elements within each set (A and B) vary on fine- ime-grained scale, while the elements across the two sets vary on a coarse- coarse-grained scale. The remaining details and additional experiments can be found in Appendix G.4. The experiments are shown in Figure 1. Notice that after initialization (index on x-axis), the first value is computed at epoch 10 (at which point previous experiments show that entanglement starts to hold with late phase).
create
of hyper-parametent
carse-grained and fine-grained.
We make three observations this experiment. First, the relative ordering of Tr(H) values for runs between sets A B stay the same after the first 10 epochs. Second, the degree of entanglement is higher between any two epochs when looking at runs across sets A and B, while is weaker when looking at runs within any one the sets. Finally, test accuracies for set B runs always higher than those set runs, but this trend not strong for rums within any one set. Note that the minimum loss values are roughly at similar scale for each dataset and they all at or below 10-2
3

---

Under review as a conference paper at ICLR 2021
B COMPUTATION OF Tr(H)
We computed Tr(H) in our experiments using the Hutchinson's 's estimator Hutchinson (1990),
Tr(H): = Tr(H- II)
TTrHH E[(z²|)
B[Tr(H zz")]
Elz H z]
1 M
H. 22
M
Y=1
M
1
Σ
0e
2
M
00
00
t=1
1
M
0
0e T
2
M
00
00
where is the identity matrix, z a multi-variate standard Gaussian random variable, and z, are iid. instances of z. The larger the value of M, the more accurate the approximation is. We used M = 30. To make the above computation efficient, note that the gradient de and it can be re-used in the summation over the M samples.
only needs to be computed once
C APPROXIMATIONS IN FISHER PENALTY
In this section, we describe the approximations made in Fisher Penalty in detail. Recall, that Tr(F) can be expressed
8 [| |10 < 0)13
Tr F) X
(3)
In the preliminary experiments, we found empirically that we can use the norm of the expected gradient rather than the expected norm of the gradient, which is direct expression of Tr(F):
N
M
i
iop (u/
(xr, ú)
~
f(ann Fam
00
M
R
m=
N
E(En Inm)
NM
00
m==1
where N and M are the minibatch size and the number of samples from po(y) respectively. This greatly improves the computational efficiency. With B and M we end up with the following learning objective function:
B Σ
B Σ
( B1.B: 0)
la U6 0) +a
g 3 Mi)
B
B
Σ y (x;; 11) which we denote by Tr(FB), and Tr(F) correlate well during training. To demonstrate this, we train SimpleCNN the CIFAR-I0 dataset with 5 different learning rates (from 1 to 10-1). The outcome is shown in Figure 12. We see that for most of the training, with the exception of the final phase, Tr(F and Tr(F) correlate extremely well. Equally importantly, we find that using a large learning affects both Tr(F and Tr(F). which further suggests the two are closely connected.
found empirically that 1
4

---

Under review as a conference paper at ICLR 2021
8
6
Tr(Fe
4
2
0
20
40
Tr(F)
Figure 12: Correlation between (F) and Tr(F for SimpleCNN trained on the CIFAR-10 dataset. Blue to red color denotes learning rates from 10-3 to 10-1 The value of (T(F) and Tr(FB) correlate strongly for the most of the training trajectory. Using large learning rate reduces both Tr(F) and
(F
on
0.
1009
10t
0.3
3.0
10'
160
162
300
Spoch
Czech
Spsch
Epoch
(a) Fisher Penalty with
10
(b) Fisher Penalty with
Figure 13: A comparison between the effect of recomputing Fisher Penalty gradient every 10 iterations (left) or every iteration (right), with respect to validation accuracy and TF(F). We denote by the frequency with which we update the gradient. Both experiments result in approximately 80% test accuracy with the best configuration.
0.8
Validation Accuracy
0.7
10a
0.6
Tr(FF)
0.5
10
0.4
0.3
10
100
200
300
100
200
Epoch
Epoch
Figure 14: Using Fisher Penalty without the approximation results in similar generalization performance. We penalize the norm of the gradient rather than norm the mini-batch gradient (as in Equation 2). We observe that this variant of Fisher Penalty improves generalization to similar degree as the version of Fisher Penalty used the paper (c.f. Figure 13.), achieving 79.7% test accuracy.
We also update the gradient of Tr(F only every 10 optimization steps. We found empirically does not affect generalization performance nor the ability regularize Tr(F) in our setting. However, we acknowledge that is plausible that this choice would have to be reconsidered training with very large learning rates with larger models.

---

Under review as a conference paper at ICLR 2021
Figure 13 compares learning curves of training with FP recomputed every optimization step, or every 10 optimization steps. For each, we tune the hyperparameter a, checking 10 values equally spaced between 10 2 and 10 logarithmic scale. We observe that for the optimal value of a, both validation accuracy and Tr(F) are similar between the two runs. Both experiments achieve approximately 80% test accuracy.
Finally, to ensure that using the approximation in Equation does not negatively affect how Fisher Penalty improves generalization reduces the value of Tr(F), we experiment with a variant of Fisher Penalty without the approximation. Please recall that we always measure Tr(F) (i.e. we do not use approximations computing (r)F) that is reported in the plots), regardless of what variant of penalty is used in regularizing the training.
Specifically, we augment the loss function with the norm of the gradient computed on the first example in the mini-batch as follows
B (xx ys: v: + (1g(21, 01)1
E' (IIIBBBLLB::0
Σ = B
1
(5)
We apply this penalty in each optimization step. We tune the hyperparameter a, checking 10 values equally spaced between 10 4 and 10 on a logarithmic scale.
Figure 14 summarizes the results. We observe that the best value of a yields 79.7% test accuracy, compared 80.02% test accuracy yielded by the Fisher Penalty. The effect on Tr(F) is also very similar. We observe that the best run corresponds to maximum value of Tr(F) of 24.16, compared to that of 21 achieved by Fisher Penalty. These results suggest that the approximation used Fisher Penalty only improves the generalization and flattening effects of Fisher Penalty.
D A CLOSER LOOK AT THE SURPRISING EFFECT OF LEARNING RATE ON THE LOSS GEOMETRY IN THE EARLY PHASE OF TRAINING
It is intuitive to hypothesize that the catastrophic Fisher explosion (the initial growth of the value of (r(F)) occurs during training with large learning rate, but is overlooked due to not sufficiently fine- fmneggained computation of Tr(F), In this section show evidence against this hypothesis based on the literature mentioned in the main text. We also run additional experiments in which we compute the value of Tr(F) each iteration.
The surprising effect of the learning rate on the geometry of the loss surface (e.g. the value of Tr(F)) was demonstrated in prior works (Jastrzebski et al., 2019; Golatkar et al., 2019; Lewkowycz et al., 2020; Leclerc & Madry, 2020). In particular, Jastrzebski et al. (2020); Lewkowycz et al. (2020) show that training with large learning rate rapidly escapes regions of high curvature, where curvature is understood as the spectral norm of the Hessian evaluated at the current point of the loss surface. Perhaps the most direct experimental data against this hypothesis can be found in Anonymous (2021) in Figure 1, where training with Gradient Descent finds regions of the loss surface with large curvature for small learning rate rapidly in the early phase of training.
We also run the following experiment to provide further evidence against the hypothesis. We train SimpleCNN on the CIFAR-10 dataset using two different learning rates, while computing the value of (FF) for every mini-batch. We use 128 random samples in each iteration to estimate (F.F).
We find that training with a large learning rate never (even for a single optimization step) enters a region with the value of Tr(F) as large as reached during training with a small learning rate. Figure 15 shows the experimental data.
We also found similar to hold when varying the batch size, see Section E, which further shows that the observed effects cannot be explained by the difference in learning speed incurred by using a small learning rate.
To summarize, both the published evidence of Jastrzebski et al. (2020); Lewkowycz et al. (2020); Anonymous (2021), well our additional experiments are inconsistent with the hypothesis that the results in this paper can be explained by differences in training speed between experiments using large and small learning rates.

---

Under review as a conference paper at ICLR 2021
100
6 10
Training Accuracy (%)
0.001
4 10a
n
0.001
n 0.01
0.01
3
60
Tr(F)
x 101
40
10a
20
100
10a
10a
10a
100
10
10
10a
Step
Step
Figure 15: Training with large learning rate never (even for a single optimization step) enters region with as large value of (FF as the maximum value of Tr(F) reached during training with small learning rate. We run the experiment using SimpleCNN on the CIFAR-10 10 dataset with two different learning rates. The left plot shows the value of Tr(F) computed at each iteration, and the right plot shows training accuracy computed on the current mini- mini-batch (curve has been smoothed for clarity).
E CATASTROPHIC FISHER EXPLOSION HOLDS IN TRAINING WITH LARGE BATCH-SIZE
In this section, we show evidence that the conclusions transfer to large batch size training. Namely, show that (I) catastrophic Fisher explosion also occurs in large batch size training, and (2) Fisher Penalty can improve generalization and close the generalization gap due to using a large batch size (Keskar et al., 2017).
= 1280
Training Accuracy (%)
= 1280
5 128
80
S 128
= 12
S = 12
60
Tr(F
10
40
20
10
10
102
100
10
102
Step
Step
Catastrophic Fisher explosion in large batch size training. Experiment run the CIFAR- 10 and dataset the SimpleCNN model. The left plot shows the value of Tr(F) computed at each iteration, and the right plot shows training accuracy computed on the current mini-batch (curve has been smoothed for clarity).
Figure 16:
We first train SimpleCNN on the CIFAR-10 dataset using three different batch sizes, while computing the value of Tr(F) for every mini-batch. We use 128 random samples each iteration to estimate Tr(F). Figure 16 summarizes the experiment. We observe that training with large batch size enters a region of the loss surface with a substantially larger value of Tr(F) than the small batch size.
Next, we run variant of one of the experiments in Table 1. Instead of using suboptimal (smaller) learning rate, we use a suboptimal (larger) batch size. Specifically, we train SimpleCNN on the CIFAR-10 dataset (without augmentation) with 10x larger batch size while keeping learning rate the same. Using a larger batch size results 3.24% lower test accuracy (76.94% compared to 73.7% test accuracy, c.f. with Table 1).

---

Under review as a conference paper at ICLR 2021
We next experiment with Fisher Penalty. We apply the penalty in each optimization step and use the first 128 examples when computing the penalty. We also use a 2x lower learning rate, which stabilizes training but does not improve generalization reaches 73.59% test accuracy). Figure 17 shows Tr(F) and validation accuracy during training for different values of the penalty. We observe that Fisher Penalty improves test accuracy from 73.59% to 78.7%. Applying Fisher Penalty also effectively reduces the peak value of (r
its own (training with this learning rate
Taken together, the results suggest that Catastrophic Fisher explosion holds in large batch size training; using a small batch size improves generalization by a similar mechanism as using a large batch size, which can be introduced explicitly in the form of Fisher Penalty.
0.8
Validation Accuracy
10a
0.6
Tr(FF
10
0.4
10-1
0.2
100
200
300
0
100
200
300
Epoch
Epoch
Figure 17: Fisher Penalty improves in large batch size training. Experiment the CIFAR-10 10 dataset (without augmentation) and the SimpleCNN model. Warmer color corresponds to larger coefficient used in Fisher Penalty.
F (H(H) AND (F( CORRELATE STRONGLY
We demonstrate a strong correlation between (r(H) and Tr(F) for DenseNet, ResNet-56 and SimpleCNN in Figure 18. We calculate Tr(F) using mini-batch. We see that Tr(F) has smaller magnitude (because we use the mini-batch gradient which has lower variance), but correlates extremely well with Tr(H).
0.02
30
0.002
15.0
4 0.01
n= 0.02
0.04
25
0.004
12.5
n=
Tr(F)
20
7.5
15
5.0
10
2.5
0.0
200
300
2000
4000
Tr(H)
Tr(H)
(a) DenseNet on CIFAR-100 100
(b) SimpleCNN on CIFAR-10
(c) ResNet-56 on CIFAR-100
Figure 18: Correlation between Tr(F) and Tr(H).
G ADDITIONAL EXPERIMENTAL DETAILS
G. EARLY PHASE Tr(F) CORRELATES WITH FINAL GENERALIZATION
Here, describe additional details for experiments in Section 3.
8

---

Under review as a conference paper at ICLR 2021
In the experiments with batch size, for 100 we use batch sizes 100, 500 and 700, and 1.2. For CIFAR-100, we use batch sizes 100, 300 and 700, and e = 3.5. These thresholds are crossed between 2 and epochs across different hyperparameter settings. The remaining details for CIFAR- 100 and CIFAR-10 10 are the same as described in main text. The optimization details for these datasets follows.
ImageNet: No data augmentation was used in order to allow training loss to converge to small values. We use a batch size of 256. Training is done using SGD with momentum set 0.9, weight decay set to le - 4, and with base learning rate as per the aforementioned details. Learning rate is dropped by a factor of 0.1 after 29 epochs and training is ended at around 50 epochs which most runs converge to small loss values. No batch normalization is used and weight are initialized using Fixup Zhang et al. (2019). For each hyperparameter setting, we run two experiments with different random seeds due to the computational overhead. We compute Tr(F) using 2500 samples (similarly to ?).
We used random flipping as data augmentation. In the experiments with variation in learning rates, we use a batch size of 256. In the experiments with variation in batch size, we use learning rate of 0.02. Training is done using SGD with momentum set to 0.9, weight decay set to le 5, and with base learning rate as per the aforementioned details. Learning rate is dropped by a factor of 0.5 at epochs 60, 120, and 170, and training is ended at 200 epochs at which most runs converge to small loss values. No batch normalization is used and weight are initialized using Arpit et al. (2019). For each hyperparameter setting, we run 32 experiments with different random seeds. We compute Tr(F) using 5000 samples.
CIFAR-10:
No data augmentation was used for CIFAR- 100 to allow training loss to converge to small values. We used random flipping as data augmentation for CIFAR-IO. In the experiments with variation in learning rates, we use a batch size of 100. In the experiments with variation in batch size, we use learning rates of 0.02. Training is done using SGD with momentum set to 0.9, weight decay set to 1e- - 5, and with base learning rate as per the aforementioned details. Learning rate is dropped by a factor of 0.5 at epochs 60, 120, and 170, and training is ended at 200 epochs which most runs converge to small loss values. No batch normalization is used and weight are initialized using Arpit et al. (2019). For each hyperparameter setting, we run 32 experiments with different random seeds. We compute Tr(F) using 5000 samples.
CIFAR-100:
G.2 FISHER PENALTY
Here, we describe the remaining details for the experiments in Section 4. We first describe how tune hyperparameters in these experiments. In the remainder of this section, we describe each setting used in detail.
Tuning hyperparameters In all experiments, we refer to the optimal learning rate 1) as the learning rate optimized using grid search. In most experiments we check 5 different learning rate values uniformly spaced logarithmic scale, usually between 10-2 and 10"h In some experiments we adapt the range to ensure that the range includes the optimal learning rate. We tune the learning rate only once for each configuration (i.e. we do not repeat it for different random seeds).
In the first setting, for most experiments involving gradient norm regularizers, we use 10x smaller learning rate than 17" For TinylmageNet, we use 30x smaller learning rate than 17" To pick the regularization coefficient we evaluate 10 different values uniformly spaced on a logarithmic scale between 10-1 x v to 10a x e with v E R4. We choose the best performing a according to best validation accuracy. We pick the value of manually with the aim that the optimal a is included in this range. We generally found that v = 0.01 works well for GP, GP,, and FP. For GP, we found in some experiments that it is necessary to pick larger values of c.
Measuring Tr(F) We measure Tr(F) using the number of examples equal to the batch size used in training. For experiments with Batch Normalization layers, we use Batch Normalization in evaluation mode due to the practical reason that computing Tr (F)) uses batch size of 1, and hence Tr(F) is not defined for a network with Batch Normalization layers in training mode.
DenseNet on the CIFAR-100 dataset We use the DenseNet (L=40, k=12) configuration from Huang al. (2017). We largely follow the experimental setting in Huang et al. (2017). We use

---

Under review as a conference paper at ICLR 2021
the standard data augmentation (where noted) and data normalization for CIFAR-100 100. We hold out random 5000 examples as the validation set. We train the model using SGD with momentum of 0.9, batch size of 128, and weight decay of 0.0001. Following Huang et al. (2017). we train for 300 epochs and decay the learning rate by a factor of 0. after epochs 150 and 225. To reduce variance, in testing we update Batch Normalization statistics using 100 batches from the training set.
Wide ResNet on the CIFAR-100 dataset We train Wide ResNet (depth 44 and width 3, without Batch Normalization layers). We largely follow experimental setting in He et al. (2015). We use the standard data augmentation and data normalization for CIFAR-100. We hold out random 5000 examples as the validation set. We train the model using SGD with momentum of 0.9, a batch size of 128, weight decay of 0,0010. Following He et al. (2015), we train for 300 epochs and decay the learning rate by a factor of 0.1 after epochs 150 and 225, We remove Batch Normalization layers. To ensure stable training we use the SkipInit initialization (De & Smith, 2020).
VGG-11 on the CIFAR-100 dataset We adapt the VGG-11 model (Simonyan & Zisserman, 2015) to CIFAR-100. We do not use dropout nor Batch Normalization layers. We hold out random 5000 examples as the validation set. We use the standard data augmentation (where noted) and data normalization for CIFAR- -000. We train the model using SGD with momentum of 0.9, a batch size of 128, and weight decay of 0.0001. We train the model for 300 epochs, and decay the learning rate by a factor of 0.1 after every 40 epochs starting from epoch 80.
SimpleCNN on the CIFAR-10 10 dataset We also experiments on the CNN example architecture from the Keras example repository (Chollet & others, 2015)', which we change slightly. Specifically, remove dropout and reduce the size of the final fully-connected layer to 128. We train it for 300 epochs and decay the learning rate by a factor of 0.1 after the epochs 150 and 225. We train the model using SGD with momentum of 0.9, a batch size of 128.
Wide ResNet on the TinyImageNet dataset We train Wide ResNet (depth 44 and width 3, with Batch Normalization layers) on TinylmageNet Le & Yang (2015). TinylmageNet consists of subset of 100,000 examples from ImageNet that we downsized to 32x32 pixels. We train the model using SGD with momentum of 0.9, a batch size of 128, and weight decay of 0.0001. We train for 300 epochs and decay the learning rate by factor of 0.1 after epochs 150 and 225. We train the model using SGD with momentum of 0.9, a batch size of 128. We do not use validation in TinylmageNet due to its larger size. To reduce variance, in testing we update Batch Normalization statisties using 100 batches from the training set.
G.3 FISHER PENALTY REDUCES MEMORIZATION
Here, we describe additional experimental details for Section 4.1. We use two configurations described in Section G.2: VGG-11 1 trained on CIFAR-10M 100 dataset, and Wide ResNe trained on the CIFAR- 100 dataset. We tune the regularization coefficient a in the range {0.01, 0.1, 0.31, 10), with the exception of GP, for which we use the range {10, 30, 100, 300, 1000}. We tuned mixup coefficient in the range {0.4, 0.8, 1.6, 6.4]. We removed weight decay in these experiments.
G.4 EARLY Tr(F) INFLUENCES FINAL CURVATURE
We used random flipping as data augmentation for CIFAR-10. We use learning rate of 0.02 for all experiments. Training is done using SGD with momentum 0.9, weight decay 1e 5, and with batch size shown in figures. Learning rate is drop by factor of 0.5 at 80, 150, and 200 epochs, and training is ended at 250 epochs. No batch normalization is used and weight are initialized using Arpit et al. (2019). For each batch size, we run 32 experiments with different random seeds. We compute Tr(F) using 5000 samples.
CIFAR-10:
No data augmentation is used. We use a batch size of 100 for all experiments. Training is done using SGD with momentum 0.9, weight decay Ie - 5, and with base learning rate as shown in figures. Learning rate is drop by a factor of 0.5 at 80, 150, and 200 epochs, and training is ended at 250 epochs. No batch normalization is used and weight are initialized using Arpit et al. (2019). For
CIFAR-100:
Accessible at htwp:///hhccceeeeeeeeeeeeeeeeeee
10

---

Under review as a conference paper at ICLR 2021
each learning rate, run 32 experiments with different random seeds. We compute Tr(F) using 5000 samples.


Reviews:
[{"review_id": "p4w7XJw71W2", "reviewer": "AnonReviewer5", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "# Summary\nThis work claims SGD regularizes the model through penalizing the trace of Fisher in the early phase. This claim is supported by the similar generalization behavior of SGD (with optimal learning rate) and Fisher penalization (for SGD with small learning rate). A series experiments are conducted to verify the understanding.\n\nI find the paper writing is rather misleading. On the one hand, no mathematical justifications, and the logical chain is weak --- hard to claim which factor is the cause and which one is the effect. On the other hand, though written as \"trace of Fisher\" in the paper, in experiments they compute a different regularizer --- norm of expected gradient. Detailed questions come in the following.\n\n# Questions\n\n1. Abstract, \"We highlight that in the absence of implicit or explicit regularization...\". How do you get rid of the implicit regularization???\n\n2. Eq.(2) is super misleading. Trace of Fisher is the expected gradient norm, but in Eq. (2) you compute norm of expected gradient. Statistically the latter has nothing to do with the former: one is the second moment, and the other is the squared first moment. Please justify.\n\n3. I have a very simple explanation to your observed phenomenon. In the beginning, gradient is large, thus your version of \"trace of fisher\" is large, as the gradient decreases, the \"trace of fisher\" decreases. Now we look at large learning rate and small learning rate. With large learning rate, SGD converges faster, thus its gradients decrease faster, which causes the \"trace of fisher\" decreases faster. But with small learning rate, SGD converges slower, thus its gradient decrease slower, and the \"trace of fisher\" appears to be large in the beginning epochs.\n\nTherefore I am not at all convinced by your arguments. \n\n4. From the above discussion, at least you need to normalize the \"trace of fisher\" by gradient norm --- which then becomes a measurement of gradient confusion. See following for references.\n\n5. FP/GP. If I understand correctly, you need to compute gradient of gradient, which expands your computation graph at least twice. Can you report the GPU memory consumption? \n\n6. Finally, let us take about the practical role of FP. According to Table 1, the improvement of FP over GP is marginal. I cannot see a potential of FP. Not only in theory, but also in practice this paper is not satisfactory. \n\n\n# Missing Refs\nTons of theory paper should be discussed.  A few of them come to my brain are listed in below. Please do a more complete literature investigation.\n\nFisher\n- Liang, Tengyuan, et al. \"Fisher-rao metric, geometry, and complexity of neural networks.\" The 22nd International Conference on Artificial Intelligence and Statistics. 2019.\n- Karakida, Ryo, Shotaro Akaho, and Shun-ichi Amari. \"Universal statistics of Fisher information in deep neural networks: Mean field approach.\" The 22nd International Conference on Artificial Intelligence and Statistics. PMLR, 2019.\n\nSGD regularization mechanism\n- Daneshmand, H., Kohler, J., Lucchi, A., and Hofmann, T. Escaping saddles with stochastic gradients. arXiv preprint arXiv:1803.05999, 2018.\n- Zhu, Zhanxing, et al. \"The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects.\" arXiv preprint arXiv:1803.00195 (2018).\n- Wu, Jingfeng, et al. \"On the Noisy Gradient Descent that Generalizes as SGD.\" arXiv preprint arXiv:1906.07405 (2019).\n\nGradient confusion is highly related to the trace Fisher. See this one and its follow-ups.\n- Sankararaman, Karthik A., et al. \"The impact of neural network overparameterization on gradient confusion and stochastic gradient descent.\" arXiv preprint arXiv:1904.06963 (2019).\n\nAdversarial regularization is also related to GP/FP:\n- Miyato, Takeru, et al. \"Virtual adversarial training: a regularization method for supervised and semi-supervised learning.\" IEEE transactions on pattern analysis and machine intelligence 41.8 (2018): 1979-1993.\n\n\n  ", "extra_scores": {}, "extra_text": {}}, {"review_id": "izFImoH1aw3", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "After the rebuttal, the authors added a section on large-batch training, which shows that the catastrophic Fisher explosion also occurs in large batch training. This makes the paper more convincing. However, the main concern that this paper lacks theoretical contribution still exists. Therefore, I keep the weak acceptance recommendation. \n\nSummary:\n\nIn this paper, the authors made a study on Fisher Information Matrix (FIM) in the training dynamics. By tracking the trace of FIM in the initial training process, the authors show that it can be strongly connected to a model’s generalization performance. This successfully explains why some hyper-parameter choices lead to significantly better generalization. Finally, empirical evaluations show that penalizing the trace FIM can reduce memorization and encourage wide minima solution, leading to better generalization.\n\nPros:\n\nThis paper takes one step forward to the question of why some hyper-parameter choices (small batch size or large learning rate) lead to better generalization. In many previous works, the implicit regularization effect of stochastic gradient descent (SGD) accounts for this observation (small batch size generalizes better). Most of the above works focused on the anisotropic noise in SGD. However, there is still a gap between the anisotropic noise and implicit regularization, i.e. why does anisotropic noise lead to implicit regularization hence better generalization. [1] explains this in a very general way: it helps to escape local minima. This paper tries to fill in the gap by leveraging FIM. \n\nFIM is also studied in the context of SGD regularization [2]. However, it only shows that the trace of FIM affects optimization performance. How it relates to the generalization performance still remains unknown. Section 4 and section 5 shed light on this question by drawing connection to memorization and final curvatures. \n\nLarger scale dataset such as Tiny-Imagenet is included in the experiments. This shows that the conclusion also stands on a more practical and noisy dataset. \n\nCons:\n\nMost of the empirical verifications in this paper are comparing large and small learning rates. To be more general, the authors can also include the study of batch size. The central claim in this paper would be more convincing if catastrophic Fisher explosion also occurs in large batch training. More importantly, reducing the trace can improve large batch training generalization. Additionally, the connection between FIM and noisy gradient descent [3] is worth mentioning. For example, how does the trace of FIM in noisy gradient descent compared to other methods?\n\nOne of the postulations is regularizing the trace of FIM reduces memorization. The paper verifies this by introducing noisy labels. This requires modification to the training dataset. Another way to verify this postulation is to make comparisons on out-of-distribution dataset (CIFAR10-C and CIFAR100-C). The method which reduces memorization should be more robust on out-of-distribution datasets.\n\nA majority of the analysis comes from empirical evaluations. The claim is more convincing if the conclusion can also be analytically proved in the convex quadratic scenario. \nThe experiments section of this paper is based on the empirical Fisher matrix (true Fisher matrix is expensive to compute). Although empirical Fisher is commonly used in practice to approximate true Fisher, the approximation is only accurate in the local minima [4]. The paper should have a discussion on the difference between true Fisher and empirical Fisher. \n\nLegends are missing in Figure 4 and Figure 6. \n\n[1]: Zhu, Zhanxing et al. “The Anisotropic Noise in Stochastic Gradient Descent: Its Behavior of Escaping from Sharp Minima and Regularization Effects.” ICML (2019).\n[2]: Wen, Yeming et al. “An Empirical Study of Stochastic Gradient Descent with Structured Covariance Noise.” AISTATS (2020).\n[3]: Wu, Jingfeng et al. “On the Noisy Gradient Descent that Generalizes as SGD.” ICML (2020)\n[4]: Kunstner, Frederik et al. “Limitations of the empirical Fisher approximation for natural gradient descent.” NeurIPS (2019).", "extra_scores": {}, "extra_text": {}}, {"review_id": "cCpCqfW5QQh", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "Summary: This paper explores the relationship between the trace of the Fisher Information Matrix (FIM) and generalization performance of deep learning models. There are a few core insights which are verified empirically across a number of different models and datasets: (1) large learning rate/small batch sizes (which tends to lead to improved generalization) can be realized as implicitly penalizing the trace of Fisher and (2) how regularizing the trace of Fisher discourages memorization and guides optimization to \"flatter minima\". \n\nMy overall assessment of this paper leans towards positive. While it is probably not surprising to deep learning optimization/generalization researchers that the trace of FIM is closely related to generalization performance, this paper does a good job from the empirical standpoint, the experiments are done in a clean and systematic way to verify the hypotheses. \n\nSome comments/remarks/questions:\n\n- The manuscript is overall well-written; I have no problems following the logic and the experimental setups. Related works for the most part are discussed throughly and well-explained. A downside is that there is completely no theoretical arguments given in the paper, I am wondering if it's possible even in the case of a convex quadratic/extremely simple networks, the authors can show how a small Tr(F) leads to a tighter/better generalization bound? \n\n- Just to verify- the Fisher matrix which is computed throughout is the empirical estimate of the true Fisher and not the \"empirical Fisher\"- as the labels are sampled from the predictive distribution? This often results in a confusion; see [1] for more details. \n\n- In Section 3, the authors compared FP with GP which is a fair comparison. I am wondering if the authors compared their approach to even more closely-related methods such as input-output Jacobian regularization [2], regularizing by the Frobenius norm of Fisher; Tr(F^2) instead of Tr(F), and the Fisher-Rao norm which has a tight relationship with capacity/generalization of neural networks as shown in [3].\n\nReferences:\n\n[1] Kunstner, Frederik, Philipp Hennig, and Lukas Balles. \"Limitations of the empirical Fisher approximation for natural gradient descent.\" Advances in Neural Information Processing Systems. 2019.\n\n[2] Novak, Roman, et al. \"Sensitivity and generalization in neural networks: an empirical study.\" arXiv preprint arXiv:1802.08760 (2018).\n\n[3] Liang, Tengyuan, et al. \"Fisher-rao metric, geometry, and complexity of neural networks.\" The 22nd International Conference on Artificial Intelligence and Statistics. 2019.", "extra_scores": {}, "extra_text": {}}, {"review_id": "IbaNJu_HM-", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper empirically investigates the effect of the trace of the Fisher Information Matrix (FIM) early in training has on the generalization of SGD. Authors demonstrate that the effect of optimally chosen learning rate and batch size for SGD can be modeled as an implicit penalty on the trace of FIM. They argue that explicitly penalizing the trace of FIM discourages memorizing noisy labels, thus leading to better generalization. Furthermore, they experimentally show that the early low value of the trace of FIM may bias the optimization towards a flat optimum which has been observed to correlate well with good generalization.\n\n*Positives:*\n+ The paper is well-written and easy to follow. Authors convey their message clearly.\n+ The paper's motivation is definitely relevant. Understanding the connection between the early stage loss landscape of neural networks and final generalization is crucial and has drawn significant attention recently.\n+ The introduced Fisher-penalty is interesting and the authors provide insights into the mechanism how regularizing the trace of FIM might improve generalization.\n\n*Negatives:*\n- The exact mechanism how FP influences learning with noisy labels could be investigated a bit more rigorously. The discussion provided in the paper is not completely clear.\n- The experiments are somewhat inconclusive in some cases (see below) and would require some extra comments/explanation.\n\nOverall, the paper has some interesting contribution by demonstrating the positive effect of various gradient norm penalties on the generalization of SGD and it is tied well into existing literature. Furthermore, the experiments are comprehensive and in-depth. Therefore I would recommend acceptance if a couple of issues are addressed.\n\nFirst, as seen in Fig. 4 generalization peaks if FP is applied starting after some positive number of epochs. The difference in validation accuracy between turning on FP from the very beginning and only turning it on somewhat later is not always negligible. This observation somewhat violates the 'earlier we regularize the better' idea. Can the authors comment on this discrepancy?\n\nSecond, the benefit of high early regularization is somewhat inconclusive based on Fig. 10. It appears that the best generalization belongs to low early regularization, whereas high early regularization leads to better average test accuracy. I would be interested to know how much FP might hinder optimization by discouraging the exploration of certain directions in the loss landscape. Can it even contribute to trapping the network in local optima by penalizing large gradients needed to escape such 'bad' optima?\n\nMinor comment: there is a typo in the caption of Fig. 6.", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: This is a tricky one, hence my low confidence rating.

The reviewers seem to agree that the paper is well written, easy to follow, and that it tests a relevant hypothesis that is of interest to the community. There was some disagreement as to whether the experiments are comprehensive, complete and/or conclusive enough, although on balance it seems reviewers were overall satisfied barring a few additional requests which the authors addressed in their feedback.

However, no reviewers support the paper strongly (borderline accepts) while R5 remains unconvinced and has raised a technical point in their review about the estimator of the Trace of the Fisher information matrix. The question R5 has raised is central to the paper's methods, arguments and conclusions. In a message to ACs and PCs the authors raised concerns about R5. I personally thought that while R5 could have worded their review more carefully and respectfully (as I pointed out in my respose) the concerns raised were otherwise motivated, the reviewer engaged in a discussion, and the arguments were laid out clearly. I side with R5 and I think that the paper should be rewritten with more clarity on this question - the problem R5 found is likely to trip up others who read or build on the paper.

The authors have raised that there are two parallel submissions closely related to this one, complicating the decision making somewhat:
[1] https://openreview.net/forum?id=rq_Qr0c1Hyo [2] https://openreview.net/forum?id=3q5IqUrkcF

Decision: Reject