Regularizing Deep Neural Networks with Stochastic Estimators of Hessian Trace
Authors: Yucong Liu
Tong Lin
Venue: ICLR 2022 Submitted (2022)
Keywords: Regularization
Hessian Trace
Stochastic Estimator
Nonlinear Dynamical System
Generalization Error

Abstract: In this paper we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. Hutchinson method is a classical unbiased estimator for the trace of a matrix, but it is very time-consuming on deep learning models. Hence a dropout scheme is proposed to efficiently implements the Hutchinson method. Then we discuss a connection to linear stability of a nonlinear dynamical system. Experiments demonstrate that our method outperforms existing regularizers such as Jacobian, confidence penalty, and label smoothing. Our regularization method is also orthogonal to data augmentation methods, achieving the best performance when our method is combined with data augmentation. 

Under review as a conference paper at ICLR 2022
REGULARIZING DEEP NEURAL NETWORKS WITH STOCHASTIC ESTIMATORS OF HESSIAN TRACE
Anonymous authors Paper under double-bling review
ABSTRACT
In this paper we develop novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by recent guar- antee bound of the generalization error. Hutchinson method is classical unbiased estimator for the trace of matrix, but it is very time-consuming ing models. Hence a dropout scheme is proposed to efficiently implements the Hutchinson method. Then discuss connection to linear stability of nonlin- ear dynamical system. Experiments demonstrate that our method outperforms ex- isting regularizers such Jacobian, confidence penalty, and label smoothing. Our regularization method is also orthogonal to data augmentation methods, achieving the best performance when our method is combined with data augmentation.
deep learn-
INTRODUCTION
Deep neural networks (DNNs) are developing rapidly and are widely used in many fields such as image classification, machine translation, language modeling and speech recognition. As more and more models are proposed in the literature, deep neural networks have shown remarkable improve- ments in performance. However, among various learning problems, wver-fiting training data is a great problem that affects the test accruacy. So a certain regularization method often needed in the training process.
In linear models, Ridge Regression (Hoerl & Kennard 1970) and Lasso [Tbssiraan] 1996) are usu- ally used avoid over-fitting. They are also called L2 and L, regularization. L2 regularization has the effect of shrinkage while L1 regularization can be conductive to both shrinkage and spar- sity. From the Bayesian perspective, L2 and L1 regularization can be interpreted with normal prior distribution and laplace prior distribution respectively.
Apart from Le and L1 regularization, there are many other forms of regularizers in DNNs. The most widely used one is Weight-Decay (Krogh & Hertz 1992). Loshchilov & Hutter (2019) also showed that L2 regularization and Weight-Decay are not identical. Dropout (Srivastava et al. 2014) is another method to avoid over-fitting by reducing co-adapting between units in neural networks. Dropout has inspired a large body of work studying its effects (Wager et al. (2013); Helmbold & Long (2015); Wei et al. (2020)). After dropout, various regularization schemes can be applied additionally.
In this paper, propose function. We refer to our regularization method as Stochastic Estimators of Hessian Trace (SEHT). On one hand, our hessian regularization is valuable to guarantee good generalization. On the other hand, from the perspective parameters move in the parameter space on the basis of training data. In our experiments, Hessian regularization shows competing test performance and low time consumption with our stochastic algorithm.
regularization by penalizing the trace of second derivative of loss
dynamical system, it influences the stability of the system, in which
RELATED WORK
There many regularization methods in previous work. Label Smoothing (Szegedy et al. 2016) estimates the marginalized effect of label-dropout and reduces over-fitting by preventing a network from assigning full probability each training example. Confidence Penalty (Pereyra et al. 2017)

---

Under review as a conference paper at ICLR 2022
prevents peaked distributions, leading to better generalization. A network appears to be overconfi- dent when it places all probability on a single class in the training set, which is often a symptom of over-fitting DropBlock (Ghiasi et al. 2018) is a structured form of dropout, it drops contiguous regions from feature map of a layer instead of dropping out independent random units.
Data augmentation methods are also used in practice to improve model's accuracy and robustness when training neural networks. Cutout (DeVries & Taylor, 2017) is a data augmentation method where parts of the input examples are zeroed out, in order to encourage the network to focus more on less prominent features, then generalize to situations like occlusion. Mixup (Zhang et al. (017) extends the training distribution by incorporating the prior knowledge that linear interpolations of feature vectors should lead to linear interpolations of the associated targets.
Sokolić al. (2017) first proposed Jacobian regularization, method focusing on the norm of Ja- cobian matrix with respect to input data. It was proved that generalization error can be bounded by the norm of Jacobian matrix. Besides that, Jacobian matrix shows improved stability of the model predictions against input perturbations according to Taylor expansion. [Hoffman et al. (2019) showed that Jacobian regularization enlarges the size of decision cells and is practically effective in improving the generalization error and robustness of the models. To simplify calculation, stochastic algorithm of Jacobian regularization was also proposed.
Motivated by Jacobian regularization, consider the generalization error and stability of the model respect to Hessian matrix. Then we combine Linear Stability Analysis and propose Hessian regu- larization with corresponding stochastic algorithms. We compare Hessian regularization with other methods and demonstrate promising performance in experiments. The main idea to estimate the trace of Hessian matrix is Hutchinson Method(Avron & Toledo 201T) and the algorithm was also discussed by Yao et al.] (2020a). We make an improvement by designing a new probability distribution dropout parameters which decrease time consumption obviously without losing gen- eralization.
Hessian information is powerful tool used on analyzing the property of neural networks. Yao al. 20206) designed AdaHessian, a second order stochastic optimization algorithm. Yu et al. (2021) used Hessian trace to measure sensitivity, developing Hessian Aware Pruning method find sensitive parameters in a neural network model and Neural Implant technique to alleviate accuracy degradation. However, their methods are static in essence, we focus on dynamical motion of param- eters in parameter space. [Sankar et al. (2021) also proposed a Hessian regularization. They forcused on the layerwise loss landscape via the eigenspectrun of the Hessian at each layer. We start from different perspectives, generalization error and dynamical system of parameters. Our experiments also shows better results than (Sankar et al. 2021 's method.
HESSIAN REGULARIZATION
Here introduce Hessian regularization method based generalization error and corresponding stochastic algorithms in details. Then we discuss linear stability analysis to explain why Hessian regularization can prevent neural networks from over-fitting.
3.1 TRACE OF HESSIAN MATRIX
In this study, we consider a multi-class classification problem. Input x is a N-dimensional vector, where I E X S R.V. with is the input space. y = {1, (I.......A) M) is the label space, which means that we have M classes. Each input æ has label y E V. Sample space is defined as = X' x y. An element of S is denoted by 8 = (ær, y). We assume that samples a from S are drawn according
probability distribution P. A training set of n samples drawn from P is denoted by S,, = { = {(x,, y:)
Our goal is to find a classification function f. which takes æ E IR' as input and outputs x = fxx) is M-dimensional score vector, where each element 2, is the score belonging to category i E y. The highest score indicates the most probable label. So the estimated label is given as 9 g(=)= arg max, 2, arg max,ey fxx);
N
2

---

Under review as a conference paper at ICLR 2022
A loss function is used to measure the discrepancy between the true label y and the estimated label g In this paper, we use the cross-entropy loss,
((f(z), v) = - log exp(f(x)y)
(1)
En exp(f(x)))
The empirical loss the classifier f(x) associated with the training set is defined as
(emp(s) = (()) n a:ES.
Σ {(((m)).).)
(2)
and the expected loss of the classifier f(x) is defined as lezp(f) [ (x), y)1 E. (l(f((=)) y)).
(3)
then the difference between kemp (f) and Lee (f) is called generalization error: GE Keep (f) (fml(f))l-
(4)
Wei et al. 2020) showed generalization bound of linear models with cross-entropy loss of M classes. Let W is the weight matrix, ru ((W) = E[[l l|] and v(W) É[tr(H)). For linear models, the Jacobian matrix J is vector defined as dz and the Hessian matrix H is defined as (az,dz,) JMxxxMx over the training examples, for all weight matrices W satisfying the norm A. the following bound holds:
ae
0
With probability bound || TW7T12. 1|2,1
(Ap(W))&((B)) Av AVBu(Wyo
BA70
E
1.01
+ (5)
n
Vn
nllgg
BA400 ((W)n
+
Here with some fixed bound B 0,
8= min B},
W 22.1 = Σ Σ (W?))
(6)
= I (nM) max ||2:112
= Bilog(1//) (1/0) + log log n)
So one guarantee good generalization when the trace of Hessian matrix and norm of Jacobian matrix are small. On one hand, when learning with gradient descent, we want to find a local or global minimum of loss function. Naturally, minimum the gradient is zero and the norm of Jacobian matrix is small near minimum. So gradient descent helps us to ensure the norm of Jacobian Matrix be small.
On the other hand, the trace of Hessian Matrix is hard to be constrained by gradient descent. From this aspect, proposed Hessian regularization for linear models as:
1
(r(H..z) n
It's the trace of second derivative of loss with respect to output of linear model z, which is also the end nodes of a linear model. DNN is consist of many layers, with each layer being viewed as a linear model (except the nonlinear activation functions). Thus. generalize the Hessian regularization to every node in DNN and define it as
(8)
72
It's the trace of second derivative of loss l with respect to parameters w.
Here define new loss with our Hessian regularization
1 tr (H((),))
Loss Cemp (f) +
(9)
where À controls the strength of Hessian regularization.
3

---

Under review as a conference paper at ICLR 2022
3.2 HUTCHINSON METHOD
In a typical DNN, there are more than millions of parameters. So the calculation of Hessian matrix
difficult. Hutchinson Method (Avron Toledo 2011 I an unbiased estimator for the trace of matrix. Let A be an entries are i.i.d Rademacher random variables (Pr(a, +1) = 2). then of Aa an unbiased
symmetric matrix with tr (A) * O. Let a be random vector whose
estimator of tr (rA),, based on the following equation:
tr (r(A) = tr(AT) tr tr(AEloo")) Eltr(Ago
(Ago") (oo=))== = 1B Aa].
(10)
In this paper. we consider the trace of Hessian matrix H, which is the second derivative matrix. Since the Rademacher random vector is irrelevant to network parameters, we expand the expression of Hutchinson estimator follow:
dl
d dl
dl
do dw
d(
dl
0² Ho
0²
dw
0²
de
(11)
dw
de
du
du
Based Equation and O, we only have to caleulate the gradient g of de a, instead of the whole Hessian matrix. The whole calculation process only include two inner products and two derivations. We refer to the Hutchinson stochastic estimator of Hessian trace as SEHT-H.
di and the derivative
dw
Algorithm 1: SEHT-H
Input: n-dimensional gradient g
Output: Estimation of tr(H)
trace= =
for i to mazIter do
σ Rademacher(n))
/x n-dim vector with each element sampled from Rademacher distribution. */
v 0: h de du t = o² h
/x inner product */
/* derivative of v +/
/* inner product +/
bruce +=
end return
Even though it's a stochastic algorithm, it cost much time because of great number of parameters. So we propose another efficient algorithm below based on the basic idea of Hutchinson Method.
3.3 DROPOUT METHOD
This stochastic algorithm inspired by Dropout (Srivastava al. 2014). Every node in the neu- ral network has probability to be igonred in the training process to reduce co-adaptations. In Hessian regularization,
H'r to lower the trace Ir (r(H) ΣS Twe the sum of diagonal elements of Hessian matrix. Based on the idea of Dropout, straining the tr( H), since reducing the partial sum of diagonal elements can have a large chance to reduce the total sum. The partial sum of diagonal element is denoted as fr(H). Considering the layer structures of neural networks, the process of randomized parameter selection be di- vided into two steps: randomly select layers in neural network with probability and randomly select parameters in the selected layers with probability p2- In other words, when carrying Hes- sian regularization, we ignore other layers with probability selected layers with probability
ignore some parameters in
and ignore other parameters in P2- In experiment, simply P1 P2-
In addition, Hutchinson method shows a technique, which avoids to calculate the whole Hessian matrix. Here we define new probability distribution Q(p) (if ~ Q(p), then Pr(æ +1) p and Pr(æ 0) = 2p). Then, let a be random vector whose entries are i.i.d Q random variables.
Elgo" |fir the position of 0 in o)
(12)
4

---

Under review as a conference paper at ICLR 2022
Here / d0ag(0, 1), diagonal matrix with diagonal elements equal to or 1.
Similar to Equation [0, if we fix the position of 0 in o, we have unbiased estimator of the partial sum of diagonal elements:
(r( tr (AT) tr (r(Aa(oo")] ")) E(tr(Aoo
BBo" Aa).
(13)
We can expand the expression same as Equation IT and transform the calculation process into two inner product and two derivation. We name this method as SEHT-D.
Algorithm 2: SEHT-D
Input: probability p. parameter w in selected layers, and corresponding n-dim gradient g
Output: estimation of (H)
trace = 0
for mar mar do Q(p): /x n a-dim vector with each element sampled from Q(p)
distribution */
0
/* inner product */
/* derivative of */
t
/* inner product +/
frace += ti
end
return
frace manlter
In our experiments, we mainly test the performance of this method. Compared with other regu- larization methods, our Hessian regularization shows improved test performance with fast training speed.
3.4 LINEAR STABILITY ANALYSIS
Training process can be regarded as a motion in the parameter space, from the initial parameter to local or global minimum. Current parameters is point the parameter space and gradient descent is the move of the parameter point each time. Then we can see gradient descent from another perspective. Original gradient descent is defined as series discrete updates:
W2+1 W/ 19t-
(14)
Here Wt is the parameters in step t, is learning rate and 9: is gradient.
If we consider learning rate as discrete time interval move w in parameter space, At = 7). then
Aw At (w, 2).
(15)
We assume time interval or learning rate is small enough, approximately we get contiunous form:
dw dl (--g(w,))
(16)
Thereafter. with initial condition, have the complete trajectory of parameter point based on ordinary differential equation (ODE) theory. The process of gradient descent transformed to a Nonlinear Dynamical System. So introduce Linear Stability Analysis of Nonlinear Dynamical Systems below.
Nonlinear Dynamical System (Thomas Witelski & Mark Bowen 2015)
defined differential (x (4), _2))))... xn ".))...,,,,,
(1)) E IR" the vector of state variables, describes properties of interest the system, evolving for times t 2 O and starting from a given initial state x(0) To- The rate functions for the rates of change of each 2j. di
function dæz = fxe f) with an initial condition æ00) Do. x(()
zt
dz, = fü, have similarly been collected in vector f = (f1, fn), where each fi can potentially depend on all of the state variables. Since fr doesn't 't have be linear function, the system is called Nonlinear Dynamical System. A classic example of dynamical system from mechanics is the system for motion of a particle. system is called non-autonomous the rate function have an explicit dependence time. In this paper, we only focus autonomous systems,
dx
f(x), x00)= = Eg
(17)
dt
5

---

Hy. And the trace and the cigenvalues of J (w+) are also negative of Under review as a conference paper at ICLR 2022
The equilibrium points z, defined by positions where rate functions vanish, f(xx) = 0. If any solution starting near an equilibrium point leaves the neighbourhood of æ, as t - oc, then 2, is called asymptotically unstable, while if all solutions starting within the neighbourhood approach æ. as - oo then the equilibrium is called asymptotically stable. Lyapunov (1992) gave more rigorous definition and discussion, known Lyapunov Stability Theory. The idea of Lyapunov Stability can be extended to infinte-dimensionall manifolds, where it is known as Structural Stability (Pugh & Peixoto 2008), which concerns the behavior of different but "nearby" solutions to differential equations.
For a basic Nonlinear Dynamical System, to ensure equilibrium point a, stable, we need to construct Jacobian matrix J 0f Given the equilibrium point I.. Jxx.) is constant matrix. 8xr
Using the conclusion from Linear State Space Model, if all eigenvalues of (a..) have real parts less than zero, then æ, is stable. If at least one of the eigenvalues has real part greater than zero, then is unstable.
Back to gradient descent, it can regarded as a Nonlinear Dynamical System, according to Equa- tion 16 The local or global minimum, which is the goal of gradient descent, is the equilibrium point in such system, since that minimum point satisfies the condition du = 0. The Jacobian ma- trix in this dynamical system is the negative of the Hessian matrix in Hessian regularization, J (w.) = Bg a 2z the trace the eigenvalues of He.
at
سw
It's easy to see that in our Hessian regularization, we lower the trace of Hessian Matrix He, thus increasing the trace of the Jacobian matrix J(w).. However, real matrix, complex eigenvalues are always conjugate and the trace are always real number. When we increase the trace, we increase the real parts of eigenvalues of J J(w)) to some extent. other word, the goal to preclude the stability of equilibrium point by our Hessian regularization.
Why we want instability in this dynamical system of gradient descent? The stability of local or global minimum shows the stability toward training data. The whole dynamical system of gradient descent is a motion based on training set since the motion of parameters is decided by Equation 17 Equation 17 is consist of two parts: one is the initial condition, which is randomized in DNN, while the other part is the differential function Equation [6] which is directly determined by training data. In other words, with different training data, the parameters have different trajectory in parameter space. It only use information about training data and the equilibrium point depends on training data. Therefore, the stability is relevant training set and reducing the stability some extent can avoid over- fitting to training data.
EXPERIMENTS
We evaluate our Hessian regularization with other regularization methods on variety of datasets. We also combine data augmentation methods with our Hessian regularization to test its efficacy.
4.1
IMAGE CLASSIFICATION
4.1.1
CIFAR-10
The CIFAR-10 10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
For our experiment, we use ResNet-18 (He et al. 2016) as the backbone neural network. For all models, we use Weight Decay 5 x -4 We set learning rate 0.01, batch size 32, momentum 0.9 and all models were trained 200 epochs with Cosine Annealing (Loshchilov & Hutter 2016). For Jacobian regularization, we set number of projections Ayro; = and weight values ÄJæ = 0.01. For DropBlock, block_size = 7 and keep-prob 0.9. For Confidence Penalty and Label Smoothing, we set weight value 0.1, which is found to work best on CIFAR-10 by Pereyra et al. (2017) in their experiments. For Hessian regularization SEHT-D, we also set weight value O. testing with probability value 0.01 and 0.05. addition, we always add output layer in the selected layer, which corresponds to Equation 7 Cutout size of 16 x 16 pixels is used in experiment, based on the

---

Under review as a conference paper at ICLR 2022
validation results mentioned by DeVriee & Taylor 2017). For mixup, 0.1. Averages and 95% confidence intervals are estimated over 5 distinct runs.
We also observe the computational cost of the training with SEHT-D (maxlter=l, prob=0.01) to be only 1.2 times that the baseline. Although increasing the probability to select parameters improve test accuracy, the time consumption will increase lot. In experiment, SEHT-D (maxlter=l, prob=0.05) costs 1.5 times of the baseline.
In experiments, find that Jacobian regularization and Dropblock have worse performance than the baseline with Weight-Decay. Confidence Penalty has slight improvement and Label Smoothing has obvious improvement. However, SEHT-D shows better results, compared with all other regularization methods which are tested. In addition, our SEHT is suitable for combination with other methods. When we combin SEHT-D with mixup, we get the best test accuracy in our experiment, 1.45 more than baseline.
In Sankar al. (2021) 's experiment, they got test accuracy 88.13 13 on CIFAR-10 with ResNet- 18, which is much worse than our result: 94.35 with SEHT-D (maxlter=1, prob=0.01) and 94.37 with SEHT-D (maxlter=l. prob=0.05). Moreover, their improvement based on their methods only 0.02 for full-network method and 0.10 10 for middle-network method. Our Hessian regularization method improves the model 0.35 on test accuracy with SEHT-D (maxlter=1, prob=0.01) and improves 0.37 on test accuracy with SEHT-D (maxIter=1, prob=0.05), which is over 3 times of their improvement.
Table I: ResNet-18 on CIFAR-10
Model
Test Accuracy
Baseline with Weight-Decay
94.00 + 0.47
Jacobian
89.23 + 1.02
DropBlock
89.23 + 0.44
Sanker's Method for Full Network
88.05 + 0.22
Sanker's Method for Middle Network Confidence Penalty
88.13 0.12
94.01 = 0.40
Label Smoothing
94.26 + 0.26
SEHT-D (maxlter=1, prob=0..01
94.35 + 0.18
SEHT-D (maxlter=1, prob=o0s==1proob0000 prob=0.05)
94.37 + 0.27
SEHT-D (maxlter=1, prob=0001) Label Smoothing
94.38 + 0.24
cutout
94.02 1 0.22
mixup
95.39 + 0.13
SEHT-D (maxlter=l, prob=0.01) (rob=00)=1,
mixup
95.45 + 0.06
4.1.2 CIFAR-100
CIFAR-100 dataset is just like the CIFAR-10, 10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.
We use Wide Residual Networks (WRN) (Zagoruyko & Komodakis 2016) as the backbone neural network. We use WRN-28-10 10 specifically, with depth 28 and fixed widening factor of 10. For all models, use Weight Decay of 10-4, We set batch size 32, momentum 0.9 and all models were trained 200 epochs. The learning rate is initially set to 0.1 and is scheduled to decrease by factor of 5 after each of the 60th. 120th. and 160th epochs. We test Dropout with a drop probability of p 0.3, determined by ZZgoruyko & Komodakis (2016)'s cross-validation. For Confidence Penalty, Label Smoothing and our Hessian regularization, we set weight value O.I. Cutout size of 8 pixels is used according to Deiesss & Taylor (2017)'s validation results. For mixup, still set a = 0.1. Averages and 95% confidence intervals of top-1 accuracy and top-5 accuracy are estimated over 5 distinct runs.
In this experiments, Confidence Penalty has better top-5 accuracy, worse top-1 1 accuracy and Label Smoothing has better t accuracy, worse top-5 accuracy, compared with baseline method. How- ever, our SEHT-D method shows better results on both top-1 accuracy and top-5 accuracy. improving 1.13 and 0.65 respectively. Our Hessian regularization method also perform better when combined

---

Under review as a conference paper at ICLR 2022
with other method this dataset. When testing together with Dropout, SEHT-D has best accu- racy, compared with Label Smoothing and Confidence Penalty. Our SEHT-D also improves cutout for 0.39 top-1 accuracy, 0.26 top-5 accuracy, and improves mixup 0.21 top-1 accuracy, 0.09 on top-5 accuracy.
Table 2: WRN-28-10 CIFAR-100
Model
Top-1 Accuracy
Top-5 Accuracy
Baseline with Seight-Deecay
73.79 = 2.68
92.01 1.32
Confidence Penalty
73.46 = 1.21
92.16 0.58
Label Smoothing
74.15 + 0.92
90.40 0.73
SEHTT-Dmmaxtter1,, prob=0.01)
74.92 + 0.77
92.66 0.54
Confidence Penalty + Dropout
74.80 + 0.91
93.09 0.51
Label Smoothing Dropout
72.89 + 1.57
90.43 + 0.97
SEHTT-Dmmaxffer=1, prob=0.01) + Dropout
7
94.38 + 0.09
cutout
76.70
93.72 0.40
SEHTTD(maxlter=1,, prob=0.01) probb=00= + cutout
77.09 0.37
93.98 + 0.21
mixup
78.38 0.31
94.37 0.31
SEHT-DDaxlier-l,probo prob=-00.0) + mixup
78.59 0.46
94.46 0.30
Our experiments Image Classification shows that our Hessian regularization method outperforms other regularization methods and can efficiently combined with data augmentation methods.
4.2
LANGUAGE MODELING
4.2.1
WIKI-TEXT2
The Wiki-Text language modeling dataset collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia.
We use 2-layer LSTM (Hochreiter & Schmidhuber 1997). The size of word embeddings 512 and the number of hidden units per layer is 512. We run every algorithm for 40 epochs, with batch size 20, gradient clipping 0.25, and Dropout ratio 0.5. We perform a grid search over Dropout ratios {0. 0.1, 0.2, 0.3, 0.4, 0.5} and find 0.5 to work best. We tune the initial learning rate from (0.001, 0.01, 0.1, 0.5, 1, 10, 20, 40} and decrease the learning rate by factor of when the valida- tion enror saturates. We find initial learning rate 20 works best. Parameters are initialized from uniform distribution [-0.1, 0.1]. For label smoothing, we perform grid search over weight values (0.001, 0.005, 0.01, 0.05, 0.1) and find 0.01 to work best. For the confidence penalty, a grid search over weight values (0.001, 0.005, 0.01, 0.05, 0.1} and find 0.01 to work best. For our Hessian regularization, probability values {0.01, 0.05J. Weight 0.01 and probability 0.05 work best. Averages and 95% confidence intervals are estimated over 5 distinct runs.
perform
perform grid search over weight values (0.001, 0.005, 0.01, 0.05, 0.1),
In this experiments with LSTM, our SEHT-D has the hest test perplexity and Label Smoothing shows best validation perplexity. SEHT-D improves the model 2.61 on test perplexity. Confidence Penalty performs only sightly better than the baseline method.
Table 3: LSTM on Wiki-Text2
Model
Validation Perplexity
Test Perplexity
Baseline with Dropout
101.82 0.32
95.65 + 0.19
Confidence Penalty
101.39 0.32
95.57 + 0.11
Label Smoothing
99.58 + 0.11
95.03 + 0.58
SEHT-D
100.69 + 0.53
94.86 + 0.50
8

---

Under review as a conference paper at ICLR 2022
We also tested with 2-layer GRU (Cho et al. 2014). The size of word embeddings is 512 and the number of hidden units per layer is 512. We run every algorithm for 40 epochs. with batch size 20, gradient clipping 0.25 and Dropout ratio 0.3. We perform grid search over Dropout ratios {0, 011, 0.2, 0.3, 0.4, 0.5) and find 0.3 to work best. We tune the initial learning rate from (0.001, 0.01, 0.1, 0.5, 1, 10, 20, 40) and decrease the learning rate by factor of 4 when the vali- dation error saturates. We find initial learning rate 20 works best, same as LSTM. Parameters initialized from a uniform distribution 1-0.1, 0.1]. For label smoothing, we perform grid search over weight values (0.001, 0.005, 0.01, 0.05, } and find 0.05 to work best. For the confi- dence penalty, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1) } and find 0.005 to work best. For our Hessian regularization, we perform grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1}. Weight 0.001 works best. We set probability values 0.01. Averages and 95% confidence intervals are estimated over 5 distinct runs.
Our Hessian regularization method has both the best validation perplexity and the best test per- plexity, improving 2.83 and 2.61 respectively compared with baseline method. Confidence Penalty surpasses Label Smoothing with GRU model, compared with LSTM. Label Smoothing also show better results than baseline.
Table 4: GRU on Wiki-Text2
Model
Validation Perplexity
Test Perplexity
Baseline with Dropout
119.04 4.67
111.64 + 3.67
Confidence Penalty
116.40 + 0.17
109.27 : 0.05
Label Smoothing SEHT-D
117.47 0.48
110.46 + 0.87
116.21 + 0.60
109.03 + 0.30
Our experiments on Language Modelling demonstrate that all these three regularization methods can improve models, while our SEHT-D is the best.
CONCLUSION
We propose a new regularization method named Stochastic Estimators of Hessian Trace (SEHT). Our method is motivated by a guarantee bound that a lower trace of the Hessian can result a lower generalization error. We also explained our method with dynamical system theory. Our ex- periment shows that SEHT-D yields promising test performance with fast training speed. SEHT-D also achieves better results when combined with data augmentation methods.
REPRODUCIBILITY STATEMENT
Our codes are available herel.
REFERENCES
Haim Avron and Sivan Toledo. Randomized algorithms for estimating the trace of an implicit sym- metric positive semi-definite matrix. Journal of the ACM (JACM), 58(2):1-34, 2011.
Kyunghyun Cho, Bart van Merriënboer, Caglar Gulcehre, Dzmitry Bahdanau, Fethi Bougares, Hol- ger Schwenk, and Yoshua Bengio. Learning phrase representations using RNN encoder-decode for statistical machine translation. In Proceedings of the 2014 Conference on Empirical Meth- ods in Natural Language Processing (EMNLP), pp. 1724-1734. Association for Computational Linguistics, October 2014.
Terrance Deiii and Graham W Taylor. Improved regularization of convolutional neural networks with Cutout. arXiv preprint arXiv: 1708. 04552, 2017.
Golnaz Ghiasi, Tsung-Yo Yi Lin, and Quoe V Le. Dropblock: A regularization method for convolu- tional networks. Advances in Neural Information Processing Systems, 31:10727-10737, 2018.

---

Under review as a conference paper at ICLR 2022
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
David P Helmbold and Philip M Long. On the inductive bias of dropout. The Journal of Machine Learning Research, 16(1):30333354, 2015.
Sepp Hochreiter and Jürgen Schmidhuber. Long short-term memory. Neural computation, 9(8): 1735-1780, 1997.
Arthur E Hoerl and Robert W Kennard. Ridge regression: Biased estimation for nonorthogonal problems. Technometrics, 12(1):55-67, 1970.
Judy Hoffman, Daniel A Roberts, and Sho Yaida. Robust learning with jacobian regularization. Conference on the Mathematical Theory of Deep Learning (DeepMath), 2019.
Anders Krogh and John Hertz. A simple weight decay can improve generalization. Advances in Neural Information Processing Systems, 4, 1992.
Ilya Loshchilov and Frank Hutter. Sgdr: Stochastic gradient descent with warm restarts. arXiv preprint arXiv:/608 03983, 2016.
Ilya Loshchilov and Frank Hutter. Decoupled weight decay regularization. International Conference on Learning Representutions, 2019.
Aleksandr Mikhailovich Lyapunov. The general problem of the stability of motion. International journal of control, 55(3)5331533, 1992.
Gabriel Pereyra, George Tucker, Jan Chorowski, Eukasz Kaiser, and Geoffrey Hinton. Regularizing neural networks by penalizing confident output distributions. arXiv preprint 17Xvv11001.0658, 2017.
Charles Pugh and Maurício Matos Peixoto. Structural stability. Scholarpedia, 3(9):4008, 2008.
Adepu Ravi Sankar, Yash Khasbage, Rahul Vigneswaran, and Vineeth N Balasubramanian. A deeper look the hessian eigenspectrum of deep neural networks and its applications regularization. In Proceedings of the AAAI Conference on Artificial Intelligence, volume 35, pp. 9481-9488 2021.
Jure Sokolić, Raja Giryes, Guillermo Sapiro, and Miguel RD Rodrigues. Robust large margin deep neural networks. IEEE Transactions on Signal Processing, 65(16))4425-42288 2017.
Nitish Srivastava, Geoffrey Hinton, Alex Krizhevsky, llya Sutskever, and Ruslan Salakhutdinov. Dropout: a simple way prevent neural networks from overfitting. The journal of machine learning research, 15(1):1929-199 1958, 2014.
Christian Szegedy. Vincent Vanhoucke, Sergey loffe, Jon Shlens, and Zbigniew Wojna. Rethink- ing the inception architecture for computer vision. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 2818-2826, 2016.
Thomas Witelski Thomas Witelski and Mark Bowen Mark Bowen. Methods of mathematical mod- elling: Continuous systems and differential equations, 2015.
Robert Tibshirani. Regression shrinkage and selection via the lasso. Journal of the Royal Statistical Suciety: Series B (Methodological), 58(1):267-288, 1996.
Stefan Wager, Sida Wang, and Percy S Liang. Dropout training as adaptive regularization. Advances in neural information processing systems, 26:351-359, 2013.
Colin Wei, Sham Kakade, and Tengyu Ma. The implicit and explicit regularization effects of dropout. In International Conference on Machine Learning, pp. 10181-10192. PMLR, 2020.
Zhewei Yao, Amir Gholami, Kurt Keutzer, and Michael W Mahoney. Pyhessian: Neural networks through the lens of the hessian. In 2020 IEEE International Conference on Big Data (Big Data), pp. 581-590. IEFF, 2020a.
10

---

Under review as a conference paper at ICLR 2022
Zhewei Yao, Amir Gholami, Sheng Shen, Mustafa Mustafa, Kurt Keutzer, and Michael W Ma- honey. Adahessian: An adaptive second order optimizer for machine learning. arXiv preprint 00f::2000600071,, 2020b.
Shixing Yu, Zhewei Yao, Amir Gholami, Zhen Dong, Michael W Mahoney, and Kurt Keutzer. Hessian-aware pruning and optimal neural implant. arXiv preprinl urXiv2200.08944 2021.
Sergey Zagoruyko and Nikos Komodakis.
Wide
residual networks.
arXiv preprint
100.006605 2016.
Hongyi Zhang. Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXix: 2017.


Reviews:
[{"review_id": "w9klzMtlzTq", "reviewer": "Reviewer_YAnV", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n\n* The motivation of using jacobian and hessian trace norm regularizations is drawn from existing theoretical work, it's quite a nice way of tying it together with the extensive literature on generalization bounds. One comment worth adding though is that (I believe) all of those bounds are unfortunately vacuous. But even in that light, I think the idea is natural/worth writing a paper about.\n* The initial downsampling (corresponding to p_1) is a clever way to make this work. Otherwise, instead of the 1.2-1.5x slowdowns, I would have expected to have seen at the very least a 2x slowdown (for two more derivatives), and practically speaking more like a 10-50x slowdown given how slow hessian vector products seem to be in practice.\n\nWeaknesses:\n\n* Unfortunately, the \"better\" results have heavily overlapping confidence intervals. In the top pane, the 2 std confidence interval is +/- 0.47/ Combined with say the seht-d + label smoothing, it's clear that none of these results are significant. I would suggest that the authors aggressively drive down the sizes of these confidence intervals. In the bottom pane, the comparison between mixup and seht-d + mixup suffers from the same problem. In fact, from these experiments, the only reasonable convlusion is that \"mixup works really well\".\n\n* For comparison's sake, I think the authors really need to try the full version of the regularizer (with multiple iterations and prob=1.0). It's obviously going to be much slower, but if that doesn't work then the existing seht-d then I think something is probably wrong i.e., if the base claim is that \"hessian regularization works\", it is much less convincing if you don't directly test that hypothesis and go directly to approximations. I'd even be interested in seeing how much variance is introduced by the trace estimator, the two rounds of downsampling etc.\n\n* I think there needs to be a careful study of hyperparameters of the regularizer i.e., prob and regularization coefficient etc.", "summary_of_the_paper": "In this paper, the authors propose an efficient regularizer for the trace of the Hessian. To circumvent the extra computation needed to do the naive Hutchinson estimator (which pretty much amounts to taking *three* derivatives of the loss), the authors subsample the weights matrices, and then a second time internal to each weights matrix). Experimental results on CIFAR-10 and CIFAR-100 are presented: the proposed regularizer helps, but unfortunately not in a statistically significant way.", "summary_of_the_review": "The idea is interesting, the approximation is nice, but the experimental results are too weak/incomplete to make this a convincing paper.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "1: The contributions are neither significant nor novel."}, "extra_text": {}}, {"review_id": "qUhS-ehLPe", "reviewer": "Reviewer_CMzT", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "Strengths:\n- The connection of Hessian spectrum and trace to generalization is an interesting and promising area of research and it is great to see results for a method regularizing the Hessian that seems to outperform other regularization methods\n- The structure of the paper is clear and the paper is understandable\n\nWeaknesses:\n- The connection between theoretical justification and practical method is not always clear and the relevant details could be expanded:\n\nMy feeling is that the connection between using the Hessian / Jacobian with respect to the logits vs with respect to the parameters is not as straight forward as it is made out to be in the paper and could be explained better. For example the paper states: \"Naturally, at the minimum the gradient is zero and the norm of the Jacobian matrix is small near minimum.\"\n\nBut if we have for e.g. a nonlinear classifier f(x) = W y(x) where W are the weights of the last layer then dl/df is what is meant as Jacobian in the bound but the gradient in optimization is with respect to parameters, e.g. dl(f(x))/dW = dl/df * df/dW = dl/df * y(x). Could there perhaps be a way for the gradient to be small if y(x) is small without dl/df being small? This question could be expanded upon.\n\nSimilarly, an explanation of the difference between the trace of the loss Hessian w.r.t. to logits vs w.r.t to the weights would benefit the paper. For example it is well-known that the Hessian w.r.t. to weights for a deep neural network can have negative eigenvalues but the Hessian w.r.t. to logits for a convex loss has only positive eigenvalues. Does this matter to the trace and associated bound somehow?\n\nRegarding the linear stability argument the paper could be improved by empirically demonstrating that regularizing the Hessian trace leads to an optimum with a Hessian that has \"less stability\", perhaps by analyzing the eigenvalues at the optimum. Furthermore, is there a connection between the idea that increasing instability helps generalization to the literature that says that flat optima help generalization?\n\n- There is no discussion of using the reverse-over-reverse autodiff method for computing the Hessian vector product instead of a forward-over-reverse autodiff method which can save memory and hence could also be faster on a GPU:\n\nIn principle you can get the directional derivative of the scalar g^T(omega) * sigma with respect to omega in the direction of sigma by evaluating\n\ndg/domega * sigma = (g(omega + eps * sigma) - g(omega)) / eps\n\nor similarly\n\nd(g*sigma)/domega * sigma = (g(omega + eps * sigma) * sigma - g(omega) * sigma) / eps\n\nat a factor two of the cost of evaluating only the gradient (gradient evaluated at omega). Alternatively, forward-mode autodiff can be used to the same effect and the dropout idea can still be applied by setting some of the sigma to zero and hence eliminating the cost further due to dropped nodes in the computational graph.\n\n- The experimental section does not analyse the regularizer itself or the connection to theory of the Hessian trace regularization and is instead focused on just demonstrating beating baselines:\n\nThere is currently no analysis of the trade-off of the bias of the trace estimator by using more dropout vs additional compute cost, just the statement \"Although increasing the probability to select parameters can improve test accuracy, the time consumption will increase a lot\". The paper could be improved by adding an empiricial analysis of the accuracy of the trace estimator.\n\nThere is also no analysis of how trace accuracy impacts generalization performance and trace itself, e.g. you could add plots showing how the true Hessian trace changes throughout training.\n\nIt would be good to see the training and cross validation curves for different hyperparams of overlayed penalty terms (like Hessian trace term, L2 norm term etc.) to see how they impact training and how stable training is to the corresponding hyperparameters. It would be good to have an analysis of how much does Hessian trace term add on top of hyper-param tuned other regularization, e.g. for Jacobian regularization the paper says just \"we set number of projections nproj = 1 and weight values λJR = 0.01\" which does not necessarily suggest that there was a sufficient hyperparameter tuning effort.\n\nAnother interesting question to analyze would be: How does the added regularizer impact the training dynamics, e.g. faster or slower convergence in the beginning vs end of training and why?\n\n\nSome more general recommendations to improve the paper:\n- Checking grammar and spelling typos will improve readability and quality of the paper\n- State the generalization bound as a theorem as in the original paper (Theorem 4.1)\n- In related work perhaps also connect to other works analyzing Hessian and Fisher matrix spectrum and related norms for generalization (e.g. \"Fisher-Rao Metric, Geometry, and Complexity of Neural Networks\")\n", "summary_of_the_paper": "The authors propose a method for estimating the trace of the Hessian of the cross-entropy loss with respect to the weights of a neural network classifier. They suggest adding the trace estimator as a regularizing penalty term for training neural networks with improved generalization.\n\nThe authors give two theoretical motivations for regularizing the Hessian trace:\n- For linear models the trace of the Hessian of the cross-entropy loss with respect to the logits appears as a factor in a term for bounding generalization error. The authors draw a connection between the Hessian with respect to the weights and the Hessian with respect to the logits and argue that penalizing the trace of the Hessian with respect to the weights also leads to a tighter bound on the generalization error.\n- For the continuous gradient dynamics around the optimum the Hessian eigenspectrum is the negative eigenspectrum of the Jacobian of the gradient dynamics which determine the stability of the dynamic system. The authors argue that decreasing Hessian trace increases Jacobian trace and hence reduces data dependend stability which will avoid overfitting.\n\nThe authors' suggested method for estimating the trace uses the stochastic Hutchinson trace estimator which samples a random vector sigma with E(sigma sigma^T) = I to get E(sigma^T H sigma) = tr(H). The authors appear to use back-propagation through a back-propagation gradient approach to compute sigma^T H sigma by first computing the gradient g = dl/domega of the loss l with respect to the weights omega, and then computing the gradient of the inner product g^T sigma which is a scalar, with respect to the weights again. This gradient-over-gradient can be inner product multiplied again with sigma.\n\nIn order to save cost the authors suggest a drop out method setting a fraction of the terms of sigma to 0 which saves a fraction of the derivative computation and correspondingly drops terms of the trace sum.\n\nThe authors then present experimental results evaluating their suggested regularization method in combination with and compared to other regularization methods on\n- CIFAR10 / Resnet18\n- CIFAR100 / Wide Residual Network\n- WikiText-2 / 2-layer LSTM", "summary_of_the_review": "I do not recommend to accept the paper in its current form since the connection between the theoretical justification and the empirical evidence is weak and the experiments in the paper are not sufficient for an empirical understanding of how the suggested regularizer really works.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "g9GveEhBHDp", "reviewer": "Reviewer_imd7", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper makes a strong case for the proposed regularization scheme: penalizing the trace of the Hessian.  The empirical results included in the paper seem to indicate that the proposed regularization scheme improves network performance (and generalizability).  The technical details are accessible, and it is easy to follow the mathematical steps presented in the paper that lead to the final \"efficient\" algorithm for estimating the trace of the Hessian.  Section 3.4 that describes the linear stability analysis cogently argues why minimizing the trace of the Hessian is one way to improve the generalizability of the network. \n\nThe paper also suffers from some shortcomings.  While reading the paper, I stumbled upon a number typos: misspellings, incorrect double quotes (very common in LaTeX users), etc.  These can be easily fixed.\n\nA more serious issue with the paper is its use of the word \"efficient,\" as in, \"to efficiently implements [sic] the Hutchinson method.\"  I did not see any experiments that showcase how efficient the proposed method is.  I would have liked to see some runtime figures that drive home the point about the efficiency of the proposed approach.  Is there a way to remedy this?\n\nThe paper actually presents two methods for estimating the trace of the Hessian.  The first method does not use dropout,  whereas the second, more efficient, method uses dropout.  The paper contains no results using the first method.  It would be nice to know the performance gap between the first and the second method.  In many situations performance and efficiency are inversely related.  Does that hold here too?  Alternately, may be deep neural networks upend this trend and the second method posts both better performance and better runtimes.", "summary_of_the_paper": "The paper develops a new regularization method for deep neural networks.  The proposed methods penalizes the trace of the Hessian.  The paper adapts Hutchinson method for estimating trace of a positive semi-definite matrix for the purposes of estimating trace of the Hessian.  This adaptation uses a dropout mechanism to efficiently compute trace of the Hessian.   The paper also studies the effects of minimizing the trace of the Hessian using linear dynamical systems theory, and shows that lowering the trace of Hessian diminishes the stability of the equilibrium points in the parameter (i.e., weight) space.  Thereby, reducing overfitting and improving generalizability of the network.  The paper concludes with a set of experiments on standard deep learning benchmarks---CIFAR-10, CIFAR-100, and WIKI-TEXT2---and in almost every case the proposed SEHT-D scheme achieves best results.\n", "summary_of_the_review": "Overall the paper is well-written.  It proposes a new regularization scheme that seems to improve deep neural networks performance on CIFAR-10, CIFAR-100, and WIKI-TEXT2 datasets.  The proposed scheme improves performance for both convolutional and recurrent neural networks.  The paper uses linear dynamical systems theory to argue that minimizing the trace of the Hessian improves the generalizability of the network.  Lastly, the mathematical description for \"efficiently\" computing the trace of the Hessian is easy to follow.\n\nMy only issue with the paper is that it doesn't include any results to support the assertion that the algorithm for estimating trace of the Hessian is efficient.\n", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "b5EBP-PgpsB", "reviewer": "Reviewer_GDik", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "The paper starts well, with clear and simple goal - to introduce a new regualrizer. It motivates it through an existing generalization bound on linear models. Although, this bound does not translate to DNN models, I think this as a motivation is clearly presented and makes sense.\nThe stochastic estimators of the Hessian through Hutchinson method are well known and used in many places in ML. \n\nEquation 11 is somewhat redundant, since anyone familiar with how modern autodiff packages perform efficient Hessian-vector products  would know that they are just restating well known facts. For the least they should be citing Perlmutter's paper on fast Hessian-vector products, which basically define this. \nIn this whole section on the Hessian estimator, I'm very unclear what is the benefit of the proposed extension SETH-D over SETH-H? Theoretically, due to dropping blocks of the whole network it should be more efficient. However, practically the way that modern autodiff packages work, it would be a significantly challenge to actually compute Hessian-vector products such that they benefit from this. E.g. if you just zero out those parts of `sigma` and perform still perform \"dense\" products on line 5 and 6 of Algorithm 2 it won't make any difference. Later in the results the authors do cite that based on the probability that they pick `p` they gain different run times. I think the authors should provide more details on how in practice they actually implement SEHT-D in order to take advantage of the structure of \\sigma? These details although might look like not so important, are probably the only reason why one would even use SEHT-D over SETH-H. \n\nI have quite a bit of issue with the whole section 3.4. The authors attempt to motivate the trace trough stability analysis. First, that they expand the idealized ODE, for which there is now some literature which uses backward analysis to derive more accurate ODEs and in practice show case that the idealized version is not well capturing what happens with the dynamical system. Second, the main point at the end of this whole section, talking about stability of equilibrium points, is to argue that the goal of the method is to have a \"more unstable dynamical system\". Why would we want that - I don't know. The last paragraph presents an argument that since standard training is a function only of the training data, we would like it to be unstable in order to avoid overfitting. However, the authors do not present any evidence or even elude to why that last statement would be true and I'm quite doubtful if it will at all. We can make training unstable in many other ways (injecting significant amount of noise for instance) which do not lead to better generalization. As a result I would advice the authors to either significantly change, or just remove this whole section, as it currently serves no purpose. \n\nFinally, on the results section I have a few major issue with some of the numbers I'm seeing, which make me a bit doubtful of the overall results.\n\nThe WRN-28-10 has a baseline with 73% top 1 accuracy. The defualt pytorch open-source results from https://github.com/meliketoy/wide-resnet.pytorch indicate a 80% accuracy (ZCA - no-dropout 5e-4 weight decay). Perhaps the authors are suing different type of preprocessing scheme, or there are other differences, but this should be cleared out as to why they get so much worse results with the same model.\n\nAnother issue I have with the presented results, is that the authors do not mention to have done any search over the regualrizer penalty for any of the competing methods (and for theirs they do only two values - 0.01 0.05). They must mention why this is done - if they pick the values from one of the papers, they should make that more clear and also should make it clear that problems considered are equivalent to those in the reference. For many regualrizers their optimal weight changes between different problems. This might or might not explain some of the degradation results we see for some of the other techniques. \n\nProbably, the final experiment is best execute, with significant hyper parameter search and good ablations, which makes it the only one that makes me trust its results, and unsurprisingly there the benefits are significantly more marginal then in the others. However, it is unclear what is Validation and Test Perplexy exactly? Do you train and cross-validate the hyper parameter to select them, and then retrain on train + validation and measure Test, or do you just measure the performance on the Test set of the best cross-validated model? Please clarify. ", "summary_of_the_paper": "The authors propose a regularizer for training DNNs - specifically adding the trace of the Hessian w.r.t. the parameters of the model.\nThey present two efficient stochastic estimators of the trace, based on the Hutchinson method and their own extension to it, specific to NNs. \nThe authors also provide a motivation of the regularizer through a known generalization bound and through the perspective of dynamical systems stability. \nIn three experiments - on CIFAR10, CIFAR100 and Wiki-Text they present results suggesting that the proposed method is performing better on average than other techniques. \n", "summary_of_the_review": "Good and simple idea. I have some issues with lacking implementation details of the method that the authors use in practice. One of the sections seems completely redundant to the narrative, unless the authors provide better explanation for their final statements there. Some of the experiments seem to have lower numbers than publicly known results and in general there is a lack of exploring hyper parameters for other regualrizers. The last experiment is probably the only one that is done really well.", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "4: All of the claims and statements are well-supported and correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}]

Author rebuttal:

Meta-review: This paper regularizes deep neural networks via the Hessian trace.  The algorithm is based on Hutchison’s method, further accelerated via dropout.  Connection to the linear stability of dynamical system is discussed.  The proposed regularization shows favorably in the experimental results.

The idea of the method is clear.  The paper’s writing needs a lot of improvement because there are a number of grammatical errors.  The major technical concerns include: a) the experimental results are still not convincing; b) the explanation of favoring instability in the dynamical system that resorts to overfitting prevention (reviewer GDik).  I’ve read the rebuttal, but remain unconvinced.
Decision: Reject