Year: 2021

Abstract:
In this paper we develop a novel regularization method for deep neural networks by penalizing the trace of Hessian. This regularizer is motivated by a recent guarantee bound of the generalization error. Hutchinson method is a classical unbiased estimator for the trace of a matrix, but it is very time-consuming on deep learning models. Hence a dropout scheme is proposed to efficiently implements the Hutchinson method. Then we discuss a connection to linear stability of a nonlinear dynamical system. Experiments demonstrate that our method outperforms existing regularizers such as Jacobian, confidence penalty, and label smoothing. Our regularization method is also orthogonal to data augmentation methods, achieving the best performance when our method is combined with data augmentation.

heading: 1 INTRODUCTION
text: Deep neural networks (DNNs) are developing rapidly and are widely used in many fields such as image classification, machine translation, language modeling and speech recognition. As more and more models are proposed in the literature, deep neural networks have shown remarkable improvements in performance. However, among various learning problems, over-fitting on training data is a great problem that affects the test accruacy. So a certain regularization method is often needed in the training process.
In linear models, Ridge Regression (Hoerl & Kennard, 1970) and Lasso (Tibshirani, 1996) are usually used to avoid over-fitting. They are also called L2 and L1 regularization. L2 regularization has the effect of shrinkage while L1 regularization can be conductive to both shrinkage and sparsity. From the Bayesian perspective, L2 and L1 regularization can be interpreted with normal prior distribution and laplace prior distribution respectively.
Apart from L2 and L1 regularization, there are many other forms of regularizers in DNNs. The most widely used one is Weight-Decay (Krogh & Hertz, 1992). Loshchilov & Hutter (2019) also showed that L2 regularization and Weight-Decay are not identical. Dropout (Srivastava et al., 2014) is another method to avoid over-fitting by reducing co-adapting between units in neural networks. Dropout has inspired a large body of work studying its effects (Wager et al. (2013); Helmbold & Long (2015); Wei et al. (2020)). After dropout, various regularization schemes can be applied additionally.
In this paper, we propose a new regularization by penalizing the trace of second derivative of loss function. We refer to our regularization method as Stochastic Estimators of Hessian Trace (SEHT). On one hand, our hessian regularization is valuable to guarantee good generalization. On the other hand, from the perspective of dynamical system, it influences the stability of the system, in which parameters move in the parameter space on the basis of training data. In our experiments, Hessian regularization shows competing test performance and low time consumption with our stochastic algorithm.

heading: 2 RELATED WORK
text: There are many regularization methods in previous work. Label Smoothing (Szegedy et al., 2016) estimates the marginalized effect of label-dropout and reduces over-fitting by preventing a network from assigning full probability to each training example. Confidence Penalty (Pereyra et al., 2017)
prevents peaked distributions, leading to better generalization. A network appears to be overconfident when it places all probability on a single class in the training set, which is often a symptom of over-fitting. DropBlock (Ghiasi et al., 2018) is a structured form of dropout, it drops contiguous regions from a feature map of a layer instead of dropping out independent random units.
Data augmentation methods are also used in practice to improve model’s accuracy and robustness when training neural networks. Cutout (DeVries & Taylor, 2017) is a data augmentation method where parts of the input examples are zeroed out, in order to encourage the network to focus more on less prominent features, then generalize to situations like occlusion. Mixup (Zhang et al., 2017) extends the training distribution by incorporating the prior knowledge that linear interpolations of feature vectors should lead to linear interpolations of the associated targets.
Sokolić et al. (2017) first proposed Jacobian regularization, a method focusing on the norm of Jacobian matrix with respect to input data. It was proved that generalization error can be bounded by the norm of Jacobian matrix. Besides that, Jacobian matrix shows improved stability of the model predictions against input perturbations according to Taylor expansion. Hoffman et al. (2019) showed that Jacobian regularization enlarges the size of decision cells and is practically effective in improving the generalization error and robustness of the models. To simplify calculation, stochastic algorithm of Jacobian regularization was also proposed.
Motivated by Jacobian regularization, we consider the generalization error and stability of the model respect to Hessian matrix. Then we combine Linear Stability Analysis and propose Hessian regularization with corresponding stochastic algorithms. We compare our Hessian regularization with other methods and demonstrate promising performance in experiments. The main idea to estimate the trace of Hessian matrix is Hutchinson Method(Avron & Toledo, 2011) and the algorithm was also discussed by Yao et al. (2020a). We make an improvement by designing a new probability distribution to dropout parameters which decrease time consumption obviously without losing generalization.
Hessian information is powerful tool used on analyzing the property of neural networks. Yao et al. (2020b) designed AdaHessian, a second order stochastic optimization algorithm. Yu et al. (2021) used Hessian trace to measure sensitivity, developing a Hessian Aware Pruning method to find insensitive parameters in a neural network model and a Neural Implant technique to alleviate accuracy degradation. However, their methods are static in essence, we focus on dynamical motion of parameters in parameter space. Sankar et al. (2021) also proposed a Hessian regularization. They forcused on the layerwise loss landscape via the eigenspectrum of the Hessian at each layer. We start from different perspectives, generalization error and dynamical system of parameters. Our experiments also shows better results than Sankar et al. (2021)’s method.

heading: 3 HESSIAN REGULARIZATION
text: Here we introduce a Hessian regularization method based on generalization error and corresponding stochastic algorithms in details. Then we discuss linear stability analysis to explain why Hessian regularization can prevent neural networks from over-fitting.

heading: 3.1 TRACE OF HESSIAN MATRIX
text: In this study, we consider a multi-class classification problem. Input x is a N-dimensional vector, where x ∈ X ⊆ RN , with X is the input space. Y = {1, 2, ...,M} is the label space, which means that we have M classes. Each input x has a label y ∈ Y . Sample space is defined as S = X × Y . An element of S is denoted by s = (x,y). We assume that samples s from S are drawn according to a probability distribution P . A training set of n samples drawn from P is denoted by Sn = {si}ni=1 = {(xi,yi)}ni=1.
Our goal is to find a classification function f , which takes x ∈ RN as input and outputs z = f(x). z is a M-dimensional score vector, where each element zi is the score x belonging to category i ∈ Y . The highest score indicates the most probable label. So the estimated label is given as ŷ = g(x)= arg maxi zi = arg maxi∈Y f(x)i.
A loss function is used to measure the discrepancy between the true label y and the estimated label g(x). In this paper, we use the cross-entropy loss,
`(f(x), y) = − log exp(f(x)y)∑ y′ exp(f(x)y′) . (1)
The empirical loss of the classifier f(x) associated with the training set is defined as
`emp(f) = Ê[`(f(x),y)] = 1
n ∑ si∈Sn `(f(xi),yi), (2)
and the expected loss of the classifier f(x) is defined as
`exp(f) = E[`(f(x),y)] = Es∼P [`(f(x),y)], (3) then the difference between `emp(f) and `exp(f) is called generalization error:
GE(f) = ||`exp(f)− `emp(f)||. (4)
Wei et al. (2020) showed generalization bound of linear models with cross-entropy loss of M classes. Let W is the weight matrix, µ(W ) = Ê[‖J‖2] and v(W ) = Ê[tr(H)]. For linear models, the Jacobian matrix J is a vector defined as ∂`
∂z and the Hessian matrix H is defined as { ∂ 2` ∂zi∂zj }M×M . With probability 1 − δ over the training examples, for all weight matrices W satisfying the norm bound ||W T ||2,1 ≤ A, the following bound holds:
E[¯̀]− 1.01Ê[¯̀] . (Aµ(W )) 2 3 (θB) 1 3
n 1 3
+ A √ Bv(W )θ√ n + BA2θ n(log2( BA 2θ v(W )n ) + 1) + ζ. (5)
Here with some fixed bound B > 0, ¯̀= min{`, B},
‖W ‖2,1 = ∑ j √∑ i (W 2ij),
θ = log3(nM) max i ‖xi‖22 ,
ζ = B(log(1/δ) + log log n)
n .
(6)
So one can guarantee good generalization when the trace of Hessian matrix and norm of Jacobian matrix are small. On one hand, when learning with gradient descent, we want to find a local or global minimum of loss function. Naturally, at minimum the gradient is zero and the norm of Jacobian matrix is small near minimum. So gradient descent helps us to ensure the norm of Jacobian Matrix to be small.
On the other hand, the trace of Hessian Matrix is hard to be constrained by gradient descent. From this aspect, we proposed Hessian regularization for linear models as:
1 n tr(H`,z). (7)
It’s the trace of second derivative of loss ` with respect to output of linear model z, which is also the end nodes of a linear model. A DNN is consist of many layers, with each layer being viewed as a linear model (except the nonlinear activation functions). Thus. we generalize the Hessian regularization to every node in a DNN and define it as
1 n tr(H`,ω). (8)
It’s the trace of second derivative of loss ` with respect to parameters ω.
Here we define a new loss with our Hessian regularization as
Loss = `emp(f) + λ · 1
n tr(H`,ω), (9)
where λ controls the strength of our Hessian regularization.

heading: 3.2 HUTCHINSON METHOD
text: In a typical DNN, there are more than millions of parameters. So the calculation of Hessian matrix is difficult. Hutchinson Method (Avron & Toledo, 2011) is an unbiased estimator for the trace of a matrix. Let A be an n × n symmetric matrix with tr(A) 6= 0. Let σ be a random vector whose entries are i.i.d Rademacher random variables (Pr(σi = ±1) = 12 ), then σ
TAσ is an unbiased estimator of tr(A), based on the following equation:
tr(A) = tr(AI) = tr(AE[σσT ]) = E[tr(AσσT )] = E[σTAσ]. (10)
In this paper, we consider the trace of Hessian matrix H , which is the second derivative matrix. Since the Rademacher random vector is irrelevant to network parameters, we expand the expression of Hutchinson estimator as follow:
σTHσ = σT d dldω dω σ = σT ( d dldω dω · σ + dl dω · dσ dω ) = σT ( d( dldω · σ) dω ). (11)
Based on Equation 11 and dσdω = 0, we only have to calculate the gradient g = dl dω and the derivative of dldω ·σ, instead of the whole Hessian matrix. The whole calculation process only include two inner products and two derivations. We refer to the Hutchinson stochastic estimator of Hessian trace as SEHT-H.
Algorithm 1: SEHT-H Input: n-dimensional gradient g Output: Estimation of tr(H)
1 trace = 0 ; 2 for i = 1 to maxIter do 3 σ ∼ Rademacher(n) ; /* n-dim vector with each element sampled from Rademacher distribution. */ 4 v = g · σ ; /* inner product */ 5 h = dvdω ; /* derivative of v */ 6 t = σT · h ; /* inner product */ 7 trace += t ; 8 end 9 return tracemaxIter
Even though it’s a stochastic algorithm, it cost much time because of a great number of parameters. So we propose another efficient algorithm below based on the basic idea of Hutchinson Method.

heading: 3.3 DROPOUT METHOD
text: This stochastic algorithm is inspired by Dropout (Srivastava et al., 2014). Every node in the neural network has a probability p to be igonred in the training process to reduce co-adaptations. In our Hessian regularization, we want to lower the trace tr(H) = ∑ i ∂2` ∂ωi∂ωi
, the sum of diagonal elements of Hessian matrix. Based on the idea of Dropout, we can ignore some parameters in constraining the tr(H), since reducing the partial sum of diagonal elements can have a large chance to reduce the total sum. The partial sum of diagonal element is denoted as t̃r(H). Considering the layer structures of neural networks, the process of randomized parameter selection can be divided into two steps: randomly select layers in neural network with probability p1 and randomly select parameters in the selected layers with probability p2. In other words, when carrying out Hessian regularization, we ignore other layers with probability 1 − p1, and ignore other parameters in selected layers with probability 1− p2. In our experiment, we simply set p1 = p2. In addition, Hutchinson method shows a technique, which avoids to calculate the whole Hessian matrix. Here we define a new probability distribution Q(p) (if x ∼ Q(p), then Pr(x = ±1) = p and Pr(x = 0) = 1 − 2p). Then, let σ be a random vector whose entries are i.i.d Q random variables,
E[σσT |fix the position of 0 in σ] = Ĩ . (12)
Here Ĩ = diag(0, 1), a diagonal matrix with diagonal elements equal to 0 or 1.
Similar to Equation 10, if we fix the position of 0 in σ, we have unbiased estimator of the partial sum of diagonal elements:
t̃r(A) = tr(AĨ) = tr(AE[σσT ]) = E[tr(AσσT )] = E[σTAσ]. (13) We can expand the expression same as Equation 11 and transform the calculation process into two inner product and two derivation. We name this method as SEHT-D.
Algorithm 2: SEHT-D Input: probability p, parameter ω in selected layers, and corresponding n-dim gradient g Output: estimation of t̃r(H)
1 trace = 0 ; 2 for i = 1 to maxIter do 3 σ ∼ Q(p) ; /* n-dim vector with each element sampled from Q(p) distribution */ 4 v = g · σ ; /* inner product */ 5 h = dvdω ; /* derivative of v */ 6 t = σT · h ; /* inner product */ 7 trace += t ; 8 end 9 return tracemaxIter
In our experiments, we mainly test the performance of this method. Compared with other regularization methods, our Hessian regularization shows improved test performance with fast training speed.

heading: 3.4 LINEAR STABILITY ANALYSIS
text: Training process can be regarded as a motion in the parameter space, from the initial parameter to a local or global minimum. Current parameters is a point in the parameter space and gradient descent is the move of the parameter point each time. Then we can see gradient descent from another perspective. Original gradient descent is defined as a series of discrete updates:
ωt+1 = ωt − ηgt. (14) Here ωt is the parameters in step t, η is learning rate and gt is gradient.
If we consider learning rate as discrete time interval to move ω in parameter space, ∆t = η, then ∆ω
∆t = −g(ω, x). (15)
We assume time interval or learning rate is small enough, approximately we get a contiunous form: dω
dt = −g(ω, x). (16)
Thereafter, with an initial condition, we have the complete trajectory of parameter point based on ordinary differential equation (ODE) theory. The process of gradient descent is transformed to a Nonlinear Dynamical System. So we introduce Linear Stability Analysis of Nonlinear Dynamical Systems below.
Nonlinear Dynamical System (Thomas Witelski & Mark Bowen, 2015) is defined as a differential function dxdt = f(x, t) with an initial condition x(0) = x0. x(t) = (x1(t), x2(t), . . . , xn(t)) ∈ R n is the vector of state variables, describes n properties of interest in the system, evolving for times t ≥ 0 and starting from a given initial state x(0) = x0. The rate functions for the rates of change of each xi, dxidt = fi, have similarly been collected in a vector f = (f1, f2, . . . , fn), where each fi can potentially depend on all of the state variables. Since fi doesn’t have to be a linear function, the system is called Nonlinear Dynamical System. A classic example of a dynamical system from mechanics is the system for motion of a particle. A system is called non-autonomous if the rate function have an explicit dependence on time. In this paper, we only focus on autonomous systems,
dx dt = f(x), x(0) = x0. (17)
The equilibrium points x∗ are defined by positions where rate functions vanish, f(x∗) = 0. If any solution starting near an equilibrium point leaves the neighbourhood of x∗ as t → ∞, then x∗ is called asymptotically unstable, while if all solutions starting within the neighbourhood approach x∗ as t→∞ then the equilibrium is called asymptotically stable. Lyapunov (1992) gave more rigorous definition and discussion, known as Lyapunov Stability Theory. The idea of Lyapunov Stability can be extended to infinite-dimensional manifolds, where it is known as Structural Stability (Pugh & Peixoto, 2008), which concerns the behavior of different but ”nearby” solutions to differential equations.
For a basic Nonlinear Dynamical System, to ensure equilibrium point x∗ is stable, we need to construct Jacobian matrix J = ∂f∂x . Given the equilibrium point x∗, J(x∗) is a constant matrix. Using the conclusion from Linear State Space Model, if all eigenvalues of J(x∗) have real parts less than zero, then x∗ is stable. If at least one of the eigenvalues has a real part greater than zero, then x∗ is unstable.
Back to gradient descent, it can be regarded as a Nonlinear Dynamical System, according to Equation 16. The local or global minimum, which is the goal of gradient descent, is the equilibrium point in such system, since that minimum point satisfies the condition dωdt = 0. The Jacobian matrix in this dynamical system is the negative of the Hessian matrix in our Hessian regularization, J(ω∗) = −∂g ∂ω = −∂ ∂`∂ω ∂ω = −H`. And the trace and the eigenvalues of J(ω∗) are also negative of the trace and the eigenvalues of H`.
It’s easy to see that in our Hessian regularization, we lower the trace of Hessian Matrix H`, thus increasing the trace of the Jacobian matrix J(ω∗). However, in real matrix, complex eigenvalues are always conjugate and the trace are always real number. When we increase the trace, we increase the real parts of eigenvalues of J(ω∗) to some extent. In other word, the goal is to preclude the stability of equilibrium point by our Hessian regularization.
Why we want instability in this dynamical system of gradient descent? The stability of local or global minimum shows the stability toward training data. The whole dynamical system of gradient descent is a motion based on training set since the motion of parameters is decided by Equation 17. Equation 17 is consist of two parts: one is the initial condition, which is randomized in a DNN, while the other part is the differential function Equation 16, which is directly determined by training data. In other words, with different training data, the parameters have different trajectory in parameter space. It only use information about training data and the equilibrium point depends on training data. Therefore, the stability is relevant to training set and reducing the stability to some extent can avoid over-fitting to training data.

heading: 4 EXPERIMENTS
text: We evaluate our Hessian regularization with other regularization methods on a variety of datasets. We also combine data augmentation methods with our Hessian regularization to test its efficacy.

heading: 4.1 IMAGE CLASSIFICATION

heading: 4.1.1 CIFAR-10
text: The CIFAR-10 dataset consists of 60000 32x32 colour images in 10 classes, with 6000 images per class. There are 50000 training images and 10000 test images.
For our experiment, we use ResNet-18 (He et al., 2016) as the backbone neural network. For all models, we use Weight Decay of 5× 10−4. We set learning rate 0.01, batch size 32, momentum 0.9 and all models were trained 200 epochs with Cosine Annealing (Loshchilov & Hutter, 2016). For Jacobian regularization, we set number of projections nproj = 1 and weight values λJR = 0.01. For DropBlock, block size = 7 and keep prob = 0.9. For Confidence Penalty and Label Smoothing, we set weight value 0.1, which is found to work best on CIFAR-10 by Pereyra et al. (2017) in their experiments. For our Hessian regularization SEHT-D, we also set weight value 0.1, testing with probability value 0.01 and 0.05. In addition, we always add output layer in the selected layer, which corresponds to Equation 7. Cutout size of 16 × 16 pixels is used in our experiment, based on the
validation results mentioned by DeVries & Taylor (2017). For mixup, α = 0.1. Averages and 95% confidence intervals are estimated over 5 distinct runs.
We also observe the computational cost of the training with SEHT-D (maxIter=1, prob=0.01) to be only 1.2 times that of the baseline. Although increasing the probability to select parameters can improve test accuracy, the time consumption will increase a lot. In our experiment, SEHT-D (maxIter=1, prob=0.05) costs 1.5 times of the baseline.
In our experiments, we find that Jacobian regularization and Dropblock have worse performance than the baseline with Weight-Decay. Confidence Penalty has slight improvement and Label Smoothing has obvious improvement. However, our SEHT-D shows better results, compared with all other regularization methods which are tested. In addition, our SEHT is suitable for combination with other methods. When we combin SEHT-D with mixup, we get the best test accuracy in our experiment, 1.45 more than baseline.
In Sankar et al. (2021)’s experiment, they got test accuracy 88.13 on CIFAR-10 with ResNet-18, which is much worse than our result: 94.35 with SEHT-D (maxIter=1, prob=0.01) and 94.37 with SEHT-D (maxIter=1, prob=0.05). Moreover, their improvement based on their methods is only 0.02 for full-network method and 0.10 for middle-network method. Our Hessian regularization method improves the model 0.35 on test accuracy with SEHT-D (maxIter=1, prob=0.01) and improves 0.37 on test accuracy with SEHT-D (maxIter=1, prob=0.05), which is over 3 times of their improvement.

heading: 4.1.2 CIFAR-100
text: CIFAR-100 dataset is just like the CIFAR-10, except it has 100 classes containing 600 images each. There are 500 training images and 100 testing images per class.
We use Wide Residual Networks (WRN) (Zagoruyko & Komodakis, 2016) as the backbone neural network. We use WRN-28-10 specifically, with depth 28 and fixed widening factor of 10. For all models, we use Weight Decay of 5 × 10−4. We set batch size 32, momentum 0.9 and all models were trained 200 epochs. The learning rate is initially set to 0.1 and is scheduled to decrease by a factor of 5 after each of the 60th, 120th, and 160th epochs. We test Dropout with a drop probability of p = 0.3, determined by Zagoruyko & Komodakis (2016)’s cross-validation. For Confidence Penalty, Label Smoothing and our Hessian regularization, we set weight value 0.1. Cutout size of 8 × 8 pixels is used according to DeVries & Taylor (2017)’s validation results. For mixup, we still set α = 0.1. Averages and 95% confidence intervals of top-1 accuracy and top-5 accuracy are estimated over 5 distinct runs.
In this experiments, Confidence Penalty has better top-5 accuracy, worse top-1 accuracy and Label Smoothing has better top-1 accuracy, worse top-5 accuracy, compared with baseline method. However, our SEHT-D method shows better results on both top-1 accuracy and top-5 accuracy, improving 1.13 and 0.65 respectively. Our Hessian regularization method also perform better when combined
with other method on this dataset. When testing together with Dropout, our SEHT-D has best accuracy, compared with Label Smoothing and Confidence Penalty. Our SEHT-D also improves cutout for 0.39 on top-1 accuracy, 0.26 on top-5 accuracy, and improves mixup 0.21 on top-1 accuracy, 0.09 on top-5 accuracy.
Our experiments on Image Classification shows that our Hessian regularization method outperforms other regularization methods and can be efficiently combined with data augmentation methods.

heading: 4.2 LANGUAGE MODELING

heading: 4.2.1 WIKI-TEXT2
text: The Wiki-Text language modeling dataset is a collection of over 100 million tokens extracted from the set of verified Good and Featured articles on Wikipedia.
We use a 2-layer LSTM (Hochreiter & Schmidhuber, 1997). The size of word embeddings is 512 and the number of hidden units per layer is 512. We run every algorithm for 40 epochs, with batch size 20, gradient clipping 0.25, and Dropout ratio 0.5. We perform a grid search over Dropout ratios {0, 0.1, 0.2, 0.3, 0.4, 0.5} and find 0.5 to work best. We tune the initial learning rate from {0.001, 0.01, 0.1, 0.5, 1, 10, 20, 40} and decrease the learning rate by factor of 4 when the validation error saturates. We find initial learning rate 20 works best. Parameters are initialized from a uniform distribution [−0.1, 0.1]. For label smoothing, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1} and find 0.01 to work best. For the confidence penalty, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1} and find 0.01 to work best. For our Hessian regularization, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1}, probability values {0.01, 0.05}. Weight 0.01 and probability 0.05 work best. Averages and 95% confidence intervals are estimated over 5 distinct runs.
In this experiments with LSTM, our SEHT-D has the best test perplexity and Label Smoothing shows best validation perplexity. SEHT-D improves the model 2.61 on test perplexity. Confidence Penalty performs only sightly better than the baseline method.
We also tested with a 2-layer GRU (Cho et al., 2014). The size of word embeddings is 512 and the number of hidden units per layer is 512. We run every algorithm for 40 epochs, with batch size 20, gradient clipping 0.25 and Dropout ratio 0.3. We perform a grid search over Dropout ratios {0, 0.1, 0.2, 0.3, 0.4, 0.5} and find 0.3 to work best. We tune the initial learning rate from {0.001, 0.01, 0.1, 0.5, 1, 10, 20, 40} and decrease the learning rate by factor of 4 when the validation error saturates. We find initial learning rate 20 works best, same as LSTM. Parameters are initialized from a uniform distribution [−0.1, 0.1]. For label smoothing, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1} and find 0.05 to work best. For the confidence penalty, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1} and find 0.005 to work best. For our Hessian regularization, we perform a grid search over weight values {0.001, 0.005, 0.01, 0.05, 0.1}. Weight 0.001 works best. We set probability values 0.01. Averages and 95% confidence intervals are estimated over 5 distinct runs.
Our Hessian regularization method has both the best validation perplexity and the best test perplexity, improving 2.83 and 2.61 respectively compared with baseline method. Confidence Penalty surpasses Label Smoothing with GRU model, compared with LSTM. Label Smoothing also show better results than baseline.
Our experiments on Language Modelling demonstrate that all these three regularization methods can improve models, while our SEHT-D is the best.

heading: 5 CONCLUSION
text: We propose a new regularization method named as Stochastic Estimators of Hessian Trace (SEHT). Our method is motivated by a guarantee bound that a lower trace of the Hessian can result in a lower generalization error. We also explained our method with dynamical system theory. Our experiment shows that SEHT-D yields promising test performance with fast training speed. SEHT-D also achieves better results when combined with data augmentation methods.

heading: 6 REPRODUCIBILITY STATEMENT
text: Our codes are available at here .

Review summaries:
The authors propose a regularizer for training DNNs - specifically adding the trace of the Hessian w.r.t. the parameters of the model. They present two efficient stochastic estimators of the trace, based on the Hutchinson method and their own extension to it, specific to NNs.  The authors also provide a motivation of the regularizer through a known generalization bound and through the perspective of dynamical systems stability.  In three experiments - on CIFAR10, CIFAR100 and Wiki-Text they present results suggesting that the proposed method is performing better on average than other techniques.  

The authors propose a method for estimating the trace of the Hessian of the cross-entropy loss with respect to the weights of a neural network classifier. They suggest adding the trace estimator as a regularizing penalty term for training neural networks with improved generalization.  The authors give two theoretical motivations for regularizing the Hessian trace: - For linear models the trace of the Hessian of the cross-entropy loss with respect to the logits appears as a factor in a term for bounding generalization error. The authors draw a connection between the Hessian with respect to the weights and the Hessian with respect to the logits and argue that penalizing the trace of the Hessian with respect to the weights also leads to a tighter bound on the generalization error. - For the continuous gradient dynamics around the optimum the Hessian eigenspectrum is the negative eigenspectrum of the Jacobian of the gradient dynamics which determine the stability of the dynamic system. The authors argue that decreasing Hessian trace increases Jacobian trace and hence reduces data dependend stability which will avoid overfitting.  The authors' suggested method for estimating the trace uses the stochastic Hutchinson trace estimator which samples a random vector sigma with E(sigma sigma^T) = I to get E(sigma^T H sigma) = tr(H). The authors appear to use back-propagation through a back-propagation gradient approach to compute sigma^T H sigma by first computing the gradient g = dl/domega of the loss l with respect to the weights omega, and then computing the gradient of the inner product g^T sigma which is a scalar, with respect to the weights again. This gradient-over-gradient can be inner product multiplied again with sigma.  In order to save cost the authors suggest a drop out method setting a fraction of the terms of sigma to 0 which saves a fraction of the derivative computation and correspondingly drops terms of the trace sum.  The authors then present experimental results evaluating their suggested regularization method in combination with and compared to other regularization methods on - CIFAR10 / Resnet18 - CIFAR100 / Wide Residual Network - WikiText-2 / 2-layer LSTM

In this paper, the authors propose an efficient regularizer for the trace of the Hessian. To circumvent the extra computation needed to do the naive Hutchinson estimator (which pretty much amounts to taking *three* derivatives of the loss), the authors subsample the weights matrices, and then a second time internal to each weights matrix). Experimental results on CIFAR-10 and CIFAR-100 are presented: the proposed regularizer helps, but unfortunately not in a statistically significant way.

The paper develops a new regularization method for deep neural networks.  The proposed methods penalizes the trace of the Hessian.  The paper adapts Hutchinson method for estimating trace of a positive semi-definite matrix for the purposes of estimating trace of the Hessian.  This adaptation uses a dropout mechanism to efficiently compute trace of the Hessian.   The paper also studies the effects of minimizing the trace of the Hessian using linear dynamical systems theory, and shows that lowering the trace of Hessian diminishes the stability of the equilibrium points in the parameter (i.e., weight) space.  Thereby, reducing overfitting and improving generalizability of the network.  The paper concludes with a set of experiments on standard deep learning benchmarks---CIFAR-10, CIFAR-100, and WIKI-TEXT2---and in almost every case the proposed SEHT-D scheme achieves best results.