Stochastic Modified Equations and Dynamics of Dropout Algorithm
Authors: Zhongwang Zhang
Yuqing Li
Tao Luo
Zhi-Qin John Xu
Venue: ICLR 2024 poster (2024)
Keywords: Stochastic modified equations
dropout
noise structure
flatness
Primary area: learning theory
TL;DR: We present a comprehensive theoretical framework and empirical evidence for studying  Stochastic Modified Equations (SME) in dropout, shedding light on the implicit regularization through revealing the noise structure introduced by dropout.

Abstract: Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and impact on achieving good generalization abilities remain to be further understood. In this work, we start by undertaking a rigorous theoretical derivation of the stochastic modified equations, with the primary aim of providing an effective approximation for the discrete iterative process of dropout. Meanwhile, we experimentally verify SDE's ability to approximate dropout under a wider range of settings. Subsequently, we empirically delve into the intricate mechanisms by which dropout facilitates the identification of flatter minima. This exploration is conducted through intuitive approximations, exploiting the structural analogies inherent in the Hessian of loss landscape and the covariance of dropout. Our empirical findings substantiate the ubiquitous presence of the Hessian-variance alignment relation throughout the training process of dropout.

Published as a conference paper at ICLR 2024
STOCHASTIC MODIFIED EQUATIONS AND DYNAMICS OF DROPOUT ALGORITHM
Zhongwang Zhang', Yuqing ; Luo 12,3444 І Zhi-Qin John Xu X343461
School of Mathematical Sciences, Shanghai Jiao Tong University
2 EMA-Shanghai, Shanghai Jiao Tong University
Institute of Natural Sciences, MOE-LSC, Shanghai Jiao Tong University
4 Qing Yuan Research Institute, Shanghai Jiao Tong University
5 Shanghai Artificial Intelligence Laboratory
6 Shanghai Seres Information Technology Company, Ltd
ABSTRACT
Dropout is a widely utilized regularization technique in the training of neural networks, nevertheless, its underlying mechanism and impact on achieving good generalization abilities remain to be further understood. In this work, we start by undertaking a rigorous theoretical derivation of the stochastic modified equations, with the primary aim of providing an effective approximation for the discrete iterative process of dropout. Meanwhile, we experimentally verify SDE's ability to approximate dropout under wider range of settings. Subsequently, we empirically delve into the intricate mechanisms by which dropout facilitates the identification of flatter minima. This exploration is conducted through intuitive approximations, exploiting the structural analogies inherent in the Hessian of loss landscape and the covariance of dropout. Our empirical findings substantiate the ubiquitous presence of the Hessian-variance alignment relation throughout the training process of dropout.
INTRODUCTION
Dropout is technique integrated into gradient-based algorithms for training neural networks (NNs) (Hinton et al. 2012 [Srivastava et al. 2014). It constitutes a pivotal component contributing to the attainment of state-of-the-art test performance in deep learning (Tan and Le 2019 Helmbold and Long 2015). The key idea behind dropout is to randomly deactivate a subset of neurons during the training process. Specifically, the output of each neuron is multiplied by random variable that takes the value 1/p with probability and zero otherwise. This random variable is independently sampled at each feedforward operation. Despite its widespread adoption and empirical success, the mechanism by which dropout enhances generalization in deep learning remains an ongoing area of research.
The noise structure introduced by stochastic algorithms plays crucial role in understanding their training behaviors. A series of recent works reveal that the noise structure inherent in stochastic gradient descent (SGD) is vital for exploring flatter solutions (Keskar et al.) 2016] Feng and Tu 2021 Zhu et al. 2018). Analogously, the dropout algorithm introduces a specific form of noise, acting as an
implicit regularizer that facilitates improved generalization abilities (Hinton et al. 2012 Srivastaval
ct al. 2014
Wei et al. 2020 Zhang and Xu 2022
Zhu et al. 2018).
In this paper, we first employ the stochastic modified equations (SMEs) Li al. 2017) framework to analyze the dynamics of the dropout algorithm applied to two-layer NNs. By application of SMEs, exhaustive quantification of the leading order dynamics governing dropout, and we fortify this analytical approach through some empirical validations. addition, we calculate the covariance matrix associated with the noise introduced by dropout. Hence our analytical exploration
we embark
" "Corresponding author: liyuqing 551 @ @sjuu.duucnn
1 'Conesponding author: luotao41 @ @jttueddu.cn !Coressonning author; xuzhiqin @ sjtu.edu.cn

---

Published as a conference paper at ICLR 2024
is further enriched by investigation of the alignment relation between this covariance matrix and the Hessian matrix, a relationship conceptually framed as the
alignment relation (Zhu et al.] 2018: [Wu et al.] 2022). We emphasize that this alignment property occupies central role in sculpting the flatness attributes inherent in the solutions favored by NN models, and has been firmly established that flatter solutions tend exhibit enhanced generalization capabilities (Keskar et al.)
2016 Neyshabur et al. 2017)
2 RELATED WORKS
A flurry of recent works aims to shed light on the regularization effect conferred by dropout. Wager
al. (2013) show that dropout performs form of adaptive regularization in the context of linear
regression and logistic problems. McAllester (2013) propose PAC- -ayessan bound, whereas Wan
jet al. (2013); Mou et al. (2018) derive some Rademmache--ompleeiityyyy
error bounds specifically
tailored for dropout. Cavazza et al. (2018); Mianjy and Arora (2020); Wei et al. (2020); Arora
ctt al.
2021 demonstrate that dropout regularizes the inductive bias under different settings. Jin et al. (2022) try to explain the generalization ability of dropout from the new perspective of weight
expansion. Finally, ZZhang and Xu (2022) establish that dropout facilitates condensation (Luo et al.
2021 Zhou al. 2021 2022) through an additional regularization term endowed by dropout.
Continuous formulations have been extensively utilized to study the dynamical behavior of stochastic algorithms. Li et al. (2017 2019) present entirely rigorous and self-contained mathematical formulation of the SME framework that applies to a wide class of stochastic algorithms. Furthermore, Feng et al. (2017) adopt semigroup approach investigate the dynamics of SGD and online PCA. Malladi et al. (2022) derive the SME approximations for the adaptive stochastic algorithms including RMSprop and Adam, additionally, they provide efficient experimental verification of the validity of square root scaling rules arising from the SMEs.
One noteworthy observation is the association between the flatness of minima and improved gen- eralization ability (Li et al., 2017: Jastrzebski et al.) [017 2018). Specifically, SGD is shown to preferentially select flat minima, especially under conditions of large learning rates and small batch sizes (Jastrzebski et al. 2017 2018 Wu et al. (0118). Paayyan (2018) (019) attribute such enhance- ment of flatness by SGD to the similarity between covariance of the noise and Hessian of the loss function. Furthermore, Zhu et al. (2018); Wu et al. (2022) unveil the Hessian-variance alignment property of SGD noise, shedding light on the role of SGD in escaping from sharper minima and locating flatter minima.
PRELIMINARY
In this section, present the notations and definitions utilized in our theoretical analysis. We remark that our experimental settings are more general than the counterparts in the theoretical analysis.
3.1 NOTATIONS
We set a special vector 1,..., 1)T by 1 1= 1,,1,,,... 1)T whose dimension varies. We set for the number of input samples and m for the width of the NN. We let [r]] 21, . We denote @ as the Kronecker tensor product, and (..)) for standard inner product between two vectors. We denote vector 2 norm as |1-112- vector function Los norm as I|- We also denote Tr(-) the trace of square matrix, Ia as the identity matrix of size d x c, and || signifies the Frobenius norm of matrix. Finally, we denote the set of continuous functions f(-)= .R. - R possessing continuous derivatives of order to and including r by C' (RD), the space of bounded measurable functions by By (RR and the space of bounded continuous functions by Co (R.)).
3.2 TWO-LAYER NEURAL NETWORKS AND LOSS FUNCTION
We consider the empirical risk minimization problem given by the quadratic loss:
min Rs(0)
= 2n
Σ (fo(z.)) )- - -v.)²
(1)
0
1=1
2

---

Published as a conference paper at ICLR 2024
where S == { Bi))i= is the training sample, fe( is the prediction function, e are the parame- ters, and their dependence is modeled by two-layer NN with m hidden neurons
fo(x)) = Σ a (.o...
where E R
vec(Oa. )w E RDD where D == mdd+mmd++ throughout this paper. We remark that is the set of parameters with 6, = voc( ( A. = vec( ) and we impose hereafther that the activation function o(-) be continuously differentiable up order 6, i.e., E C (R). More precisely, = vec( 9-- [p-)mm1 ), where for each r E [m], 9, := (a,, w.) and the bias term b, be incorporated by expanding æ and w, (x 1)T and (wT,b.)
0
3.3 DROPOUT
For a fixed learning rate > 0, then at the N-th iteration where EN Nn,, a scaling vector d, E R sampled with independent random coordinates: For each lt E [m]]
with probability p. with probability p.
(ŌN)
and we observe that N62 is an i.i.d. Bernoulli sequence with EŠN = 1. With slight abuse of
notations, the a-fields JN : {ō .. .r)] forms natural filtration. We then apply dropout
to the two-layer NNs by computing
m
fo 8)
(()))a,
and we denote the empirical risk associated with dropout by
2
1 n
m
ReRoP (0:
(Jo(
v:)²
Σ
(00 y1
(5)
2n
2n
We remark that the parameters at the N-th step are updated as follows:
Ox = ON a Redrop (0N-1;ōN)
(6)
where 0p- = (0). Finally, we denote hereafter that for all E [n]]
== (0N=1;;0)) = Sax (mg 8N)
y/-
STOCHASTIC MODIFIED EQUATIONS FOR DROPOUT
In this section, we approximate the iterative process of dropout (6) in the weak sense (Definition I..
4.1 MODIFIED LOSS
As the dropout iteration O reads
n n Σ c Voc,"
ON 1N-1 Vo
Rdroop (0N-1:ōN)
Since 0 vec( ( m ) vec (((ar, w. w.)) then given ON for each A [m], the expecta- tion of the increment restricted to q reads
n
n
Egy-1
V N
Box-1
ee (6x)) Va (ano(w]Zi))
1
E, V qe (a+o(w)x;1)
(.o (aoo(w)x)))
P
1=1
3

---

Published as a conference paper at ICLR 2024
where we denote for simplicity that e; ca) Ex= a,T((.xx) - ys- Compared with e," Ca does not depend the random variable Òn. Hence, we define the modified loss Ls(-) RP R for dropout:
(s(
2n
2mp
1=1
We observe that for each & E [m], the gradient of Ls restricted to qu reads
1 1-P
V. (0)
Va. (ano(w)
agd aw Va, (ano(w(a;)),
np
which indicates that given - the conditional expectation of the increment of the parameter at the N-th step reads
ON -N--NNI=-n1a Bo
V R (0N-1;ōN)
Then in the sense of expectations, x N20 follows close to the gradient descent (GD) trajectory of (O) with fixed learning rate 7). In the above procedure, we focus on the drift of dropout and disregard its fluctuation term as we merely consider the first conditional moment of the parameter increment. Please refer to Appendix G. for the detailed derivation of L.s.
4.2 STOCHASTIC MODIFIED EQUATIONS
In pursuit of more comprehensive understanding of the dynamics of dropout, we integrate the fluctuation term of dropout into our analysis. Firstly, as shown above, we observe that given N
O Vo __ (0N(ON-)),
(8)
where Ls(-): R + R is the modified loss defined in ( and V()) R - RD represents the fluctuation term of dropout. When given d V has mean and covariance n1E(O where (-)) IR' + Rbx, whose expression is deferred to Section 5.1
Consider the stochastic differential equation (SDE),
d (O,) dt + o dW,, O, e
(9)
where W, standard D-dimensional Brownian motion. Its Marrr-Maruyama discretization with step size 77 > 0 at the N-th step reads
N NN- + nb (o (O)(N-I) + vio (0))N-1)) ZN.
where EN NO, Ip) and Og (()). Thus, if we set
( = - VeLs(e).
a = ()))
(10)
O Oo.
then we would expect (9) to be "good" approximation of with time identification n/N. Based on the previous work (Li et al. 2017), we use approximations in the weak sense (Kloeden and Platen 201T Section 9.7) since the path of dropout and the corresponding SDE are driven by noises sampled in different spaces.
To compare different discrete time approximations, we need to take the rate of weak convergence into consideration. and also need choose appropriate class of functions the space of test functions. We introduce the following set of smooth functions:
{
CM (R²)
I CM (RP)
Σ CID <0o) 18ISM
(11)
where D is the usual differential operator. We remark that C,M (IR") is subset of ()R the class of functions with polynomial growth, which is chosen to be the space of test functions in previous works (Li et al. 2017 Kloeden and Platen 2011). To ensure validity of our analysis, assume that
4

---

Published as a conference paper at ICLR 2024
Assumption 1. There exists T > O, such that for any t [0, I there exists unique t-continuous solution θ to SDE (9). Furthermore, for each E [3]. there exists C(T", Ou) > 0. such that
E (11x(018) S C(TT,00)
sup 054$T"
(12)
Moreover, for the dropout iterations No O, such that given any learning rate n S no. then for all N | Nr. and for each [3].
6 let 0 < < 1, T O and set NI.- == 1 There exists
there exists C(T*, Go no) > O, such that
sup O
(10N 12) & CC(T, .0 mo)-
(13)
We remark that local existence of the solution to SDE and estimates of all 2/- 21-moments of the solution to SDE can guaranteed for smooth coefficients and sufficiently small time T > 0. Moreover, as the constants C(T Qn) and C(T 0, ?/o) exponential in time, the 2l-moments of the solution might blow up for large enough T*; which is unavoidable since we are unable to impose the uniform Lipschitz condition on VL and Σ. However, our empirical findings suggest that the SME still possess the desired approximation ability to dropout even for large learning rate, as shown in Fig, [ (a). We also remark that if (IR P) chosen be the functions in Li et al. (2019), then similar relations to (12) and (13) shall be imposed, except that in our cases, we only require the second, fourth and sixth moments be uniformly bounded.
Definition 1. The SDE ( is order weak approximation to the dropout (6) if for every E C₁ PR. there exists C > O and y/o > 0, such that given any S to and T T", then for all N N..
[ - [g((O)) C(T",,99 ,0.
(14)
We now state formally our approximation results.
Theorem 1. Fix time T < T and learning rate 1j > 0. Ifa c (R), then for all [0, T), the stochastic processes θ satisfying
dO, = bi (O,) a (O,)
(15)
is an order-1 approximation of dropoul
6 where
b (e) = - (oLs(O),
σ ( V (E)
and the expression of L.(()) is located in (7). and the expression of E() can be found in Appendix Moreover, a a E (B). then for all t e [),T), the stochastic processes O, satisfying
do, b2 (O,) + 02 (0:)
(16)
an order-2 approximation of dropout 6 where
bz(O) ( Ve (Ls( + "
[V
2 + vπ ())
It is noteworthy that our findings reproduce the explicit regularization effect attributed to dropout (Wei al. 2020 Zhang and Xu 2022). This regularization effect modifies the expected training objective from the empirical risk Rs 0 to (0). and it stems from the inherent stochastic nature of dropout. Unlike SGD, where the noise arises from the stochasticity involved in the selection of training
samples, dropout introduces noise by means of the stochastic removal of parameters.
4.3 NUMERICAL SIMULATION OF STOCHASTIC MODIFIED EQUATIONS
In this subsection, we conduct an empirical validation of the effectiveness of SMEs for dropout. This validation is conducted through exploration of the resemblance between the numerical simulation of the SME and the real-time training process of dropout. For the numerical simulation the SME, unless otherwise specified, we employ the Faler-Maruyama method to approximate its dynamic
5

---

Published as a conference paper at ICLR 2024
evolution by the order-1 approximation. It is worth noting that the noise term (0) in Equ. (o involves the computation of the square root of the covariance matrix E Consequently, the size of the covariance matrix significantly affects the speed and accuracy of the numerical simulation process. To mitigate the computational demands associated with the covariance matrix, we resize the MNIST data to 7 thereby reducing the number of network parameters involved in the simulations. Additional details of the experimental setup can be found in Appendix
6.9
201
0.9
10a
50t, 1.0033
dropout pm2pp-03
10%
drppot, pm33 pwib
[[n0] rubll:
30-1
0.5%
30"1
0.5
0.06
[[P0]]
10-1
30
skpp
Ktope
0
2000
4000
6000
6000
10060
10
10
10
10"
10F
spech
Searning
(a) loss trajectory
(b) order-1 approximation
(c) order-2 approximation
Figure I: We train two-layer fully connected networks on MNIST. The curves and points are derived from the average results of individual trials, each of which utilized the same initialization distribution. (a) The training loss trajectory obtained by SME simulation or dropout training under four cases of different learning rates and dropout rates. The error bands, portrayed with greater transparency. are derived from the maximum and minimum loss values observed across these six random trials each training step. (b, c) Convergence-ordee verification of first-order and second-order SME approximations. Each point represents the value of [[((,)) =E(0,) under given learning rate (abscissa) and dropout rate (color).
Fig. [illustrates the close correspondence between the dynamics of the SME and dropout throughout the training process. This similarity is examined from two perspectives: the trajectory of loss functions and the approximation order. In Fig. we emphasize that although dropout introduces a noise component that modifies the loss function from (0) to Ls (0), when contrasted with SMEs, the training behavior of gradient descent utilizing Ls (o) as the loss function distinet from dropout. This distinction becomes apparent when large learning rates are employed in the optimization process.
To comprehensively assess the similarity between dropout and SME simulations, we first consider four distinct cases, each characterized by various dropout rates and learning rates shown in Fig. D Fig. a) depicts the evolution of loss values under these distinct settings for both dropout and SME simulations. To ensure the robustness of our analysis, for each configuration, we conduct six independent trials of dropout and SME simulations, all initialized with identical distribution. The displayed curves represent the means of these six random trials. Moreover, the error band, indicated by lighter colors, covers the range between the maximum and minimum loss values obtained from the six trials. The observed alignment of loss trajectories between the SME simulation and dropout, as evident in Fig. [ a)) underscores a prominent resemblance in their respective loss trajectories.
To further evaluate the similarity of their parameters, we verify the approximation orders different SME simulations. Figs. [b, c) numerically verify the approximation orders of the first-order and the second-order approximation equation in Theorem [ resppecivvey. Each point represents the value of [IE)))) E(@:)ll, under a given learning rate (abscissa) and dropout rate (color). The expectation obtained by calculating the mean of 10 independent experiments with the same initialization for both dropout and SME simulation. The logarithmic plots clearly illustrate the experimental validation of the theoretical approximation order of SME, for both order- 1, shown in Fig. Tb), and order-2, shown in Fig. c). Additionally, under the same learning rate, larger values of p exhibit enhanced approximation capabilitics. This improvement is attributed to the reduction in noise with increasing p. consequently minimizing the impact of noise on the training process.
We also conduct experiments to validate the applicability of the SME approximation in complex networks and SGD settings. In the former, simulate complex networks through numerical approximation of the drift term, while in the latter, we rely on the fact that SGD noise is unbiased. For thorough discussion and detailed numerical results, please refer to Appendix

---

Published as a conference paper at ICLR 2024
880
dropout SDE
0.065
Lest acc
10-1
0.060
Ditt acc
bestt
best
0.064
0.055
9700
990g
10:
6 10
20
4000
2000 opach
opoch
(a) 0.005, 0.9
(b) 17 1, 0.3
Figure 2: The test loss and test accuracy trajectory obtained by SME simulation, dropout training, and gradient descent training with loss function Ls(0) under two settings. The curves are derived from the average results of six individual trials, each of which utilized the same initialization distribution. The error bands, portrayed with greater transparency, are derived from the maximum and minimum loss values observed across these six random trials each training step.
Fig. 2 depicts the test loss and test accuracy associated with 0.005, 0.9 and 1, p 0.3. In Fig. [[a). the trajectories of loss and accuracy, generated using three different training methods, exhibit remarkable degree of concurrence. This phenomenon can be primarily attributed to small learning rate, where the diffusion component significantly diminishes, consequently endowing the drift term Ls(0) with dominant influence. In contrast, illustrated in Fig. bb) a notable divergence becomes evident in the loss and accuracy trajectories. This discrepancy arises from the impact of the diffusion term during training, particularly with the application of large learning rates. Notably, in contrast to the training behavior of gradient descent utilizing (0( as the loss function, the trajectory generated by SME simulation exhibits closer alignment with the trajectory of dropout.
the utilization
It is noteworthy that large learning rate and dropout contribute of diffusion, methods incorporating noise such as SME and dropout tend to exhibit enhanced generalization performance. As demonstrated in Figure bb.). the test accuracy attained by Ls (0) consistently remains below the lower threshold of test accuracy attained by noise-inclusive methods for the major portion of the training duration. the sequel, we delve into an in-depth exploration of the influence exerted by noise on our learning outcomes.
increased amplitude
THE EFFECT OF DROPOUT NOISE STRUCTURE
We begin this section by examining the noise structure of dropout.
5.1 EXPLICIT FORM OF THE NOISE STRUCTURE OF DROPOUT
In this subsection, present the expression for the covariance E()). Once again, 0 vec( (q: 34=)))) vec (a,, w.)). then as denote covariance of Voltdrop (0 8N) by Σ(ON-1),
Exr Emr(0N-1) = Cov V Edtoo
(0N-1;6N), V ReS (0N-1;ōN))
then
Σ1
Σ1
Σim Σzm
Σn
Σ22
Σ
Σmi
Σmm
Σnm
Such expression of Σ arises from the inherent decoupling properties among neurons within the two-layer neural network. Due to space limitation, we defer the detailed expression of Egr to Appendix [
7

---

Published as a conference paper at ICLR 2024
5.2 INTUITIVE EXPLANATION FOR THE HESSIAN-VARIANCE VARIANCE ALIGNMENT RELATIONS
In this subsection, endeavor to show the structural similarity between the covariance and the Hessian in terms of deessian-variane alignment relations. Under the assumption that close to global minimum, we intuitively derive the structural similarity between the Hessian and the covariance at the final stage of the training process as follows:
H 1
1- p
Volo (2:;) 8 Volo (x:)
(a, (a, @0 VV. (1,0(1))):)))
1
0(0)
4,1 Vo.Vol Volo + 4422
V aa (a.o(w)a;;)) & Va. (a.o(
P
(17)
where H()) = V2 (0), and 4,1 == (c.)2 1-P Σm- 0, 4,2 == (e:) A detailed
derivation 17 provided in Appendix With the establishment structural similarity through
the aforementioned intuitive approximations outlined in (17). we proceed to empirically investigate the intricate relationship between the Hessian and the covariance, and details of the experimental settings can be found in Appendix A
5.3 EXPERIMENTAL RESULTS ON THE HESSIAN-VARIAS ALIGNMENT RELATIONS
Motivated by the relation 17,, we empirically demonstrate the structural similarity between the Hessian and the covariance of dropout, and this demonstration serves validate the alignment relation. Based on this relation, the introduction of dropout noise has the potential to expedite the escape of the model from locating sharp minima, thereby effectively enhancing the flatness of the solution. Furthermore, in Appendix B we also explore another relationship between the Hessian and the covariance known as the inverse variance-flatnes relation (Feng and Tu 2021). which also contributes to aiding the model in avoidance of the sharp minima during its optimization process.
1.0
0.8
e103
608
0.7
0.9
P- --0.
0.6
p p-
0.6.
p=0.0. n.1
0.5
2 0.8. 0.02
0-0.8 n-00
p 0.8. 0.02
0.4
1000
2000
3000
50
100
150
epach
epoch
FNN, MNIST
(b) VGG16, CIFAR-10
Figure 3: The cosine similarity (0,) between the Hessian of the loss function and the covariance of the dropout noise at each training epoch for different choices of dropout rate and learning rate. (a) The FNN with size 784-50-50-10 is trained on the MNIST dataset using the first 10000 examples as the training dataset. The dropout layer is added after the first hidden layer. (b) The VGG16 is trained on the CFIAR-10 dataset using the full examples the training dataset. The dropout layers are added after the first two convolutional layers of each block and the first fully-connected layer. The calculation of (0,,) is performed every five epochs.
To investigate the Hessian-Variance alignment relation, we study the cosine similarity quantity a(0,) between the covariance matrix Σ, (0,) and the Hessian matrix H; == H (0.) at each time step t. Σ, is the covariance matrix of 'D Derad. a collection of gradients calculated with different dropout variables 5 sampled the tth step, whose detailed definition can be found in Section B.I On the other hand, H, is the Hessian of the loss function evaluated at the tth iteration. Then the crucial
This variable also used in Wu al. 2022 for studying SGD.
8

---

Published as a conference paper at ICLR 2024
cosine similarity metric a(Oi) is formally expressed as:
Tr(H)E))
ax(O.) Hell Exll
(18)
As depicted in Fig. it is evident that throughout the training process, a())) consistently attains
values surpassing 0.85 in Fig. (aa and 0.7 in Fig. [3)b). and these observations hold true across varying learning rates and dropout rates. It's worth noting that, based on 100 samples, the average cosine similarity between the two random matrices based on the selected parameters is only 7.8 x 10-1 The cigenvalues of the two random matrices are derived from the eigenvalues of the Hessian matrix and covariance matrix of the selected parameters, and the corresponding eigenvectors are sampled from a normal distribution and normalized. The model parameters are derived from the final model represented by the blue line in Fig. (a. Consequently, the introduced noise is highly anisotropic in that it aligns well with the Hessian matrix across all directions. We acknowledged that due to computational constraints, this experiment limits the trace calculation to a subset of parameters.
6 CONCLUSIONS AND DISCUSSIONS
Our main contribution comprises two key aspects. First, we derive the SMES that provide a weak approximation to the dynamics of the dropout algorithm applied to two-layer NNs. Second, we con- duct an empirical inquiry that demonstrates the persistent validity of the Hessian-variance alignment relation throughout the training process of dropout. The Hessian- variance alignment relation has been established to be beneficial for the model to locate flatter minima, thus indicating that dropout acts as an implicit regularizer that enhances the generalization power possessed by the model.
Extension of the SME framework to multi-layer networks and SGD. While our theoretical analysis has predominantly centered around the dropout algorithm applied to two-layer neural networks and GD, it is important to note that the derivation of SMEs is not confined exclusively to two-layer neural networks, GD, or even to the dropout algorithm. For various types of neural networks, the feasibility of constructing such modified equations remains viable, provided that the stochastic algorithm iteratively updates the parameters a recursive manner, i.c., iterations form a hime-homogencous Markov chain. Furthermore, this applicability holds as long as Taylor's 's theorem with the Lagrange form of the remainder remains valid for sufficiently small learning rates. It is worth acknowledging that the complexity introduced by multi-layer networks primarily arises from the presence of dropout layers within the activation functions. This introduces high degree of non-linearity to the loss with respect to the dropout variable, rendering it challenging explicitly calculate the drift and diffusion components of the SME. We numerically verify the SDE approximation capability of complex network structures and SGD in Appendix C
The effect of learning rate on dropout. In the small learning rate regime. wherein the noise term exerts slight influence, the loss trajectories of (s( and ( 8) exhibit a notable degree of congruence. This observation has been affirmed through theoretical and empirical validations. However, it remains imperative to maintain the diffusion term is important if we aspire to gain deeper insights into the nature of dropout algorithms or other stochastic algorithms. As illustrated in Fig. (b), in the large learning rate regime, the trajectory derived from the SME simulation aligns more closely with its dropout counterpart, in stark contrast to the trajectory arising from GD training on (0). Furthermore, SMEs consistently exhibit better generalization capability in comparison to GD. Therefore, a comprehensive analytical framework that duly accommodates both drift and diffusion terms stands as a more informative tool for the insightful analysis of dropout algorithms.
More refined analysis of noise structures. In addition to the Hessian-Variie Variance alignment relation, the structural similarity between the Hessian and the covariance engenders yet another intriguing relationship known as the inverse variance-flatness flatness relation (Feng and Tu 2021). Different from the Hessian- Variance alignment relation, it focuses more on the similarity of the two feature directions. In Appendix B an investigation has been conducted to examine the correlation between the noise structure introduced by dropout and the nature of the loss landscape. This relationship also plays pivotal role in assisting the model to steer clear of sharp minima. The high similarity of the eigenvectors of two matrices is a natural extension of the inverse variance- flatness relation, please refer to Appendix (orr for detailed validation results. In Appendix D) compare the effect of noise on the model in three training strategies, dropout, SGD, and parametric noise injection (Orvieto et al. 2023), which all appear to be helpful for flatness.

---

Published as a conference paper at ICLR 2024
ACKNOWLEDGMENTS
This work sponsored by the National Key R&D Program China Grant No. 0022YA1008220 (Z. X., T. L.). the National Natural Science Foundation of China Grant No. 92270001 (Z. X.), 12371511 I (Z. X.), 12101401 (T. L.). Shanghai Municipal of Science and Technology Major Project No. 202ISHZDZX0102 (Z. X., T. L.). Shanghai Municipal Science and Technology Key Project No. 22IC1401500 (T. L.). and the HPC of School of Mathematical Sciences and the Student Innovation Center, and the Siyuan-1 cluster supported by the Center for High Performance Computing at Shanghai Jiao Tong University.
REFERENCES
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, R. R. Salakhutdinov, Improving neural networks by preventing co-adaptation of feature detectors, arXiv preprint arXiv:1207. 1207.0580 (2012).
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, R. Salakhutdinov, Dropout: a simple way to prevent neural networks from overfitting, The journal of machine learning research 15 (2014) 1929-1958.
M. Tan, Q. Le. Efficientnet: Rethinking model scaling for convolutional neural networks, in: International conference on machine learning, PMLR, 2019, pp. 6105-6114.
D. P. Helmbold, P. M. Long, On the inductive bias of dropout, The Journal of Machine Learning Research 16 (2015) 3403-3454.
N. S. Keskar, D. Mudigere, J. Nocedal, M. Smelyanskiy, deep learning: Generalization gap and sharp minima, arXiv preprint arXiv: 04486 (2016).
P. Tang, On large-batch training for
Y. Feng, Y. Tu, The inverse periancce-llatessy relation in stochastic gradient descent is critical for finding flat minima, Proceedings of the National Academy Sciences 118 (2021).
Z. Zhu, Wu, B. Yu, L. Wu, J. Ma, The anisotropic noise in stochastic gradient descent: Its behavior of escaping from sharp minima and regularization effects, arXiv preprint arXiv:1803.00195 (2018).
C. Wei, S. Kakade, T. Ma, The implicit and explicit regularization effects dropout, in: International Conference on Machine Learning, PMLR, 2020, pp. 10181-10192.
Z. Zhang, Z.-Q. J. Xu, Implicit regularization of dropout, arXiv preprint arXiv:2207.05993 (2022).
Q. Li, C. Tai, Weinan, Stochastic modified equations and adaptive stochastic gradient algorithms, in: International Conference on Machine Learning, PMLR, 2017, pp. 2101-2110.
I. Wu, M. Wang, W. Su, The alignment property of sgd noise and how it helps select flat minima: A stability analysis, Advances in Neural Information Processing Systems 35 (2022)
B. Neyshabur, S. Bhojanapalli, D. McAllester, N. Srebro, Exploring generalization in deep learning, arXiv preprint arXiv: 1iXiv:1706.066 776.08894 (2017).
S. Wager, S. Wang, P. Liang, Dropout training as adaptive regularization, Advances in neural information processing systems 26 (2013) (0013)351-355 351-359.
D. McAllester, A pac-bayesian tutorial with dropout bound, arXiv preprint
(2013).
I. Wan, M. Zeiler, S. Zhang, Y. Lecun, R. Fergus, Regularization of neural networks using drop- connect, in: In Proceedings of the International Conference on Machine learning, Citeseer, 2013.
W. Mou, Y. Zhou, J. Gao, I.. Wang, Dropout training, data-dependent regularization, and generaliza- tion bounds, in: International conference on machine learning, PMLR, 2018, pp. 3645-3653.
J. Cavazza, P. Morerio, B. Haeffele, C. Lane, V. Murino, R. Vidal, Dropout a low-rank regularizer for matrix factorization, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2018, pp. 435-444.
10

---

Published as a conference paper at ICLR 2024
P. Mianjy, R. Arora, On convergence and generalization of dropout training, Advances in Neural Information Processing Systems 33 (2020).
R. Arora, P. Bartlett, P. Mianjy, N. Srebro, Dropout: Explicit forms and capacity control, in: International Conference on Machine Learning, PMLR, 2021, pp. 351-361.
G. Jin, X. Yi, P. Yang, L. Zhang, S. Schewe, X. Huang, Weight expansion: A new perspective on dropout and generalization, arXiv preprint urXiv:2201.09209 (2022).
T. Luo, Z.-Q... Xu, Z. Ma, Y. Zhang, Phase diagram for two-layer relu neural networks infinite- width limit, Journal of Machine Learning Research 22 (2021) 1-47.
H. Zhou, Q Zhou, T. Luo, Y. Zhang, Z.-Q. J. Xu, Towards understanding the condensation of neural networks at initial training, arXiv preprint arXiv:2105.11686 (2021).
H. Zhou, Q. Zhou, Z. Jin, T. Luo, Y. Zhang, Z.-Q. J. Xu, Empirical phase diagram for three-layer neural networks with infinite width, Advances Neural Information Processing Systems (2022).
Q. Li, C. Tai, E. Weinan, Stochastic modified equations and dynamics of stochastic gradient algorithms i: Mathematical foundations, The Journal of Machine Learning Research 20 (2019) 1474-1520.
Y. Feng, L. Li, J.-G. Liu, Semi-groups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations, arXiv preprint
(2017).
S. Malladi, Lyu, Panigrahi, S. Arora, On the SDEs and scaling rules for adaptive gradient algorithms, in: A. H. Oh, A. Agarwal, D. Belgrave, K. Cho (Eds.), Advances in Neural Information Processing Systems, 2022. URL: https: // openreview net forum? id=F 22mh jHKQP
H. Li, Z. Xu, G. Taylor, C. Studer, T. Goldstein, Visualizing the loss landscape of neural nets, arXiv preprint arXiv:1712.0991 (2017).
S. Jastrzebski, Z. Kenton, D. Arpit, Ballas, A. Fischer, Y. Bengio, Storkey, Three factors influencing minima in sgd, arXiv preprint arXiv:171 1.1117111.44233 (2017).
S. Jastrzebski, Z. Kenton, N. Ballas, A. Fischer, Y. Bengio, A. Storkey, On the relation between the sharpest directions of dnn loss and the sgd step length, arXiv preprint
(2018).
L. Wu, C. Ma, W. E, How sgd selects the global minima in over parameterized learning: A dynamical stability perspective, Advances in Neural Information Processing Systems (2018).
V. Papyan, The full spectrum of deepnet hessians at scale: Dynamics with sgd training and sample size, arXiv preprint arXiv: 1et.111111.00 (2018).
V. Papyan, Measurements of three-level hierarchical structure in the outliers in the spectrum of deepnet hessians, arXiv preprint arXiv:1901.00 1901.08244 (2019).
P. Kloeden, E. Platen, Numerical Solution of Stochastic Differential Equations, Stochastic Modelling and Applied Probability, Springer Berlin Heidelberg, 2011. URL: /RtL/htttp://books..ggggg.
com. hk/books d=BCvtsscm CMC
A. Orvieto, A. Raj, Kersting, F. Bach, Explicit regularization in overparametrized models via noise injection, in: International Conference on Artificial Intelligence and Statistics, PMLR, 2023, pp. 7265-7287.
A. Vaswani, N. Shazeer, N. Parmar, J. Uszkoreit, L.. Jones, A. N. Gomez, t.. Kaiser, I. Polosukhin, Attention is all you need, in: Advances in neural information processing systems, 2017, pp- 5998-6008.
R. Shwartz-Ziv, N. Tishby, Opening the black box of deep neural networks via information, arXiv preprint 10..:i11103.0011111 (2017).
S. P. Meyn, R. L. Tweedie, Markov chains and stochastic stability, Springer Science Business Media, 2012.
11

---

Published as a conference paper at ICLR 2024
Y. Feng, L. J.-G. Liu, Semigroups of stochastic gradient descent and online principal component analysis: properties and diffusion approximations, Communications in Mathematical Sciences 16 (2018) 777-789.
M. Hairer, Ergodic theory for stochastic pdes, preprint (2008).
B. Oksendal, Stochastic differential equations: an introduction with applications, Springer Science Business Media, 2013.
12

---

Published as a conference paper at ICLR 2024
A EXPERIMENTAL SETUPS
For Fig. D Fig. 2 Fig. 7b), we use the FNN with size 49-40-10 for the MNIST classification task, where the imput data is resized to 7 x 7 to reduce the amount of calculation of the root of the parameter covariance matrix. For Fig. Fig. 2 we train the network using GD with the first 1000 images as the training set. We add a dropout layer behind the hidden layer. The dropout rate and learning rate are specified and unchanged in each experiment. For Fig. 7 we train the network using SGD with the whole training set, the batch size is 32. We add dropout layer behind the hidden layer. The learning rate is 0.1 and p = 0.8. For the accuracy of SDE simulation, all parameters are used to calculate the Hessian matrix and covariance matrix.
For Fig. Ba), Fig. Fig. 6 Fig. B(a), we use the FNN with size 784-50-50-10 for the MNIST classification task. We add a dropout layer behind the second layer. The dropout rate and learning rate are specified and unchanged in each experiment. We only consider the parameter matrix corresponding to the weight and the bias of the fully-connected layer between two hidden layers. Therefore, for experiments in Fig. B a). D = 2500. For Fig. 3(a) Fig. 4
Fig. G we train the network
using GD with the first 10000 images the training set. For Fig. a), train the network using SGD with the whole training set with batch size 32.
For Fig. B). Fig. Bb), we use the VGG16 for the CIFARIO classification task. We train the network using SGD with batch size 128 for Fig. ()), and 32 for Fig. (b). We add the dropout layer after the first two convolutional layers of each block and the first fully-connected layer. The dropout rate and learning rate are specified and unchanged in each experiment. We only consider the parameter matrix corresponding the weight and the bias of the first convolutional layer. Therefore, D 1728 for the two experiments. Due to the computational cost of the Hessian matrix and the covariance matrix, we calculate cosine similarity every five epochs.
For Fig. 5 c, e, g), we add dropout layers after the convolutional layers, and for each dropout layer, p- = 0.8. We only consider the parameter matrix corresponding to the weight of the first convolutional layer of the first block of the ResNet-20. Models are trained using full-batch GD on the CIFAR 100 classification task for 1200 epochs. The learning rate is initialized at 0.01. Since the Hessian calculation of ResNet takes much time, only perform it at specific dropout rate and learning rate.
For Fig. bb d. f. h), we use transformer Vaswani et al. (017) with dmodel 50, de de 20, dg = 256, h = 4, 3, the meaning of the parameters is consistent with the original paper. We only consider the parameter matrix corresponding to the weight of the cully-connected layer whose output is queried in the Multi-Head Attention layer of the first block of the decoder. We apply dropout to the output of each sub-layer before it is added to the sub-layer input and normalized. In addition, we apply dropout to the sums of the embeddings and the positional encodings in both the encoder and decoder stacks. For each dropout layer, p 0.9. For the English-Germall translation problem, we use the cross-entropy loss with label smoothing trained by full-batch Adam based on the Multi30k dataset. The learning rate strategy is the same as that in Vaswani et al. (2117). The warm-up step is 4000 epochs, the training step is 10000 epochs. We only use the first 2048 examples for training to compromise with the computational burden.
For Fig. aa). we use the FNN with size 49-40-40-10-10 for the MNIST classification task, where the input data resized to 7 x to reduce the amount of calculation of the root of the parameter covariance matrix. We train the network using GD with the first 1000 images as the training set. We add a dropout layer behind each hidden layer. The learning rate is 0.1 and p = 0.5. For the accuracy of SDE simulation, all parameters are used to calculate the Hessian matrix and covariance matrix. The drift term is simulated by sampling the random variable 2000 times.
For Fig. 9 a). use the ReLU FNN with the width of 1000 to fit the target function follows,
f(x) = 2 o0z }
1
1 o( 3
where (xx) = Rz)U(x). The learning rate is 1 x 10 30 For SGD, the batch size is 1. For dropout, the dropout layer is added after the hidden layer with p 0.8. For parameter noise injection, use the layer noise with the noise standard deviation a = 0.001. We initialize the parameters in the linear regime, 0- ~ N (0, mdz)) )r where 1000 is the width of the hidden layer.
13

---

Published as a conference paper at ICLR 2024
For Fig. b). Fig. [c,, the ReLU FNN with the size of 784-5500-500-50- to classify the MNIST dataset, and the learning rate 0.01. The batch size 128. For dropout, the dropout layer is added after the hidden layers with p 0.8. For parameter noise injection, the layer noise with the noise standard deviation a = 0.03.
14

---

Published as a conference paper at ICLR 2024
B EXTENDED EXPERIMENTS ON NOISE STRUCTURES.
B.1 RANDOM DATA COLLECTION METHODS
We first introduce two types of dynamical datasets collected during dropout training to study the noise structure of dropout. These datasets are different from the training sample S.
Random trajectory data. The training process of NNs usually consists of two phases: the fast convergence phase and the exploration phase (Shwartz- Ziv and Tishby, 2017). In the exploration phase, the network is often considered to be near minimum, and the movement of parameters is largely affected by the noise structure. Based on the previous work (Feng and Tu 2021). we collect parameter sets D paraa = (0, ).V from N consecutive training steps the exploration phase, where 0, is the network parameter set at i-th sample step. This sampling method requires large number of training steps, so model parameters often have large fluctuations during the sampling process. To improve the sampling accuracy, we propose another type of random data to characterize the noise structure of dropout as follows.
Random gradient data. We train the network until the loss is near zero and then freeze the training process, then we sample N realizations of the dropout variable to get the random gradient dataset, i.e., D Darad
[g The i-th sample point gi is obtained as follows: i) Firstly, we generate a realization of the dropout variable ó; under a given dropout rate; ii) Then, we compute the gradient
of the loss function with respect to the parameters, denoted by g:(+) := V (:; 6,). Each element in Pgrad represents an evolution direction of network parameters, determined by the dropout variable. Therefore, studying the structure of 'D grad can help us understand how the dropout noise exerts an impact throughout the training process.
B.2 INVERSE VARIANCE-FLATNEE RELATION
The alignment relation studied above also implies the inverse variance- flatnesss relation, i.e.. the noise variance is large along the sharp direction the loss landscape, and small along the flat direction. In this subsection, we verify this relation by two sets of experiments. Firstly, we present two different approaches to characterize the flatness of loss landscape and the covariance of noise from the random trajectory data Ppara and random gradient data D gradd, then we numerically demonstrate the inverse fitince-fllatess relation. For convenience, D refers to either the dataset Dpara or the dataset D Ppaaa depending on its context, so is the case for their corresponding covariance E and Hessian H. We then proceed to the definitions of noise variance and interval flatness.
Definition 2 (noise variance). For dataset D and its covariance E, we denote X; [ eigenvalue of Σ and its corresponding eigen direction (EE)). Then term X Σ variance of 'D at the eigen direction 1y (E).
as the ith the noise
The interval flatness below characterizes the flatness of the landscape around local minimum.
Definition 3 (interval flatness For a local minimum 0% the loss function profile Rw along direction e reads:
(T, = Rs +v),,
where y represents the distance moved in the u direction. The interval flatness flannes F. is then defined
as the width of the region within which (1) < (0). We determine F. by finding two closest points 6', < 0 and 8 > 0 on each side of the minimum that satisfy R (e',) (l.).,) ) = (0).
The interval flatness is defined as:
F. 0-=0--0-
(19)
The experiments show that the result is not sensitive to the selection of the pre-factor 2. A larger value of Fú, means a flatter landscape in the direction v.
Remark,
We use PCA to study the weight variations when the training accuracy is nearly 100%. The networks are trained with full-batch GD for different learning rates and dropout rates under the same random seed. When the loss is small enough, we sample the parameters gradients of parameters N times A (N 3000 for this experiment) and study the relationship between (X) and ( (F) for both weight dataset D para and gradient dataset D grad
"This definition is also used Feng and Tu 2021
15

---

Published as a conference paper at ICLR 2024
F2 FAG
15-1
SIRA 1h
10-1
10
1:03
-30
18 -23
10 17
20-12
00pe
P.. F21
10 Frg
Fez
10
10
13
20
100
184
(a) FNN, Para
(b) FNN, D D
grad
(c) FNN, D D ppara
(d) FNN, D
grad
Figure 4: (a, b) The inverse relation between the variance {}{( and the interval flatness (F) liz for different choices of p and learning rate Ir with different network structures. The PCA is done for different datasets D sampled from parameters for the top line and sampled from gradients of parameters for the bottom line. The dashed lines give the approximate slope of the scatter. (c, d) The relation between the variance (Var(Proj, (Hr) ) and the eigenvalue {), (H) 1 for different choices of and learning rate ir with different network structures. The projection is done for different datasets sampled from parameters for the top line and sampled from gradients of
parameters for the bottom line. The dashed lines give the approximate slope of the scatter.
For different learning rates and dropout rates, Fig. Ma, b) reveal an inverse relationship between the interval flatness of the loss landscape denoted as ( and the noise variance represented by the PCA spectrum {Ag { Notably, a power-law relationship can established between (F))))) and { Specifically, in the low flatness region, the dropout-induced noise exhibits a large variance. As the loss landscape transitions into the high flatness regime, the linear relationship between variance and flatness becomes more evident. Overall, These findings consistently demonstrate the inverse relation between variance and flatness, exemplified in Fig. 4[a b). Subsequently, we delve into the definitions of Projected variance and Hessian flatness.
Definition (projected variance). For a given direction v R.. and dataset D ( where 0, TRD, the inner product of and 0; denoted by Projø (0:) (0;, then define the projected variance for 'D at the direction as follows,
Var(Proj..D)) = EN En (0.) 12)
N
where μ is the mean value of (Proj.
Definition 5 (Hessian flatness). For Hessian H, as we denote XeHH) by the i-th eigenvalue of H corresponding to the eigenvector v (H), we term ); ) the Hessian flatness along direction u, H)).
The eigenvalues of the Hessian evaluated at a local minimum often serve as indicators of the flatness of the loss landscape, and larger eigenvalues correspond to sharper directions. In our investigation, analyze the interplay between the eigenvalues of Hessian H at the final stage of the training process and the projected variance of dropout at each of the corresponding eigen directions. i.c., A (H) v.s. [Var(Proi (D)) N Specifically, we sample the parameters or gradients of parameters N times (N 1000 for this experiment), and examine the relationship between (): and [Var(Proje, (H) { for both the parameter dataset D Pars and the gradient dataset D grad.
Under various dropout rates and learning rates, Fig. [c. d) presents establishes a consistent power- law relationship between { and [Var(Proje, (#n) (D)) and this relationship remains robust irrespective of the choice between parameter dataset Dparn or the gradient dataset ' Dgrad- The positive correlation observed between the Hessian flatness and the projection variance provides insights into the structural characteristics of the dropout-induced noise. Specifically, these character- istics have the potential to facilitate the escape from sharp minima and enhance the generalization capabilities of NNs. Additionally, Fig. 4 highhlights the distinct linear structure exhibited by gradient sampling in comparison to parameter sampling, which corroborates the discussions outlined in Section B.I
Furthermore, we verify the inverse relation between the covariance matrix and the Hessian matrix of dropout through different data collection methods and projection methods larger network
16

---

Published as a conference paper at ICLR 2024
200
R22
10-
10
A.D
20p8- -27
Abpa-
Aar
Poce
(a) ResNet20.. para
(b) Transformer. D para
ResNet20, grad
Transformer.Ju grad
10
10-4
pibds I1:
80-4
20-4
Kapp
12-11
alpu-1
34w0 2.57
13:17
10-7
20
12:3
10
10:11
10-4
AP91
AdMI
AdH
AP0
(e) ResNet20.
(f) Transformer, para
gg) ResNet20,, Deend
(h) Transformer, Dgrad
Figure 5: (a, b, c, d) The inverse relation between the variance {Λ<E)) and the interval flatness (x) / for different choices of p and learning rate Ir with different network structures. The PCA is done for different datasets sampled from parameters for the top line and sampled from gradients of parameters for the bottom line. The dashed lines give the approximate slope of the scatter. (c, g, h) The relation between the variance {Var(Proj, (rrooo..)) (D)) and the eigenvalue {A((H)) 3- for different choices of and learning rate with different network structures. The projection is done for different datasets D sampled from parameters for the top line and sampled from gradients of parameters for the bottom line. The dashed lines give the approximate slope of the scatter.
structures, such ResNet-20 and transformer, and more complex datasets, such as CIFAR- and Multi30k, as shown in Fig.
B.3 THE SIMILARITY OF EIGENVECTORS BETWEEN THE HESSIAN MATRIX AND THE COVARIANCE MATRIX
1.0
oopeecoor indeo covariance LD
1005
0.8
Hestinn eigenvetio Index
0.6
projection
0.4
eigenvector
olgvector Hessian
(a) inner product
(b) projection ratio
Figure 6: (a) The cosine similarity between the first 15 eigenvectors of the Hessian matrix and the covariance matrix. The color represents the value of cosine similarity, and the lines on the top and left represent the energy ratio of the first & eigenvalues of the covariance matrix and the Hessian matrix, respectively, (b) The projection ratio of the first 9 eigenvectors of the covariance matrix in different eigendirections of the Hessian matrix.
We study the similarity of eigenvectors between the Hessian matrix and the covariance matrix. The model parameters are derived from the final model represented by the blue line in Fig. B(a). In Fig. 6 a), we study the cosine similarity between the first 15 eigenvectors of the Hessian matrix and
17

---

Published as a conference paper at ICLR 2024
the covariance matrix. The color represents the value of cosine similarity, and the lines on the top and left represent the energy ratio ((k) of the first & eigenvalues of the covariance matrix and the Hessian matrix respectively, that is, ((k) Σ (2.) NV (A,)², where X; is the i-th eigenvalue, and N is the number of the eigenvalues for the two matrices. It is easy to find that the energy ratio the first 9 eigenvectors of the two matrices far exceeds the energy ratio of other eigenvectors. Meanwhile, the first 9 eigenvectors of the two matrices are highly similar, which implies the alignment property between the Hessian matrix and the covariance matrices in the eigenspace.
In order to further characterize the similarity between the first few eigenvectors of the two matrices, we study the projection of the first 9 eigenvectors the covariance matrix different eigendirections of the Hessian matrix. As shown in Fig. bb). we calculate the value of a b (H), 2y (2))² for the index of the Hessian eigenvector and the index j of the covariance eigenvector, where a 42) is the cosine similarity between the two vectors V2 0ý). The energy proportions of the first nine eigenvectors of the covariance matrix in the first nine directions of the Hessian matrix are the first
all greater than 0.95, which means that the similarity of the eigenvectors corresponding few eigenvalues of the Hessian matrix and the covariance matrix is much higher than the similarity between them and other feature directions, which further confirms the alignment properties between the Hessian matrix and the covariance matrix in the eigenspace.
18

---

Published as a conference paper at ICLR 2024
C EXTENDED ANALYSIS OF DROPOUT UNDER COMPLEX NETWORK STRUCTURES AND SGD SETTING
In the main text, the theoretical part and experimental verification mainly focus on the settings of the two-layer network and GD. This is mainly due to the derivation of the specific forms of the drift term and covariance matrix. In fact, the SME framework in this work can be applied to more complex network structures and SGD settings without being limited to the explicit calculation of drift terms and covariance matrices. In the following, we conduct a detailed analysis of the feasibility of SME under complex network structures and SGD settings, and verify it through SME numerical simulation and the alignment between the Hessian matrix and covariance matrix under broader settings.
C.I FEASIBILITY ANALYSIS FOR NETWORK STRUCTURES AND SGD SETTING
The formulation of our stochastic modified equations relies on two critical components. Firstly, the algorithm updates parameters iteratively in recursive manner, i.e., O F(0x-1, 8x), thereby forming iterations as time-homogeneous Markov chain. This characteristic is foundational for applying the stochastic modified equation approach to weakly approximate the dynamics of dropout. The second crucial component involves the utilization of Taylor's theorem with the Lagrange form of the remainder. In light of these two foundational components, we assert that the derivation and analysis of stochastic modified equations extend beyond the confines of two-layer neural networks and dropout. This approach is applicable to broader spectrum of stochastic algorithms, including but not limited to SGD and ADAM, and extends naturally to deeper more complex neural network architectures.
One might wonder why we only demonstrate results limited to two-layer neural networks. For one thing, two-layer neural networks are indeed a meaningful step towards a thorough understanding of dropout. For another, the specific structure of two-layer neural networks, wherein parameters are 'decoupled', i.e.. 0 q. = (a,, wr), facilitates the demonstration of dropout's 's effect through the computation of the network output. Finally, for two-layer neural networks, we are able to give out an explicit expression for the modified loss, since we are able to calculate the first and second moment of the random variable (5)r. In the case of multi-layer neural networks, if dropout is applied only to the outermost layer, we can still calculate the explicit expression for the modified loss, since its calculation involves solely the first and second moments of the random variable (8)r- However, the situation becomes significantly more challenging when dropout is applied to the inner layers of the multi-layer neural network. In this scenario, obtaining a closed-form expression for the expectation [6)], where h is highly nonlinear function with respect to 6, becomes nearly impossible. Moreover, the computation of the covariance is even more infeasible in multi-layer neural networks compared to two-layer neural networks.
While the structural dynamics of dropout within nonlinear activations for deep neural networks remain uncertain, assert that the same structure is retained if we apply dropout in linear manner to the deep neural networks by computing fo (x) Er= a (ō), (w: (.(")), where != is the output function of a L-layer neural network for 2 2, the same structure remains, and the modified loss Ls(-) reads:
(s
2mp ΣΣ
( x 1/
2n
As for the SGD setting, given the nature of SGD as an unbiased estimator with respect to the full sample, we conjecture that an order-1 approximation utilizing SME for SGD combined with dropout shall be in the form:
d0, = - LoLs((,))d + (0.) dW.,
wherein the drift term remains invariant regardless of GD or SGD, while the diffusion term Σ (0.) combines noise from both dropout and SGD.
As for the scenario where dropout is applied within the nonlinear activations, this exact form of the SME remains unexplored. However, design numerical experiments to verify the approximation ability of deeper network SMEs in the next subsection, where the drift term here is calculated numerically through multiple samplings dropout variable 8.
19

---

Published as a conference paper at ICLR 2024
C.2 EXTENDED EXPERIMENTS FOR NETWORK STRUCTURES AND SGD SETTING
In this section, we verify the feasibility of the above analysis through numerical experiments. We all use first-order SME for approximation. Please refer to the Appendix A for specific experimental settings.
C.2.1 SME SIMULATION
In this subsection, we conduct an empirical validation of the feasibility analysis of SMEs for dropout. This validation is conducted through an exploration of the resemblance between the numerical simulation of the SME and the real-time training process of dropout. For the numerical simulation of the SME, unless otherwise specified, we employ the Euler-Mant Maruaa-aauyyaaa method to approximate its dynamic evolution by the order-1 approximation. To mitigate the computational demands associated with the covariance matrix, we resize the MNIST data to 7 x 7, thereby reducing the number of network parameters involved in the simulations.
cropout
10"4
dropouti
SDE
10-1
0014
0.43
0011
0.07
0.13
1002
4x10-
0.00
3x100
7x
opach
opoch
(a) deep network
(b) SGD setting
Figure 7: We train fully connected networks on MNIST. The curves and points are derived from the average results of 20 individual trials, each of which utilized the same initialization distribution. The training loss trajectory obtained by SME simulation or dropout training under four cases of different learning rates and dropout rates. The error bands, portrayed with greater transparency, are derived from the maximum and minimum loss values observed across these 20 random trials at each training step. (a) Four-layer network, each layer width is 40. The learning rate is 0.1 and p = 0.5. (b) Two-layer network training with SGD, the batch size is 32. The learning rate is 0.1 and p = 0.8.
As shown in Fig. 7 we compare the SME simulation results under deep network and SGD settings with dropout. For deep networks, we add a dropout layer after each hidden layer. Since the drift term cannot be calculated explicitly, we approximate the expectation by sampling the noise multiple times. This also limits the number of parameters in our model. For dropout under the SGD setting, we use the same drift term as under the GD setting, and calculate the noise structure from the combined noise the two random algorithms, dropout and SGD.
C.2.2 HESSIAN- VARIANCE ALIGNMENT
In this section, we mainly study the impact of dropout of complex network structures on Hessian- variance alignment under the SGD setting. In the main text, we have conducted relevant experiments on complex network structures under the GD setting. To investigate the Hessian- Variance alignment relation for dropout with SGD, study the cosine similarity quantity a0,)) between the covariance matrix E; == E(,)) and the Hessian matrix H. == (O))) at each time step t under the SGD setting. Σ, is the covariance matrix of D grrad. a collection of gradients calculated with different dropout variables and input data æ sampled at the 7th step. On the other hand, H, is the Hessian of the loss function evaluated at the tth iteration. Then the crucial cosine similarity metric a(0)) is formally expressed as:
Tr H,E H. E
ax (x,).)
(20)
As depicted in Fig. it is evident that throughout the training process, a(+)) consistently attains values surpassing 0.85 in Fig. a)) and 0.65 in Fig. g (), and these observations hold true across
20

---

Published as a conference paper at ICLR 2024
varying learning rates and dropout rates. It's worth noting that, based 100 samples, the average cosine similarity between the two random matrices based on the selected parameters is only 8.6x 10 The cigenvalues of the two random matrices are derived from the eigenvalues of the Hessian matrix and covariance matrix the selected parameters, and the corresponding eigenvectors are sampled from normal distribution and normalized. The model parameters derived from the final model represented by the blue line in Fig. 8(a). Consequently, the introduced noise is highly anisotropic in that it aligns well with the Hessian matrix across all directions. We acknowledged that due to computational constraints, this experiment limits the trace calculation to a subset of parameters, which can be effectively regarded as the projection of the Hessian and the noise into specific directions.
0.9,8 0. 0.01
0.8
1.0
= D.8, 0.00
p D.8. 11 0.005
0.7-
ate.)
0.6
0.9
0.5
0.6, 41
0.8, 00.0.
0=0.6.1=0.02
0.8
300
100
150
epoch
epoch
(a) three-layer FNN, MNIST
(b) VGGI6, CIFAR-10
Figure 8: The cosine similarity ax())) between the Hessian of the loss function and the covariance of the dropout noise at each training epoch for different choices of dropout rate and learning rate. (a) The FNN with size 784-50-50-10 10 trained on the MNIST dataset with a batch size of 32. The dropout layer is added after the first hidden layer. (b) The VGGI6 is trained the CFIAR-10 dataset with batch size of 32. The dropout layers are added after the first two convolutional layers of each block and the first fully- cnlleceonnected layer. The calculation of (0)) is performed every five epochs.
21

---

Published as a conference paper at ICLR 2024
D EFFECT OF NOISE FROM DIFFERENT TRAINING STRATEGIES ON RESULTS.
In this subsection, we compare the effect of noise from different training strategies on model output and generalization. We mainly compared three training strategies: SGD, dropout, and parameter noise injection. Parameter noise injection is well known as the explicit regularizer on the trace of the hessian (Orvieto et al. 2023). We first study the results of ReLU NNs training under three training strategies, as shown in Fig. 9 aa. It is easy to see that the model output of GD+dropout has better smoothness and is more in line with expectations. three randomized algorithms. We follow the settings in Orvieto et al. (2023) and conduct experiments using MI.P1 the corresponding code, with the network size size of 128 for the three algorithmss. The mean (solid line) and standard deviation (shaded range) were calculated from five independent experiments for each setting. For dropout, we add the dropout layer under each hidden layer, and we set p 0.8. For parameter noise injection, standard deviation o = 0.03. At the same time, we compared the traces of the Hessian matrix of the three training strategies, which is metric widely used characterize flatness, As shown in Fig. [c), the trace of the Hessian matrix of dropout is smaller than that of the other two training strategies during the entire training process, suggesting a stronger tendency for the flat solution.
Fig. 9b), we compare the generalization capabilities of
We use a batch
set the noise
0.30
GDedropour
$8.41
4220
5CD
5CD spout
20.2
0.15
20.01
2220
37.0
5CD dropout SCD mcloc (epprt
2020
0.00
27.8
(a) two-layer FNN, output
(b) deep FNN, accuracy
(c) deep FNN, Hessian trace
Figure 9: (a) Two-layer ReLU NN output under different training strategies. The width of the hidden layer 1000, and the learning rate is x 10-3 For SGD, the batch size For dropout, the dropout layer is added after the hidden layer with p = 0.8. For parameter noise injection, we use the layer noise with the noise standard deviation o 0.001. (b, c) The deep FNN trained on MNIST under different training strategies. The network size is 144-500-500-500-
and the learning rate is 0.01. The batch size is 128. For dropout, the dropout layer is added after the hidden layers with p= 0.8. For parameter noise injection, we use the layer noise with the noise standard deviation 0.03. (b) Test accuracy. (c) Trace of the Hessian matrix.
22

---

Published as a conference paper at ICLR 2024
E EXTENDED EXPERIMENTS OF DROPOUT UNDER MEAN-FIELD LIMIT
In the above, we do not treat the network width as the variable under study, but treat it as a constant value order one. However, if take the mean-field limit of the two-layer network, i.e.,
fo( = Σ
0 (w)
r=1
where m + oo, then the modified equation has the following form:
1
1-p
7 12
Ls(0)
2m
2mpm2
The additional regularity of the dropout modified equation disappears in the infinite width limit, which contrary to our common sense. Intuitively speaking, we take each neuron to be independent and identically distributed, even if we 'drop' some neurons, dropout does seem exert impact on the training process. We carefully verified this by experimentally studying different wide networks with and without dropout. As shown in Fig. TO for narrow networks, dropout will have certain impact on the training process, and this impact will become insignificant when the network is wide enough.
We design experiments with and without dropout at different hidden layer widths under the mean- field setting. The quantity Ri we study is defined as 2mmm"p. >i E (FT(w. As shown in Fig. I0 for the wide network, the model loss trajectory is almost consistent with or without dropout, and the cosine similarity between the two model parameter vectors i.c., flatten all the parameters of the model into one vector, is 0.9999998, but there is no such similarity for the narrow network.
100
m=10>, dropout
m=100, dropout m=100, GD
10 GD
Rs(
10-1 1
10-2
10-1
R1 ())
10-2
10-3
0
500
1000 epoch
1500
2000
Figure 10: The two-layer network with and without dropout different hidden layer widths under the mean-fice field setting. The learning rate 0.01 for experiments, and p = 0.9 for experiments with dropout. These settings can make the dropout noise have little effect on the results.
23

---

Published as a conference paper at ICLR 2024
F PRELIMINARIES
F.1 NOTATIONS
We adhere wherever possible to the following notation. Dimensional indices are written as subscripts with bracket to avoid confusion with other sequential indices (e.g. time, iteration number), which do not have brackets. When more than one indices are present, we separate them with a comma, c.g. (i) is the i-th coordinate of the vector æt, the hith member of a sequence.
We set special vector (1, 1....... ..l.....,1"..... T)T by 1 := := 11,1,1,.... 1)T whose dimension varies. We set n for the number of input samples, m for the width of the neural network, and == mm(ddd + 1) hereafter in this paper. We let [n]] {1, 2.............. n.). We set set. N(u, Σ) as the normal distribution with mean il and covariance Σ. We denote @ as the Kronecker tensor product, (.,-) for standard inner product between two vectors, and A B for the Frobenius inner product between two matrices A and B. We denote vector L norm as 11.. vector or function L norm as II-Ilac function L norm |1-1l1. matrix infinity norm as bo-lo matrix spectral (operator) norm as I1-112--2- and matrix Frobenius norm ||-1pp Finally,
denote the set of continuous functions f(-) R - R possessing continuous derivatives of order up to and including by C* C"((RR PTR and for Polish space X. we denote the space of bounded measurable functions by (X²) and the space of bounded continuous functions by (x))). In the mathematical discipline of general topology, a Polish space is a separable complete metric space.
F.2 PROBLEM SRTUP
For the empirical risk minimization problem given by the quadratic loss:
min Rs( 0
= 2n Σ (fe(x.) (((xx)))-- - yi)²
(21)
where S := (ax, M/) ) is the training sample, fa()) is the prediction function, O are the parameters be optimized over, and their dependence is modeled by two-layer neural network (NN) with m hidden neurons
fo(w) = Σ a (w)Tx)
(22)
where E Rd, 0 vec(Oa, (w) with 02 vec( (ar }m Ow vee( (, }m 1) is the set of parameters, o (-) is the activation function applied wisdlly to its input, and o is 1-Lipschitz with E C (IR). More precisely, 0 = veol (g }mm,) whereas for each E [m]]. qr (ar, wT)T. We remark that the bias term by can be incorporated by expanding æ and w, to (æt, 1)T and (wI,6,)²
Given fixed learning rate 7 > O, then at the N-th iteration, where
tN Nn.
and scaling vector dx E R sampled with independent random coordinates: For each k E [m],
1
with probability p.
(ōx) k
p
(23)
with probability 1-p.
and observe that (ōv 1221 is an i.i.d. Bernulli sequence with E5; and naturally, with slight abuse of notations, the o-fields FN == {o(6), -2.) forms a filtration.
We then apply dropout to two-layer NNs by computing
f) = E (w)x),
(24)
r=l
24

---

Published as a conference paper at ICLR 2024
and we denote the empirical risk associated with dropout by
Rs
Rdrop (0:
(fo(2;; 6) - y²))
2n
(25)
2
m
Σ
(6), ( yt
2n
We observe that the parameters the N-th step updated via back propagation follows:
N = x~ - e Edrop (0N-1;ōN),
(26)
where 0, := (0). Finally, we denote hereafter that for all E [n].
= e (0 == f
(æ;; 8x) - yi
hence the empirical risk associated with dropout Rdrop (0x-11 6,)) can be written into
Retrop (0N-1;ôN)
1
(eN)
2n
thus the dropout iteration 26 reads
O~ - -
Rehrop (0N-1;6x) = - Σ
Vo
1=1
and we may proceed to the introduction of the SME approximation.
25

---

Published as a conference paper at ICLR 2024
G
STOCHASTIC MODIHIED EQUATIONS FOR DROPOUT
G.1
MODIFIED Loss
Recall that the parameters at the N-th step are updated as follows:
7 Vee
N = N-
(27)
π 1=1
and since {ōx 22 is i.i.d. sequence, then the dropout iteration (7 updates the parameters in recursion form of
0N - F(ON-I,
(28)
where F(..)) RD R' DD is smooth (C function, and {N21 is disturbance sequence on Rmm whose marginal distribution possesses a density supported on an open subset of
RT Then, based on the results in Meyn and Tweedie (2012), the dropout iterations (27) forms time-homogeneous Markov chain. Thus, we may misuse EE| [r] the conditional expectation given FN. with Eøx- [t]. the conditional expectation given ~ Then, for each [m1). the conditional expectation of the increment restricted to reads
n
n
RØN-1
EON-1
(6N Vaa (aza(w]a:))
qu
and since
Ea [e, (5,)))
= EoN-
Σ ((.
Eow-1 [(ōN)a]
r=1,ryek
+ Eow-1 [(xx)]] (wo.a))
Σ a,o(wTx;)) yi
1 azo
+
p
r= L
(--1)
0 (w).a: - Vi + p
aso (
For simplicity, given fixed & E [m], for any E [n]] we denote hereafter that
e: = e0) = Σ (wTTa Vi.
== = (0) = Σ a,o (.o.wa - yis
r=L.rs
remark that compared with eiV, e, and 6,,1 do not depend the random variable 8n. Then Eox-1 (e: (8)) can be written in short by
1 -apo([xx
Eov-1 [c (6N)a) C1,AA
P
(29)
= + (p-1) a= (=o(w]a.))
Hence for each [m], expectation of the increment restricted to qu reads
a
EØx-1
Σ
e (8N)) Vo (azo(w[x:))
V 9n (alo(w(x;)) +
Σ
1 -1
(ro(w[W; (aLo(w(x;)),
p
26

---

Published as a conference paper at ICLR 2024
then define the modified loss Ls(-) Rm((+1)
IR for dropout:
1
11
11
m
(0)
Σ
Σ
a (.)
(30)
2n
2mp
since as ON-1 is given, then by taking the conditional expectation, increment of the dropout iteration reads
ON - ON-1 -n Volts (0N-1; -VVoLs(0))
which implies that in the sense of expectations, follows close to the gradient descent trajectory of (0) with fixed learning rate 17.
G.2 STOCHASTIC MODIFIED EQUATIONS
We then follow the strategy of Li al. (017) to derive the stochastic modified equations (SME) for dropout. Firstly, from the results in Section G.1 we observe that given ON-1,
ON -oo===-Vooss(0eee
(ON
(31)
where Ls(-): Rm((+1) + R is the modified loss defined in 330, and V()) Rm(d+1) Rm((+1) is a mmdd -)-dimensional random vector, and when given Ox V (0x-1) has mean 0 and covariance ( 1), where E(-) gm((d+)) + II [l= Recall that 0 = vec( ) vec ({(ar, w.) ( and for any k, r E [m]. we denote that
is the covariance of V Volt (0N=1:6N).
E (0N-1) = Cov (Var H HeSrop (0N-1:8N) Vg, Rdrop (0N-1;dN))
then
Σn
E
Σ mm
Σn
Σzz
Σ Ezm
Σ
Σm1
Σn
Σ Emm
For each 1: E [m]] we obtain that
Cov (V Rg (ON-1;ōN), Vg, neroop (ON-1;ōN)
1 (alo(w)za))) Vya (aFo(w))x)))
Ciii +
p
72
p
n
Cs + (aco(((( Van (aro(unTa:))
x
0:0 m (w) w;) V. (ano(w((ii))
12
1.
alo (w) x.) V Vg. (w))))
27

---

Published as a conference paper at ICLR 2024
and for each k; r E [m]] with l & fr,
Exr 10
Cov
avn Rs Re (ON-1;ōN). Vg. Reshor (0N- (0N-1;
-
CAAAAr + - (aLo(w(Ts)
1 (w) Van (ano(w))):))
11
p
ako (r Vg. (a,0(
1
1)
x; Vga (alo(w[a)))
p
71
n
CiAV/irr + ano(w(xs)) a ( a (ano(w)
where we denote hereafter that
m
1- =
Vir
111,
and compared with ef" 6, Ar still does not depend on the random variable 8N. We remark that the expression above is consistent in that for the extreme case where = 1, dropout 'degenerates' to gradient descent (GD), hence the covariance matrix degenerates to a zero matrix, i.e., Σ 0DXD- We remark that details for the derivation of Σ is deferred to Section
Now, as we consider the stochastic differential equation (SDE).
d (O,) dt
dW, O @0),
(32)
where W, is standard m(d + 1) -dimensional standard Wiener process, whose Buler-Maruyeer discretization with step size 17 >> at the -V step reads
N = θ nn 1) + nb (Ow(N-I) + Vio (0w(N-1) ZN,
where ZN ~ NO, I m()1)) and O, = (0)). Thus, we set
b ( = Vels(()),
σ + = (
(33)
c := Oo.
then would expect (32) be a 'good' approximation of 31 with the time identification n/N.
Based on the earlier work of Li et al. (2017), since the path of dropout and the counterpart of SDE
are driven by noises sampled in different spaces. Firstly, notice that the stochastic process (0,) } N20
induces a probability measure on the product space IR x R X ... x R x ..., whereas (0r)22
can form piece-wise induces probability measure on C (IO, co), R.). To compare them, linear interpolation of the former. Alternatively, as we do in this work, we sample a discrete number of points from the latter. Secondly, the process {0x) N20 is adapted the filtration generated by FN whereas the process (O.).20 is adapted an independent Wiener filtration Ft. Hence, not appropriate to compare individual sample paths. Rather, we define below a sense of weak approximations (Kloeden and Platen 2011 Section 9.7) by comparing the distributions of the two processes.
To compare different discrete time approximations, we need to take the rate of weak convergence into consideration, and we also need choose an appropriate class of functions as the space of test functions. We introduce the following set of smooth functions:
C. (Im(ddtii)
={
E CM (Rm(a--))
[fllew = Σ | fl < {
1M
28

---

Published as a conference paper at ICLR 2024
where D is the usual differential operator. We remark that C,2, (RR) is subset of ( the class of functions with polynomial growth, which is chosen to be the space of test functions in previous works (Li et al. 2017 Kloeden and Platen 2011 Malladi et al. 2022).
Before we proceed to the definition of weak approximation, to ensure the rigor and validity of our analysis, shall assert assumption regarding the existence and uniqueness of solutions to the SDE (32
Assumption There exists T > O, such that for any time t E [0,T"), there exists a unique t-continuous solution O of the initial value problem:
d = b dt + a dW,) O = (0);
with the property that O is adapted to the filtration Ft generated by W. for all time 8 S t. Furthermore, for any t [0, T
% 10.0)) < co.
Moreover, assume that the second, fourth and sixth moments of the solution SDE (32) uniformly bounded with respect to time t, i.e., for each E [3], there exists C(T*, . > 0, such that
sup SS
E (-112 C((T", ,o).
(34)
As for the dropout iterations 27 we assume further that the second, fourth and sixth moments of the dropout iterations (7) are uniformly bounded with respect to the number of iterations N. i.e.,
let < < 1, T and set Nr,- = n ]. then for each E [3], there exists T > and 1/o > 0,
such that for any given learning rate S 1o and all N E [0 Nr there exists C(T 0 7o) > 0, such that
21 < C(T",,0,0 no). E 0
(35)
We remark that if (RDP) is chosen to be the test functions in Li et al. (201)) then similar relations to 64 and (35) shall be imposed, except that in our cases, we only require the second, fourth and sixth moments be uniformly bounded, while in their cases, all 2/-moments are required for I 2 L.Establishments of the validity of A 2 regarding the existence and uniqueness of the SDE will be exhibited in Section
The definition of weak approximation is stated out as follows.
Definition The SDE (32) is an order a weak approximation to the dropout (27), if for every g E C.vl (Rm(d+1)), there exists C and nj > such that given any rj S vop and T S T then for all N E [Nr.a)
( [0( s C(T.g.m))7.
(36)
29

---

Published as a conference paper at ICLR 2024
H SEMIGROUP AND PROOF DETAILS FOR THE MAIN THEOREM
In this section, we use a semigroup approach (eng et al. 2018) to study the time-homogeneous Markov chains (processes) formed by dropout.
H.1 DISCRETE AND CONTINUOUS SEMIGROUP
Definition 7. A Markov operator over a Polish space X is bounded linear operator P (x( 7 (x) satisfying
P1=1. =;
Pip is positive whenever is positive;
If sequence {fn.} C (x) converges pointwise to element (x()), then Pipn converges pointwise to Py:
To demonstrate further inequalities that Markov operators satisfy, offer the following proposition
Proposition 1. A Markov operator P BB(X)) - Ba(X)) over a Polish space X' satisfies
(Pf(x))) <Pf+)))
(P/(x))- P/
-Pf(x)) =P((a))>((((a))
Moreover, if the Polish space X is equipped with measure μs, a function f : X B. is said be element C) (X) if
Jxe
Sidu
Then for every f (X), the following holds
|PSI < 11/1l--
In mathematics, the positive part of real function is defined by the formula
ft (x) max(f(=)).0) {00 f(x) if fxx) > O, otherwise.
Similarly, the negative part of f is defined as
- min(f(i)))))))))) (-((x)
if fax) < <0, otherwise.
= = (aaax(f((x))).)) =
0
We proceed to the proof for Proposition
I
Proof. From the definition of #addd f it follows that
(Pf)+ (Pft - P max (0,Pft - PF
< max (0,Pf*) = Pf+
Similarly, we obtain that
= -Pf) max (0.Pf Pft)
< max (0,P/-) PST.
Hence for the last inequality
PP = (Pf)+ (Pf)
$ Pft +PF
= = =
30

---

Published as a conference paper at ICLR 2024
Finally, by integrating the above relation over X' we obtain that
IPfll1 pp
(37)
{ P PlS|dp=
-J1 1/id I1
Inequality (37) is extremely important, and any operator P that satisfies it is called contraction. This relation is known as the contractive property of P. To illustrate its power, note that for any (X)) we have
fll = ||P p"-1 fll, - ||P--
As we consider Markov processes with continuous time, it is natural to consider a family of Markov operators indexed by time. We call such family Markov semigroup (Hairer 2008), provided that satisfies the relation
P.+.
Ps. for any time 8, >
(38)
And if given ((2), where (()) the Borel o-algebra X, and given any two times t, the following holds almost surely
P ( E A X,) =
then we call X₁ homoeeooogss Markov process with semigroup (P:).20-
In our case for dropout, we set the Polish space X' = RD,, and since C.¹ (IRP) C B,(RP), then
WLOG we fix g E C.²d DI and define
P (ng(() == E 9 @ (0- Vo Vo Rstoo (0: [
(39)
We conclude that the dropout iterations 277 forms time-homogeneous Markov chain with discrete Markov semigroup {P" J n20'
As for the SDE (32), based on Assumption and combined with the results in (Hairer 2008
Example 1), the Markov semigroup (P: (Phh20 associated to the solutions of the SDE reads: For any
g E (R.)).
0.Pig LPig.
where L is termed the generator of the diffusion process 32 which reads
Cg = (b, Veg) oat Vaa.
(40)
Moreover, for a fixed test function g E C. P), then for any two times s, 2 0,
P.g(O.) == exp(tC)g = Ee. [g ((++))) and (Pr)e2o forms continuous Markov semigroup for the SDE 32
(41)
H.2 SEMIGROUP EXPANSION WITH ACCURACY OF ORDER ONE
Our results are essentially based on Itô- Itô-Taylor expansions Kloeden and Platen 2011 or Taylor's theorem with the Lagrange form of the remainder (Li et al. 2019 Lemma 27).
Theorem (Order-1 accuracy). Fix time T < T if we choose
( = - (e)
(+) / (=))
then for all E [0, the stochastic processes θ satisfying
de, = b (G,) dt + a
O O().
(42)
an order-1 approximation of dropout (27). i.e. given any test function g E (R.)), there exists
No > 0 and CT,, ||gl| 7)) > 0, such that for any nj S no and T < T", and for all N E [Nr.)).
the following holds:
[g(ON) Eg(O, (1T,119 0o. 10)5
(43)
where 0, O
31

---

Published as a conference paper at ICLR 2024
Proof. By application of Taylor's theorem with the Lagrange form of the remainder, we have that for some a 2 1,
D
009
g(0) g())
Σ
II
9 (C)
}(5)) ()))) ... (1,)))
(1)
11:rody =1 j=1
a+1
gat1g (70 (1 7),
Σ
II
0 (.) {())
(a + 1)!
(51)
))((1) (1,)
for some y E (0, We adopt the Einstein's summation convention, where repeated (spatial) indices summed, i.e.,
()) = Σ ())
As we choose & == 0,, 2- 0g and a L, then we obtain that
g(0)) g(0o) = (Veg(Oo), 01 - 0o)
Vag(n01 + +1--7)00) (0, 0o) @ (0) 0o) (Veg(O.), 01 Oo) Vog(Ou) (01 0o) @ (0) 0o),
where 0o = -0: + (1 0o. and we observe that since
01 0 = = + vTVV(
then
Eg(O,) - (0)) = (Vog(Oo). E0: -Bo) + E Vog(Oo) (0) Ob) & (0) 8b)
2
=-n (o)( (o o + E (0,),
where the remainder term (.) RdD R, whose expression reads
(o) == _ Vog(dn) (0, 0o) @ (0, (0,-0n)
0n)
(44)
and we remark that Og and 0g are implicitly defined by Oo- Then, directly from Assumption obtain that
2
F (00) E Vog(Oo) (01 00) s (0, O.)]
S 2 | E [|0, 0o112 = ,² 19 E Vo drop (Oo: 5, 6,))[]]
$ >² |aal|c C((T", 0. 70),
since VoLs(O) ()) and Σ (0) can be bounded above by the second and fourth moments of the dropout iteration
We observe that
,, - 0, = /-" " b dss + [' (O.)dw.
As we choose 2 := -,,, θ = O, and 1, then we obtain that
g ())) ())) (eg((nn) O, Qa)
1
2 vo)(O,) (O,, -.) (O,, -O))
where
2 = 10, + (1 7).
32

---

Published as a conference paper at ICLR 2024
for some 7 (0, 1). Then
,,( - Eg(@n)
(Vog(On),BBB - o) + 2 [og(O)) ( (0,- 0n] (Veg(Oo), / Ebb(..)dd + E Vor(Gn) (o,, 0n) 8 (0, -Go)
and since
+ / c (Veg(())).)) Ju,
Veg((O)) Ebb()))) = (Veg(O.).
then we obtain that
O,, - =) (Veg E(b(O + [-' [ L ((Vog(O,), b) (0,)
=
1
+ 2
1 V (9, 4,) @ ( 00)
= 1 (Ve b(O)) + (O)),
where the remainder term (. : RDD > R, whose expression reads
( = /.''
" /'"
L Ves( b) (e, dudds
(45)
E V (lg( (0, θ.) (o,, 0
and we remark that @, and , are implicitly defined by O As choose
(()) = -s) o (0) ( = v1 (E
then we carry out the computation for L b) (O.).
L L(Veg(Oo), b) (0,)= = (VeLs( (Ve((O))) VeLs(O)) le=e.)
+ 2 + v (veg( VeLs( le=
since Ve (O), Ls and Σ ()) can be bounded above by the second, fourth
and sixth moments of the solution to SDE 32) hence we may apply the mean value theorem to (45 and obtain that
/ SC (Veg(Go)) b) ( )ds B (an((o) : :0,, -0,- o 0n)
(o))
1
s +1191e c C(T" On)ds + 2 |19l| E O 0
7
/ b(O.)ds + o(O))dW,
x 2 |lallc C(T 0n) + Iglll E
12 |lllea C(TT On) + +2 Bollcc E
1." ((O))ds
5 2
+2llolles E 1-" (OO)ddW
2
B
2
x 2 lllll C(T", Oa) + llllec n [
f f )l
+ lllllc E.
x 2 llallc C(T", Qa) + 2lllolla~ 1 vols /
+ 2 11glle nB | - ||] S l1alll C ((T",0)
33

---

Published as a conference paper at ICLR 2024
To sum up for now,
[B((1) - Eg((0,) = Bg((O)) - Vols( + (0)
- (g)) (Ver( + (00)
since 0g = G and b ((0)) = -V thus
(p)g(00) - Png((n)) [m)(()) - Eg( ,))
< (00) + E (b)]
(46)
r l1gllo. C(T 8. no) +1 Illlo C(T*, 1a)
(11²).
For the N-th step iteration, since [E)(Ox) Eg( ,,,) ) = |Pxy g(0)) ( (
and the RHS of the above equation can be written into telescoping sum as
PNN (0o) - (O)) hence by application of Proposition D
(PN- o P(--1)9((0)) - -N Ping(On)),
we obtain that
[Eg(Ox) Bg(O
nN)
PPN. ---1111 PP(-I)n((0)) - -P PPngg(oo)
1=1
N
PN a (P)) P(--1a- - -P, P(--mm) g(On)),
since (P) o Pa-D1m - Pr P(--1))) O))) can be regarded as (IB if we choose measure μz to be the delta measure concentrated Og. i.e.,
;= 60.
hence by the conctration property of Markov operators, obtain further that
N
[Bg(Ox) Bg(( S
Σ
(P)) P(--1) P,, P(--1))) g(Oo)
[=1
N
By taking expectation conditioned on θ (1-1p then similar to the relation (46). the following holds
()) (0)9(( E [g(()) - Eg(O,, (1-1))
(0 + E ( (111))
< 2 || C(T", Og. no) + of lI C(T*, Op) = -11²)
We remark that the last line of the above relation is essentially based Assumption 2 since E EE (O)(1)))) and E E. ( can be bounded above by the second, fourth and sixth mo- ments of the solution to SDE (32). hence may apply dominated convergence theorem obtain the last line of the above relation.
To sum up,
N Σ
P g(0o)
Pyng((Oo)
N-41
1+1
P(--1)n((O)) D'N-'' Pingg(e)) NO
hence for N Nrw IP, ((0)) - PaNg((o) NO = Nn0()) TT) = O(n).
34

---

Published as a conference paper at ICLR 2024
H.3 SEMIGROUP EXPANSION WITH ACCURACY OF ORDER TWo
Theorem 3 (Order-2 accuracy). Fix time T TTT, if we choose
b( = Ve (Ls(0)+ + [VoLs(0)1
( = vī (
then for all t [), the stochastic processes + satisfying
do, = b( o
6 - @
(47)
is an order-2 approximation of dropout (27). i.e., given any test function E C2 (R.P), there exists no 0 and CTT, ||gllou n7) 0, such that for any 1j S 170 and T S TT, and for all N [N
the following holds:
Eg(O xx)) Eg(OnN)] S C(T, 119l|cs c, no)6
(48)
where 0p- -
Proof. By application of Taylor's theorem with the Lagrange form of the remainder, we have that for some a 2 1,
D
80g (0)
g(0) g(
Σ
II
(
((s)
s! Hypny
20 ... 00
(1 (is)
1
Σ
a+1
II
0 (1))
0()) (00())) ... 30 000(1)
gatlg
(70 (1-1))7)) -))
(a+ + 1}!
for some (0,1). 1).
As we choose + := 01, 0 Og and 2, with slight misuse of the Frobenius inner product notation, we obtain that
9(0)) ((0)) = (Veg(Co), 0, - 0o) + (og(On) (0) 0o) @ (0) 0n) 2
+ Vàg(n0 + +(1-7)0p (0) 0o) @ (0 0o) @ (0, 0o)
(Vog(On) 0 0o) + Vog (0)) (0, 0o) @ (0) 0o)
Vog(Oo) (0) 0o) x 00 0o) @ (0) 0o),
where 0, = ;000 + (1 7) 00. and we observe since
0, -0-00-1 0, = - + (O,),
then
O1) (g)(O)) = 10, - BB,) - - : [E 0o) @ (0) 0o)]
1
E vāg(Oo) (0) - 0o) @ (0, - 00) 8 (0) - -0n]
(Ver(O)), VoLs(0)
1 Vog(On) (VoIs( R-0 X (0) + I(6)))
F2
where the remainder term E,(.) : RD + R, whose expression reads
1
(3(0)) : 6
vāg( ( - 0b) @ (0) - -Bu) @ (01 - 0b)
(49)
35

---

Published as a conference paper at ICLR 2024
and we remark that Og and 01 are implicitly defined by du- Then, directly from Assumption 2 obtain that
(00) - | B100, 0ol12 - 1/² l E || ddrop (Oo: 51 |
S n7 I C(TT, On no).
since VoLs () and Σ (0) can be bounded above by the second and fourth moments the dropout iteration
observe that
, O, [-" b(O,)ds + ['''......
As we choose 2 = , 9 = O and = 3, then we obtain that g(On)) ((Op) (Veg(On), O On
1 (o)(On) , 2o) @ (0,, On) +
2
+ (O)( ( . @ (0, On) @ (0,, 2p)
6
1
24 ( O ) 8 (O, 2 (O, Oa) (o, Oo).
where
0 O = , + (1 7)
for some 7 E (0, 1). Then
Eg(O,, (g(()))
(Veg(()), BO,) + Vo. (O)) 00)) 000
ve Ton E[ -.) ( 8 (0, O.)
E (on())) ( 2.) 8 ( ()0 6a) @ (O, - 0n) -b) Oo)(Oo), Eb -.. ( ( 6o) @ (Ø,, On)
+ 24
v Op) E On) 8 (0, m,0 8 ( 0n)
[o)(O.) (,, ..) 0 ( θ₂) 2 (O, O,) (,, θ.)
and since
VVeg On), = (V) + / (Veg((.)).)) b) (e,)
then obtain that
Eg(O,) - Eg(On)= = (Vo)))) E(b(O)))) + j [" [ a (Veg((O)) b) (O,)
(or(On) ( 1a) (O,, 2n)
+ (g)(On) θ₂) @ ( 2a) 8 (O,, -b)]
+ 24 ( (0,, θ 0 ( 2₂) & (O,, θo) s (0,, - 0b0
and once again since
(0,) + / C ((C(Veg(O)). 6)) (O.)du,
L (Veeg(O)) (V)
36

---

Published as a conference paper at ICLR 2024
then obtain that
(,)),, (g(())) - (o) /-' [ L ((Veg())),)) b) (0. dudds
+ (o)(O)) : 4,) @ (0, [o)]
+ V
).0 8 (0, 2a) 8 ( -
1
+
E (og( , m ( θn0 2 (O, O 2 (o,, 0n1
24
[' L ((eeg())) b) (oo)
n (Veg(On) ((b
0 / ((C(Ve)( b)) (o.. duddds
+ Vag(O)) O mb) 8 , -n)l
+ Vog(On) ( θ₂) ( On) @ -0,--b Op)
1
E V ( 3,) O θa) 2 (O, O₂) 2 (e,, - -Oo)
+ 24
7 (Veg(())) (b) (b + 2 (Veg 9o) (9))
(0))))) O e.) ( 0a)l + (u))
where the remainder term (.): RDD + I, whose expression reads
(0)) JT [. (() b)) O n, ddudds
+ (
: ( - O,) @ ,0,, O,) @ ( -n)l
(50)
6
+ "
1
E
Oo) (O,, 0₂) 8 (O, O₂) @ ( ( ())
24
and we remark that θ and O are implicitly defined by Og. As we choose
b = Ve (Ls(O)+ + | =Voos(t)))))) ( = ())
then we carry out the computation for LCC (c (e.).
CC (Ven(()))) b)) (e.) =C
(e.)
(b. Ve (b. Vo (V
و V (b. (VV)
" Σ ( (Vor
Σ: v2 Σ (Ver b))
2
=b ( (bVoo (0. (R,)
=(Vols(0..)Ve Vo ( Vo Vo (IVoLS
( +,
37

---

Published as a conference paper at ICLR 2024
since VoLs(O), (o VoLs(O). Σ (e), R,, ( and Ri (Q.) can be bounded above by the second, fourth and sixth moments of the solution to SDE (32 Moreover, we observe that
E ( " - O @ , 2 @ (o -o0l
)o + / o(O.)a..)) @ (1 b(O.)dss + S ['o.).)ww,
( [(' { ((O.)d.. )
and its entry can be categorized into four types. The first one the pure drift part, i.c.,
L. ((0
[' b(O.)ds [ b(O.)ds,
then by application of the mean value theorem and the fact that VoLs( (o (g)s())) and E (@) can be bounded above by the second, fourth and sixth moments of the solution to SDE 32
we obtain that
E / (( 8 I ((O.))d @ /'" b(O.)ds
- Eb( b( & ((..) O
The second one is the pure noise part, i.c.,
(L (O.)a.. & (o.)).))..) & (ooo).)a..).
and as the odd moments of zero mean Gaussian variables are zero, hence we have
[[. ((.)aa.)))) & ((oo(o))).)) & (L. (()o(o.))w,))
) = 0,
the third and fourth one are both of the mixed part, for the third one
J. b(e.)ds @ [ b(O.)ds & (o.)).)).
whose expectation of course zero since the drift part and the noise part is independent, and the fact the odd moments of zero mean Gaussian variables are zero, and for the fourth one
S" b(o.)ds @ (f"o + ( (('olo)aa.))
we obtain that
E b(O.)ds (f" (('o(o.)aw.)) & (l" [(o.)aa.))
m Eb( 0 E ((O.)d..)
[(.))..)..))) O
As denote
(²(O)) = = - θ,) 3 ( - 2a) (o,, θa)],
then we obtain that
(e (en))l C(T" An).
38

---

Published as a conference paper at ICLR 2024
Hence may apply the mean value theorem to (50) and obtain that
J /// 7L (C (Veg( b)) (o.) deds
(
V
E O . 8 ( O₂) 2 (O, -nn) V
6
(o)( (0, mb) 2 ( 2a) @ O 2 ( 0.)
+
24
! "
<
"Ilsllce 1 + 6 gg 1/ C(T",
0
l ||0,, 0
Igllc C(T", 90) + Ilallc C((T,,
l E //''''' b(O.)ds+ ['" o(O.)d
$ 8 lllllc C(T', 1gllce C' .u)
VOLs( 2
1gl E I'"' o(e.)a
+ 24 ||ll '
1
24
$ x7 lllllo C(T O + 1gl|c= CT Qn)
|
2 + 24 Igllco E / (l.()
C
+ 24 | F Vols((n))
S R l C(T O,)
1 n C(T" Oa)
+ C nE n2 ( [
+ n 19 E (),)
2
S " Iglllc C(T*, Oo)
We remark that for the last but third line we apply the Burkholder-Davis-Guuute Davis-Gundy inequality. To sum up for now,
Eg(0,) - Eg(O,) -
VoLs(0))
1/2
Vor(O0): (VoLs(0) 0-o. @ VoLs(0))ess) o=a. Σ((b)) + E2)(0))
+
2
39

---

Published as a conference paper at ICLR 2024
and
(g((,,) - (g) = 1 x- (0n)
1
: : - - (0)
2
2 (g) E3
" (Ves((.).
2 L (Ve(( (9))
1 v
b(O.)ds+ ["o(O.)dw,)
2
(L, (" + ['ooo.) + (0))
= (V T (Vee(()))) b) (9o)
+ - E I' b(O.)ds @ b((.)ds
(o0())) E '' a ( d., [ ['..o.).. (O))
2
o
n (Veg( E(b(
2 () (eo)
2 ( E L b(O.) nppp).oaaa
w..) / [(..) + (0))) Vog(On) E 2
+
we observe that
V og Ob): E S. (e,) J [....aa.
EE
(0) o (O.)ds (d)
2
I Vog(On) O
E
thus
n " (V (Oo)
Og) Eg(O,) (V
+ 2 v :
I/'' [ b(O.) @ ((O,.)ddd
+ 7
[ /VV :())dss (2)) 2
Since
Ver(O,): Ebb() b(O,)
= -Von(On) :()(e))) @ b(On)) + /-' L (Vān(00) b(O.) @ b(O.)) do
=VZr(On) : Eb)) 8 b(Oo)) /'' c (V ((.)) 8 b(O))) dw
+ +ʃ% c ((Vor( ((O.) b(O.)) du,
40

---

Published as a conference paper at ICLR 2024
and since
Vo)(Oo) E[
=V (o) E [00) + [ ((Vg((O)) E(+))) de,
we are one step away to finish our proof,
(v) n (Vee b) (Oo)
Eg((,,) Eg(O,)
1 / b(Oo) @ (
(ar(
2
7
E
(o)( ((oo)ds (0))
2
where we misuse notations for E.
O) and the term
7
(V b()..) @ b(On)) dwdsdu
7
[
+
L (V)r(())) b(..) @ b(.))) dudsdu
[. [ c ( E(O,) dods,
included, and (0)) is still of order (17²) by similar reasoning and omit its demonstration. Thus
Og( - -Eg(On) = (Veg Oo): E(b(O)))) n/² -/r (b(Oo). Ve (Veg(@o). b) (0n))
11" I(On) V (eer(Oo))b) (9n)
2
Vor
O1)
((a))
7
+ 2
(o)(()) E() + E (o).
and recall that since we choose
(()) = Ve (ss(O) " (oe[s((0)1
a ( = VT (+))
then
(9,) Eg((,,)
7 (V))(O,), Vo (Ls())) le=
2
Veg(Op), Ve (|VoLs lo-0.)
4
n
(Ve (Ts(()) . Ve (Vey Oo). (es(()))) le-e.)
2
2
(or(O)) (Ve (L)( ( Vo (Ls(O)) le
2
+ (og(O,): E + +
2
-
(Veg(())) Vo
(L) (L
(-
+ Ver( (On) (V ) 8 Ve (Ls(O)) le
2
122
(O)) O())) + (0,)
2
41

---

Published as a conference paper at ICLR 2024
thus, we have
[E((0,) Eg((,,) (g((n) 77 Vols()) 0-
VoLs(())
+ 112 Vag(On): (VoLs(0) 0-0 @ VoLs(0) 0-0 + I(Oo)
2
+ (00)
Eg(Oo) + + (Ve (4s le
r² (O))) (V ). @ Ve (Ls(()) l=
2
2 V (eg(O)) E())) E (b))
E2 + E
n² 1 co C(T So no) + n²² lllllc C(T",
O
For the N-th step iteration, since
[Eg(Ox) ( ,,N) (P)) g(0b) -P Pnog((o))
and the RHS of the above equation can be written into a telescoping sum as
P. (0o) Pnngg( ( (P. 1+1 o P(--))n((0)) - P.V Ping(
hence by application of Proposition D [Bg(Ox) (,,) S
we obtain that
P.N. N. 1+1 PP(-1)n((U)) - -NN (i))
1=1
ppy. (P) P(--1) P g(On))
1=1
since (P)) Put-ii - Pa 1(-1)) g( (0) be regarded (IR if we choose measure p to be the delta measure concentrated on O i.e.,
; 6
hence by the conctration property of Markov operators, we obtain further that
N
[Eg(Ox) - 10 nN) &
Σ
(P)) P(I-1) - Pn 0 P(--1j)) g(
1=1
N
1)
-
l=1
By taking expectation conditioned on (-11)) then similar to the relation (46). the following holds
()) -P = [B((,)) - (e()) θ (1
< (²(00 + F3 (11))
< n l|gllco C(T" Oo. no) + r7 llglec C(T*, Op)
On7²).
We remark that the last line of the above relation is essentially based on Assumption 2 since E (Q and E E (m))) can be bounded above by the second, fourth and sixth mo- ments the solution to SDE (32). hence we may apply dominated convergence theorem to obtain the last line of the above relation.
42

---

Published as a conference paper at ICLR 2024
To sum up,
N
[P g(06) Pnngg(oo| Σ
1 ((-1))0(00)) -Pin-
o (n))(On)) NO
1=1
hence for N = NT. -r
|PNN g(0o) - Pnogg(Oo) = NNO = NnO(1) = STO()) = =0()))
43

---

Published as a conference paper at ICLR 2024
1 VALIDATION FOR ASSUMPTION 1
In this section, we endeavor to demonstrate the validity of Assumption We begin this section by making some estimates on the modified loss Ls and covariance Σ.
I.1 ESTIMATES ON MODIFIED Loss AND COVARIANCE
For the modified loss, recall that 0 = veo( (gr))=1) = vec (((ar; w. (r)) as we have
V qu (s( 1 "1 Vg. q. (ato( + 1 Σ axo(w[a) Van (wx)(
ni i=1
and under the usual convention that for all i E [n].
c l2:
1/41
where e is some universal constant, and that o(0) = 0, we obtain that
ed
Σ a a (w a;;) - y/
m
$ 1 +
|rrl |rrl22
<1
1
[|,11 + ||,,, 112
2
S1+
|00
2
hence
VV Ls0)ll2 S (1+ [01
|qell2 +
1 P
|1
p
thus we have
VoLs(0)112 - (1+ ||0|) |01|
1 P |
$ Cp(1+ + | (.)
Moreover, since
1 Σ Vee s Voa+eeV
V (s()
n
1 diag V2 (
+
np
as we denote only for now as matrix multiplication,
v
1 P diag ( (afo(w]Z;))) np
Veg &
+o
x
V
1-P
Ve (aza(w)a;)3)
np
:=1
then the components in V Ls O
Vols) can be categorized into six different types: Firstly,
II(Ve @ Veg) e, Voeil2
< |ey Vec Vecel2
11 |
|0
1
| !
44

---

Published as a conference paper at ICLR 2024
Secondly,
Ve.. eg Voeel2
< (1 ||0| B) Verell2
(1+ | b) |012
$ (1+11015)
Thirdly,
|| [diag { VV2 (afo e Vocil2
$ [1+ |013) ||daag V
|01
x (1+ | (3) (1+ |1+101 |0|2
$ + 1112)
Fourthly,
|| (e & Vec) Ve (2)) 12
| 0
(1+ 1101)
$
Fifthly,
(& 22
Vo (,)o((T2,))) )
<
[ 1 V |
K O
|
<
O
8 2
Finally,
|| (diag V2a (2fo(w]x:)))) Ve (1)a(w)z,)))1
< diag { [ (afa(w]<;))) 99) |0
+ | |01 s
< + |(0l)]
To sum up. for the drift term b()), regardless of the choice of first order or second order accuracy, we obtain that
bb - 1 ||@||
As for the covariance Σ, recall that 0 = veo pm = vec (f(ar, wm Imm)). )). then we obtaín that the covariance Σ reads
Σ1
E1
Σ pm mm
Σg
Enz
Σ=
Em1
Em
Σ Σmm
45

---

Published as a conference paper at ICLR 2024
For each lo E [m/] we obtain that
[e) = ( -1--
[alo(wZz))) ae (aro(w0(x;i))
Ci + p
i=1
CiAt + (so(w((ai)) Ve (ano(w)aa;))
010 vz (axo(un(aii)
p2
1=1,1
1=1
n
alo un n; V.. (u)o(())))))
and for each r E [m]] with k + r,
(--1)
1
G + =oro(waaa 1 (a.o(w)f(s)) Va (alo(w(a;))
(e)
p
71
p
a. Va. (a,o(w)xx)))
-111
1)
(.o(w. V (axo(w[x:))
p
CAN (wo(()))) a.o(w)))))) ) Va (aro
p
hence we obtain that
(1,1 (erie 1
(0) s.
+ Σ
afo(w]):;) ar:) |
1=1,4
Cp(1 + | 101 |
< (1 O [
and by similar reasoning
[E)( s (1 |010 1.)
I.2 EXISTENCE, UNIQUENESS AND MOMENT ESTIMATES OF THE SOLUTION TO SDE
Existence of the solution to SDE the truncation function
32 is proved by a truncation procedure: For each M 2 1, define
((())
if O < M, b(M ( if + > M.
by (+)
We also perform similar truncation (Q) and obtain its truncation xM (+). Then by and aw satisfy the Lipschitz condition and the linear growth condition, hence by application of the classical results (Oksendal 2013 Theorem 5.2. in SDE, there exists unique solution e ar (-) to the truncated SDE
dO, bus (0,) dt + as (0:) dW, O, = (0).
(51)
We may choose M large enough, such that
| < M,
and the solution to SDE 332) coincides with the solution to SDE 51 least for period of time T* > since @ < M. We remark that T is the desired time in Assumption 2 We also remark
that not only for any time & [o, T*), the second, fourth and sixth moments of the solution to SDE
46

---

Published as a conference paper at ICLR 2024
(32) uniformly bounded with respect to time t, but also that for any time t [0, all moments of the solution to SDE (32) are uniformly bounded with respect to time t.
At this point, it is important discuss that prove is that for fixed time T, rate r > 0 small enough so that the SME is a good approximation of the distribution the dropout iterates. What particular, is not hard to construct systems where for fixed 1/, both the SME and the asymptotic expansion fails when time T is large enough.
take the learning
did not prove is that for fixed 17, the approximations hold for arbitrary time T. In
1.3 MOMENT ESTIMATES THE DROPOUT ITERATION
Recall that the dropout iteration reads
N ON-1 - 100--VV VeRs (0N-1;6N),
then obtain that
E
1
21
21 11y 11
10N-1112 112--2 (ON-1. (0N-- Rord (0N-1:dx)) + O
then for learning rate small enough, observe that { || N22 follows close to the trajectory of ordinary differential equation (ODE). Moreover, from the estimates obtained in Section 1.1
21 ||0
0 N1 Voll
Poop (ON-1;ôN))
< 5 1/
V VORSOO (0N (0N-1;ōN)
= =110x= 12
ce Vo ll2
< 110 2/
Cp 0 12) 0N-1112
<Cp(1 + 110~~11 2(+2)
we remark that as the above estimates hold almost surely, then for learning rate v small enough, we may apply Gronwall inequality to {E ||0x 11 N20 and shows that for some N all moments of the dropout iterations are uniformly bounded with respect to N, since for the ODE
du dt
1 1+t+"
ue u()).
(52)
with A 0. There exists time T O, such that for any time [0, T"), its solution {we is
uniformly bounded with respect to time t. And since for small enough learning rate, all moments of the dropout iterations { ||ON|] ||2 N20 follows close to the trajectory of ODEs of (22) type, hence all these moments are also uniformly bounded with respect to N.
47

---

Published as a conference paper at ICLR 2024
J EXPLICIT FORM AND DERIVATION ON THE COVARIANCE
In this section, present the expression for Σ. As 0 vec((gr) vec ((a,,,,,)))) then covariance of V Rerrop 11 6N) equals to - We denote
Ekr (0N = Cov (V Reroo (0N-1; RN), Va. Re (0x-1;dx)),
then
Σ11
E12
Eim
Σ2
E22
Emm
Σ
Σ Emi
Em2
Σ Emm
For each 1: E [m]] we obtain that
(0 =Cov (Var qe retton (0N-1:ox), V Restoo (0N-1:0x))
-
1 CGiA + (
Van (ano(
C,i + 1 p
( ae (ano(w(a:))
n
1
+
aNd (, Ve (ako(
p²
p
n
a a (1 Vg. (ato(w[a;;)
i=
where E2iA = (0) == m
Li=li (w) æ:)
Vı, and for each k, [m]] with k +
(0 =Cov (V., (V Rddoo (ON-1:ōN) V n
Rdrop (0N-1;
1 -1
m
a (1) Va (azo(())))))
a. (a.T( 1=1
(1-1)
ex1Airr + -alo([Ta:) + 1 (u.o( Val (aro(w((:))
p
(uo(w V (a,o(
+ -1
(1-1)
1
1)
1
a x, V (ato(w)xx;)
p
n
p a a.o(wa) Vg (a.0(
where GAAAA (A) = Σmh Liml, ao(w)x;; Vi- We remark that such expression is consistent in that for the extreme case where p 1, dropout 'degenerates' to GD, hence the covariance matrix degenerates to a zero matrix, i.c., Σ = DD The following part is the specific derivation process of the covariance matrix.
48

---

Published as a conference paper at ICLR 2024
J.1 ELEMENTS ON THE DIAGONAL
In this part, we compute Enk for all k E [m]]
Eea(ON Cov
(V R (ON-1;ōN). V R (0N-1;ōN))
Σ +J=1
Cov (e (ōN)a) Vg (a.o(w<)3;) @ Va (aza(w(ar,))
n
in order to compute Cov (e, (Ōx) ej (ōx)k), we need to compute firstly E lef" eg (8w)² and
since E
[ej" e (6x)2] consists of four parts, one which
E
Σ
(6x) (w) V/
Σ (w)))) y5 (ōx) l
A 1,
=E
(wx))) ys
Σ (w)) Yo
E [[Nw)]]
M=1,k/
1
E
(dn (1)
+ E
(ŌN) (wn
p
1/=Kk #
Wt &e' det
y:
(ōn) (w
Mj
Σ
(ŌN) (wx)
+ yill
k' 1,K t
R A yhh
m
a (w
Σ
p²
P WAt, Witk
1/1
(wo(w))
Vi
Σ (wo(w)) + yilli
P W=lK&'
p
W #'
m
( -
Σ
a Vs
p
Σ
k'
and the second part reads
E
(6w))a
Σ 11111
(8x (w) y5
(8
a (WTw
a.oow Vi
p2
#=
and by symmetry, the third part reads
m Σ
E
(6x)xa)((u0
(6x))a)((w) vi (8N)
1=1.E
Σ
Vi
p2
W=
49

---

Published as a conference paper at ICLR 2024
and finally, the fourth part reads
E ((dxh/aactuaa//00///rrtt////o//d
2 (
To sum up.
E [ 2g (6N)
Σ 20:
=
p2
k=1,k''
(so(waar)) (
a (un
C.AMeSi +
+
p²
p2
1
+
1
and
E Te N (dN )k E [e (5x)
1 =(ov +
egil + [axo(w[a;;) P
p
ano(wFx;)
Bg + (
1
e
p
hence
Cov (e: (8N) e5 (8N)))
=E [e eg (8N ji] - E Te (ŌN)a) [e (ōN)a)
1
m Σ
p2
/
# 1=
(+
axo (oowweeeees
+
p
) aoo +
(1
1
p2 p
1r
p2
(--1) E (5x)a) E (6N)k) +
m
Σ
=
A'
by summation over the indices and j, for each k E [m]. the covariance matrix reads:
( Cov
Hs (0N-1;ôN). V R (0N-1:ôN)
-
1 e + [ano(w)as))
Vaz (alo(w]xi))
p
1
C + 1 p
(alo(w(a aa (axo(w]a,))
1
mmw xx) Van (alo(W]x:))
12
p
I= ky
1
n
8
alo [w) x.) X Vg (aro(w]x;)))
72
i=1
50

---

Published as a conference paper at ICLR 2024
J.2 ELEMENTS OFF THE DIAGONAL
In this part, we compute Eer for all k E [mm]. where k +
(0 = Cov (V ns (0x-1;ŌN). V Rs (0N-1;
Cov (6x) (ōx))) a (azo(w]xi)) @ Var (a
n 4J=1
in order to compute Cov (ef* (5x)x: , of (ōx)r) we need to compute firstly [ [ (6x))((wx)) and since E [ej ef (ŌN)a(0N))) consists of nine parts, one of which is
E
Σ
(w))) - Vi
Σ
(6x)a)((()))))
(ōx)x(6x))
W = ke
m
=F
Σ
(u - 91
Σ (6x)a)((1))2z)) -ys
ys [[(())((NN))
1
E
P
k
ligh and .tk
Y
Yi
awo(w Vj
aNo(w y.9j
aNo
Σ
(xo(w) - Vi
W= W'
1 (1-1)
(w
(-11)
= -AAANEAMM +
p
W- Wf
and the second part reads
m
E
((w)))) - ys (6x
I of ke fo
m
aso (wo CiA.RAAr
=E
- Vi (w
[6x
p
by similar reasoning and symmetry, the third part reads
m
E
(8x) (( - Vi
m
a (,. (6N) (wo(wF
(6x) Vi
ex e..Amm
P
K=Lkk4kk
51

---

Published as a conference paper at ICLR 2024
also by similar reasoning and symmetry, the fourth part reads
E
(w ys (0x)oao(ur2))))))))00.)
W 1, kr
(xo
=E
(xx)))))(w2aaa) Vj [B
p
W'=lk'<k k f
and the fifth part reads
E [ /|tp:///0....aar//////.//0/.1//0
1 p a (w
and the sixth part reads
E [ B[(b:///aarrrnn///////aaaaaa//ad
= (W (100:
= "00 (10
also by similar reasoning and symmetry, the seventh part reads
m
E
(ōx) (w) y1 (6x))a..wt///))xxx)0...
M Ke
m
((x))))-)w(... Vs ((6,),0 a.o(w..))) ke
=E
P
and the eighth part reads
E htap:///haaaauaaa//o////o///////
and the ninth part reads
E
E [60:::00/N/60000mmmoooooaa.aaaaa
) p
To sum up,
E
-ENA) (2-1)
ano(wT e.
WEL
a.o(wTx))
(ko(w,a,)
BAAkAr D² (ooow((
p eilRl
P
1
(20(w
+
1
P
+
+ p (o
pa
52

---

Published as a conference paper at ICLR 2024
and
E
( + _ (euMi + + - _a
P
+ +AM + + + a (
+ ( + ( + bat (a)
1 ( p² (
hence
Cov e (6x),)
=E [ef eg¹ (6N))(6NN)) - -B [ef [ee (ōx)r]
1)
( + (1-1) a
1
(1- 1) p (o
u. +
p
p
af a1
-111
p2
p
by summation over the indices and] j, the covariance matrix reads
Cov V Rs (ON-1;ôN), V Ret (0N- Exr Exr(ON-1) (ON-1)
-
[as((a(a)) 1 ( Van (aro(w((a))
CAAAAr + p
p
(.o(w[x Va- (a,o(wFxx))
(--11) - p
1)
(oT[w]x Vg. (ato(w]x,1))
C.yAAr + + p ( ) Vg. (1)0)
53

---

Published as a conference paper at ICLR 2024
K
THE STRUCTURAL SIMILARITY BETWEEN HESSIAN AND COVARIANCE
We can derive the Hessian of the loss function in the expectation sense with respect to the dropout noise ò and the covariance matrix of dropout noise under intuitive approximations. We first show assumptions as follows:
Assumption 1. The NN piece-wise linear activation.
Assumption 2. The parameters of NN's ouiput layer are fixed during training.
Assumption 3. We study the loss landscape after training reaches a stable stage, i.e. the loss function in the sense of expectation is small enough,
Rdrop Eg Vo (0: 0.
Hessian matrix with dropout regularization Based on the Assumption 2 the Hessian matrix of the loss function with respect to J0.8 (x) can be written in the mean sense as:
H ~ ! Σ Vofo (2.) (ofo
1-P P V g. (a,o(w]xx;) @ q. (a,o(wIx:)))
P
where H0) (O).
We first compute the Hessian matrix after taking expectations with respect to the dropout variable,
Proof.
a
m
(0) Rs))(
V2
(u.o(w
(53)
2np
The first and second terms on the RHS of the Eq. (53) are as follows,
Rs() = - Σ (Volo (xx:) & Vola (x):) + (So (2x)) vi) VoSo (2:))
- p
V2 (a, (a,o(1
2np
(=1
r=
1-P
a (a,o(wTx;)) & Va (aTo(wTx;)) + (a,o(w]x:)) V2 (a.o(w])))))))
Note that for linear activate function, Vale (2;) = v2 (a,o( 0, a.e. Vi [n], [m]. we have
(OR
ooo (2:) @ (a;)
n
p
(a.o(w)xx)))
p
q. (a.o & Vg. (a.o(w).;;)
2np
q.
np
1=1
Thus the Eq.
(3)
can be rewritten as
1
p m
H(0)
Vofo (ofo(m:) +
Vg. (a.o(wTa:)) Vg (ara (uTTa:))
n
p
Covariance matrix with dropout regularization Based on the Assumption 3 the covariance matrix of the loss function under the randomness of dropout variable 5 and data can be written
E( ~ 4.1 (eleeeaaa + 4.2 -
V d.
@ Va. (a.o(wTx)))
p
1=1
where 41 (e 12 Emm
1:2 (e:)2
54

---

Published as a conference paper at ICLR 2024
Proof. For simplicity, we approximate the loss function through Taylor expansion, which is also used in Wei et al. (2020),
m
((Sola;;)) R [(f((x)). + (fo( - y/b)
Σ a (), r=1
where {[fe(=;;)) 8), v = = (fo 5) {(f(mm))) v.) = (So(x.) ni)² The covariance matrix under dropout regularization
Es (Vol(fo(zz:)) v.) @ Vol(fo(z;;)) V VOESRS (0:8)) @ VoEaR' (0: 6)
Es (Vo((fo(x;; 8) n.) @ Vollfo 6), v.))
Combining the properties the dropout variable we have,
E(0)
vo((fo(x:)..) u.) & Vol(fo(zi)).))
Y2
m
m (6
Es
(8 1), (1 @ Σ
V) (a
r=1
p m
vol(fo(z;))n) vi) @ vol(fo(x:)..))
+
q. (a @ (a
71
P
1
1 --P (
Ex y1)
n
p
(54)
We calculate the two terms on the RHS of the Eq. 54 separately:
E, (x = (e.)² Volo(o;) Volo(a.),
m
Eg
(e:)²
Σ
v (w,T( @ VV (a.(wxx Volo(z)) Vole(a)) Σ (a.ow))
+ 2
esa, (w
Voc (a.
(e,)²
& Vg (a,o(w qr
+ & Σ (ao(wxx;;))
Vo(e:) (ao
Under the assumption that 2 vol(fe(z;) 0. Vi E [n], we have
m
m
Eg 9:) (e,)² Va
Σ
a (w.x:))
Thus the Eq.
14) can be rewritten as
Σ(0) Volo(m.) & volo(z:)
1 1- (ct)
Σ (wTT(u)
p
+ np
(e.)² V (a.o( 8 V. (a.o(()))))
55

---

Published as a conference paper at ICLR 2024
Note that
(e:)² +
p
(aro(1 = [oo2([ 8),
P
we have
2 Σ
E(0)
EsE (So (fo( 0), Volo(w)) 8 Volo(æ;)
71 i=1
2(1 p)
((Jo(as) Vg. (10 Va. (a.((1))))))
np
56


Reviews:
[{"review_id": "UedRP6LDbr", "reviewer": "Reviewer_25p8", "summary": "This paper aims to deepen the understanding of dropout regularization in neural network training. The authors firstly derive stochastic modified equations (SMEs) that approximate the iterative process of the dropout algorithm. This provides valuable insights into dropout's dynamics. The study then conducts empirical investigations to explore how dropout assists in identifying flatter minima. By employing intuitive approximations and drawing analogies between the Hessian and covariance of dropout, the authors probe the mechanisms behind dropout's effectiveness. The empirical findings consistently demonstrate the presence of the Hessian-variance alignment relation throughout dropout's training process. This alignment relation, known for aiding in locating flatter minima, highlights dropout's implicit regularization effect, enhancing the model's generalization power.", "questions": "1. Assumption 1 seems to be quite strong? Any concrete example for this?\n\n2. Do the results hold for any activation functions? Considering the impact of different activation functions on the behavior of dropout regularization would contribute to a more comprehensive understanding\n\n3. The main theoretical result presented in this paper is an informal theorem, which may look a little bit weird.", "limitations": "", "rating": 6, "confidence": 2, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The authors present a rigorous theoretical derivation of the stochastic modified equations that approximate the iterative process of the dropout algorithm. This theoretical framework enhances the understanding of the underlying mechanisms behind dropout regularization.\n\n2. The empirical findings support the idea that dropout serves as an implicit regularizer by facilitating the identification of flatter minima. This discovery contributes to a more profound comprehension of dropout's intrinsic characteristics and its ability to improve the model's generalization capabilities.", "weaknesses": "1. The results presented in this paper are specifically applicable to shallow neural networks. The analysis and findings may not directly extend to deeper or more complex neural network architectures.\n\n2. The findings and conclusions derived from the theoretical analysis using GD may not fully reflect the behavior and performance of dropout regularization when applied in practice with SGD.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "fQpKUAbr7u", "reviewer": "Reviewer_Dhk2", "summary": "This paper examines the dropout algorithm in deep learning. The authors provide a theoretical framework by deriving stochastic modified equations to analyze the dynamics of dropout. In addition, the paper presents experimental findings that explore the relationship between the Hessian matrix and the covariance of dropout's noise", "questions": "- In Wu et al (2022), they provide an upper bound for the Hessian of the solution found by SGD based on the analysis of SGD noise, using dynamic stability analysis. Is it possible to develop a similar analysis that can offer an estimate of the Hessian for the solution found by dropout?\n\n- As shown in Figure 3, the alignment measure $\\alpha(\\theta_t)$ appears to strengthen after the initial training phase. What could be the potential factors contributing to this observed phenomenon?\n\n\nWu, Wang, and Su (2022). The alignment property of SGD noise and how it helps select flat minima: A stability analysis.", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "The optimization dynamics and generalization benifit of dropout is lack of understanding. This paper offers a rigorous theoretical analysis of the Stochastic Modified Equations associated with dropout. In addition, they conduct comprehensive experiments to explore the relationship between the Hessian matrix and the covariance of dropout's noise, which can unveil the genralization benifit of dropout. In summary, this article makes a substantial contribution to the understanding of dropout.", "weaknesses": "Theoretical analysis focused on two-layer neural networks, but it is indeed a meaningful step towards understanding dropout.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "8ubQY35qGn", "reviewer": "Reviewer_h4F1", "summary": "The authors study dropout in a one hidden layer neural network. They first compute the gradient after one step averaged over the dropout randomness, and show that this is the gradient of a modified loss which penalizes the mean squared value of each neuron's contributions to the output. The authors then write down an SDE which attempts to establish an approximation of dropout dynamics which is locally consistent with the real dynamics to second order. They conclude with theory and experiments that suggest the dropout noise covariance is aligned with the Hessian.", "questions": "How do the results change for deeper networks? Is there evidence that the same structure should remain?\n\nThe theorems in the paper rely on small learning rate; however, typically neural networks are trained with large stepsizes (which are far from gradient flow for example). Which aspects of the theory will hold at larger learning rates?\n\nFigure one and its related experiments could use improvement. For one, it would be better to plot e.g. panel 2 as a 2-D heatmap versus $\\epsilon$ and $p$, so one can understand the effects of varying each (and any interaction that occurs). Additionally, I think its important to probe some smaller $p$ values (1e-2, 1e-1 perhaps) as the effects of dropout get more interesting with sparser activations.\n\nFor Figure 2, it is better to label each panel by the parameters directly rather than (setting 1, setting 2).\n\nFor the discussion of matrix alignment, it is worth pointing out that random PSD matrices have non-trivial correlation even in high dimensions. In particular, for random PSD matrices the alignment is related to the average eigenvalue. Therefore the statement\n\n```\nIt is noteworthy that the cosine similarity between two random matrices of the same dimensions is highly improbable to exceed the threshold of 0.001.\n```\n\nis not the right one. It would be helpful to add a comparison to the plots of the covariance of two matrices with random eigenvectors, but the same eigenvalues as each of the relevant matrices. This will provide a better null model.\n\nI would suggest finding a different notation for the stepsize (currently $\\epsilon$). This will bring the notation more in line with what practitioners use, which I think will help the paper find a broader audience.\n\nAs a minor point: this line in the abstract:\n\n```\nThese dual facets of our research, encompassing both theoretical derivations and empirical observations, collectively constitute a substantial contribution towards a deeper understanding of the inherent tendency of dropout to locate flatter minima.\n```\n\ndoesn't really add anything for the reader - the content of the paper is what will convince me if the contribution is substantial or not!", "limitations": "", "rating": 6, "confidence": 4, "soundness": 4, "presentation": 3, "contribution": 2, "strengths": "The setup and analysis of dropout is well presented. The work uses relevant techniques to paint a picture of the inductive biases and some of the dynamical effects of the dropout procedure. The modified loss is easy to interpret, and it seems that at least at low learning rate the SDE performs quite similarly to the actual dynamics.", "weaknesses": "Overall, there is a question of the impact of the contribution. Everything within the paper is well executed (up to some minor comments addressed in questions), but the main result seems to be writing down the modified loss and SDE. The results about the alignment of the Hessian and dropout noise seem somewhat incomplete; I have given suggestions for improving those analyses as well.\n\nIn particular I wonder if the conclusions will generalize to the case of deeper networks, or if the qualitative picture changes, both in terms of the modified version of the loss, and the Hessian alignment. A related question is whether or not the results hold for the larger learning rates which are common in practical ML settings, and in the SGD+dropout setting. Even additional numerical evidence would be sufficient here in my opinion. Perhaps focusing experiments more on sweeps over $\\epsilon$ and $p$ can improve this.\n\nSome of the figures can also use improvement; see \"Questions\" for more detailed comments.\n\nI'm looking forward to comments from the authors on these points and am open to changing my review score if they are sufficiently addressed.\n\nUpdate: after author responses, many of the weaknesses were addressed and I have updated my review score.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "OmguAEWXBa", "reviewer": "Reviewer_hhFg", "summary": "This work analyzes the generalization capabilities of dropout through the lens of stochastic modified equations. The authors derive a weak-sense stochastic continuous time limit approximation of the discrete dropout algorithm and use this modified ODE to explain the phenomenon of dropout. This SDE is driven by a deterministic modified loss (due to dropout) and a stochastic Brownian motion with a covariance vector that also depends on the dropout parameter. The authors derive their conclusion from the similarity between the strcuture of the Hessian and the stochastic noise covariance matrix which could aid in getting flatter minimas as sharper directions would have more stochastic noise (due to similarity between Hessian structure and covariance matrix).", "questions": "Answers to questions posed in 3 to 7 are critical in my opinion as they form the basic claims and crux of the paper. \nSome sections in the manuscript need to be modified to make reading easier, see point 1 and 2.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1) The paper is moderately well written. \n2) Analyzing the effect of droput for generalization is an important problem in ML.", "weaknesses": "Although the paper seems to be easy to read at first glance, there are some portions where the authors do not clearly mention how they obtain certain equations, for example:\n\n1) In section-4.1, how the modified loss was derived is not very clear. It seems clear from the appendix. Better to refer it to the appendix or provide a statement on how it was derived. \n\n2) It's not evident how the authors get equation 8 from section 4.1. I missed the part where the stochasticity V comes into the picture from section 4.1. From Theorem-1, it is not clear, what effect dropout is having as the author's do not clearly mention the source of stochasticity. \nIt is well known a stochastic discrete algorithm can be well approximated by Euler-Maruyama discretization of an SDE, but in the manuscript the connection/derivation is not clear. \n\nNow here are some technical concerns I had:\n\n3) The role of the modified loss $L_{S}$ from the SDE is not clear in terms of generalization. Authors only use the flatness argument from the Hessian-covaraince alignment, but the deterministic part driving the SDE seems to not play any role ?\n\n5) It's not quite clear how the Hessian alignment with the covariance matrix is aiding to flatter minimas. Does the eigenvector directions of the covariance matrix also correspond exactly as that of the Hessian? Given, the equations in 5.2, it is not clear how the structure of the Hessian and the Cov matrix are the same. The coefficients (involving $e_{i}$) seem to suggest that the alignment may not be same. Infact near convegence, $l_{i,2}$ tends to 0, making the second term in the covariance term vanish. \n\n6) How the effect of dropout is in aiding to flat minima correspond to that of SGD and parameter noise injection. In some sense, dropout has some correspondace to both. Parametrer noise injection is well known as the explicit hessian regularizer (more explicitly on the trace of the hessian https://arxiv.org/abs/2206.04613). In my opinion, a comparison between SGD, parameter noise-injection and dropout are critical. \n\nminor: \n\n4) Which SME is used in the experiments?  The order-1 approximation or the order-2 approximation .", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (poster)