Journal of Machine Learning Research 1 (2017) 1-24

Submitted 11/16; Published 8/17

Surprising properties of dropout in deep networks
David P. Helmbold

dph@soe.ucsc.edu

Department of Computer Science
University of California, Santa Cruz
Santa Cruz, CA 95064, USA

Philip M. Long

plong@google.com

Google1600 Amphitheatre Parkway
Mountain View, CA 94043, USA

Editor: Sanjoy Dasgupta

Abstract
We analyze dropout in deep networks with rectified linear units and the quadratic loss. Our
results expose surprising differences between the behavior of dropout and more traditional
regularizers like weight decay. For example, on some simple data sets dropout training
produces negative weights even though the output is the sum of the inputs. This provides
a counterpoint to the suggestion that dropout discourages co-adaptation of weights. We also
show that the dropout penalty can grow exponentially in the depth of the network while theweight-decay penalty remains essentially linear, and that dropout is insensitive to various
re-scalings of the input features, outputs, and network weights. This last insensitivity
implies that there are no isolated local minima of the dropout training criterion. Our work
uncovers new properties of dropout, extends our understanding of why dropout succeeds,
and lays the foundation for further progress.
Keywords: Dropout, deep neural networks, regularization, learning theory.

Acknowledgments
We would like to acknowledge support for this project from the National Science Foundation(NSF grant IIS-9988642) and the Multidisciplinary Research Program of the Departmentof Defense (MURI N00014-00-1-0637).

c 2017 David P. Helmbold and Philip M. Long.
License: CC-BY 4.0, see https://creativecommons.org/licenses/by/4.0/. Attribution requirements are provided
at http://jmlr.org/papers/v1/helmbold17a.html.

Helmbold and Long

1. Introduction
The 2012 ImageNet Large Scale Visual Recognition challenge was won by the Universityof Toronto team by a surprisingly large margin. In an invited talk at NIPS, Hinton (2012)
credited the dropout training technique for much of their success. Dropout training isa variant of stochastic gradient descent (SGD) where, as each example is processed, thenetwork is temporarily perturbed by randomly “dropping out” nodes of the network. The
gradient calculation and weight updates are performed on the reduced network, and thedropped out nodes are then restored before the next SGD iteration. Since the ImageNet
competition, dropout has been successfully applied to a variety of domains (Dahl, 2012;
Deng et al., 2013; Dahl et al., 2013; Kalchbrenner et al., 2014; Chen and Manning, 2014),
and is widely used (Schmidhuber, 2015; He et al., 2015; Szegedy et al., 2015; Yang et al.,2016; Havaei et al., 2017); for example, it is incorporated into popular packages such as
TensorFlow, Torch, and Caffe. It is intriguing that crippling the network during training
often leads to such dramatically improved results, and dropout has also sparked substantial
research on related methods (for example, (Goodfellow et al., 2013; Wan et al., 2013; Gal
and Ghahramani, 2016)).
In this work, we examine the effect of dropout on the inductive bias of the learningalgorithm. A match between dropout’s inductive bias and some important applications
could explain the success of dropout, and its popularity also motivates the study of its
properties.
Weight decay training optimizes the empirical error plus an L2 regularization term,
λ2 , so we call λ ||w||2 the L penalty of w since it is the difference between training cri||w||
2
2
2
2
2
terion evaluated at w and the empirical loss of w. By analogy, we define the dropout penalty
of w to be the difference between the dropout training criterion and the empirical loss of w
(see Section 2). Dropout penalties measure how much dropout discriminates against weightvectors, so they are key to understanding dropout’s inductive bias.
Even in one-layer networks, conclusions drawn from (typically quadratic) approximations of the dropout penalty can be misleading (Helmbold and Long, 2015). Therefore wefocus on exact formal analysis of dropout in multi-layer networks. Theoretical analysis ofdeep networks is notoriously difficult, so we might expect that a thorough understanding ofdropout in deep networks must be achieved in stages. In this paper we further the processby exposing some of the surprising ways that the inductive bias of dropout differs from L2and other standard regularizers. These include the following:• We show that dropout training can lead to negative weights even when the output is apositive multiple of the the inputs. Arguably, such use of negative weights constitutes
co-adaptation – this adds a counterpoint to previous analyses showing that dropout
discourages co-adaptation (Srivastava et al., 2014; Helmbold and Long, 2015).
• Unlike weight decay and other p-norm regularizers, dropout training is insensitive
to the rescaling of input features, and largely insensitive to rescaling of the outputs;
this may play a role in dropout’s practical success. Dropout is also unaffected if theweights in one layer are scaled up by a constant c, and the weights of another layer
are scaled down by c; this implies that dropout training does not have isolated local
minima.
2

Dropout in Deep Networks• The dropout penalty grows exponentially in the depth of the network in cases where
the L2 regularizer grows linearly. This may enable dropout to penalize the complexityof the network in a way that more meaningfully reflects the richness of the network’s
behaviors. (The exponential growth with d of the dropout penalty is reminiscent ofsome regularizers for deep networks studied by Neyshabur et al. (2015).)• Dropout in deep networks has a variety of other behaviors different from standard
regularizers. In particular: the dropout penalty for a set of weights can be negative;the dropout penalty of a set of weights depends on both the training instances andthe labels; and although the dropout probability intuitively measures the strength ofdropout regularization, the dropout penalties are often non-monotonic in the dropout
probability. In contrast, Wager et al. (2013) show that when dropout is applied togeneralized linear models, the dropout penalty is always non-negative and does notdepend on the labels.
Our analysis is for multilayer neural networks with the square loss at the output node.
The hidden layers use the popular rectified linear units (Nair and Hinton, 2010) outputting
σ(a) = max(0, a) where a is the node’s activation (the weighted sum of its inputs). We
study the minimizers of a criterion that may be viewed as the objective function when using
dropout. This abstracts away sampling and optimization issues to focus on the inductive
bias, as in some previous work (Breiman, 2004; Zhang, 2004; Bartlett et al., 2006; Long andServedio, 2010; Helmbold and Long, 2015). See Section 2 for a complete explanation.
Related work
A number of possible explanations have been suggested for dropout’s success. Hinton et al.
(2012) suggest that dropout controls network complexity by restricting the ability to coadapt weights and illustrate how it appears to learn simpler functions at the second layer.
Others (Baldi and Sadowski, 2013; Bachman et al., 2014) view dropout as an ensemble
method combining the different network topologies resulting from the random deletion ofnodes. Wager et al. (2014) observe that in 1-layer networks dropout essentially forces
learning on a more challenging distribution akin to ‘altitude training’ of athletes.
Most formal analysis of the inductive bias of dropout has concentrated on the singlelayer setting, where a single neuron combines the (potentially dropped-out) inputs. Wager
et al. (2013) considered the case that the distribution of label y given feature vector x is amember of the exponential family, and the log-loss is used to evaluate models. They pointed
out that, in this situation, the criterion optimized by dropout can be decomposed into theoriginal loss and a term that does not depend on the labels. They then gave approximations
to this dropout regularizer and discussed its relationship with other regularizers. As we haveseen, many aspects of the behavior of dropout and its relationship to other regularizers are
qualitatively different when there are hidden units.
Wager et al. (2014) considered dropout for learning topics modeled by a Poisson generative process. They exploited the conditional independence assumptions of the generative
process to show that the excess risk of dropout training due to training set variation has
a term that decays more rapidly than the straightforward empirical risk minimization, but
3Helmbold and Long

also has a second additive term related to document length. They also discussed situations
where the model learned by dropout has small bias.
Baldi and Sadowski (2014) analyzed dropout in linear networks, and showed how dropout
can be approximated by normalized geometric means of subnetworks in the nonlinear case.
Gal and Ghahramani (2015) described an interpretation of dropout as an approximation toa deep Gaussian process.
The impact of dropout (and its relative dropconnect) on generalization (roughly, howmuch dropout restricts the search space of the learner) was studied in (Wan et al., 2013).
In the on-line learning with experts setting, Van Erven et al. (2014) showed that applying
dropout in on-line trials leads to algorithms that automatically adapt to the input sequence
without requiring doubling or other parameter-tuning techniques.
The rest of the paper is organized as follows. Section 2 introduces our notation andformally defines the dropout model. We prove that dropout enjoys several scaling invariances that weight-decay doesn’t in Section 3, and that dropout requires negative weights
even in very simple situations in Section 4. Section 5 uncovers various properties of thedropout penalty function. Section 6 describes some simulation experiments. We provide
some concluding remarks in Section 7.

2. Preliminaries
Throughout, we will analyze fully connected layered networks with K inputs, one output,d layers (counting the output, but not the inputs), and n nodes in each hidden layer.
We assume that n is a positive multiple of K and that K is a perfect square and apower of two to avoid unilluminating floor/ceiling clutter in the analysis. We will callthis the standard architecture. We use W to denote a particular setting of the weights
and biases in the network and W(x) to denote the network’s output on input x using W.
The hidden nodes are ReLUs, and the output node is linear. W can be decomposed as(W1 , b1 , ..., Wd−1 , bd−1 , w, b), where each Wj is the matrix of weights on connections fromthe j − 1st into the jth hidden layer, each bj is the vector of bias inputs into the jth hidden
layer, w are the weights into the output node, and b is the bias into the output node.
We will refer to a joint probability distribution over (x, y) pairs as an example distribution.
We focus on square loss, so the loss of W on example (x, y) is (W(x) − y)2 . The risk isthe expected loss with respect to an example distribution P , we denote the risk of W asdef
RP (W) = E(x,y)∼P (W(x) − y)2 . The subscript will often be omitted when P is clear
from the context.
The goal of L2 training is to find weights and biases minimizing the L2 criterion withdef

regularization strength λ: J2 (W) = R(W)+ λ2 ||W||2 . Here and throughout, we use ||W||2 todenote the sum of the squares of the weights of W. (As usual, the biases are not penalized.)
We use WL2 to denote a minimizer of this criterion. The L2 penalty, λ2 ||W||2 , is nonnegative. This is useful, for example, to bound the risk of a minimizer WL2 of J2 , since
R(W) ≤ J2 (W).
Dropout training independently removes nodes in the network. In our analysis each
non-output node is dropped out with the same probability q, so p = 1 − q is the probability
that a node is kept. (The output node is always kept; dropping it out has the effect ofcancelling the training iteration.) When a node is dropped out, the node’s output is set to4Dropout in Deep Networks

output
1

1
1

1

1

1
1
x1

-1
x2

Figure 1: A network where the dropout penalty is negative.
0. To compensate for this reduction, the values of the kept nodes are multiplied by 1/p.
With this compensation, the dropout can be viewed as injecting zero-mean additive noiseat each non-output node (Wager et al., 2013). 1
A dropout pattern is a boolean vector indicating the choices, for each node in the network, of whether the node is kept (1) or dropped out (0). For a network W, an input
x, and dropout pattern R, let D(W, x, R) be the output of W when nodes are dropped
out or not following R (including the 1/p rescaling of kept nodes’ outputs). The goal ofdropout training on an example distribution P is to find weights and biases minimizing the
dropout criterion for a given dropout probability:def
JD (W) = ER E(x,y)∼P (D(W, x, R) − y)2 .
This criterion is equivalent to the expected risk of the dropout-modified network, and we
use WD to denote a minimizer of it. Since the selection of dropout pattern and example
from P are independent, the order of the two expectations can be swapped, yielding
JD (W) = E(x,y)∼P ER (D(W, x, R) − y)2 .
(1)Equation (1) is a key property of the dropout criterion. It indicates when something istrue about the dropout criterion for a family of distributions concentrated on single examples, then (usually) the same thing will be true for any mixture of these single-example
distributions.
Consider now the example network in Figure 1. The weight parameters W1 and w are all1’s, and all of the biases are 0. W(1, −1) = 0 as each hidden node computes 0. Each dropout
pattern indicates the subset of the four lower nodes to be kept, and when q = p = 1/2 eachsubset is equally likely to be kept. If R is the dropout pattern where input x2 is dropped andthe other nodes are kept, then the network computes D(W, (1, −1), R) = 8 (recall that whenp = 1/2 the values of non-dropped out nodes are doubled). Only three dropout patterns
produce a non-zero output, so if P is concentrated on the example x = (1, −1), y = 8 thedropout criterion is:
JD (W) =1
2
13(8 − 8)2 + (4 − 8)2 + (0 − 8)2 = 54.
16
16
16

1. Some authors use a similar adjustment where the weights are scaled down at prediction time instead ofinflating the kept nodes’ outputs at training time.

5Helmbold and Long

As mentioned in the introduction, the dropout penalty of a weight vector for a given
example distribution and dropout probability is the amount that the dropout criterion
exceeds the risk, JD (W) − R(W). Wager et al. (2013) show that for 1-layer generalized
linear models, the dropout penalty is non-negative.
Since W(1, −1) = 0, we have R(W) = 64, and the dropout penalty is negative in our
example. This is because the variance in the output due to dropout causes the network tobetter fit the data (on average) than the network’s non-dropout evaluation. In Section 5.2,
we give a necessary condition for this variance to be beneficial. As with the dropout criterion, the dropout penalty decomposes into an expectation of penalties over single examples:
JD (W) − R(W) = E(x,y)∼P ER (D(W, x, R) − y)2 − (W(x) − y)2 .
Definition 1 Define P(x,y) as the distribution with half of its weight on example (x, y) andhalf of its weight on example (0, 0).
Unless indicated otherwise, we assume p = q = 1/2 for simplicity, although this is not
crucial for our results.

3. Scaling inputs, weights and outputs
This section compares the sensitivity of dropout and weight decay on the scale of the trainingdata.
3.1 Dropout is scale-free
Here we prove that dropout regularizes deep networks in a manner that is independent of
the scale of the input features. In other words, training under dropout regularization does
not penalize the use of large weights when needed to compensate for small input values.
Definition 2 For any example distribution P , define the dropout aversion of P to be themaximum, over minimizers WD of the dropout criterion JD (WD ), of RP (WD )−inf W RP (W).
The dropout aversion of P measures the extent to which P is incompatible with theinductive bias of dropout, measured by the risk gap between the true risk minimizer andthe optimizers of the dropout criterion.
Definition 3 For example distribution P and square matrix A, denote by A ◦ P the distribution obtained by sampling (x, y) from P , and outputting (Ax, y).
When A is diagonal and has full rank, then A ◦ P is a rescaling of the inputs, likechanging one input from minutes to seconds and another from feet to meters.
Theorem 4 For any example distribution P , and any diagonal full-rank K × K matrix A,
the dropout aversion of P equals the dropout aversion of A ◦ P .
Proof: Choose a network
W = (W1 , b1 , ..., Wd−1 , bd−1 , w, b).
6

Dropout in Deep Networks

Let
W 0 = (W1 A−1 , b1 ..., Wd−1 , bd−1 , w, b).
For any x, W(x) = W 0 (Ax), as A−1 undoes the effect of A before it gets to the restof the network, which is unchanged. Furthermore, for any dropout pattern R, we have
D(W, x, R) = D(W 0 , Ax, R). Once again A−1 undoes the effects of A on kept nodes (since
A is diagonal), and the rest of the network W 0 is modified by R an a manner paralleling
W. Thus, there is bijection between networks W and networks W 0 with JD (W 0 ) = JD (W)and R(W 0 ) = R(W), yielding the theorem.
Theorem 4 indicates that some common normalizations of the input features (for example, to have unit variance) do not affect the quality of the dropout criterion minimizers,
but normalization might change the speed of convergence and which minimizer is reached.
Centering the features has slightly different properties. Although it is easy to use the biases
to define a W 0 that “undoes” the centering in the non-dropout computation, different W 0
appear to be required for different dropout patterns, breaking the bijection exploited in
Theorem 4.
As we will see in Section 3.4, weight decay does not enjoy such scale-free status.
3.2 Dropout’s invariance to parameter scaling
Next, we describe an equivalence relation among parameterizations for dropout networks ofdepth d ≥ 2. Basically, scaling the parameters at a level creates a corresponding scaling of
the output. (A similar observation was made in a somewhat different context by Neyshabur
et al. (2015).)
Theorem 5 For any input x, dropout pattern R, any network
W = (W1 , b1 , ..., Wd−1 , bd−1 , w, b),
and any positive c1 , ..., cd , if




d−1
Y

W 0 = c1 W1 , c1 b1 , c2 W2 , c1 c2 b2 , ..., cd−1 Wd−1 , 

cj  bd−1 , cd w, 

j=1

then D(W 0 , x, R) =Qd
j=1 cj

D(W, x, R). In particular, if

d
Y

 
cj  b ,

(2)

j=1

Qd

j=1 cj = 1, then for any ex0
ample distribution P , networks W and W have the same dropout criterion, dropout penalty,

and expected loss.
Note that the re-scaling of the biases at layer j depends not only on the rescaling of theconnection weights at layer j, but also the re-scalings at lower layers.
Proof: Choose an input x and a dropout pattern R. Define W 0 as in (2). For eachhidden layer j, let (hj1 , ..., hjn ) be the jth hidden layer when applying W to x with R, andlet (h̃j1, ..., h̃jn ) be the jth hidden layer when applying W 0 instead. By induction, for all i,Qh̃ji =`≤j c` hji ; the key step is that the pre-rectified value used to compute h̃ji has thesame sign as for hji , since rescaling by cjQpreserves the sign. Thus the
Qsame units are zeroedout in W and W 0 and D(W, x, R) = ( j cj )D(W 0 , x, R). When j cj = 1, this implies
7

Helmbold and Long

D(W, x, R) = D(W 0 , x, R). Since this is true for all x and R, we have JD (W) = JD (W 0 ).
Since, similarly, W(x) = W 0 (x) for all x, so R(W) = R(W 0 ), the dropout penalities for W
and W 0 are also the same.
Theorem 5 implies that the dropout criterion never has isolated minimizers, since one
can continuously up-scale the weights on one layer with a compensating down-scaling at
another layer to get a contiguous family of networks computing the same function andhaving the same dropout criterion. It may be possible to exploit the parameterizationequivalence of Theorem 5 in training by using canonical forms for the equivalent networks
or switching to an equivalent networks whose gradients have better properties. We leavethis question for future work.
3.3 Output scaling with dropout
Scaling the output values of an example distribution P does affect the aversion, but in avery simple and natural way.
Theorem 6 For any example distribution P , if P 0 is obtained from P by scaling the outputsof P by a positive constant c, the dropout aversion of P 0 is c2 times the dropout aversion ofP.
Proof: If a network W minimizes the dropout criterion for P , then the network W 0 obtainedby scaling up the weights and bias for the output unit by c minimizes the dropout criterion
for P 0 , and for any x, y, and dropout pattern R, (D(W, x, R)−y)2 = (D(W 0 , x, R)−cy)2 /c2 .

3.4 Scaling properties of weight decay
Weight decay is not scale-free. Define the weight-decay aversion analogously to the dropout
aversion: the weight-decay aversion with P is the maximum, over minimizers W ∗ of J2 (W ∗ ),of RP (W ∗ ) − inf W RP (W).
We analyze the L2 criterion for depth-2 networks in Appendix B, resulting in the following theorem. Our proof shows that, despite the non-convexity of the L2 criterion, it isstill possible to identify a closed form for one of its optimizers.
Theorem 7 Choose an arbitrary number n of hidden nodes, λ > 0 and x ∈ RK other than(0, 0, ..., 0).
λ2
The weight-decay aversion of P(x,1) is min( 14 , x·x).
Theorem 7 shows that, unlike dropout, the weight decay aversion does depend on thescaling of the input features.
√
Furthermore, when 2λ > x · x, the weight-decay criterion for P(x,1) has only a single
isolated optimum weight setting2 – all weights set to zero and bias 1/2 at the output node.
2. Since the output node puts weight 0 on each hidden node and the biases are unregularized, this optimum
actually represents a class of networks differing only in the irrelevant biases at the hidden nodes. One
can easily construct other cases when weight-decay has isolated minima in this sense, for example whenn = 2 and there is equal probability on x and −x, both with label 1.

8Dropout in Deep Networks

This means that weight-decay in 2-layer networks can completely regularize away significant
signal in the sample even when λ is finite, contrasting starkly with weight-decay’s behavior
in 1-layer networks.
The “vertical” flexibility to rescale weights between layers enjoyed by dropout (Theorem 5) does not hold for L2 : one can always drive the L2 penalty to infinity by scaling one
layer up by a large enough positive c, even while scaling another down by c. On the other
hand, the proof of Theorem 7 shows that the L2 criterion has an alternative “horizontal”
flexibility involving the rescaling of weights across nodes on the hidden layer (under thetheorem’s assumptions). Lemma 31 shows that at the optimizers each hidden node’s contributions to the output are a constant (depending on the input) times their contribution
to the the L2 penalty. Shifting the magnitudes of these contributions between hidden nodes
leads to alternative weights that compute the same value and have the same weight decay
penalty. This is a more general observation than the permutation symmetry between hidden nodes because any portion of a hidden node’s contribution can be shifted to anotherhidden node.

4. Negative weights for monotone functions
If the weights of a unit are non-negative, then the unit computes a monotone function, inthe sense that increasing any input while keeping the others fixed increases the output. The
bias does not affect a node’s monotonicity. A network of monotone units is also monotone.
We first present our theoretical results for many features (Section 4.1) and few features(Section 4.2), and then discuss the implication of these results in Section 4.3.
4.1 The basic case – many features
In this section, we analyze the simple distribution P(1,1) that assigns probability 1/2 tothe example (0, ..., 0), 0, and probability 1/2 to the example (1, ..., 1), 1. The support ofP(1,1) is arguably the simplest monotone function. Nevertheless, we prove that dropout
uses negative weights to fit this data.
The key intuition is that optimizing the dropout criterion requires controlling the variance. Negative weights at the hidden nodes can be used to control the variance due todropout at the input layer. When there are enough hidden nodes this becomes so beneficial
that every minimizer of the dropout criterion uses such negative weights.
Theorem 8 For the standard architecture, if K > 18 and n is large enough relative to K
and d, every optimizer of the dropout criterion for P(1,1) uses at least one negative weight.
To prove Theorem 8, we first calculate JD (Wneg ) for a network Wneg that uses negative
weights, and then prove a lower bound greater than this value that holds for all networks
using only non-negative weights.
All of the biases in Wneg are 0.
A key building block in the definition of Wneg is a block of hidden units that we callthe first-one gadget. Each such block has K hidden nodes, and takes its input from the K
input nodes. The ith hidden node in the block takes the value 1 if the ith input node is 1,
and all inputs xi0 for i0 < i are 0; otherwise it takes the value 0. This can be accomplished9Helmbold and Long

with a weight vector w with wi0 = −1 for i0 < i, with wi = 1, and with wi0 = 0 for i0 > i.
The first hidden layer of Wneg comprises n/K copies of the first-one gadget.
Informally, this construction removes most of the variance in the number of 10 s in the
input, as recorded in the following lemma.
Lemma 9 On any input x ∈ {0, 1}n except (0, 0, ..., 0), the sum of the values on the first
hidden layer of Wneg is exactly n/K.
The weights into the remaining hidden layers of Wneg are all 1, and all the weights intodef

the output layer take a value c =K2, chosen to minimize the dropout
1 d−2
2nd−1 (1+ K
1+ n
)
n )(

criterion for the network. The following lemma analyzes Wneg .
(1−2−K )1
Lemma 10 JD (Wneg ) = 2 1 −.
d−2(1+ Kn )(1+ n1 )d−2When n is large relative to K and d, the 1 + K
1 + n1denominator in Lemma 10n−K
approaches 1, so JD (Wneg ) approaches 2 /2 in this case. Lemma 16 below gives a larger
lower bound for any network with all positive weights. In the concrete case when d = 2 andn = K 3 , then Lemma 10 implies JD (Wneg ) < 1/K 2 .
Proof (of Lemma 10): Consider a computation of Wneg (1, 1, ..., 1) under dropout andlet ŷ be the (random) output. Let k0 be the number of input nodes kept, and, for each j ≥ 2,
let kj be the number of nodes in the jth hidden layer kept. Call the node in each first-one
gadget that computes 1 a key node, and if no node in the gadget computes 1 because theinput is all dropped, arbitrarily make the gadget’s first hidden node the key node. This
ensures there is exactly one key node per gadget, and every non-key node computes 0. Let
k1 be the number of kept key nodes
Q on the first hidden layer. If k0 = 0, the output ŷ of the
network is 0. Otherwise, ŷ = c2d d−1
j=1 kj .
Note that k0 is zero with probability 2−K . Whenever k0 ≥ 1, k1 is distributed as
B(n/K, 1/2). Each other kj is distributed as B(n, 1/2), and k1 , k2 , ..., kd−1 are independentof one another.
dE(ŷ) = Pr(k0 ≥ 1)c2 E[k1 |k0 ≥ 1]

d−1
YE[kj ]

j=2= (1 − 2−K )c2d

 n   n d−2
2K

2

=

2c
(1 − 2−K )nd−1 .
K

Using the value of the second moment of the binomial, we getE(ŷ 2 ) = E 1k0 ≥1 c2d

d−1
Y

2 
kj   = 4c2 (1 − 2−K )

j=1




4c2 (1 − 2−K ) 2(d−1)
K
1 d−2
=
n
1+
1+
.
K2
n
n
10

 n  n
KK
+ 1 nd−2 (n + 1)d−2

Dropout in Deep Networks

Thus,

1
1 − 2E(ŷ) + E(ŷ 2 )2d−2 !
2 (1 − 2−K )
1
4c
K
1
4c
=
1 − (1 − 2−K )nd−1 +n2(d−1) 1 +
1+
2
K
K2
n
n
!
1
(1 − 2−K )=1−d−2 ,
2
1+ K 1+ 1

JD (Wneg ) =nsince c =n

K
, completing the proof.
1 d−2
2nd−1 (1+ K
1+ n
)
n )(

Next we prove a lower bound on JD for networks with nonnegative weights. Let W be
an arbitrary such network.
Our lower bound will use a property of the function computed by W that we now define.
Definition 11 A function φ : RK → R is supermodular if for all x, δ 1 , δ 2 ∈ RK whereδ1, δ2 ≥ 0
φ(x) + φ(x + δ 1 + δ 2 ) ≥ φ(x + δ 1 ) + φ(x + δ 2 ),or equivalently:
φ(x + δ 1 + δ 2 ) − φ(x + δ 2 ) ≥ φ(x + δ 1 ) − φ(x)The latter indicates that adding δ 1 to the bigger input x + δ 2 has at least as large an effectas adding it to the smaller input x.
Since W has all non-negative weights, it computes a supermodular function of its inputs.
(This fact may be of independent interest.)
Lemma 12 If a network has non-negative weights and its activation functions σ(·) are convex, continuous, non-decreasing, and differentiable except on a finite set, then the network
computes a supermodular function of its input x.
Proof: We will prove by induction over the layers that, for any unit h in the network, ifh(x) is the output of unit h when x is the input to W, then h(·) is a supermodular functionof its input.
The base case holds since each input node h outputs the corresponding component of
the input, and (x + δ 1 ) − x = (x + δ 1 + δ 2 ) − (x + δ 2 ).
Now, for the inductive step, let w be the weight vector for node h, let b be its bias, andσ(·) be its activation function. Let I(x), I(x + δ 1 ), I(x + δ 2 ), and I(x + δ 1 + δ 2 ) be theinputs to node h when the inputs to the network are x, x + δ 1 , x + δ 2 and x + δ 1 + δ 2
respectively.
By induction, these inputs to node h (componentwise) satisfy
I(x + δ 1 ) − I(x) ≤ I(x + δ 1 + δ 2 ) − I(x + δ 2 ).
Therefore, since w, δ 1 , and δ 2 are non-negative, the interval [w · I(x + δ 2 ) + b, w · I(x + δ 1 +δ 2 )+b] is at least as long and starts at least as high as the interval [w·I(x)+b, w·I(x+δ 1 )+b].
11

Helmbold and Long

Since σ is continuous and differentiable except on a finite set, we have
Z w·I(x+δ1 )+bh(I(x + δ 1 )) − h(I(x)) =σ 0 (z)dz
w·I(x)+b

Z w·I(x+δ1 +δ2 )+b≤

σ 0 (z)dz (since σ 0 is non-decreasing)w·I(x+δ 2 )+b= h(I(x + δ 1 + δ 2 )) − h(I(x + δ 2 )).

Definition 13 Let r0 ∈ {0, 1}K be the dropout pattern concerning the input layer, and let
R0 be the dropout pattern concerning the rest of the network, so that the dropout pattern
R = (r0 , R0 ).
For each ` ∈ {0, ..., K}, let ψW (`) be the average output of W under dropout when ` ofthe inputs are kept: that is,XψW (`) = E D(W, 1K , (r0 , R0 ))r0j = ` .
jLemma 14 For any ` ∈ {1, ..., K − 1},ψW (` + 1) − ψW (`) ≥ ψW (`) − ψW (` − 1).
Proof: Generate u, i and j randomly by, first choosing u uniformly at random from among
bit vectors with ` ones, then choosing i uniformly from the 0-components of u, and j
uniformly from the 1-components of u. By Lemma 12,
W(u + ei ) − W(u) ≥ W(u − ej + ei ) − W(u − ej )(3)always holds. Furthermore, u + ei is uniformly distributed among bit vectors with ` + 1
ones, u − ej is uniformly distributed among bit vectors with ` − 1 ones, and u + ei − ej isuniformly distributed among bit vectors with ` ones. This is true for W, but it is also true
for any network obtained by dropping out some of the hidden nodes of W. ThusψW (` + 1) − ψW (`)= E(D(W, 1K , (r0 , R0 )|X

r0j = ` + 1)) − E(D(W, 1K , (r0 , R0 )|jK

0= E(D(W, r0 , (1 , R )|X
X

r0j = `))jK0

r0j = ` + 1)) − E(D(W, r0 , (1 , R )|j

≥ E(D(W, r0 , (1K , R0 )|X
X

r0j = `))jr0j = `)) − E(D(W, r0 , (1K , R0 )|jX

r0j = ` − 1)),jby (3)) and the distributions of u + ei , u − ej + ei and u − ej . Since
X
X
E(D(W, r0 , (1K , R0 )|r0j = `))−E(D(W, r0 , (1K , R0 )|r0j = `−1)) = ψW (`)−ψW (`−1),
j

j

12

Dropout in Deep Networks

this completes the proof.
We will use the following lower bound on the tail of the binomial. (Many similar lower
bounds are known.)
Lemma 15 If X is distributed according to Bin(n, 1/2), then√√
Pr(X < n/2 − n/4) = Pr(X > n/2 + n/4) ≥ 1/4.
√√
Proof: Using the fact that, for any i, Pr(X = i) ≤ 1/ n, we get Pr(|X − n/2| < n/4) ≤√1/2, so Pr(X < n/2 − n/4) ≥ 1/4.
Now we are ready for the lower bound on JD (W).
1
Lemma 16 If K > 18 and the weights in W are non-negative, then JD (W) ≥ 36K.
q
1
1. First, note that ψW (0) ≤Proof: Assume to the contrary that JD (W) < 36K
18K ,1
or else the contribution to JD (W) due to the (0, 0), 0 example is at least 36K. Applying
Lemma 15, we have
r
r
√
√
2
2
ψW (K/2 − K/4) > 1 −and ψW (K/2 + K/4) < 1 +
(4)
9K
9K
1
as otherwise the contribution of one of the tails to JD (W) will be at least 36K
for the(1, . . . , 1), 1 example. We will contradict
this small variation of ψW (`) around K/2. The√
bounds on ψW (0) and ψ(K/2 − K/4) and Lemma 14 imply that ψW (`) grows rapidly
when ` is around K/2, in particular:
q
q
2
1
1
−
−
√
√
1
9K
18K
√
ψ(K/2 − K/4 + 1) − ψ(K/2 − K/4) ≥>p,
K/2 − K/4
9/32K

since K > 18. Now using Lemma 14 repeatedly shows that
r
√
√
√
K
1
2
ψ(K/2 + K/4) − ψ(K/2 − K/4) >
×p
=2
,
2
9K
9/32K
which contradicts (4), completing the proof.
Putting together Lemmas 10 and 16 immediately proves Theorem 8, since for K > 18
and large enough n, the criterion for Wneg must be less than the criterion for any networkwith all non-negative weights.
4.2 The case when K = 2
Theorem 8 uses the assumption that K > 18 and n is large enough; is the lower bound on
K really necessary? Here we show that it is not, by treating the case that K = 2.
Theorem 17 For the standard architecture, if K = 2, for any fixed d and large enough n,every optimizer of the dropout criterion for P(1,1),1 uses negative weights.
13

Helmbold and Long

Proof: Define Wneg as in the proof of Lemma 10, except that the output layer has a biasof 1/5.
We claim that
lim JD (Wneg ) = 1/10.
(5)n→∞

To see this, consider the joint input/label distribution under dropout:
Pr((0, 0), 0) = 1/2
Pr((0, 0), 1) = 1/8
Pr((2, 0), 1) = 1/8
Pr((0, 2), 1) = 1/8
Pr((2, 2), 1) = 1/8.
Due to the bias of 1/5 on the output, Wneg (0, 0) = 1/5. Thus, contribution to JD fromexamples with x = (0, 0) in this joint distribution is 1/2 × (1/5)2 + 1/8 × (4/5)2 = 1/10.
Now, choose x 6= (0, 0). If, after dropout, the input is x, each node in the hidden layer
closest to the input computes 1. Arguing exactly as in the proof of Lemma 10, in suchcases,1
E((ŷ − 1)2 ) = 1 −d−2 .
2
1 + n 1 + n1
This proves (5).
Now, let W be an arbitrary network with non-negative weights.
For our distribution,
ER (D(W, (0, 0), R) − 0)2 + ER (D(W, (1, 1), R) − 1)2.
JD (W) =2
Let V00 = E(W(0, 0)), V22 = E(W(2, 2)), V20 = E(W(2, 0)), V02 = E(W(0, 2)) where
the expectations are taken with respect to the dropout patterns at the hidden nodes (withno dropout at the inputs). Since each dropout pattern over the hidden nodes defines aparticular network, and Lemma 12 holds for all of them, the relationships also hold for the
expectations, so
V22 ≥ V20 + V02 − V00 .
Using this V notation, handling the dropout at the input explicitly, and the bias-variancedecomposition keeping just the bias terms we get:(V00 − 0)2 + (V00 − 1)2 + (V22 − 1)2 + (V20 − 1)2 + (V02 − 1)2 /4
JD (W) ≥2
8JD (W) ≥ 4(V00 − 0)2 + (V00 − 1)2 + (V22 − 1)2 + (V20 − 1)2 + (V02 − 1)2 .
We will continue lower bounding the RHS. We can re-write V22 as V20 + V02 − V00 + 
where  ≥ 0. This is convex and symmetric in V02 and V20 so they both take the same value
at the minimizer of the RHS, so we proceed using V20 for this common minimizing value.
14

Dropout in Deep Networks

2
8JD (W) ≥ (2V20 − V00 +  − 1)2 + 2(V20 − 1)2 + (V00 − 1)2 + 4V00.

Differentiating with respect to V20 , we see that the RHS is minimized when V20 =(2 + V00 − )/3, giving
8JD (W) ≥(V00 −  − 1)22+ (V00 − 1)2 + 4V00.
3

If V00 ≥ 1, then JD ≥ 1/2 (just from the (0,0),0 example) and when V00 < 1 the RHSof the above is minimized for non-negative  when  = 0. Using this substitution, theminimizing value of V00 is 1/4 giving
8JD (W) ≥ 1
JD (W) ≥ 1/8.
Combining this with (5) completes the proof.

4.3 More general distributions and implications
In the previous sub-section we analyzed the distribution P over the 2-feature examples(1, 1), 1 and (0, 0), 0. However, these two examples can be embedded in a larger feature spaceby using any fixed vector of additional feature values, creating, for instance, a distribution
over (0, 1, 0, 0, 0, 1/2, 1), 0 and (0, 1, 1, 1, 0, 1/2, 1), 1 (with the additional features underlined)
The results of Section 4.2 still apply to distribution over these extended examples after
defining Wneg network to have zero weight on the additional features, and noticing that any
weight on the additional features in the positive-weight network W can be simulated using
the biases at the hidden nodes.
It is particularly interesting when the additional features all take the value 0, we callthese zero-embeddings. Every network W with non-negative weights has JD (W) ≥ 1/8 oneach of these zero-embeddings of P . On the other hand, a single Wneg network with n/K
copies of the K-input first-one gadget has JD (Wneg ) ≈ 1/10 simultaneously for all of thesezero-embeddings of P (when n >> K).
Any source distribution over {0, 1}K × {0, 1} that puts probability 1/2 on the the 0, 0
example and distributes the other 1/2 probability over examples where exactly two inputs
are one is a mixture of zero-embeddings of P , and thus JD (W) ≥ 1/8 while JD (Wneg ) ≈ 1/10
for this mixture and optimizing the dropout criterion requires negative weights.
In our analysis the negative weights used by dropout are counterintuitive for fitting
monotone behavior, but are needed to control the variance due to dropout. This suggeststhat dropout may be less effective when layers with sparse activation patterns are fed into
wider layers, as dropout training can hijack part of the expressiveness of the wide layer tocontrol the artificial variance due to dropout rather than fitting the underlying patterns in
the data.
15

Helmbold and Long

5. Properties of the dropout penalty
This section examines some properties of the dropout penalty applied to deep networks with
ReLUs and the quadratic loss, which are often different than the case of networks without
hidden layers.
5.1 Growth of the dropout penalty as a function of d
Weight-decay penalizes large weights, while Theorem 5 shows that compensating rescaling of
the weights does not affect the dropout penalty or criterion. On the other hand, dropout canbe more sensitive to the calculation of large outputs than weight decay, and large outputs
can be produced in deep networks using only small weights. We make this observation
concrete by exhibiting a family of networks where the depth and desired output are linked
while the size of individual weights remains constant. For this family, the dropout penalty
grows exponentially in the depth d (as opposed to linearly for weight-decay), suggestingthat dropout training is less willing to fit the data in this kind of situation.
Theorem 18 If x = (1, 1, ..., 1) and 0 ≤ y ≤ Knd−1 , for Px,y there are weights W for the
standard architecture with R(W) = 0 such that (a) every weight has magnitude at most one,2
2/dbut (b) JD (W) ≥ yK , whereas (c) J2 (W) ≤ λy2 (Kn + n2 (d − 2) + n).
1/dProof: Let W be the network whose weights are all c = K 1/dyn(d−1)/d and biases are all 0,
so that the L2 penalty is the number of weights times λc2 /2. It is a simple induction toshow that, for these weights and input (1, 1, ..., 1), the value computed at each hidden nodeon level j is cj Knj−1 , so the the network outputs cd Knd−1 , and has zero square loss (sinceW(x) = cd Knd−1 = y).
Consider now dropout on this network. This is equivalent to changing all of the weightsfrom c to 2c and, independently with probability 1/2, replacing the value of each nodewith 0. For a fixed dropout pattern, each node on a given layer has the same weights, andreceives the same (kept) inputs. Thus, the value computed at every node on the same layer
is the same. For each j, let Hj be the value computed by the units in the jth hidden layer.
If k0 is the number of input nodes kept under dropout, and, for each j ∈ {1, ..., d − 1},
kj is the number of hidden nodes kept in layer j, a straightforward induction shows that,Q
Qd−1dfor all `, we have H` = (2c)` `−1
j=0 kj , so that the output ŷ of the network is (2c)j=0 kj .
Using a bias-variance decomposition,
E((ŷ − y)2 ) = (E[ŷ] − y)2 + Var(ŷ).
Since each kj is binomially distributed, and k0 , ..., kd−1 are independent, we haveE(ŷ) = (2c)d (K/2)(n/2)d−1 = cd Knd−1 = y,so
E((ŷ − y)2 ) = Var(ŷ).
SinceE(ŷ 2 ) = (2c)2d (K(K + 1)/4)(n(n + 1)/4)d−1 = y 2 (1 + 1/K)(1 + 1/n)d−1 ,16Dropout in Deep Networks

we have
Var(ŷ) = E(ŷ 2 ) − E(ŷ)2 = y 2 ((1 + 1/K)(1 + 1/n)d−1 − 1) ≥ y 2 /K,
completing the proof.
Theorem 18 shows that if y = exp(Θ(d)), the dropout penalty grows exponentially ind, whereas the L2 penalty grows polynomially. Since the dropout penalty of a distribution
decomposes into the expectation over single examples, the dropout penalty for any distribution P that puts positive probability on this ((1, 1, . . . , 1), y) example has a term thatgrows exponentially in the depth.
5.2 A necessary condition for negative dropout penalty
Section 2 contains an example where the dropout penalty is negative. The following theorem
provides a necessary condition.
Theorem 19 The dropout penalty can be negative. For all example distributions, a necessary condition for this in rectified linear networks is that either a weight, input, or bias isnegative.
Proof: Baldi and Sadowski (2014) show that for networks of linear units (as opposed to thenon-linear rectified linear units we focus on) the network’s output without dropout equals
the expected output over dropout patterns, so in our notation: W(x) equals ER (D(W, x, R)).
Assume for the moment that the network consists of linear units and the example distribution is concentrated on the single example (x, y). Using the bias-variance decomposition
for square loss and this property of linear networks,

JD (W) = ER (D(W, x, R)−y)2 = (ER (D(W, x, R)−y)2 +VarR (D(W, x, R)) ≥ (W(x)−y)2
and the dropout penalty is again non-negative. Since the same calculations go throughwhen averaging over multiple examples, we see that the dropout penalty is always nonnegative for networks of linear nodes. When all the weights, biases and inputs in a networkof rectified linear units are positive, then the rectified linear units behave as linear units, so
the dropout penalty will again be non-negative.
5.3 Multi-layer dropout penalty does depend on labels
In contrast with its behavior on a variety of linear models including logistic regression(Wager et al., 2013), the dropout penalty can depend on the value of the response variable
in deep networks with ReLUs and the quadratic loss. Thus in a fundamental and important
respect, dropout differs from traditional regularizers like weight-decay or an L1 penalty.
Theorem 20 There are joint distributions P and Q, and weights W such that, for all
dropout probabilities q ∈ (0, 1), (a) the marginals of P and Q on the input variables areequal, but (b) the dropout penalties of W with respect to P and Q are different.
17Helmbold and Long

We will prove Theorem 20 by describing a general, somewhat technical, condition that
implies that P and Q are witnesses to Theorem 20.
For each input x and dropout pattern R, let H(W, x, R) be the values presented to the
output node with dropout. As before, let w ∈ Rn be those weights of W on connections
directly into the output node and let b be the bias at the output node. Let r ∈ {0, 1}nbe the indicator variables for whether the various nodes connecting to the output node are
kept.
Proof (of Theorem 20): Suppose that P is concentrated on a single (x, y) pair. We willthen get Q by modifying y.
Let h be the values coming into the output node in the non-dropped out network.
Therefore the output of the non-dropout network is w · h + b while the output of the
network with dropout is w · H(W, x, R) + b. We now examine the dropout penalty, whichis the expected dropout loss minus the non-dropout loss. We will use δ as a shorthand forw · (H(W, x, R) − h).
dropout penalty = E (w · H(W, x, R) + b − y)2 − (w · h + b − y)2= E (w · H(W, x, R) + b − w · h + w · h − y)2 − (w · h + b − y)2= E δ 2 + 2(w · h + b − y)E(δ)
which depends on the label y unless E(δ) = 0.
Typically E(δ) 6= 0. To prove the theorem, consider the case where• there are two inputs and one hidden layer,• x = (1, −2),• all weights in the network are 1, and• all biases are 0.
Without dropout h = (0, 0), but with dropout the hidden nodes are never negative andcompute positive values when only the negative input is dropped, so that the expectationof δ is positive.

6. Experiments
To complement our theoretical results we performed two sets of experiments. The first set
tests the scale dependence of dropout and weight decay, while the the second set examines
its promotion of negative weights even when learning monotone functions. The code isaccessible athttps://www.dropbox.com/sh/6s2lcfrq17zshmp/AAAQ06uDa4gOAuAnw2MAghEMa?dl=0
6.1 Scale (in)sensitivity
The scale dependence simulations were implemented using Torch. Our experiment used
the standard architecture with K = 5 inputs, depth d = 2, n = 5 hidden nodes. We used18

Dropout in Deep Networks

1.1

Dropout
Weight Decay

1

Training error

0.9

0.8

0.7

0.6

0.5
0.4

0.6

0.8

1
Input Scale

1.2

1.4

1.6

Figure 2: Training error as a function of the scale of the inputs for Dropout and Weight
Decay in the experiment of Section 3.

0.01
stochastic gradient using the optim package for Torch, with learning rate 1+0.00001t
andmomentum of 0.5, and a maximum of 100000 iterations.
We performed 10 sets of training runs. In each run:

• Ten training examples were generated uniformly at random from [−1, 1]K .
Q• Target outputs were assigned using y = i sign(xi ).
• Five training sets S1 , ..., S5 with ten examples each were obtained by rescaling the
inputs by {0.5, 0.75, 1, 1.25, 1.5} and leaving the outputs unchanged.
• The weights of a network Winit were initialized using the default initialization from
Torch.
• For each Si :– Winit was cloned three times to produce WD , W2 and Wnone with identical starting parameters.
– WD was trained with dropout probability 1/2 and no weight decay.
– W2 was trained with weight decay with λ = 1/2 and no dropout.
– Wnone was trained without any regularization.
The average training losses of WD and W2 , over the 10 runs, are shown in Figure 2.
(The average training loss of Wnone was less than 0.05 at all scales.)The theoretical insensitivity of dropout to the scale of the inputs described in Theorem 4
is also seen here, along with the contrast with weight decay analyzed in Theorem 7.
The scale of the inputs also affects the dynamics of stochastic gradient descent. With
very small inputs, convergence is very slow, and with very large inputs, SGD is unstable.
The effects of the scale of the inputs on inductive bias analyzed in this paper are visible atthe scales where optimization can be done effectively.
19

Helmbold and Long

Table 1: Summary of weights resulting from noisy “all same” initialization
weights
num below −1.0
−0.1 −0.001
0 0.001 0.1 above
from
neg
−1.0
to
to
to
to
to
to
1.0
−0.1 −0.001
0 0.001
0.1 1.0
initial wts
76
0
0
75
1
1
166 5970weight-decay
7
0
0
0
7
1
276 5560dropout
82
0
70
12
0
0
146 6120
6.2 Negative Weights
Here we demonstrate dropout’s propensity to use negative weights even when the function
to be learned is monotonic. These experiments were implemented with Keras on top of
TensorFlow and using SGD optimization with a learning rate of 0.005. Weight decay learning used a parameter of 0.05, and dropout training used a dropout rate of 0.5. We used thestandard architecture with six inputs, 12 hidden nodes, and one output. The training set
consists of six inputs as follows:
inputs(0,0,0,0,0,0)(1,1,0,0,0,0)(0,0,1,1,0,0)(0,0,0,0,1,1)

label
0
1
1
1

multiplicity
×3
×1
×1
×1

p
A natural network for this data uses all weights set to the same value w = 1/24 (withthe biases set to 0). This network computes the correct labels for all of the training inputs.
We ran each of dropout and weight-decay for 10,000 epochs from 10 different initial weight
settings created by by sampling the weights independently from the uniform distribution
over [−0.2w, 2.2w]. This randomization gives a one-in-twelve chance of changing the weight’s
sign.
Table 1 summarizes the initial weights, weights learned by weight-decay, and weights
learned by dropout aggregated over the 10 initializations. This table shows that dropout not
only (slightly) increases the number of negative weights, but also tends to greatly increase
their magnitude. As expected, weight-decay eliminates almost all negative weights, and the
few remaining negative weights have very small magnitudes.
We also ran a similar experiment based on the “first-one” gadget construction of Section 4.1. The hidden nodes are organized into six groups of two, with each group implementing three first-one gadgets – one for each of the input “pairs”. Now the uncorrupted
weights into the hidden layer are 0, 1, or −1, and those at the output node are 1/6 sothat the network computes the labels in the training set. As before, 10 noisy initial weight
settings are chosen by selecting each weight uniformly from [−0.2w, 2.2w], where w is thecorresponding uncorrupted weight.
Table 2 summarizes the initial weights, weights learned by weight-decay, and weights
learned by dropout aggregated over the 10 initializations. From this initialization, dropout
tends to preserve the magnitudes of the negative weights. Although dropout increases thenumber of small-magnitude negative weights, much of this increase can be explained by20

Dropout in Deep Networks

Table 2: Summary of weights resulting from noisy “first-one” initialization
weights
num below −1.0
−0.1 −0.001
0 0.001 0.1 above
from
neg
−1.0
to
to
to
to
to
to
1.0
−0.1 −0.001
0 0.001
0.1 1.0
initial wts
204
90
80
34
180
1
47 232176
weight-decay 111
0
0
57
68
68
298 3490dropout
303
91
93
119
5
0
168 189175

half of the 180 weights initialized to exactly zero becoming negative. Again, weight-decay
dramatically decreases both the number of the negative weights and their magnitudes.

7. Conclusions
The reasons behind dropout’s surprisingly good performance in training deep networks
across a variety of applications are somewhat mysterious and there is relatively little existing
formal analysis. A variety of explanations have been offered (Bachman et al., 2014; Baldi
and Sadowski, 2014, 2013; Gal and Ghahramani, 2015; Wager et al., 2014), including thepossibility that dropout reduces the amount of coadaptation in a network’s weights (Hinton
et al., 2012).
The dropout criterion is an expected loss over dropout patterns, and the variance in the
output values over dropout patterns contributes to this expected loss. Therefore dropout
may co-adapt weights in order to reduce this (artificial) variance. We prove that thishappens even in very simple situations where nothing in the training data justifies negative
weights (Theorem 8). This indicates that the relationship between dropout and co-adaption
is not a simple one.
The effects of dropout in deep neural networks are rather complicated, and approximations can be misleading since the dropout penalty is very non-convex even in 1-layer
networks (Helmbold and Long, 2015). In Section 3 we show that dropout does enjoy several scale-invariance properties that are not shared by weight-decay. A perhaps surprisingconsequence of these invariances is that there are never isolated local minima when learninga deep network with dropout. Further exploration of these scale invariance properties iswarranted to see if they are a contributor to dropout’s empirical success or can be exploited
to facilitate training. While contrasting dropout to weight-decay in simple situations, wefound that a degenerate all-zero network results (Theorem 7) when the L2 regularization
parameter is above a threshold. This is in dramatic contrast to our previous intuition fromthe 1-layer case.
In (Wager et al., 2013), dropout was viewed as a regularization method, adding a data
dependent penalty to the empirical loss of (presumably) undesirable solutions. Section 5shows that, unlike the generalized linear models case analyzed there, the dropout penalty
in deeper networks can be negative and depends on the labels in the training data, andthus behaves unlike most regularizers. On the other hand, the dropout penalty can growexponentially in the depth of the network, and thus may better reflect the complexity of
the underlying model space than L2 regularization.
21

Helmbold and Long

This paper uncovers a number of dropout’s interesting fundamental properties using
formal analysis of simple cases. However, the effects of using dropout training in deep
networks are subtle and complex, and we hope that this paper lays a foundation to promotefurther formal analysis of dropout’s properties and behavior.

Acknowledgments
We are very grateful to Peter Bartlett, Seshadhri Comandur, and anonymous reviewers forvaluable communications.

Appendix A. Table of Notation
Notation
1set(x, y)
σ(·)
W
w, v
W(x)
P
Px,y
RP (W)q, pRr, sD(W, x, R)
JD (W)J2 (W)
λ
WD
WL2
n, d
K

Meaning
indicator function for “set”
an example with feature vector x and label y
the rectified linear unit computing max(0, ·)an arbitrary weight setting for the network
specific weights, often subscripted
the output value produced by weight setting W on input x
an arrelbitrary source distribution over (x, y) pairs
the source distribution concentrated on the single example (x, y)
the risk (expected square loss) of W under source P
probabilities that a node is dropped out (q) or kept (p) by the dropout processa dropout pattern, indicates the kept nodes
dropout patterns on subsets of the nodes
Output of dropout with network weights W, input x, and dropout pattern R
the dropout criterion
the L2 criterion
the L2 regularization strength parameteran optimizer of the dropout criterion
an optimizer of the L2 criterion
the network width and depth
the number of input nodes

Appendix B. Proof of Theorem 7
Here we prove Theorem 7, showing that the weight-decay aversion depends on the valuesof the inputs and the number of input nodes K. Furthermore, unlike the single-layer case,the L2 regularization strength has a threshold where the minimizer of the L2 criterion
degenerates to the all-zero network.
We will focus on the standard architecture with depth d = 2. Recall that we are
analyzing the distribution P(x,1) that assigns probability 1/2 to (x, 1) and probability 1/2
to (0, 0).
22

Dropout in Deep Networks

Also, recall that, for P(x,1) , the weight-decay aversion is the maximum risk incurred bya minimizer of J2 .
The proof of Theorem 7 involves a series of lemmas, in which we show that there is aminimizer of J2 of a special form, and then relate any hidden node’s effect on the output
to the regularization penalty on the weights in and out of that node. This will allow us totreat optimizing the L2 criterion as a one-dimensional problem, whose solution yields thetheorem.
For some minimizer WL2 of J2 , let vj∗ denote the vector of weights into hidden node j
and wj∗ denote the weight from j to the output node. Let a∗j be the bias for hidden nodej and let b∗ be the bias for the output node. Let hj be the function computed by hidden
node j.
Lemma 21 We may assume without loss of generality that for each hidden node j, there
is an input x̃ ∈ {0, x} such that hj (x̃) = 0.
Proof: Suppose neither of hj (0) or hj (x) was 0. If wj∗ = 0, then replacing both with 0 does
not affect the output of WL2 , and does not increase the penalty. If wj∗ 6= 0, then subtracting
min{hj (0), hj (x)} from a∗j and adding wj∗ · min{hj (0), hj (x)} to b does not affect the output
of WL2 or the penalty, but, after this transformation, min{hj (0), hj (x)} = 0.
Lemma 22 We may assume without loss of generality that for each hidden node j, we have|vj∗ · x| = max{hj (0), hj (x)}.
Proof: If hj (x) ≥ hj (0) = 0, then bias a∗j = 0, and hj (x) = vj∗ · x.
If hj (0) > hj (x) = 0. Then, a∗j > 0, and vj∗ · x ≤ −a∗j . If needed, the magnitude of vj∗
can be decreased to make vj∗ · x = −a∗j . This decrease does not affect WL2 (x) or WL2 (0),and can only reduce the L2 penalty.
Lemma 23 We may assume without loss of generality that for each hidden node j, we havehj (0) = 0.
Proof: Suppose hj (0) > 0. Let z be this old value of hj (0). Then hj (x) = 0 andz = hj (0) = −vj∗ · x. If we negate v∗ and set aj = 0, then Lemma 22 implies that we swap
the values of hj (x) and hj (0).
Then, by adding zwj∗ to b∗ and negating wj∗ , we correct for this swap at the output nodeand do not affect the function computed by WL2 or the penalty.
Note that Lemma 23 implies that, without loss of generality, a∗1 = ... = a∗K = 0. We
continue with that assumption.
Lemma 24 For all j, vj∗ · x ≥ 0.
Proof: Since a∗j = 0, if vj∗ · x < 0, we could make WL2 compute the same function with asmaller penalty by replacing vj∗ with 0.
Lemma 24 implies that the optimal WL2 computes the linear function:
WL2 (x̃) = (w∗ )T V ∗ x̃ + b∗ ,23

Helmbold and Long

when x̃ is a non-negative multiple of x. Later we will call (w∗ )T V ∗ x̃ the activation at theoutput node.
∗ T∗

Lemma 25 b∗ = 1−(w 2) V x .
Proof: Minimize J2 (wrt distribution P(x,1) ) as a function of b using Calculus.
Now, for x̃ a non-negative multiple of x,
WL2 (x̃) =1+ (w∗ )T V ∗ (x̃ − x/2)
2(6)which immediately implies
WL2 (0) = 1 − WL2 (x).

(7)

Lemma 26 Each vj∗ is a (positive) rescaling of x.
Proof: Projecting vj∗ onto the span of x does not affect hj , and cannot increase the penalty.

Lemma 27 WL2 (x) ≤ 1.
Proof: By (7), if WL2 (x) > 1 then WL2 (0) < 0 and the loss and the penalty would bothbe reduced by scaling down w∗ .
Lemma 28 WL2 maximizes W(x) over those weight vectors W that have the same penalty
as WL2 and compute a function of the form W(x̃) = 12 + (w)T V (x̃ − x/2) (that is, Equation 6).
Proof: Let W maximize W(x) over the networks considered. If WL2 (x) < W(x) ≤ 1, thenW would have the same penalty as WL2 but smaller error, contradicting the optimality ofWL2 .
f obtained by scaling down the weights in the output
If W(x) > 1, then the network W
flayer so that W(x)= 1 has a smaller penalty than WL2 and smaller error, again contradicting WL2 ’s optimality.
Informally, Lemmas 27 and 28 engender a view of the learner straining against the yolkof the L2 penalty to produce a large enough output on x. This motivates us to ask how
large W(x) can be, for a given value of ||W||22 (recall that Lemma 24 allows us to assume
that the biases at the hidden nodes are all 0).
Definition 29 For each hidden node j, let αj be the constant such vj∗ = αj x, so thathj (x) = αj x · x.
Recall that the activation at the output node on input x is the weighted sum of thehidden-node outputs, (w∗ )T V ∗ x.
24

Dropout in Deep Networks

node j is
Definition 30 The contribution to the activation at the output due to hidden λ∗∗wj hj (x) = wj αj x·x and the contribution to the L2 penalty from these weights is 2 (wj∗ )2 + αj2 x · x .
We now bound the contribution to the activation in terms of the contribution to theL2 penalty. Note that as the L2 “budget” increases, so does the the maximum possible
contribution to the output node’s activation.
Lemma 31 If B is hidden node j’s weight-decay contribution, (wj∗ )2 + αj2 x · x, then hiddenqB∗
node j’s contribution to the output node’s activation is maximized when wj =
2 and
q
√
B
αj = 2x·x, where it achieves the value B x · x/2
q
Proof: Since αj2 x · x + (wj∗ )2 = B, we have wj∗ = B − αj2 x · x, so the contribution toq
the activation can be re-written as αj x · x B − αj2 x · x. Taking the derivative with respectq
B
and we want the positive solution (otherwise theto αj , and solving, we get αj = ± 2x·x
q
q
B
B
∗
node outputs 0). When αj =2x·x we have wj =2 and thus the node’s maximum
contribution to the activation is
r r
√
B
B
B x·x
(x · x) =.
2 2x · x2

Lemma 32 The minimum sum-squared weights for a network W (without biases at thehidden nodes) that has an activation A at the output node on input x is √2A.
x·x
Proof: When maximized, the contribution of each hidden node to the activation at the√output is x · x/2 times the hidden node’s contribution to the sum of squared-weights.
Since each weight in W is used in exactly one hidden node’s contribution to the output
node’s activation, this completes the proof.
Note that this bound is independent of n, the number of hidden units, but does dependon the input x.
Proof (of Theorem 7): Let A ≥ 0 be the activation at the output node for WL2 on
input x. From Lemma 25 we get that b∗ = 1−A
2 . Combining Lemmas 27 , 28 and 32, we
can re-write the J2 criterion for WL2 and distribution P(x,1) in terms of A as follows.
1 ∗1
λ(b − 0)2 + (WL2 (x) − 1)2 + ||W||22
2
2
2
!

2 
2
1
1−A
1+A
2A
=
+
− 1 + λ√
.
2
2
2
x·x

J2 (WL2 ) =

25

(8)

Helmbold and Long

Differentiating with respect to A, we see that the criterion is minimized when
A=1− √

2λ
x·x

2λ
≤1
x·x
2λ
when √>1x·xwhen √

A=0

since we assumed A ≥ 0; when A = 0 then WL2 has all zero weights with a bias of 1/2 atthe output.
The risk part of (8) simplifies to(1 − A)2
λ2
=
,
4
x·x

so the overall the risk of WL2 , which minimizes J2 , is min


1 λ2
,
.
4 x·x

References
P. Bachman, O. Alsharif, and D. Precup. Learning with pseudo-ensembles. NIPS, 2014.
P. Baldi and P. Sadowski. The dropout learning algorithm. Artificial intelligence, 210:
78–122, 2014.
Pierre Baldi and Peter J Sadowski. Understanding dropout. In Advances in Neural Information Processing Systems, pages 2814–2822, 2013.
P. L. Bartlett, M. I. Jordan, and J. D. McAuliffe. Convexity, classification, and risk bounds.
Journal of the American Statistical Association, 101(473):138–156, 2006.
L. Breiman. Some infinity theory for predictor ensembles. Annals of Statistics, 32(1):1–11,
2004.
Caffe. Caffe, 2016. http://caffe.berkeleyvision.edu.
Danqi Chen and Christopher D Manning. A fast and accurate dependency parser usingneural networks. In EMNLP, pages 740–750, 2014.
G. E. Dahl.
Deep learning how I did it:
http://blog.kaggle.com.

Merck 1st place interview, 2012.

G. E. Dahl, T. N. Sainath, and G. E. Hinton. Improving deep neural networks for LVCSR
using rectified linear units and dropout. ICASSP, 2013.
L. Deng, J. Li, J. Huang, K. Yao, D. Yu, F. Seide, M. L. Seltzer, G. Zweig, X. He,
J. Williams, Y. Gong, and A. Acero. Recent advances in deep learning for speech research at microsoft. ICASSP, 2013.
Yarin Gal and Zoubin Ghahramani. Dropout as a Bayesian approximation: Representing
model uncertainty in deep learning. arXiv:1506.02142, 2015.
26

Dropout in Deep Networks

Yarin Gal and Zoubin Ghahramani. A theoretically grounded application of dropout inrecurrent neural networks. In Advances in Neural Information Processing Systems, pages
1019–1027, 2016.
Ian J. Goodfellow, David Warde-Farley, Mehdi Mirza, Aaron C. Courville, and Yoshua
Bengio. Maxout networks. In ICML, pages 1319–1327, 2013.
Mohammad Havaei, Axel Davy, David Warde-Farley, Antoine Biard, Aaron Courville,
Yoshua Bengio, Chris Pal, Pierre-Marc Jodoin, and Hugo Larochelle. Brain tumor segmentation with deep neural networks. Medical image analysis, 35:18–31, 2017.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Delving deep into rectifiers:
Surpassing human-level performance on imagenet classification. In ICCV, pages 1026–
1034, 2015.
D. P. Helmbold and P. M. Long. On the inductive bias of dropout. JMLR, 16:3403–3454,
2015.
G. E. Hinton. Dropout: a simple and effective way to improve neural networks, 2012.
videolectures.net.
G. E. Hinton, N. Srivastava, A. Krizhevsky, I. Sutskever, and R. R. Salakhutdinov. Improving neural networks by preventing co-adaptation of feature detectors, 2012. Arxiv,
arXiv:1207.0580v1.
Nal Kalchbrenner, Edward Grefenstette, and Phil Blunsom. A convolutional neural network
for modelling sentences. In ACL, pages 655–665, 2014.
P. M. Long and R. A. Servedio. Random classification noise defeats all convex potential
boosters. Machine Learning, 78(3):287–304, 2010.
Vinod Nair and Geoffrey E Hinton. Rectified linear units improve restricted boltzmann
machines. In ICML, pages 807–814, 2010.
Behnam Neyshabur, Ryota Tomioka, and Nathan Srebro. Norm-based capacity control inneural networks. In COLT, pages 1376–1401, 2015.
Jürgen Schmidhuber. Deep learning in neural networks: An overview. Neural Networks,
61:85–117, 2015.
N. Srivastava, G. Hinton, A. Krizhevsky, I. Sutskever, and R. Salakhutdinov. Dropout: A
simple way to prevent neural networks from overfitting. JMLR, 15:1929–1958, 2014.
Christian Szegedy, Wei Liu, Yangqing Jia, Pierre Sermanet, Scott Reed, Dragomir
Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper
with convolutions. CVPR, 2015.
TensorFlow. Tensorflow, 2016. https://www.tensorflow.org.
Torch. Torch, 2016. http://torch.ch.
27

Helmbold and Long

T. Van Erven, W. Kotowski, and M. K. Warmuth. Follow the leader with dropout perturbations. COLT, pages 949–974, 2014.
S. Wager, S. Wang, and P. Liang. Dropout training as adaptive regularization. NIPS, 2013.
S. Wager, W. Fithian, S. Wang, and P. S. Liang. Altitude training: Strong bounds for
single-layer dropout. NIPS, 2014.
L. Wan, M. Zeiler, S. Zhang, Y. Le Cun, and R. Fergus. Regularization of neural networks
using dropconnect. In ICML, pages 1058–1066, 2013.
Zichao Yang, Xiaodong He, Jianfeng Gao, Li Deng, and Alex Smola. Stacked attention networks for image question answering. In Proceedings of the IEEE Conference on Computer
Vision and Pattern Recognition, pages 21–29, 2016.
T. Zhang. Statistical behavior and consistency of classification methods based on convex
risk minimization. Annals of Statistics, 32(1):56–85, 2004.

28