"6.3 Hidden Units
So far we have focused our discussion on design choices for neural networks that
are common to most parametric machine learning models trained with gradientbased optimization. Now we turn to an issue that is unique to feedforward neural
networks: how to choose the type of hidden unit to use in the hidden layers of the
model.
The design of hidden units is an extremely active area of research and does not
yet have many definitive guiding theoretical principles.
Rectified linear units are an excellent default choice of hidden unit. Many other
types of hidden units are available. It can be difficult to determine when to use
which kind (though rectified linear units are usually an acceptable choice). We
191
CHAPTER 6. DEEP FEEDFORWARD NETWORKS
describe here some of the basic intuitions motivating each type of hidden units.
These intuitions can help decide when to try out each of these units. It is usually
impossible to predict in advance which will work best. The design process consists
of trial and error, intuiting that a kind of hidden unit may work well, and then
training a network with that kind of hidden unit and evaluating its performance
on a validation set.
Some of the hidden units included in this list are not actually differentiable at
all input points. For example, the rectified linear function g (z) = max{0, z} is not
differentiable at z = 0. This may seem like it invalidates g for use with a gradientbased learning algorithm. In practice, gradient descent still performs well enough
for these models to be used for machine learning tasks. This is in part because
neural network training algorithms do not usually arrive at a local minimum of
the cost function, but instead merely reduce its value significantly, as shown in
figure 4.3. These ideas will be described further in chapter 8. Because we do not
expect training to actually reach a point where the gradient is 0, it is acceptable
for the minima of the cost function to correspond to points with undefined gradient.
Hidden units that are not differentiable are usually non-differentiable at only a
small number of points. In general, a function g(z) has a left derivative defined
by the slope of the function immediately to the left of z and a right derivative
defined by the slope of the function immediately to the right of z. A function
is differentiable at z only if both the left derivative and the right derivative are
defined and equal to each other. The functions used in the context of neural
networks usually have defined left derivatives and defined right derivatives. In the
case of g(z) = max{0, z}, the left derivative at z = 0 is 0 and the right derivative
is 1. Software implementations of neural network training usually return one of
the one-sided derivatives rather than reporting that the derivative is undefined or
raising an error. This may be heuristically justified by observing that gradientbased optimization on a digital computer is subject to numerical error anyway.
When a function is asked to evaluate g(0), it is very unlikely that the underlying
value truly was 0. Instead, it was likely to be some small value  that was rounded
to 0. In some contexts, more theoretically pleasing justifications are available, but
these usually do not apply to neural network training. The important point is that
in practice one can safely disregard the non-differentiability of the hidden unit
activation functions described below.
Unless indicated otherwise, most hidden units can be described as accepting
a vector of inputs x, computing an affine transformation z = W x + b, and
then applying an element-wise nonlinear function g(z). Most hidden units are
distinguished from each other only by the choice of the form of the activation
function g(z).
192
CHAPTER 6. DEEP FEEDFORWARD NETWORKS
6.3.1 Rectified Linear Units and Their Generalizations
Rectified linear units use the activation function g(z) = max{0, z}.
Rectified linear units are easy to optimize because they are so similar to linear
units. The only difference between a linear unit and a rectified linear unit is
that a rectified linear unit outputs zero across half its domain. This makes the
derivatives through a rectified linear unit remain large whenever the unit is active.
The gradients are not only large but also consistent. The second derivative of the
rectifying operation is 0 almost everywhere, and the derivative of the rectifying
operation is 1 everywhere that the unit is active. This means that the gradient
direction is far more useful for learning than it would be with activation functions
that introduce second-order effects.
Rectified linear units are typically used on top of an affine transformation:
h = g(Wx + b). (6.36)
When initializing the parameters of the affine transformation, it can be a good
practice to set all elements of b to a small, positive value, such as 0.1. This makes
it very likely that the rectified linear units will be initially active for most inputs
in the training set and allow the derivatives to pass through.
Several generalizations of rectified linear units exist. Most of these generalizations perform comparably to rectified linear units and occasionally perform
better.
One drawback to rectified linear units is that they cannot learn via gradientbased methods on examples for which their activation is zero. A variety of
generalizations of rectified linear units guarantee that they receive gradient everywhere.
Three generalizations of rectified linear units are based on using a non-zero
slope αi when zi < 0: hi = g(z, α)i = max(0, zi) + αi min(0, zi). Absolute value
rectification fixes αi = −1 to obtain g(z) = |z|. It is used for object recognition
from images (Jarrett et al., 2009), where it makes sense to seek features that are
invariant under a polarity reversal of the input illumination. Other generalizations
of rectified linear units are more broadly applicable. A leaky ReLU (Maas et al.,
2013) fixes αi to a small value like 0.01 while a parametric ReLU or PReLU
treats αi as a learnable parameter (He et al., 2015).
Maxout units (Goodfellow et al., 2013a) generalize rectified linear units
further. Instead of applying an element-wise function g(z ), maxout units divide z
into groups of k values. Each maxout unit then outputs the maximum element of
193
CHAPTER 6. DEEP FEEDFORWARD NETWORKS
one of these groups:
g(z)i = max
j∈G(i)
zj (6.37)
where G(i) is the set of indices into the inputs for group i, {(i − 1)k + 1, . . . ,ik}.
This provides a way of learning a piecewise linear function that responds to multiple
directions in the input x space.
A maxout unit can learn a piecewise linear, convex function with up to k pieces.
Maxout units can thus be seen as learning the activation function itself rather
than just the relationship between units. With large enough k, a maxout unit can
learn to approximate any convex function with arbitrary fidelity. In particular,
a maxout layer with two pieces can learn to implement the same function of the
input x as a traditional layer using the rectified linear activation function, absolute
value rectification function, or the leaky or parametric ReLU, or can learn to
implement a totally different function altogether. The maxout layer will of course
be parametrized differently from any of these other layer types, so the learning
dynamics will be different even in the cases where maxout learns to implement the
same function of x as one of the other layer types.
Each maxout unit is now parametrized by k weight vectors instead of just one,
so maxout units typically need more regularization than rectified linear units. They
can work well without regularization if the training set is large and the number of
pieces per unit is kept low (Cai et al., 2013).
Maxout units have a few other benefits. In some cases, one can gain some statistical and computational advantages by requiring fewer parameters. Specifically,
if the features captured by n different linear filters can be summarized without
losing information by taking the max over each group of k features, then the next
layer can get by with k times fewer weights.
Because each unit is driven by multiple filters, maxout units have some redundancy that helps them to resist a phenomenon called catastrophic forgetting
in which neural networks forget how to perform tasks that they were trained on in
the past (Goodfellow et al., 2014a).
Rectified linear units and all of these generalizations of them are based on the
principle that models are easier to optimize if their behavior is closer to linear.
This same general principle of using linear behavior to obtain easier optimization
also applies in other contexts besides deep linear networks. Recurrent networks can
learn from sequences and produce a sequence of states and outputs. When training
them, one needs to propagate information through several time steps, which is much
easier when some linear computations (with some directional derivatives being of
magnitude near 1) are involved. One of the best-performing recurrent network
194
CHAPTER 6. DEEP FEEDFORWARD NETWORKS
architectures, the LSTM, propagates information through time via summation—a
particular straightforward kind of such linear activation. This is discussed further
in section"
Summarize the above basic principle of ReLU actication function for neural networks with equations, try to use your own words, do not directly copy the sentence from the above paper.