Title: MOLLIFYING NETWORKS

Abstract: The optimization of deep neural networks can be more challenging than the traditional convex optimization problems due to highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape from for algorithms based on simple gradient descent. In this paper, we attack the problem of optimization of highly non-convex neural networks objectives by starting with a smoothed -or mollified -objective function which becomes more complex as the training proceeds. Our proposition is inspired by the recent studies in continuation methods: similarly to curriculum methods, we begin by learning an easier (possibly convex) objective function and let it evolve during training until it eventually becomes the original, difficult to optimize objective function. The complexity of the mollified networks is controlled by a single hyperparameter that is annealed during training. We show improvements on various difficult optimization tasks and establish a relationship between recent works on continuation methods for neural networks and mollifiers.

1. INTRODUCTION:

In the last few years, deep neural networks -i.e. convolutional networks (LeCun et al., 1989) , LSTMs (Hochreiter & Schmidhuber, 1997a) or GRUs (Cho et al., 2014) -set the state of the art on a range of challenging tasks (Szegedy et al., 2014; Visin et al., 2015; Hinton et al., 2012; Sutskever et al., 2014; Bahdanau et al., 2014; Mnih et al., 2013; Silver et al., 2016) . However when trained with variants of SGD (Bottou, 1998) deep networks can be difficult to optimize due to their highly non-linear and non-convex nature (Choromanska et al., 2014; Dauphin et al., 2014) .

A number of approaches were proposed to alleviate the difficulty of optimization: addressing the problem of the internal covariate shift with Batch Normalization (Ioffe & Szegedy, 2015) , learning with a curriculum (Bengio et al., 2009) , recently an approach to train RNNs with diffusion process (Mobahi, 2016) , and graduated optimization (Hazan et al., 2015) . The impact of noise injection on the behavior of modern deep learning methods has been explored by Neelakantan et al. (2015a) . Hazan et al. (2015) have shown that injecting a particular noise and scheduling it carefully, can guarantee the convergence in O(1/σ 2 2 ) steps for -optimal and σ-nice functions. Similar to our work graduated optimization optimizes a smoothed objective function without performing expensive convolutions. Injecting noise to the activation functions and scheduling it have been recently shown to improve the performance on a wide variety of tasks (Gulcehre et al., 2016) .

We connect the ideas of curriculum learning and continuation methods with those arising from models with skip connections and using layers that compute near-identity transformations. Skip connections allow to train very deep residual and highway architectures (He et al., 2015; Srivastava et al., 2015) by skipping layers or block of layers. Similarly, it has been shown that stochastically changing the depth of a network during training (Huang et al., 2016b) does not prevent convergence and enables better generalization performance.

We discuss the idea of mollification for neural networks -a form of differentiable smoothing of the loss function connected to noisy activations -which in our case can be interpreted as a form of adaptive noise injection which is controlled by a single hyperparameter. Inspired by Huang et al. (2016b) , we use a hyperparameter to stochastically control the depth of our network. This allows us to start the optimization from a convex objective function (as long as the optimized criterion is convex, e.g. linear or logistic regression) and to slowly introduce more complexity into the model by annealing the hyperparameter, thus making the network deeper and increasingly non-linear. An important difference of our work compared to injecting noise to the gradients as it is explored in (Hazan et al., 2015; Neelakantan et al., 2015b ) is that we inject the noise in the forward computation of the graph and we shape the cost function directly. As a result the cost function for the mollified network both at the test time and during the training are consistent and this makes the early-stopping much easier.

2. MOLLIFYING OBJECTIVE FUNCTIONS:

2.1 CONTINUATION AND ANNEALING METHODS Continuation methods and simulated annealing provide a general strategy to reduce the impact of local minima and deal with non-convex, continuous, but not necessarily everywhere differentiable objective functions by smoothing the original objective function and gradually reducing the amount of smoothing during training (Allgower & Georg, 1980 ) (see Fig. 1 ).

In machine learning, approaches based on curriculum learning (Bengio et al., 2009) are inspired by this principle and define a sequence of gradually more difficult training tasks (or training distributions) that eventually converge to the task of interest.

In the context of stochastic gradient descent, we use a stochastic estimation of the gradient for the smoothed objective function. This is convenient because it may not be analytically feasible to compute the smoothed function, but a Monte-Carlo estimate can often be obtained easily.

In this paper we construct a sequence of smoothed objective functions obtained with a form of mollification and we progressively optimize them. The training procedure iterates over the sequence of objective functions starting from the simpler ones -i.e. with a smoother loss surface -and moving towards more complex ones until the last, original, objective function is reached.1 

2.2. MOLLIFIERS AND WEAK GRADIENTS:

We smooth the loss function L, which is parametrized by θ ∈ R n , by convolving it with another function K(•) with stride τ ∈ R n :

EQUATION

Although there are many choices for the function K(•), we focus on those that satisfy the definition of a mollifier.

A mollifier is an infinitely differentiable function that behaves like an approximate identity in the group of convolutions of integrable functions. If K(•) is an infinitely differentiable function, that converges to the Dirac delta function when appropriately rescaled and for any integrable function L, then it is a mollifier:

EQUATION

If we choose K(•) to be a mollifier and obtain the smoothed loss function L K as in Eqn. 1, we can take its gradient with respect to θ using directly the result from Evans (1998):

EQUATION

To relate the resulting gradient ∇ θ L K to the gradient of the original function L, we introduce the notion of weak gradient, i.e. an extension to the idea of weak/distributional derivatives to functions with multidimensional arguments, such as loss functions of neural networks.

For an integrable function

EQUATION

where K(τ ) is an infinitely differentiable function vanishing at infinity, C ∈ [a, b] n and τ ∈ R n .

As long as the chosen K(•) fulfills the definition of a mollifier we can use Eqn. 3 and Eqn. 42 to rewrite the gradient as:

∇ θ L K (θ) = (L * ∇K)(θ) by Eqn. 3 (5) = C L(θ -τ )∇K(τ )dτ (6) = - C g(θ -τ )K(τ )dτ by Eqn. 4 (7)

For a differentiable almost everywhere function L, the weak gradient g(θ) is equal to ∇ θ L almost everywhere. With a slight abuse of notation we can therefore write:

EQUATION

2.3. GAUSSIAN MOLLIFIERS:

It is possible to use the standard Gaussian distribution N (0, I) as a mollifier K(•), as it satisfies the desired properties: it is infinitely differentiable, a sequence of properly rescaled Gaussian distributions converges to the Dirac delta function and it vanishes in infinity. With such a K(•) the gradient becomes:

EQUATION

Exploiting the fact that a Gaussian distribution is a mollifier, we can focus on a sequence of mollifications indexed by scaling parameter introduced in Eqn. 2. A single element of this sequence takes the following form:

EQUATION

Replacing with σ yields a sequence of mollifications indexed by σ:

EQUATION

with the following property (by Eqn. 2):

EQUATION

An intuitive interpretation of the result is that σ determines the standard deviation of a mollifying Gaussian and is annealed in order to construct a sequence of gradually less "blurred" and closer approximations to L. This is consistent with the property that when σ is annealed to zero we are optimizing the original function L.

So far we obtained the mollified version L K (θ) of the cost function L(θ) by convolving it with a mollifier K(θ). The kernel K(θ) corresponds to the average effect of injecting noise ξ sampled from standard Normal distribution. The amount of noise controls the amount of smoothing.

Gradually reducing the noise during training is related to a form of simulated annealing (Kirkpatrick et al., 1983) . Similarly to the analysis in Mobahi (2016) , we can write a Monte-Carlo estimate of

L K (θ) = (L * K)(θ) ≈ 1 N N i=1 L(θ -ξ (i)

). We provide the derivation and the gradient of this equation in Appendix A.

The Monte-Carlo estimators of the mollifiers can be easily implemented with neural networks, where the layers typically have the form:

EQUATION

with h l-1 a vector of activations from the previous layer in the hierarchy, W l a matrix representing a linear transformation and f an element-wise non-linearity of choice.

A mollification of such a layer can be formulated as:

h l = f((W l -ξ l )h l-1 ), where ξ l ∼ N (µ, σ 2 ) (16)

From Eqn. 16, it is easy to see that both weight noise methods proposed by Hinton & van Camp (1993) and Graves (2011) can be seen as a variation of Monte-Carlo estimate of mollifiers.

2.4. GENERALIZED AND NOISY MOLLIFIERS:

We introduce a generalization of the concept of mollifiers that encompasses the approach we explored here and that is targeted during optimization via a continuation method using stochastic gradient descent.

Definition 2.1. (Generalized Mollifier). A generalized mollifier is an operator, where T σ (f ) defines a mapping between two functions, such that

EQUATION

EQUATION

In addition, we consider noisy mollifiers which can be defined as an expected value of a stochastic function φ(x, ξ) under some noise source ξ with variance σ:

EQUATION

Definition 2.2. (Noisy Mollifier). We call a stochastic function φ(x, ξ σ ) with input x and noise ξ a noisy mollifier if its expected value corresponds to the application of a generalized mollifier T σ , as per Eqn. 20.

The composition of two noisy mollifiers sharing the same σ is also a noisy mollifier, since the three properties in the definition (Eqns. 17,18,19) are still satisfied. When σ = 0 no noise is injected and therefore the original function will be optimized. If σ → ∞ instead, the function will become an identity function. Thus, for instance, if we mollify each layer of a feed-forward network except the output layer, when σ → ∞ all the mollified layers will become identity function and the objective function of the network with respect to its inputs will be convex.

Consequently, corrupting separately the activation function of each level of a deep neural network (but with a shared noise level σ) and annealing σ yields a noisy mollifier for the objective function. This is related to the work of Mobahi (2016) , who recently introduced a way of analytically smoothing of the non-linearities to help the training of recurrent networks. The differences of that approach from our algorithm is two-fold: we use a noisy mollifier (rather than an analytic smoothing of the network's non-linearities) and we introduce (in the next section) a particular form of the noisy mollifier that empirically proved to work well.

Shaping the cost function to define a sequence of costs that are progressing from easier to more difficult ones can be related to the reward shaping (Ng et al., 1999; Ng, 2003) algorithms. In our algorithm, we shape the cost and the model architecture itself, rather than rewards or the targets in order to make the optimization easier. In that sense, reward shaping can be considered to be more closer to curriculum learning.

3. METHOD:

We propose an algorithm to mollify the cost of a neural network which also addresses an important drawback of the previously proposed noisy training procedures: as the noise gets larger, it can dominate the learning process and lead the algorithm to perform a random walk on the energy landscape of the objective function. Conversely in our algorithm, as the noise gets larger gradient descent minimizes a simpler (e.g. convex) but still meaningful objective function.

We define the desired behavior of the network in the limit cases where the noise is very large or very small, and modify the model architecture accordingly. Specifically, during training we minimize a sequence of increasingly complex noisy objectives

L = (L 1 (θ; ξ σ1 ), L 2 (θ; ξ σ2 ), • • • , L k (θ; ξ σ k ))

that we obtain by annealing the scale (variance) of the noise σ i . Let us note that our algorithm satisfies the fundamental properties of the generalized and noisy mollifiers that we introduced earlier.

We use a noisy mollifier based on our definition in Section 2.4. Instead of convolving the objective function with a kernel:

1. We start the training by optimizing a convex objective function that is obtained by configuring all the layers between the input and the last cost layer to compute an identity function, i.e., by skipping both the affine transformations and the blocks followed by nonlinearities. 2. During training, the magnitude of noise which is proportional to p is annealed, allowing to gradually evolve from identity transformations to linear transformations between the layers. 3. Simultaneously, as we decrease the p, the noisy mollification procedure allows the element-wise activation functions to gradually change from linear to be nonlinear.

4. SIMPLIFYING THE OBJECTIVE FUNCTION FOR FEEDFORWARD NETWORKS:

For every unit of each layer, we either copy the activation (output) of the corresponding unit of the previous layer (the identity path in Figure 2 ) or output a noisy activation hl of a non-linear transformation of it ψ(h l-1 , ξ; W l ), where ξ is noise, W l is a weight matrix applied on h l-1 and π is a vector of binary decisions for each unit (the convolutional path in Figure 2 ):

EQUATION

EQUATION

)

To decide which path to take, for each unit in the network, a binary stochastic decision is taken by drawing from a Bernoulli distribution with probability dependent on the decaying value of p l :

EQUATION

If the number of hidden units of layer l -1 and layer l +1 is not the same, we can either zero-pad layer l -1 before feeding it into the next layer or apply a linear projection to obtain the right dimensionality.

For p l = 1, the layer computes the identity function leading to a convex objective. If p l = 0 the layer computes the original non-linear transformation unfolding the full capacity of the model. We call the connections that are introduced this way as unitwise stochastic skip connections(USSC).

In DropIn layers (Smith et al., 2016) a binary random vector is sampled from a Bernoulli distribution to decide whether to introduce skip a connection from the layer below l -1 for each layer l and they use this as a regularization. As opposed to the USSC, DropIn layers do not necessarily do not necessarily achieve a convex objective function as the DropIn ratio (p l ) increases.

The pseudo-code for the mollified activations is reported in Algorithm 1. In the top path, the input is processed with a convolutional block followed by a noisy activation function, while in the bottom path the original activation of the layer l -1 is propagated untouched. For each unit, one of the two paths in picked according to a binary stochastic decision π.

Algorithm 1 Activation of a unit i at layer l.

1:

x i ← w i h l-1 + b i an affine transformation of h l-1 2: ∆ i ← u(x i ) -f(x i ) ∆ i is a measure of a saturation of a unit 3: σ(x i ) ← (sigmoid(a i ∆ i ) -0.5) 2

std of the injected noise depends on ∆ i 4: ξ i ∼ N (0, 1)

sampling the noise from a basic Normal distribution 5:

s i ← p l c σ(x i )|ξ i |

Half-Normal noise controlled by σ(x i ), const. c and prob-ty

p l 6: ψ(x i , ξ i ) ← sgn(u * (x i ))min(|u * (x i )|, |f * (x i ) + sgn(u * (x i ))|s i ||) + u(0)

noisy activation 7: π l i ∼ Bernoulli(p l ) p l controls the variance of the noise AND the prob of skipping a unit 8: hl i = ψ(x i , ξ i ) hl i is a noisy activation candidate 9: φ(h l-1 , ξ i , π l i ;

w i ) = π l i h l-1 i + (1 -π l i ) hl

i make a HARD decision between h l-1 i and hl i

5. LINEARIZING THE NETWORK:

In Section 2, we show that convolving the objective function with a particular kernel can be approximated by adding noise to the activation function. This method may suffer from excessive random exploration when the noise is very large.

We address this issue by bounding the element-wise activation function f(•) with its linear approximation when the variance of the noise is very large, after centering it at the origin. The resulting function f * (•) is bounded and centered around the origin.

Note that centering the sigmoid or hard-sigmoid will make them symmetric with respect to the origin. With a proper choice of the standard deviation σ(h), the noisy activation function becomes a linear function of the input when p is large, as illustrated by Figure 10 .

Let u * (x) = u(x)-u(0), where u(0) is the offset of the function from the origin, and x i the i-th dimension of an affine transformation of the output of the previous layer h l-1 : x i = w i h l-1 + b i . Then:

ψ(x i , ξ i ; w i ) = sgn(u * (x i ))min(|u * (x i )|, |f * (x i ) + sgn(u * (x i ))|s i ||) + u(0) (25)

The noise is sampled from a Normal distribution with mean 0 and whose standard deviation depends on c: s i ∼ N (0, p c σ(x i ))

5.1. LINEARIZING RELU ACTIVATION FUNCTION:

We have a simpler form of the equations to linearize ReLU (Nair & Hinton, 2010) achieve the linearization of the activation function when we have a very large noise in the activation function:

s i = minimum(|x i |, pσ(x i )|ξ|) (26) ψ(x i , ξ i , w i ) = f(x i ) -s i (27)

6. MOLLIFYING LSTMS AND GRUS:

In a similar vein it is possible to smooth the objective functions of LSTM and GRU networks by starting the optimization procedure with a simpler objective function such as optimizing a word2vec, BoW-LM or CRF objective function at the beginning of training and gradually increasing the difficulty of the optimization by increasing the capacity of the network.

For GRUs we set the update gate to 1 t -where t is the time-step index -and reset the gate to 1 if the noise is very large, using Algorithm 1. Similarly for LSTMs, we can set the output gate to 1 and input gate to 1 t and forget gate to 1 -1 t when the noise is very large. The output gate is 1 or close to 1 when the noise is very large. This way the LSTM will behave like a BOW model. In order to achieve this behavior, the activations ψ(x t , ξ i ) of the gates can be formulated as: ψ(x l t , ξ) = f(x l t + p l σ(x)|ξ|) By using a particular formulation of σ(x) that constraints it to be in expectation over ξ when p l = 1, we can obtain a function for γ ∈ R within the range of f(•) that is discrete in expectation, but still per sample differentiable:

EQUATION

We provide the derivation of Eqn. 28 in Appendix B. The gradient of the Eqn. 28 will be a Monte-Carlo approximation to the gradient of f(x l t ).

7. ANNEALING SCHEDULES FOR p:

We used a different schedule for each layer of the network, such that the noise in the lower layers will anneal faster. This is similar to the linearly decaying probability of layers in Huang et al. (2016b) .

Exponential Decay:

In our experiments, we focused on using an annealing schedule similar to the inverse sigmoid rule in Bengio et al. (2015) with p l t ,

EQUATION

with hyper-parameter k ≥ 0 at t th update for the l th layer, where L is the number of layers of the model. We stop annealing when the expected depth p t = L i=1 p l t reaches some threshold δ. In our experiment we set v t to be a moving average of the loss3 of the network, but for some of our experimnets that resulted unstable behaviors in the training and thus we have to fix v t to 1. An advantage of using running average of loss for the v t is because the behavior of the loss/optimization can directly influence the annealing behavior of the network. Because we will have: lim vt→∞ p l t = 1 and, lim vt→0

EQUATION

This has a desirable property: when the training-loss is high, the noise injected into the system will be large as well. As a result, the model is encouraged to do more exploration, while if the model converges the noise injected into the system by the mollification procedure will be zero.

Square-root Decay:

The square root decaying of the learning rate is very popular. In our experiments we have tried to decay the p l t according to min(p min , 1 -t N epochs ) where t is the index of the iteration/epoch over the data and N epochs corresponds to the total number of iterations that the model will do on the dataset. p min is hyperparameter to define the lower bound on the p l t .

Linear Decay We have also experimented with linearly decaying the p l t via min(p min , 1-t N epochs ). We have compared the plot of different annealing methods described in this paper as in Figure 4 . Furthermore, in our experiments we observe that training with noisy mollifiers can potentially be helpful for the generalization. This can be due to the noise induced to the backpropagation through the noisy mollification, that makes SGD more likely to converge to a flatter-minima (Hochreiter & Schmidhuber, 1997b) because the noise will help it escape from sharper local minima.

8. EXPERIMENTS:

In this section we mainly focus on training of difficult to optimize models, in particular deep MLPs with sigmoid or tanh activation functions. The details of the experimental procedure is provided in Appendix C.

8.1. EXPLORATORY STUDY ON MNIST:

We train a thin deep neural network on MNIST (LeCun & Cortes, 1998) dataset with 72 hidden layers and 100 hidden units. We train our model with Adam (Kingma & Ba, 2014) optimizer and fix the learning rate of all the models to 3e-4. We have used the same learning-rate for all the models in order to factor out the possibility that a model converges faster, due to the fact of using a larger learning rate.

Firstly, in Figure 5 , we investigate the effect of using different annealing schedules. Exponential decay converges faster compared to linear decay and square root decay of p. We find it to be very unstable to train our model with linear and square root decay in particular for large c values. Thus we have to use smaller c value (20 instead of 100) to be able to train the model with causing it to diverge. In Figure 6 , we show the effect of using the noisy training procedure that we have introduced by sampling a mask from Bernoulli and Gaussian distributions versu using the deterministic approximation of this noisy procedure which we also use for the test time but during the training as well. In Figure 7 , we compare the results obtained for the model using mollification obtained with or without batch normalization and feed-forward residual networks. The mollified model performs very closely to the MLP trained with residual connections and the batchnorm. However, using residual connections and batch-norm does not seem to improve the results.

We have tried to run experiment with the Monte-Carlo approximation of the mollification which is derived in Appendix A, however when we start with large noise and anneal the noise during the training, the model was very unstable and training was diverging. If we start with small noise and anneal the magnitude of the noise during the training, we could not observe any effect of it on the training. (Graves, 2016; Kalchbrenner et al., 2015) . We experiment on forty dimensional (bits) parity problem with 6-layer MLP using sigmoid activation function. All the models are initialized with Glorot initialization Glorot et al. (2011) and trained with SGD with momentum. We compare an MLP with residual connections using batch normalization and a mollified network with sigmoid activation function. As can be seen in Figure 8 , the mollified network converges faster.

Deep Pentomino Pentomino is a toy-image dataset where each image has 3 Pentomino blocks.

The task is to predict whether if there is a different shape in the image or not (Gülçehre & Bengio, 2013) . The best reported result on this task with MLPs is 68.15% accuracy (Gulcehre et al., 2014) 

9. LSTM EXPERIMENTS:

Predicting the Character Embeddings from Characters Learning the mapping from sequences of characters to the word-embeddings is a difficult problem. Thus one needs to use a highly non-linear function. We trained a word2vec model on Wikipedia with embeddings of size 500 (Mikolov et al., 2014) with a vocabulary of size 374557.

LSTM Language Modeling:

We evaluate our model on LSTM language modeling. Our baseline model is a 3-layer stacked LSTM without any regularization. We observed that mollified model converges faster and achieves better results. We provide the results for PTB language modeling in Table 2 .

10. CONCLUSION:

We propose a novel method for training neural networks inspired by an idea of continuation, smoothing techniques and recent advances in non-convex optimization algorithms. The method makes learning easier by starting from a simpler model, solving a well-behaved problem, and gradually transitioning to a more complicated setting. We show improvements on very deep models, difficult to optimize tasks and compare with powerful techniques such as batch-normalization and residual connections. We also show that the mollification procedure improves the generalization performance of the model on two tasks.

Our future work includes testing this method on large-scale language tasks that require long training time, e.g., machine translation and language modeling. Moreover, (Kaiser & Sutskever, 2015) observed that the training of Neural-GPU model can be improved significantly by using gradient noise which can be related to the smoothing of the loss surface, it would be interesting to try mollification on this model to see if the training of Neural GPU can be made easier by using mollification procedure.

:

We plan to release the source code of the models and experiments under, http://github.com/ caglar/molly_nets/.

We omit for brevity the algebraic details involved with a translation of the argument.

Depending on whether the model overfits or not, this can be a moving average of training or validation loss.

http://deeplearning.net/software/theano/

Figures and Tables:

Figure 1: A sequence of optimization problems of increasing complexity, where the first ones are easy to solve but only the last one corresponds to the actual problem of interest. It is possible to tackle the problems in order, starting each time at the solution of the previous one and tracking the local minima along the way.

Figure 2: Top: Stochastic depth. Bottom: mollifying network. The dashed line represents the optional residual connection.In the top path, the input is processed with a convolutional block followed by a noisy activation function, while in the bottom path the original activation of the layer l -1 is propagated untouched. For each unit, one of the two paths in picked according to a binary stochastic decision π.

Figure 3: The figures show how to evolve the model to make it closer to a linear network. Arrows denote the direction of the noise pushing the activation function towards the linear function. a) The quasi-convex envelope established by a |sigmoid(•)| around |0.25x|. b) A depiction of how the noise pushes the sigmoid to become a linear function.

Figure 4: We compare different annealing schedules with respect to the time in y-axis(iterations).

Figure 5: We compare the training performance of different types of annealing methods used with mollification procedure to anneal the parameter of the mollification p. Decaying the p exponentially achieves both better training and validation performance.

Figure 6: We show the learning curves of the model where we don't inject the noise during the training and instead use the deterministic approximation for the mollification during the training as well. The differerence in terms of speed of learning is very small.

We investigate the effect of using batch norm and residual connection for mollifcation and compare against to the network with residual connections and batch-norm. The effect of batch norm on this task for mollification seems to be very small and training convergence performance of the all the approaches are very close to each other.

Table content: <table><tr><td/><td/><td/><td/><td colspan="2">10 1 10 0 10 1</td><td/><td>mollified net with batchnorm mollified net without batch norm resnet model</td></tr><tr><td/><td/><td/><td>NLL</td><td colspan="2">10 3 10 2</td><td/></tr><tr><td/><td/><td/><td/><td colspan="2">10 4</td><td/></tr><tr><td/><td/><td/><td/><td colspan="2">10 5</td><td/></tr><tr><td/><td/><td/><td/><td colspan="2">10 6</td><td>0</td><td>100</td><td>200 Iterations 300</td><td>400</td><td>500</td></tr><tr><td>0.2 0.4 0.6 0.8 1.0 1.2 Figure 7: 0 Train NLL 0.0</td><td>50</td><td>100</td><td colspan="2">150</td><td colspan="3">6-layers Mollified Sigmoid MLP 6-layers Residual Sigmoid MLP with Batch Normalization 6-layers of Sigmoid MLP 200 250 300 350 400 x250 updates</td><td>450</td><td>Test Accuracy 93.25 Mollified Convnet 92.45 Stochastic Depth ResNet 91.78</td></tr><tr><td colspan="8">Figure 8: The learning curves of a 6-layers MLP</td><td>Table 1: CIFAR10 deep convolutional</td></tr><tr><td colspan="8">with sigmoid activation function on 40 bit parity</td><td>neural network.</td></tr><tr><td>task.</td><td/><td/><td/><td/><td/><td/></tr><tr><td colspan="7">8.2 DEEP MLP EXPERIMENTS</td></tr><tr><td colspan="8">Deep Parity Experiments Training neural networks on a high-dimensional parity problem can</td></tr><tr><td>be challenging</td><td/><td/><td/><td/><td/><td/></tr></table>

. The same model as ours trained without noisy activation function and vanilla residual connections scored 69.5% accuracy, while our mollified version scored 75.15% accuracy after 100 epochs of training on the 80k dataset.CIFAR10We experimented with deep convolutional neural networks of 110-layers with residual blocks and residual connections comparing our model against ResNet and Stochastic depth. We adapted the hyperparameters of the Stochastic depth network fromHuang et al. (2016a)  and we used the same hyperparameters for our algorithm. We report the training and validation curves of the three models in Figure10and the best test accuracy obtained early stopping on validation accuracy over 500 epochs in Table1. Our model achieves better generalization than ResNet. Stochastic depth achieves better generalization, but it might be possible to combine both and obtain better results.

Table content: <table><tr><td/><td>0.50</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td/><td/><td/><td/><td/><td/><td colspan="3">Mollified Deep LSTM</td></tr><tr><td/><td>0.45</td><td/><td/><td/><td/><td/><td colspan="2">Original Model</td><td/></tr><tr><td/><td>0.40</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.35</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td>Loss</td><td>0.30</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.25</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.20</td><td/><td/><td/><td/><td/><td/><td/><td/><td>Test PPL</td></tr><tr><td/><td>0.15</td><td/><td/><td/><td/><td/><td/><td/><td/></tr><tr><td/><td>0.10</td><td/><td/><td/><td/><td/><td/><td/><td/><td>LSTM</td><td>119.4</td></tr><tr><td/><td>0</td><td>10000</td><td>20000</td><td>30000</td><td>40000 # Updates</td><td>50000</td><td>60000</td><td>70000</td><td>80000</td><td>Mollified LSTM 115.7</td></tr><tr><td colspan="10">Figure 9: The training curve of a bidirectional-</td><td>Table 2: 3-layered LSTM net-</td></tr><tr><td colspan="10">RNN that predicts the embedding corresponding</td><td>work on word-level language</td></tr><tr><td colspan="6">to a sequence of characters.</td><td/><td/><td/><td/><td>modeling for PTB.</td></tr></table>

Peer review (iclr2017):
This paper first discusses a general framework for improving optimization of a complicated function using a series of approximations. If the series of approximations are well-behaved compared to the original function, the optimization can in principle be sped up. This is then connected to a particular formulation in which a neural network can behave as a simpler network at high noise levels but regain full capacity as training proceeds and noise lowers. The idea and motivation of this paper are interesting and sound. As mentioned in my pre-review question, I was wondering about the relationship with shaping methods in RL. I agree with the authors that this paper differs from how shaping typically works (by modifying the problem itself) because in their implementation the architecture is what is *shaped*. Nevertheless, the central idea in both cases is to solve a series of optimization problems of increasing difficulty. Therefore, I strongly suggest including a discussion of the differences between shaping, curriculum learning (I*m also not sure how this is different from shaping), and the present approach. The presentation of the method for neural networks lacks clarity in presentation. Improving this presentation will make this paper much easier to digest. In particular: - Alg. 1 can not be understood at the point that it is referenced. - Please explain the steps to Eq. 25 more clearly and connect to steps 1-6 in Alg. 1. - Define u(x) clearly before defining u*(x) There are several concerns with the experimental evaluations. There should be a discussion about why doesn*t the method work for solving much more challenging network training problems, such as thin and deep networks. Some specific concerns: - The MLPs trained (Parity and Pentomino) are not very deep at all. An experiment of training thin networks with systematically increasing depth would be a better fit to test this method. Network depth is well known to pose optimization challenges. Instead, it is stated without reference that *Learning the mapping from sequences of characters to the word-embeddings is a difficult problem.* - For cases where the gain is primarily due to the regularization effect, this method should be compared to other weight noise regularization methods. - I also suggest comparing to highway networks, since there are thematic similarities in Eq. 22, and it is possible that they can automatically anneal their behavior from simple to complex nets during training, considering that they are typically initialized with a bias towards copying behavior. - For CIFAR-10 experiment, does the mollified model also use Residual connections? If so, why? In either case, why does the mollified net actually train slower than the residual and stochastic depth networks? This is inconsistent with the MLP results. Overall, the ideas and developments in this paper are promising, but it needs more work to be a clear accept for me.