Title: Mollifying Networks
Authors: Caglar Gulcehre
Marcin Moczulski
Francesco Visin
Yoshua Bengio
Venue: ICLR 2017
Keywords: Deep learning
Optimization

Abstract: The optimization of deep neural networks can be more challenging than the traditional convex optimization problems due to highly non-convex nature of the loss function, e.g. it can involve pathological landscapes such as saddle-surfaces that can be difficult to escape from for algorithms based on simple gradient descent. In this paper, we attack the problem of optimization of highly non-convex neural networks objectives by starting with a smoothed -- or mollified -- objective function which becomes more complex as the training proceeds.  Our proposition is inspired by the recent studies in continuation methods: similarly to curriculum methods, we begin by learning an easier (possibly convex) objective function and let it evolve during training until it eventually becomes the original, difficult to optimize objective function. The complexity of the mollified networks is controlled by a single hyperparameter that is annealed during training. We show improvements on various difficult optimization tasks and establish a relationship between recent works on continuation methods for neural networks and mollifiers.


TL;DR: We are proposing a new continuation method for neural networks, that starts from optimizing a convex objective function and gradually during the training the function evolves into more non-convex function.

Review discussion:
It appears to me that apart from work in regularization through noise, this is most directly related to the large amount of research in shaping in reinforcement learning, e.g. see Andrew Ng's PhD thesis and references therein and lots of follow up work. Do you agree with this?
Why is comparing the learning/generalization of two methods using the same hyperparameters meaningful? Does this really tell us if one method is better than the other?
>It appears to me that apart from work in regularization through noise, this is most directly related to the large amount of research in shaping in reinforcement learning, e.g. see Andrew Ng's PhD thesis and references therein and lots of follow up work. Do you agree with this?
We agree that our work is related to the reward shaping methods in reinforcement learning.
However, in that sense, our method is not as tightly related to shaping as curriculum learning. Because curriculum learning is interested in changing the input and output distributions to make the task easier to optimize.
On the other hand, in our proposed algorithm, we are smoothing out the cost function, by changing the capacity/complexity of the model without changing the reward function or input-output distributions of the network.
>Why is comparing the learning/generalization of two methods using the same hyperparameters meaningful? Does this really tell us if one method is better than the other?
In terms of making the comparisons fair, keeping the capacity related hyperparameter same for all models is crucial.
However, optimal values for learning rate and other hyperparameters can be different for different methods. In that sense, on some tasks, we have tuned the learning rate for our baseline and used the same learning rate for our proposed mollification algorithm. For our algorithm, we have just finetuned the algorithm/method specific hyperparameters (such as k in annealing). In that sense, we have tried to avoid doing more hyperparameter search for our own model than our baselines.
The paper only shows results on toy tasks (MNIST, very weak PTB baseline). Do you have any results on real-world tasks, e.g., ImageNet or a large language model? Any comparison to SOTA networks?
Thanks for your remark and questions.
In addition to providing results on challenging deep-LSTM language modeling results on PTB(3 layers of LSTM network without regularization), Pentomino, 40 bits Parity Task and a difficult language task which we have constructed: "Mapping the sequence of characters to word2vec embeddings" results. We have shown results on CIFAR10 and compared them against to SOTA results on it , e.g.: "Stochastic Depth" with a deep residual convnet (120 layers).
Note that we have tried to focus on tasks that are challenging to optimize such as very deep LSTMs on "Characters to word2vec embeddings" task since we believe that this method would be more useful, i.e. in the cases where regular SGD fails in terms of optimization.
However, we will include larger-scale experiments with mollification procedure such as language modeling on text8 in the next update of the paper.
Are you annealing the Bernoulli parameter from 1 to 0. If so, it seems that the variance of the Bernoulli variable, and thus, also of the function, would actually increase first.
That is a very good point, we will fix this in the paper. We are annealing the Bernoulli parameter from 1 to 0. We are going to fix this in the paper.
The authors show that the idea of smoothing a highly non-convex loss function can make deep neural networks easier to train.
The paper is well-written, the idea is carefully analyzed, and the experiments are convincing, so we recommend acceptance. For a stronger recommendation, it would be valuable to perform more experiments.
In particular, how does your smoothing technique compare to inserting probes in various layers of the network?
Another interesting question would be how it performs on hard-to-optimize tasks such as algorithm learning.
For example, in the "Neural GPU Learns Algorithms" paper the authors had to relax the weights of different layers of their RNN to make it optimize -- could this be avoided with your smoothing technique?
We would like to thank you for your valuable feedback.
>  In particular, how does your smoothing technique compare to inserting probes in various layers of the network?
Indeed, inserting probes at each layer can make both the training and optimization easier.
However, the effect of probes and the effect of mollification on the training can be different, e.g. inserting probes will not necessarily make the objective function smoother. It will mainly help the optimization by dealing with the vanishing gradients problem in the lower layers.
Nonetheless, there is a similarity with inserting probes into the network and our mollification procedure. In the early stages of the training, mollifiers can also act like probes by introducing random linear connections from the units of the layer to the cost. Note that this behavior will change during the course of training as we anneal the hyperparameter of the mollification.
Finally, it is important to point out that probes affect the network layerwise, while our mollification procedure is more fine-grained, as it can introduce probes for each unit individually.
>   For example, in the "Neural GPU Learns Algorithms" paper the authors had to relax the weights of different layers of their RNN to make it optimize -- could this be avoided with your smoothing technique?
Thanks for pointing out this very interesting connection. Mollification also relies on an optimization schedule which starts from an easier task and ends with the task of interest, so both approaches are particular forms of continuation methods.
Interestingly, both papers exploit gradient noise to ease the optimization. The relationship between the gradient noise and smoothing has been demonstrated in [1,2] and its adoption in the Neural GPU provides further empirical evidence of its effectiveness. It would indeed be interesting to see if mollification could replace the soft-sharing annealing completely.
[1] Gulcehre C, Moczulski M, Denil M, Bengio Y. Noisy Activation Functions. arXiv preprint arXiv:1603.00391. 2016 Mar 1.
[2] Hazan, E., Levy, K.Y. and Shalev-Shwartz, S., 2015. On graduated optimization for stochastic non-convex problems. arXiv preprint arXiv:1503.03712.
--CMFY
This paper first discusses a general framework for improving optimization of a complicated function using a series of approximations. If the series of approximations are well-behaved compared to the original function, the optimization can in principle be sped up. This is then connected to a particular formulation in which a neural network can behave as a simpler network at high noise levels but regain full capacity as training proceeds and noise lowers.
The idea and motivation of this paper are interesting and sound. As mentioned in my pre-review question, I was wondering about the relationship with shaping methods in RL. I agree with the authors that this paper differs from how shaping typically works (by modifying the problem itself) because in their implementation the architecture is what is "shaped".
Nevertheless, the central idea in both cases is to solve a series of optimization problems of increasing difficulty.
Therefore, I strongly suggest including a discussion of the differences between shaping, curriculum learning (I'm also not sure how this is different from shaping), and the present approach.
The presentation of the method for neural networks lacks clarity in presentation. Improving this presentation will make this paper much easier to digest.
In particular:
- Alg. 1 can not be understood at the point that it is referenced.
- Please explain the steps to Eq. 25 more clearly and connect to steps 1-6 in Alg. 1.
- Define u(x) clearly before defining u*(x)
There are several concerns with the experimental evaluations. There should be a discussion about why doesn't the method work for solving much more challenging network training problems, such as thin and deep networks. Some specific concerns:
- The MLPs trained (Parity and Pentomino) are not very deep at all. An experiment of training thin networks with systematically increasing depth would be a better fit to test this method. Network depth is well known to pose optimization challenges.
Instead, it is stated without reference that "Learning the mapping from sequences of characters to the word-embeddings is a difficult problem."
- For cases where the gain is primarily due to the regularization effect, this method should be compared to other weight noise regularization methods.
- I also suggest comparing to highway networks, since there are thematic similarities in Eq. 22, and it is possible that they can automatically anneal their behavior from simple to complex nets during training, considering that they are typically initialized with a bias towards copying behavior.
- For CIFAR-10 experiment, does the mollified model also use Residual connections? If so, why?
In either case, why does the mollified net actually train slower than the residual and stochastic depth networks? This is inconsistent with the MLP results.
Overall, the ideas and developments in this paper are promising, but it needs more work to be a clear accept for me.
We would like to thank our reviewer for his/her important remarks.
> Therefore, I strongly suggest including a discussion of the differences between shaping, curriculum learning (I'm also not sure how this is different from shaping), and the present approach.
Thank you for pointing it out. We are going to add a discussion about the relationship between mollifying the cost function and shaping in RL. As mentioned by the reviewer there is a similarity in that although we shape the architecture and the cost, rather than the reward or the targets, still those two approaches are both forms of continuation methods. They are very related to each other and ideally they can be combined together to achieve better performance.
> The presentation of the method for neural networks lacks clarity in presentation. Improving this presentation will make this paper much easier to digest.
In particular: - Alg. 1 can not be understood at the point that it is referenced.
We are sorry for not putting enough details in the paper. Following the suggestion, we will add more on implementing our generalized noisy mollification approach for different types of neural networks.  Section 5 can be improved by explaining the Eqn 25 more clearly. We will introduce the algorithm after Section 5.1 and explain each step.
> Please explain the steps to Eq. 25 more clearly and connect to steps 1-6 in Alg. 1.
Let us point out that a simpler formulation for ReLU is provided in Section 5.1.
Nonetheless, we agree that Eqn 25 (for sigmoid and tanh activation functions) looks complicated.
In Eqn 25, in a nutshell, we add noise into the activation function which is sampled from Half-Normal distribution to the absolute value of the centered activation function (for tanh and ReLU, this step is not necessary). The direction of the noise points towards the absolute value of the function as the standard deviation of the noise grows.
Lastly, we multiply it with the sign of the input x again and uncenter the activation function. We illustrate this idea more thoroughly in Figure 3.
We will clarify the steps to Eq. 25 and explain it in more details.
> Define u(x) clearly before defining u*(x)
Thank you for noticing this shortcoming. u(x) is the first order Taylor approximation of the original activation function around zero and u*(x) stands for the centered u(x) which is obtained by shifting u(x) towards the origin. We will carefully define u(x) before u*(x).
> There are several concerns with the experimental evaluations. There should be a discussion about why doesn't the method work for solving much more challenging network training problems, such as thin and deep networks.
Please note that the network in our experiments on CIFAR10 can be considered very deep and quite thin. In those experiments, we used a model which has 110 layers with a relatively small layer width (we used 16, 32 or 64 as the kernel size, depending on the layer, following the architecture reported in [1]).
For a meaningful benchmark, we compare our approach with very strong baselines such as ResNet and Stochastic Depth models and the experiments show that Mollifying Networks perform at least comparably well.
We apologize if the paper conveys the impression that the method doesn't work in this kind of setting, we will make sure this information is stated more clearly.
> The MLPs trained (Parity and Pentomino) are not very deep at all. An experiment of training thin networks with systematically increasing depth would be a better fit to test this method.
Thanks, this is a very interesting suggestion. We are going to add further experiments contrasting the use of mollification with respect to a different number of layers for a neural network.
> For cases where the gain is primarily due to the regularization effect, this method should be compared to other weight noise regularization methods.
Thank you for pointing it out. We ran some preliminary experiments with the weight noise and annealing the weight noise by using the Monte Carlo estimate of the mollifier on MNIST dataset and compared it with our method. We did not notice significant improvements adding weight noise in terms of optimization of the deep network.
On the contrary, in some cases, it was making the training more difficult and causing numerical stability issues.
It seems that when the noise is very large for the weight noise, it dominates the training and the optimizer does random exploration on the loss surface instead of efficiently minimizing the objective function. We will add these results to our paper.
> Instead, it is stated without reference that "Learning the mapping from sequences of characters to the word-embeddings is a difficult problem.”
The reason why learning to map a sequence of characters to word-embeddings is difficult is because of the nature of the task. The sequence of characters operates in the orthographic space, where each individual character doesn’t have any particular meaning in English.
However, word embeddings live in semantic space (where similar-meaning words are nearby), and the mapping from orthographic space (where similarly written words are nearby) to this space is highly nonlinear (since a single character change can yield a totally different meaning).
In our experiments, the slow convergence and high training error on this task with deep bidirectional LSTM and deep MLP are an empirical evidence of this phenomenon. We will further elaborate on this in the paper.
>  I also suggest comparing to highway networks, since there are thematic similarities in Eq. 22, and it is possible that they can automatically anneal their behavior from simple to complex nets during training, considering that they are typically initialized with a bias towards copying behavior.
Indeed our mollification procedure is related to highway networks. As mentioned by the reviewer, highway networks can be trained with a particular mollifier where the activations of the gates are starting very close to 1.0 and during training the model can learn to anneal the gating activation. Consider that in the setting where the sigmoids are almost saturated, training can be difficult/slow.
Note that the above would correspond to a deterministic mollifier. In our preliminary experiments deterministic mollifiers are more prone to overfitting.
Nevertheless we agree that it would be an interesting comparison. We are planning to add results on a small dataset such as MNIST/Parity task to compare both approaches.
>  For CIFAR-10 experiment, does the mollified model also use Residual connections? If so, why?
Yes. As we mention in appendix C3, we use residual connections in our mollified convnet. The main reason is that for such a deep network, when we anneal the noise very fast, the mollified network without residual connections becomes more difficult to train and finding a good annealing schedule was harder. Coincidentally, mollified resnet achieves better results.
> In either case, why does the mollified net actually train slower than the residual and stochastic depth networks? This is inconsistent with the MLP results.
We refer to the Figure 6 b) for the curves of the training. At the beginning, the learning of the mollifying network is slower because the amount of noise injected into the network is much bigger.
However, as the training progresses the mollifying network catches up (around epoch 250) and achieves very similar train-phase negative log-likelihood to the residual network.
[1] Huang, G., Sun, Y., Liu, Z., Sedra, D. and Weinberger, K., 2016. Deep networks with stochastic depth. arXiv preprint arXiv:1603.09382.
--CMFY
The paper shows the relation between stochastically perturbing the parameter of a model at training time, and considering a mollified objective function for optimization. Aside from Eqs. 4-7 where I found hard to understand what the weak gradient g exactly represents, Eq. 8 is intuitive and the subsequent Section 2.3 clearly establishes for a given class of mollifiers the equivalence between minimizing the mollified loss and training under Gaussian parameter noise.
The authors then introduce generalized mollifiers to achieve a more sophisticated annealing effect applicable to state-of-the-art neural network architectures (e.g. deep ReLU nets and LSTM recurrent networks). The resulting annealing effect can be counterintuitive: In Section 4, the Binomial (Bernoulli?) parameter grows from 0 (deterministic identity layers) to 1 (deterministic ReLU layers), meaning that the network goes initially through a phase of adding noise. This might effectively have the reverse effect of annealing.
Annealing schemes used in practice seem very engineered (e.g. Algorithm 1 that determines how units are activated at a given layer consists of 9 successive steps).
Due to the more conceptual nature of the authors contribution (various annealing schemes have been proposed, but the application of the mollifying framework is original), it could have been useful to reserve a portion of the paper to analyze simpler models with more basic (non-generalized) mollifiers.
For example, I would have liked to see simple cases, where the perturbation schemes derived from the mollifier framework would be demonstrably more suitable for optimization than a standard heuristically defined perturbation scheme.
We would like to thank the reviewer for interesting remarks.
>  The paper shows the relation between stochastically perturbing the parameter of a model at training time, and considering a mollified objective function for optimization. Aside from Eqs. 4-7 where I found hard to understand what the weak gradient g exactly represents, Eq. 8 is intuitive and the subsequent Section 2.3 clearly establishes for a given class of mollifiers the equivalence between minimizing the mollified loss and training under Gaussian parameter noise.
Thank you for this important comment. We will work on improving the section on the weak gradient.
Nevertheless, we appreciate that the reviewer found our introduction of equivalence between minimizing the mollified loss and training while injecting Gaussian noise intuitive and clear. It is a fundamental contribution for the rest of the paper and for the whole method.
>  The authors then introduce generalized mollifiers to achieve a more sophisticated annealing effect applicable to state-of-the-art neural network architectures (e.g. deep ReLU nets and LSTM recurrent networks). The resulting annealing effect can be counterintuitive: In Section 4, the Binomial (Bernoulli?) parameter grows from 0 (deterministic identity layers) to 1 (deterministic ReLU layers), meaning that the network goes initially through a phase of adding noise.
Thanks for pointing this out. You are indeed right, Eq. 24 refers to a single neuron rather than a layer. In that context, the variable is coming from a Bernoulli distribution whose parameter "p" starts from 1 (according to our formulation, the model has deterministic identity layers at the beginning) and anneals to 0 (deterministic nonlinear layers) during training.
This indeed causes the variance of Bernoulli’s distribution to be small at the beginning.
Then it grows and starts decreasing again once the parameter “p” of Bernoulli reaches 0.5.
However, the variance of the noisy activation $\phi(.,.)$ will constantly decrease (starting with high noise).
A parameter of the stochastic noisy mollifier that we have used in our experiments is controlled by p^l. It is being annealed according to the method we have introduced in Section 7.
>  Annealing schemes used in practice seem very engineered (e.g. Algorithm 1 that determines how units are activated at a given layer consists of 9 successive steps).
In Algorithm 1 the lines from 1 to 8 are about our stochastic approach to linearize activation functions. The annealing method is described in Section 7 and the same method is being used in all our experiments. This annealing approach is very general - we do not specifically engineer this method to be able to use it with a particular architecture or model.
>  Due to the more conceptual nature of the authors contribution (various annealing schemes have been proposed, but the application of the mollifying framework is original), it could have been useful to reserve a portion of the paper to analyze simpler models with more basic (non-generalized) mollifiers.
For example, I would have liked to see simple cases, where the perturbation schemes derived from the mollifier framework would be demonstrably more suitable for optimization than a standard heuristically defined perturbation scheme.
Thank you for the suggestion. We will conduct more experiments in that direction. We will add more results on Monte-Carlo estimate of the mollifiers which we introduced earlier in the paper on MNIST/parity task. We will also add some experiments with our mollification procedure where we do not anneal the parameters of the mollifier. These experiments will provide us with more insights on the importance of annealing.
--CMFY

Meta-review:
role: PC
data: comment: The paper presents a nice idea for using a sequence of progressively more expressive neural networks to train a model. Experiments are shown on CIFAR10, parity, language modeling to show that the methods performs well on these tasks.
 However, as noted by the reviewers, the experiments do not do a convincing enough job. For example, the point of the model is to show that optimization can be made easier by their concept, however, results are presented on depths that are considered shallow these days. The results on PTB are also very far from SOTA. However, because of the novelty of the idea, and because of the authors ratings, I'm giving the paper a pass. I strongly encourage the authors to revise the paper accordingly for the camera ready version.
 
 Pros:
 - interesting new idea
 - shows gains over simple baselines.
 Cons:
 - not a very easy read, I think the paper was unnecessarily dense exposition of a relatively simple idea.
 - experiments are not very convincing for the specific type of problem being addressed.

Decision: Accept (Poster)