Venue: ICLR 2017 conference

Reviews:

final_score: rating: null
review_content: The paper only shows results on toy tasks (MNIST, very weak PTB baseline). Do you have any results on real-world tasks, e.g., ImageNet or a large language model? Any comparison to SOTA networks?
review_title: Real-world results?

final_score: rating: 7: Good paper, accept
review_content: The authors show that the idea of smoothing a highly non-convex loss function can make deep neural networks easier to train.

The paper is well-written, the idea is carefully analyzed, and the experiments are convincing, so we recommend acceptance. For a stronger recommendation, it would be valuable to perform more experiments. In particular, how does your smoothing technique compare to inserting probes in various layers of the network? Another interesting question would be how it performs on hard-to-optimize tasks such as algorithm learning. For example, in the "Neural GPU Learns Algorithms" paper the authors had to relax the weights of different layers of their RNN to make it optimize -- could this be avoided with your smoothing technique?
review_title: Good paper that could use a few more experiments

final_score: rating: 6: Marginally above acceptance threshold
review_content: This paper first discusses a general framework for improving optimization of a complicated function using a series of approximations. If the series of approximations are well-behaved compared to the original function, the optimization can in principle be sped up. This is then connected to a particular formulation in which a neural network can behave as a simpler network at high noise levels but regain full capacity as training proceeds and noise lowers.

The idea and motivation of this paper are interesting and sound. As mentioned in my pre-review question, I was wondering about the relationship with shaping methods in RL. I agree with the authors that this paper differs from how shaping typically works (by modifying the problem itself) because in their implementation the architecture is what is "shaped". Nevertheless, the central idea in both cases is to solve a series of optimization problems of increasing difficulty. Therefore, I strongly suggest including a discussion of the differences between shaping, curriculum learning (I'm also not sure how this is different from shaping), and the present approach.

The presentation of the method for neural networks lacks clarity in presentation. Improving this presentation will make this paper much easier to digest. In particular:
- Alg. 1 can not be understood at the point that it is referenced. 
- Please explain the steps to Eq. 25 more clearly and connect to steps 1-6 in Alg. 1.
- Define u(x) clearly before defining u*(x)

There are several concerns with the experimental evaluations. There should be a discussion about why doesn't the method work for solving much more challenging network training problems, such as thin and deep networks. Some specific concerns:

- The MLPs trained (Parity and Pentomino) are not very deep at all. An experiment of training thin networks with systematically increasing depth would be a better fit to test this method. Network depth is well known to pose optimization challenges. Instead, it is stated without reference that "Learning the mapping from sequences of characters to the word-embeddings is a difficult problem."

- For cases where the gain is primarily due to the regularization effect, this method should be compared to other weight noise regularization methods.

- I also suggest comparing to highway networks, since there are thematic similarities in Eq. 22, and it is possible that they can automatically anneal their behavior from simple to complex nets during training, considering that they are typically initialized with a bias towards copying behavior.

- For CIFAR-10 experiment, does the mollified model also use Residual connections? If so, why? In either case, why does the mollified net actually train slower than the residual and stochastic depth networks? This is inconsistent with the MLP results.

Overall, the ideas and developments in this paper are promising, but it needs more work to be a clear accept for me.
review_title: Interesting direction but requires improvements

final_score: rating: null
review_content: It appears to me that apart from work in regularization through noise, this is most directly related to the large amount of research in shaping in reinforcement learning, e.g. see Andrew Ng's PhD thesis and references therein and lots of follow up work. Do you agree with this?

Why is comparing the learning/generalization of two methods using the same hyperparameters meaningful? Does this really tell us if one method is better than the other?

review_title: Relations to shaping, Evaluation protocols

final_score: rating: null
review_content: Are you annealing the Bernoulli parameter from 1 to 0. If so, it seems that the variance of the Bernoulli variable, and thus, also of the function, would actually increase first.
review_title: annealing

final_score: rating: 6: Marginally above acceptance threshold
review_content: The paper shows the relation between stochastically perturbing the parameter of a model at training time, and considering a mollified objective function for optimization. Aside from Eqs. 4-7 where I found hard to understand what the weak gradient g exactly represents, Eq. 8 is intuitive and the subsequent Section 2.3 clearly establishes for a given class of mollifiers the equivalence between minimizing the mollified loss and training under Gaussian parameter noise.

The authors then introduce generalized mollifiers to achieve a more sophisticated annealing effect applicable to state-of-the-art neural network architectures (e.g. deep ReLU nets and LSTM recurrent networks). The resulting annealing effect can be counterintuitive: In Section 4, the Binomial (Bernoulli?) parameter grows from 0 (deterministic identity layers) to 1 (deterministic ReLU layers), meaning that the network goes initially through a phase of adding noise. This might effectively have the reverse effect of annealing.

Annealing schemes used in practice seem very engineered (e.g. Algorithm 1 that determines how units are activated at a given layer consists of 9 successive steps).

Due to the more conceptual nature of the authors contribution (various annealing schemes have been proposed, but the application of the mollifying framework is original), it could have been useful to reserve a portion of the paper to analyze simpler models with more basic (non-generalized) mollifiers. For example, I would have liked to see simple cases, where the perturbation schemes derived from the mollifier framework would be demonstrably more suitable for optimization than a standard heuristically defined perturbation scheme.
review_title: interesting view on improving the optimization of neural networks, proposed practical mollifiers seem quite engineered

Meta-review: The paper presents a nice idea for using a sequence of progressively more expressive neural networks to train a model. Experiments are shown on CIFAR10, parity, language modeling to show that the methods performs well on these tasks.
 However, as noted by the reviewers, the experiments do not do a convincing enough job. For example, the point of the model is to show that optimization can be made easier by their concept, however, results are presented on depths that are considered shallow these days. The results on PTB are also very far from SOTA. However, because of the novelty of the idea, and because of the authors ratings, I'm giving the paper a pass. I strongly encourage the authors to revise the paper accordingly for the camera ready version.
 
 Pros:
 - interesting new idea
 - shows gains over simple baselines.
 Cons:
 - not a very easy read, I think the paper was unnecessarily dense exposition of a relatively simple idea.
 - experiments are not very convincing for the specific type of problem being addressed.
Decision: Accept (Poster)