Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: The work was prompted by  an interesting observation: a phase transition can be observed in deep learning with stochastic gradient descent and Tikhonov regularization. When the regularization parameter exceeds a (data-dependent) threshold, the parameters of the model are driven to zero, thereby preventing any learning. The authors then propose to moderate this problem by letting the regularization parameter to be zero for 5 to 10 epochs, and then applying the "strong" penalty parameter. In their experimental results, the phase transition is not observed anymore with their protocol. This leads to better performances, by using penalty parameters that would have prevent learning with the usual protocol.

The problem targeted is important, in the sense that it reveals that some of the difficulties related to non-convexity and the use of SGD that are often overlooked. The proposed protocol is reported to work well, but since it is really ad hoc, it fails to convince the reader that it provides the right solution to the problem. I would have found much more satisfactory to either address the initialization issue by a proper warm-start strategy, or to explore standard optimization tools such as constrained optimization (i.e. Ivanov regularization) , that could be for example implemented by stochastic projected gradient or barrier functions. I think that the problem would be better handled that way than with the proposed strategy, which seems to rely only on a rather limited amount of experiments, and which may prove to be inefficient when dealing with big databases.

To summarize, I believe that the paper addresses an important point, but that the tools advocated are really rudimentary compared with what has been already proposed elsewhere.

Details :
- there is a typo in the definition of the proximal operator in Eq. (9) 
- there are many unsubstantiated speculations in the comments of the experimental section that do not add value to the paper 
- the figure showing the evolution of the magnitude of parameters arrives too late and could be completed by the evolution of the data-fitting term of the training criterion
review_title: An interesting diagnosis but a rudimentary cure

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper is well motivated and written. However, there are several issues.
1. As the regularization constant increases, the performance first increases and then falls down -- this specific aspect is well known for constrained optimization problems. Further, the sudden drop in performance also follows from vanishing gradients problem in deep networks. The description for ReLUs in section 2.2 follows from these two arguments directly, hence not novel. Several of the key aspects here not addressed are: 
1a. Is the time-delayed regularization equivalent to reducing the value (and there by bringing it back to the 'good' regime before the cliff in the example plots)? 
1b. Why should we keep increasing the regularization constant beyond a limit? Is this for compressing the networks (for which there are alternate procedures), or anything else. In other words, for a non-convex problem (about whose landscape we know barely anything), if there are regimes of regularizers that work well (see point 2) -- why should we ask for more stronger regularizers? Is there any optimization-related motivation here (beyond the single argument that networks are overparameterized)? 
2. The proposed experiments are not very conclusive. Firstly, the authors need to test with modern state-of-the-art architectures including inception and residual networks. Secondly, more datasets including imagenet needs to be tested. Unless these two are done, we cannot assertively say that the proposal seems to do interesting things. Thirdly, it is not clear what Figure 5 means in terms of goodness of learning. And lastly, although confidence intervals are reported for Figures 3,4 and Table 2, statistical tests needs to be performed to report p-values (so as to check if one model significantly beats the other).
review_title: The authors propose a modification on regularization weights in an attempt to impose very strong regularization on backpropataion. There are several issues with the proposal as well as its efficacy in practice. The current statues of the paper needs more work.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The authors studied the behavior that a strong regularization parameter may lead to poor performance in training of deep neural networks. Experimental results on CIFAR-10 and CIFAR-100 were reported using AlexNet and VGG-16. The results seem to show that a delayed application of the regularization parameter leads to improved classification performance.

The proposed scheme, which delays the application of regularization parameter, seems to be in contrast of the continuation approach used in sparse learning. In the latter case, a stronger parameter is applied, followed by reduced regularization parameter. One may argue that the continuation approach is applied in the convex optimization case, while the one proposed in this paper is for non-convex optimization. It would be interesting to see whether deep networks can benefit from the continuation approach, and the strong regularization parameter may not be an issue because the regularization parameter decreases as the optimization progress goes on.

One limitation of the work, as pointed by the authors, is that experimental results on big data sets such as ImageNet is not reported. 

review_title: The authors proposed to apply L1/L2 regularization in training of deep neural networks after 5-10 epochs. 

Meta-review: The submission is motivated by an empirical observation of a phase transition when a sufficiently high L1 or L2 penalty on the weights is applied.  The proposed solution is to optimize for several epochs without the penalty followed by introduction of the penalty.  Although empirical results seem to moderately support this approach, there does not seem to be sufficient theoretical justification, and comparisons are missing.  Furthermore, the author response to reviewer concerns contain unclear statements e.g.
"The reason is that, to reach the level of L1 norm that is low enough, the model needs to go through the strong regularization for the first few epochs, and the neurons already lose its learning ability during this period like the baseline method."
It is not at all clear what "neurons already lose its learning ability" is supposed to mean.
Decision: Reject