Venue: ICLR 2017 conference

Reviews:

final_score: rating: 7: Good paper, accept
review_content: It is interesting to derive such a bound and show it satisfies a regret bound along with empirical evidence on the CIFAR-10 for cross entropy loss and auto encoder for MSE loss. At least empirically, by comparing the observed training loss and taylor loss, the better a particular optimizer performs (training loss statement, not a validation or observed test statement) the smaller the difference between these two. Also shown is the regret loss is satisfied at different scales of the network, by layer, neuron and whole network. 

The Taylor approximation can be used to investigate activation configurations of the network, and used to connect this to difficulty in optimizing  at kinks in the loss surface, along with an empirical study of exploration of activation surface of the SGD/Adam/RMSprop optimizers, the more exploration the better the resulting training loss.

Not that it impacts the paper but the weaker performance of the SGD could be related to the fixed learning rate, if we anneal this learning rate, which should improve performance, does this translate to more exploration and tightening between the actual loss and the Taylor loss? 

- It might be useful to use a cross validation set for some of the empirical studies, in the end we would like to say something about generalization of the resulting network

- Is there a reason the subscript on the Jacobian changes to a_l in the <Gl,V> definition?

review_title: Authors make use of a first order Taylor expansion to reach a first order approximation to the output of a neural network with convex loss function using the backpropagated gradients. Shows that difference in actual training loss and Taylor loss satisfies a regret bound. Exploration of the optimization surface is discussed

final_score: rating: 3: Clear rejection
review_content: This paper develops a theoretical guarantee for the convergence of the training error. The result is quite general that covers the training of a wide range of neural network models. The key idea of the paper is approximate the training loss by its linear approximation. Since its linearity in the variables (thus convex), the authors plug in results that has been developed in the literature of online learning. 

This paper has good novelty in using the Taylor approximation thus greatly simplifying the analysis of the behaviour of the model. However, there are two problems about the main result of this paper, Theorem 2.

1. It is not clear if the Taylor optimum would converge or not. 
As noticed by the authors, the upper bound is path dependent. Appendix 3 tries to claim that this Taylor optimum indeed converges, but the proof is buggy. In the proof of Lemma 2, it is proved that the difference between two sequential Taylor optimum is approaching 0. Note that this is actually weaker than being Cauchy sequence and insufficient to guarantee convergence.

2. The lefthand side of Equation (3) (I will denote it by L3 in this review) is not equivalent to training error. 
An upper bound on this average error is not sufficient to guarantee the convergence of the training error neither. Take the gradient descent for example (thus each minibatch x_0^n is the whole training set), the convergence of the training error should be lim_{n -> \infty} l(f_{w^n}(x_0^n), y^n). The convergence of L3 is necessary but not sufficient to imply the convergence of the training error.

Another concern about Theorem 2 (but it is minor compared to the two problems mentioned above) is that to achieve the O(1/\sqrt{n}) rate, the algorithm has to pick a particular learning rate. Larger or smaller learning rate (in the order of n) will lead to significantly worse regret. But in the experiments of the paper, the learning rates are not picked according to the theorem.

Overall, this paper has a good motivation and good novelty. It could be further developed into a good paper. But due to the two problems and a buggy proof mentioned above, I think it is not ready for publish yet.

review_title: Good Novelty. But the main theorem of the paper is problematic.

final_score: rating: null
review_content: Hi,

In the paper, it is pointed out that the Taylor optimum is path-dependent. Theoretically, is it possible that the Taylor optimum does not converge at all? In the experiment, it seems not the case, but do we have theoretical guarantee?
review_title: The convergence of the Taylor optimum

final_score: rating: 7: Good paper, accept
review_content: This paper adopts Taylor approximations of neural nets for separating convex and non-convex components of the optimization. This enables them to bound the training error by the Taylor optimum and regret (theorem 2). This is a nice theoretical result applicable to popular deep nets. The empirical studies back up the theoretical claim.
review_title: Review

Meta-review: The paper presents a theoretical analysis of the convergence of the training error. The presented result is rather general and can potentially apply to many neural networks. 
 
 Reviewers pointed out several important concerns regarding the mathematical rigor and precision of the claims. The authors' response partially addressed concerns about the scope of the claims and unclear arguments of the proofs. 
 
 We invite the authors to submit a revision of the paper, where all statements and proofs are mathematically clear, to the workshop track.
Decision: Invite to Workshop Track