Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper formulates the deep ResNet as a boosting algorithm. Based on this formulation, the authors prove that the generalization error bound decays exponentially with respect to the number of residual blocks. Further, a greedy block-wise training procedure is proposed to optimized ResNet-like neural networks. The authors claim that this algorithm is more efficient than standard end-to-end backpropagation (e2eBP) algorithm in terms of time and memory consumption.
Overall, the paper is well organized and easy to follow. I find that using the boosting theory to analyze the ResNet architecture quite interesting. My concerns are mainly on the proposed BoostResNet algorithm.
1.	I don’t quite understand why the sequentially training procedure is more time efficient than e2eBP.  It is true that BoostResNet trains each block quite efficiently. However, there are T blocks need to be trained sequentially. In comparison, e2eBP updates *all* the blocks at each training iteration.
2.	The claim that BoostResNet is memory efficient may not hold in practice. I agree that the GPU memory consumption is much lower than in e2eBP. However, this only holds *under the assumption that the intermediate outputs of a previous block are stored to disk*. Unfortunately, this assumption is not practical for real problems: the intermediate outputs usually requires much more space of the original datasets. What makes thing worse, the widely used data augmentation techniques (horizontal flip, shift, etc.) would further make the space requirement hundreds of or even thousands of times larger.
3.	The results in Figure 2 seem quite surprising to me, as the ResNet architectures is supposed to be quite robust when the network goes deeper. Have you tried the convolutional ResNet structure used in their original paper?
4.	In Figure 3, how did you measure the number of gradient updates? In the original ResNet paper, the number of iterations required to train a model is 164(epochs)*50000(training samples)/128(batch-size)=6.4x10^5, which is far less than that showing in this figure.
5.	In Figure 3, it seems that the algorithms are not fully converged, and on CIFAR-10 the e2eBP outperforms BoostResNet eventually. Is there any explanations? 
review_title: theoretical analysis is interesting, experiments are relatively weak

final_score: rating: 5: Marginally below acceptance threshold
review_content: Disclaimer: I reviewed this paper for NIPS as well and many of comments made by reviewers at that time still apply to this version of the paper as well, although presentation has overall improved.

The paper presents a boosting-style algorithm for training deep residual networks. Convergence analysis for training error is presented and analysis of generalization ability is also provided. Paper concludes with some experimental results.

The main contribution of this work is interpretation of ResNet as a telescoping sum of differences between the intermediate layers and treating these differences as weak learners that are then boosted. This indeed appears to an interesting insight about ResNet training.

On the other hand, one of the main objections during NIPS reviews was the relation of this work to work of Cortes et al. on Adanet. In particular, generalization bounds presented in this work are results taken from that paper (which authors admit). What is less clear is the distinction between the algorithmic approaches which makes it hard to judge the novelty of this work. There is a paragraph at the end of section 2 but it seems rather vague.

One other objection during NIPS reviews was experimental setup explanation of which is omitted from the current version. In particular, same learning rate and mini-batch size was used both for boosting and backprop algorithms which seems strange since boosting is supposed to train much smaller classifiers.

Another concern is practicality of the proposed method which seems to require maintaining explicit distribution over all examples which would not be practical for modern datasets where NNs are typically applied.

review_title: An ineresting approach to boosting residual networks but problems from NIPS reviews are still not resolved.

final_score: rating: 4: Ok but not good enough - rejection
review_content: Summary:
This paper considers a learning method for the ResNet using the boosting framework. More precisely, the authors view the structure of the ResNet as a (weighted) sum of base networks (weak hypotheses) and apply the boosting framework. The merit of this approach is to decompose the learning of complex networks to that of small to large networks in a moderate way and it uses less computational costs. The experimental results are good. The authors also show training and generalization error bounds for the proposed approach.

Comments: 
The idea of the paper is natural and interesting. Experimental results are somewhat impressive. However, I am afraid that theoretical results in the paper contain several mistakes and does not hold. The details are below.

I think the proof of Theorem 4.2 is wrong. More precisely, there are several possibly wrong arguments as follows:
- In the proof, \alpha_t+1 is chosen so as to minimize an upper bound of Z_t, while the actual algorithm is chosen to minimize Z_t. The minimizer of Z_t and that of an upper bound are different in general. So, the obtained upper bound does not hold for the training error of the actual algorithm. 
- It is not a mistake, but, there is no explanation why the equality between (27) and (28) holds. Please add an explanation. Indeed, equation (21) matters. 

Also, the statement of Theorem 4.2 looks somewhat cheating: The statement seems to say that it holds for any iteration T and the training error decays exponentially w.r.t. T. However, the parameter T is determined by the parameter gamma, so it is some particular iteration, which might be small and the bound could be large. 

The generalization error bound Corollary 4.3 seems to be wrong, too. More precisely, Lemma 2 of Cortes et al. is OK, but the application of Lemma 2 is not. In particular, the proof does not take into account of the function \sigma. In other words, the proof considers the Rademacher complexity R_m(\calF_t), of the class \calF_t, but, acutually, I think it should consider R_m(\sigma(\calF_t)), where the class \sigma(\calF_t) consists of the composition of functions \sigma and f_t in \calF_t. Talagrand’s lemma (see, e.g., Mohri et al.’ s book: Foundation of Machine Learning) can be used to analyze the complexity of the composite class. But, the resulting bound would depend on the Lipschizness of \sigma in an exponential way. 

The explanation of the generalization ability is not sufficient. While the latter weak hypotheses are complex enough and would have large edges, the complexity of the function class of weak hypotheses grows exponentially w.r.t. the iteration T, which should be mentioned. 

As a summary, the paper contains nice ideas and experimental results are promising, but has non-negligible mistakes in theoretical parts which degrade the contribution of the paper.

Minor Comments:
-In Algorithm 1, \gamma_t is not defined when a while-loop starts. So, the condition of the while-loop cannot be checked.

 

review_title: The paper contains nice ideas and experimental results are promising, but has non-negligible mistakes in theoretical parts which degrade the contribution.

Meta-review: All three reviewers felt that the paper was just below the acceptance threshold, with scores of 5,4,5. R1 felt there were problems in the proofs, but the authors rebuttal satisfactorily addressed this. R3 and the authors had an extended discussion with the authors, but did not revise their score from its initial value (5). R4 had concerns about the experimental evaluation, that wasn't fully addressed in the rebuttal. With no reviewers advocating acceptance, the paper will have to rejected unfortunately. 
Decision: Reject