Venue: ICLR 2020 Conference

Reviews:

final_score: rating: 3: Weak Reject
review_content: The paper presents a new gradient update rule for neural net weights, which is a result of relaxing the usual SGD to a regularized linear regression between the descent direction and the original gradient. The method is interpreted as some kind of input normalization similar to Batch Norm later in the paper.

Although the method seems to work and the idea is interesting, I’m not entirely convinced by the experimental evidence. There are some technical concerns:

- In section 2.1, I’m only familiar with the MNIST dataset. Although the proposed method achieves slightly better performance than vanilla SGD, it’s pretty weird the accuracy is only 98.xx% as a very simple CNN could easily achieve >99% accuracy on MNIST. This might be due to the simple network architecture used. A more modern architecture would be preferable in those experiments. Also the improvement on CIFAR seems to close to tell.

- The writing seems to need some improvement. It’s sometimes hard to follow the text, and some result tables (Table A1,A2,A3) are in the appendix not the main text. It might be better to reorganize the text and present everything about the method (fast approximation, CNN, interpretation as normalization) together.

- The authors name the proposed method “deep gradient boosting”, but I’m not entirely sure how it is related to gradient boosting. Eq. 3 concerns sum of weak predictors, but how is this related to Eq. 4? The neural net F(x) doesn’t seem to be exactly a sum of weak learners. I don’t quite follow Eq. 5 – There seems to be missing something.

Typos:
- Eq. 5, d L_t instead of d L_T, and should there be a d \phi / d w_ij term?
- Eq. 7, v_t => v on the right-hand side.
- Eq. 12, should be \sum_i=1^B

review_title: Official Blind Review #2

final_score: rating: 3: Weak Reject
review_content: The paper proposes an a new idea of treating the back propagated gradients using chain rule as pseudo residual targets of a gradient boosting problem. Then the weight update is done by solving the boosting problem using a linear base learner. Furthermore, to reduce computational cost incurred by solving the boosting problem, an idea proposed is only keep the diagonal terms of the matrix inversion involved. 

It is an interesting idea to look at gradients different and update weight accordingly. The experimental results shows some improvements. 

The paper is written in a way that it looks like two papers and connected with the same idea DGB. It is a bit confusing during the first pass of reading. 

In all experiments, can you also report running times? how much is the overhead? and  how much reduction in training time if only using diagonal terms? 

Does only using diagonal terms hurt model performance?

For 2.1.1 MNIST, why is alpha the larger the better while the others are not this case?

For 2.1.5 AIR, why spit training and test equality?

For 4.3 why choose heavy model VGG? not ResNet? 

In Figure 1, is the spike of BN normal?


review_title: Official Blind Review #1

final_score: rating: 3: Weak Reject
review_content: Summary
This paper propose an extention method of SGD, deep gradient boosting (DGB), which views the back-propagation procedure as a pseudo-residual targets of a gradient boosting problem. To apply DGB to the real CNNs, DGB is simplified to a input normalization layer, conditioned on the assumption that the convolution kernels should be small. After applying the input normalization layer to CNNs, the model could achieve comparable performance to the model with BN on CIFAR-10 and ImageNet recognition.

There are several concerns influencing my rating:
* I cannot catch the advantages of this input normalization layer compared to BN. For example, could this input normalization layer help to address the problem that BN performs bad when batch size is small? The authors mention that this layer does not have additional parameters. But as I know, the parameter size of BN is small, which downgrades the significance of the proposed method. 

* In the CIFAR-10 and ImageNet experiments, only the VGG model is adopted, which obviously limits the application scope. Could the proposed method work well on ResNet, DenseNet or other more recent deep architectures?

* In the DGB experiments, the improvements of DGB compared with SGD in four datasets all seem marginal, in which DGB is slower than SGD. 

Overall, I recognize the exploration of this method. But the advantages of DGB compared to SGD seem marginal.

review_title: Official Blind Review #3

Meta-review: The paper introduces a neat idea that an SGD update can be written as a solution of the linear least squares problem with a given backpropagated output; this is generalized to a larger batch size, giving a sort of "block" gradient-type update. Some notes that the columns of $O_t$ have to be scaled are made, but not clear why. The paper then goes into the experiments, and then gets back to the fast approximation of DGB. It really looks like bad organization of the paper, which was noted. 
The reviewers agree that the actual computational improvements are marginal, and all recommend rejection. As a recommendation, I would suggest to restructure the paper for a more coherent view, and also the improvements in Top-1 are not very stimulating. The general view is interesting, but it is not clear what insight it brings.
Decision: Reject