Venue: ICLR 2018 conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes a variant of ADAM optimization algorithm that normalizes the weights of each hidden unit. They further suggest using batch normalization on the output of the network before softmax to improve the generalization. The main ideas are new to me and the paper is well-written. The arguments and derivations are very clear. However, the experimental results suggest that the proposed method is not superior to SGD and ADAM.

Pros: 

- The idea of optimizing the direction while ignoring the magnitude is interesting and make sense.
- Using batch normalization before softmax is interesting.

Cons:

- In the abstract, authors claim that the proposed method has good optimization performance of ADAM and good generalization performance of SGD. Such a method could be helpful if one can get to the same level of generalization faster (less number of epochs). However, the experiments suggest that optimization advantages of the proposed method do not translate to faster generalization. Figures 2,3 and Table 1 indicate that the generalization performance of this method is very similar to SGD.

- The paper is not coherent. In particular, direction-preserving ADAM and batch-normalized softmax trick are completely orthogonal ideas. 

- In the introduction and Section 2.2, authors claim that weight decay has a significant effect on the generalization performance of DNNs. I wonder if authors can refer to any work on this. My own experience and several empirical works have suggested that weight decay does not improve generalization significantly.


review_title: A variant of ADAM optimization algorithm that normalizes the weights of each hidden unit

final_score: rating: 5: Marginally below acceptance threshold
review_content: Method:

The paper is missing analysis of some important related works such as

"Beyond convexity: Stochastic quasi-convex optimization" by E. Hazan et al. (2015) 

where Stochastic Normalized Gradient Descent (SNGD) was proposed. 

Then, normalized gradient versions of AdaGrad and Adam were proposed in 

"Normalized Gradient with Adaptive Stepsize Method for Deep
Neural Network Training" by A. W. Yu et al. (2017).

Another work which I find to be relevant is 

"Follow the Signs for Robust Stochastic Optimization" by L. Balles and P. Hennig (2017).

From my personal experiments, restricting w_i to have L2 norm of 1, i.e., to be +-1 
leads to worse generalization. One reason for this is that weight decay is not 
really functioning since it cannot move w_i to 0 or make its amplitude any smaller. 
Please correct me if I misunderstand something here. 

The presence of +-1 weights moves us to the area of low-precision NNs, 
or more specifically, NNs with binary / binarized weights as in 

"BinaryConnect: Training Deep Neural Networks with
binary weights during propagations" by M. Courbariaux et al. (2015)

and 

"Binarized neural networks: Training deep neural networks with weights and activations constrained to+ 1 or-1" by M. Courbariaux et al. (2016). 

Regarding
"Moreover, the magnitude of each update does not depend on themagnitude of the gradient. Thus, ND-Adam is more robust to improper initialization, and vanishing or exploding gradients."

If the magnitude of each update does not depend on the magnitude of the gradient, then the algorithm heavily depends on the learning rate. Otherwise, it does not have any means to approach the optimum in a reasonable number of steps *when* it is initialized very / unreasonably far from it. The claim of your second sentence is not supported by the paper. 

Evaluation:

I am not confident that the presented experimental validation is fair. First, the original WRN paper and many other papers with ResNets used weight decay of 0.0005 and not 0.001 or 0.002 as used for SGD in this paper. It is unclear why this setting was changed. One could just use \alpha_0 = 0.05 and \lambda = 0.0005.

Then, I don't see why the authors use WRN-22-7.5 which is different from WRN-28-10 which was suggested in the original study and used in several follow-up works. The difference between WRN-22-7.5 and WRN-28-10 is unlikely to be significant, 
the former might have about only 2 times less parameters which should barely change the final validation errors. However, the use of WRN-22-7.5 makes it impossible to easily compare the presented results to the results of Zagoruyko who had 3.8\% with WRN-28-10. I believe that the use of the setup of Zagoruyko for WRN-22-7.5 would allow to get much better results than 4.5\% and 4.49\% shown for SGD and likely better 4.14\% shown for ND-Adam. I note that the use of WRN-22-7.5 is unlikely to be due to the used hardware because later in paper the authors refer to WRN-34-7.5.

My intuition is that the proposed ND-Adam moves the algorithm back to SGD but with potentially harmful constraints of w_i=+-1. Even the values of \alpha^v_0 found for ND-Adam (e.g., \alpha^v_0=0.05 in Figure 1B) are in line of what would be optimal values of \alpha_0 for SGD. 

I find it uncomfortable that BN-Softmax is introduced here to support the use of an optimization algorithm, moreover, that the values of \gamma_c are different for CIFAR-10 and CIFAR-100. I wonder if the proposed values are optimal (and therefore selected) for all three tested algorithms  or only for Adam-ND. I expect that hyperparameters of SGD and Adam would also need to be revised to account for BN-Softmax.
review_title: Some related works should be analyzed. The experimental validation should to be revised. 

final_score: rating: 4: Ok but not good enough - rejection
review_content: The paper extended the Adam optimization algorithm to preserve the update direction. Instead of using the un-centered variance of individual weights, the proposed method adapts the learning rate for the incoming weights to a hidden unit jointly using the L2 norm of the gradient vector. The authors empirically demonstrated the method works well on CIFAR-10/100 tasks.

Comments:

- I found the paper very hard to follow. The authors could improve the clarity of the paper greatly by listing their contribution clearly for readers to digest. The authors also combined the proposed method with a few existing deep learning tricks in the paper. All those tricks that, ie. section 3.3 and 4, should go into the background section.

- Overall, the only contribution of the paper seems to be the ad-hoc modification to Adam in Eq. (9). Why is this a reasonable modification? Do we expect this modification to fail in any circumstances? The experiments on CIFAR dataset and one CNN architecture do not provide enough evidence to show the proposed method work well in general.


review_title: review

Meta-review: The paper proposes a modification to Adam which is intended to ensure that the direction of weight update lies in the span of the historical gradients and to ensure that the effective learning rate does not decrease as the magnitudes of the weights increase.  The reviewers wanted a clearer justification of the changes made to Adam and a more extensive evaluation, and held to this opinion after reading the authors' rebuttal and revisions.

Pros:
+ The basic idea of treating the direction and magnitude separately in the optimization is interesting.

Cons:
- Insufficient evaluation of the new method.
- More justification and analysis needed for the modifications.  For example, are there circumstances under which they will fail?
- The modification to Adam and batch-normalized softmax idea are orthogonal to one another, making for a less coherent story.
- Proposed method does not have better generalization performance than SGD.
- Concern that constraining weight vectors to the unit sphere can harm generalization.

Decision: Reject