Venue: ICLR 2018 conference

Reviews:

final_score: rating: 6: Marginally above acceptance threshold
review_content: The manuscript proposes to increase the norm of the last hidden layer to promote better classification accuracy. However, the motivation is a bit less convincing. Here are a few motivations that are mentioned.
(1) Increasing the feature norm of correctly classified examples helps cross entropy, which is of course correct. However, it only decreases the training loss. How do we know it will not lead to overfitting?
(2) Increasing the feature norm of mis-classified examples will make gradient larger for self-correction. And the manuscript proves it in property 2. However, the proof seems not complete. In Eq (7), increasing the feature norm would also affect the value of the term in parenthesis. As an example, if a negative example is already mis-classified as a positive, and its current probability is very close to 1, then further increasing feature norm would make the probability even closer to 1, leading to saturation and smaller gradient.
(3) Figure 1 shows that examples with larger feature norm tend to be predicted well. However, it is not very convincing since it is only a correlation rather than causality. Let's use simple linear softmax regression as a sanity check, where features to softmax are real features rather than hidden units. Increasing the feature norm seems to be against the best practice of feature normalization in which each feature after normalization is of variance 1.

The manuscript states that the feature norm won't be infinitely increased since there is an upper bound. However, the proof of property 3 seems to only apply to the certain cases where K<2D. In addition, alpha is in the formula of upper bound, but what is the upper bound of alpha?

The manuscript does comprehensive experiments to test the proposed method. The results are good, since the proposed method outperforms other baselines in most datasets. But the results are not impressively strong.

Minor issues:
(1) For proof of property 3, it seems that alpha and beta are used before defined. Are they the radius of two circles?
review_title: The idea is interesting, but motivation requires more justification. Results are good, but not very impressive

final_score: rating: 6: Marginally above acceptance threshold
review_content: Pros:
1. It provided theoretic analysis why larger feature norm is preferred in feature representation learning.

2. A new regularization method (feature incay) is proposed.

Cons:
It seems there is not much comparison between this proposed method and the concurrent work "COCO(Liu et al. (2017c))".
review_title: This paper investigates how to finetune feature norms of correctly-classified and mis-classified feature vectors to improve learning process. Based on the analysis, they proposed feature incay to encourage larger feature norm. Experimental results on four datasets demonstrate the effectiveness of the proposed method.

final_score: rating: 6: Marginally above acceptance threshold
review_content: The analyses of this paper (1) increasing the feature norm of correctly-classified examples induce smaller training loss, (2) increasing the feature norm of mis-classified examples upweight the contribution from hard examples, are interesting. The reciprocal norm loss seems to be reasonable idea to improve the CNN learning based on the analyses. 

However, the presentation of this paper need to be largely improved. For example, Figure 3 seems to be not relevant to Property2 and may be show the feature norm is lower when the samples is hard example. Therefore, the author used reciprocal norm loss which increases feature norm as shown in Figure 4. However, both Figures are not explained in the main text, and thus hard to understand the relation of Figure 3 and 4. The author should refer all Figures and Tables. 

Other issues are:
-Large-margin Soft max in Figure 2 is not explained in the introduction section. 
-In Eq.(7), P_j^I is not defined. 
- In the Property3, The author wrote “ where r is lower bound of feature norm”. 
 However, r is not used.
-In the experimental results, “RN” is not defined.
-In the Table3, the order of \lambda should be increasing or decreasing order. 
- Table 5 is not referred in the main text. 

== Updated review == 
The presentation has been improved, I have increased the rate from 5 to 6. 
Following are further comments for presentation. 

-	Fig.2 “ the increasing L2 norm “ seems to  “the order of L2 norm ”
-	Pp.4 the first sentence above Eq.(7) “According to definition …”  should be improved . 
-	pp.5, the first sentence of the second paragraph “The feature norm can be optimized ..” is not clear. 
-	It would be better put Figure 5 under Property3. 
-	D should be defined in Property3. 
-	pp.8 wrote “However, 259-misclassfied examples are further introduced”. However, in Table 5, it seems to be 261. 
-	Section 5. is “Conclusion and future work”. However, future work is not mentioned. 

review_title: Interesting analysis of feature norm, but the paper needs improvements

Meta-review:  + An intriguing novel regularization method: encouraging larger norms for the feature vector input to the last softmax layer of a classifier.
 + Resonably extensive experimental validation shows that it improves test accuracy to some degree.
 - While a motivation is given, the formal analysis of what is really going on remains very superficial and limited.

Technical note: Simply scaling the softmax layer's input would not change class rankings, so any positive effect of this regularizer on classification performance is due to it changing the learning dynamic in the upper layers as well. The paper could be much stronger if it did provide an analysis regarding how the global learning dynamic is affected in all layers, by the interaction between weight decay and the last layer's feature incay.

Decision: Invite to Workshop Track