Title: FEATURE INCAY FOR REPRESENTATION REGULARIZA-TION

Abstract: Softmax-based loss is widely used in deep learning for multi-class classification, where each class is represented by a weight vector and each sample is represented as a feature vector. Inspired by that weight decay is a common practice to regularize the weight vectors, we investigate how to regularize the feature vectors since representation is also tunable in deep learning. One main observation is that elongating the feature norm of both correctlyclassified and mis-classified feature vectors improves learning: ( 1) increasing the feature norm of correctly-classified examples enlarges the probability margin among different classes and ensures better generalization. (2) increasing the feature norm of mis-classified examples can up-weight the contribution from hard examples. Accordingly, we propose feature incay to regularize feature vectors by encouraging larger feature norm. Extensive empirical results on MNIST, CI-FAR10, CIFAR100 and LFW demonstrate the effectiveness of feature incay.

1. INTRODUCTION:

Deep Neural Networks (DNNs) with softmax-based loss have achieved state-of-the-art performance on numerous multi-class classification related tasks. In DNNs, both representations and classifiers are learned within a unified network concurrently, where the final representation for a sample is the feature vector f outputted from the penultimate layer, while the last layer outputs scores z i = w i • f for each category i, where w i is the weight vector for category i. Before defining the loss, the scores for each category are normalized into probability via softmax function, i.e., pi = e z i j e z j . A well-trained DNN should output significant larger probability for the correct label than other labels, which requires the score for the correct label is significantly larger than other labels. Since z i = w i • f = w i f cos(θ), where θ is the angle between w i and f , the goal of significant larger score for the correct label than other labels can be achieved by tuning w i , f and θ. While increasing the weight norm w i is constrained by weight decay for regularization, thus f and θ become the two main factors for optimization. Although softmax loss can tune both of them, there is still much room to improve either or both factors.

Recently, Liu et al. (2016) , Wang et al. (2017) and Liu et al. (2017a) propose different approaches to further optimize the factor of angular θ, and all of them have achieved obviously better performance. To further emphasize the factor of angular, Ranjan et al. (2017) and Liu et al. (2017c) propose to use normalized feature vectors for softmax loss where the factor of feature norm is totally ignored. In this work, we make the effort to optimize the feature norm. Firstly, we analyze the connections between the feature norm and the classification accuracy within softmax loss, they are highly correlated as illustrated in Figure 1 . Features with larger norm tend to be correctly classified with higher probability. Here we propose to optimize the feature norm by augmenting the softmax loss with feature incay.

In contrast to weight decay that shrinks weight vectors to be of small norm, feature incay tends to stretch out the feature vectors. From the computational perspective, larger feature norm results in larger score differences among categories, which can better separate the categories. From the perspective of pattern detection, larger feature norm encourages model to learn and detect more prominent patterns.

Figure 2 (a), 2(b) and 2(c) show the results of comparison experiments by adding feature incay to softmax loss, where feature incay achieves larger feature norm, smaller loss value and higher accuracy on test set. The geometric interpretation of feature incay is illustrated in Figure 2 (d), 2(e) and 2(f), we can achieve the larger inter-class separability by explicitly optimizing both the f and θ compared with the other methods. Besides, the proposed feature incay (implemented as Reciprocal Norm Loss) is designed to increase the feature norm adaptively according to the original feature norm, which can also help reduce the intra-class variances as discussed in section 3.4.

Our main contributions are summarized as follows:

• The benefits of increasing the feature norm for both correctly-classified examples and misclassified examples are analyzed theoretically. • We verify the effectiveness of feature incay empirically on four widely used classification datasets(i.e., MNIST, CIFAR10, CIFAR100 and LFW) using various network architectures.

With feature incay, we achieve consistent performance improvements on all of them. • We also conduct experiments to study the weights' distribution w/ feature incay and w/o feature incay and investigate where the improvements come from.

2. RELATED WORK:

Large-margin Softmax Loss. Liu et al. (2016) proposed to improve softmax loss by incorporating an adjustable margin m multiplying the angle between a feature vector and the corresponding weight vector. Compared with the softmax loss, it pays more attention to the angular decision margin between classes as illustrated in Figure 2 (e). Large-margin softmax loss appends stronger constraint to the angular, while feature incay considers constraint to feature norm. As illustrated in Figure 2 (f), feature incay is orthogonal to large-margin softmax loss.

Center Loss. Wen et al. (2016) presented the center loss to learn centers for deep features of each class and penalize the distances between the deep features and their corresponding class centers.

Combining softmax loss with center loss actually uses two sets of classifiers, where representation is learned based on both the inner product to weight vector and the Euclidean distance to class center. The added center loss helps minimize the intra-class distances also by influencing the feature norm, namely, small feature norm will be increased and large feature norm will be decreased during the process of pushing feature vectors to class centers. Different from center loss, feature incay also increases the large feature norm instead of penalizing feature vectors with large norm as center loss.

Weight/Feature Normalization. Inspired by the fact that feature normalization before calculating the sample distances usually achieves better performance for retrieval tasks, Ranjan et al. (2017) proposed to use normalized feature vectors in softmax loss during training, thus the feature norm has no effect on softmax loss and angle is the main factor to be optimized. Congenerous cosine loss (Liu et al. (2017b) ), NormFace (Wang et al. (2017) ), and cosine normalization (Chunjie et al. (2017) ) take a step further to normalize the weight vectors which replace inner product with cosine similarity within softmax loss, and only optimize the factor of angle. Recently, Wang et al. (2018b) , Wang et al. (2018a) and Deng et al. (2018) all proposed to add the angle margin after normalizing both the weight vectors and feature vectors during nearly the same period. Although normalization mechanism achieves much lower intra-class angular variability by emphasizing more on the angle during training, they ignore that feature norm is another useful factor worth to optimize.

Feature Scale. COCO (Liu et al. (2017c) ) and L 2 -softmax (Ranjan et al. (2017) ) are the most relevant to our work, and especially COCO is a concurrent work. Here we compare our method with COCO theoretically, Relations: Both COCO and feature incay increase the feature norm, which is the common reason for the performance improvement. Differences: COCO is optimizing the feature embedding on a hypersphere while feature incay optimizes feature embedding located between two hyperspheres with different radiuses(Property 3 in section 3.4). COCO proposes a novel con-generous cosine loss while feature incay uses the original softmax loss. Feature incay is simpler than COCO and it can be easily plugged into almost all the related work that use softmax loss.

Empirically comparision is provided in section 4.2.

3 OUR WORK

3.1. REVISITING SOFTMAX LOSS:

Let X = {(x i , y i )} N i=1 be the training set contains N samples, where x i is the raw input to the DNN, y i ∈ {1, 2, • • • , K} is the class label that supervises the output of the DNN. Denote f i as the feature vector for x i learned by the DNN, {w j } K j=1 represent weight vectors for the K categories. Then, softmax loss is defined as,

EQUATION

Bias terms are ignored following the discussions in recent works Liu et al. (2016) and Wang et al. (2017) . Denote the angle between w j and f i as θ wj ,fi , the inner product between w j and f i can be rewritten as

EQUATION

By combining the above two equations, we get Proof. Assuming f i is correctly-classified into category y i , here we define the probability margin between category y i and any other category j(j = y i ) as:

L softmax = - 1 N N i=1 log e wy i fi cos(θ wy i ,f i ) K j=1 e wj fi cos(θ w j ,f i ) (3) 3.2 FEATURE NORM

EQUATION

Recall that when f i is correctly classified, we have w T yi f i > w T j f i for j = y i . Then, for any t > 0, we have, According to the above inequations, we have

e (w T j -w T y i )((1+t)fi) < e (w T j -w T y i )fi (5) 1 -e (w T j -w T y i )(1+t)fi > 1 -e (w T j -w T y i )fi K k=1 e (w T k -w T y i )(1+t)fi < K k=1 e (w T k -w T y i )fi (6)

M yi,j ((1 + t)f i ) > M yi,j (f i ) (7)

In summary, the probability margin increases by increasing the feature norm and larger probability margin ensures better generalization.

Property 2 [Feature Norm Matters for Mis-classified Examples] Assuming the feature vector f i is mis-classified into category k(k = y i ), increasing the feature norm of f i can up-weight the gradients of the weight vector w k .

Proof. According to definition of softmax loss in Eq.( 3), the gradient with respect to weight vector

EQUATION

where

P i j = e w T j f i K k=1 e w T k

f i , and h(i) is an indicator function, h(i) = 1 if y i = j otherwise h(i) = 0. Specifically, we analyze how the value of ∂Lsoftmax ∂w k changes, which contains two factors: (1)(P i kh(i)) increases with larger f i when f i is mis-classified as category k. By increasing f i we will always have larger P i k and h(i) = 0 when computing w k . So the absolute value of (P i k -h(i)) is increased. (2) f i is increased. So increasing the feature norm f i can up-weight the gradients for w k . Especially for hard examples f i with P i k close to 1 and P i yi close to 0, (P i yi -h(i)) is closer to -1 by increasing f i , thus increasing f i can also increase the gradients for weight vector w yi .

We investigate the distribution of feature norm over correctly-classified and mis-classified examples on CIFAR10 in Figure3, and the gradients contributed by the mis-classified features is suppressed due to most of the mis-classified examples are of small feature norm. Thus it is necessary to increase the feature norm of mis-classified examples. By optimizing the feature norm of mis-classified vectors, the errors can be fixed as illustrated in the fifth column of Table 6 . (e.g., 336 examples from the test set of CIFAR10 become correctly-classified by increasing their average feature norm from 169.3 to 195.9) Besides, the proposed feature incay increases the originally small feature norm faster than the originally large feature norm. Thus the feature norm of the mis-classified examples will increase faster than the feature norm of the correctly-classified examples, which can further up-weight the contribution of the hard examples.

3.3. RECIPROCAL NORM LOSS:

The superiorities of increasing feature norm have been investigated in the previous subsection. Here we explore several methods that increase the feature norm end-to-end by penalizing an additional term, such as -f 2 , -log( f 2 ) and 1 f 2 . The comparison analysis is provided in supplementary. Specifically, we choose 1 f 2 and propose the Reciprocal Norm Loss, where the definition of Reciprocal Norm Loss is,

L = - 1 N N i=1 log e w T y i fi K j=1 e w T j fi softmax loss + µ K k=1 w k 2 weight decay + λ 1 N N i=1 1 f i 2 + feature incay (9)

where is a small positive value to prevent dividing by value close to zero, λ is a hyper-parameter used to control the influence of feature incay. The whole loss function consists of three parts: softmax loss, weight decay and feature incay, of which the later two items prefer small weight norm and large feature norm. Feature incay can be considered during the whole training procedure or after some training iterations with softmax loss. Large feature norm brings large inter-class separability under the constraint of small weight norm, otherwise large feature norm can be trivially achieved by increasing the weight norm.

The simple reciprocal form of feature norm has an important property that moves feature vectors with small norm faster along weight vectors than feature vectors with large norm and results intra-class compactness, which is illustrated in Figure 4 . Specifically, denote feature incay of f i as

F(f i ) = 1 fi 2 + , the gradient is ∂F ∂fi = - 2fi ( fi 2 + ) 2 .

For any two feature vectors f p and f q satisfying f q > f p , we always have ∂F ∂fp > ∂F ∂fq . Thus vectors with small norm increase fast along their original directions while feature vectors with large norm increase slowly along their original directions. Property 3 [Feature Norm Bound] Given (a) the angle between any feature vector f i and its corresponding weight vector w yi is zero, (b) the angles between any two neighbor weight vectors of different classes are θ, we have (1) the minimal inter-class distance is 2α sin( θ 2 ), where α is lower bound of feature norm, (2) to ensure the maximal intra-class distance is smaller than the minimal inter-class distance, the upper bound of feature norm is 3α, especially when K < 2D(K is the number of classes and D is the features' dimension), the upper bound in the range of

[(1 + √ 2)α, 3α].

Proof. Figure 5 shows the 2-dimensional case satisfying conditions (a) and (b), where black arrows named with W i represent weight vectors for each class. As we have assumed that all feature vectors are lying on the directions of their corresponding W i , blue circle and purple circle denote the lower bound and upper bound of feature norm respectively. Thus the maximal intra-class distance is d 2 = β -α and the minimal inter-class distance is d 1 = 2α sin( θ 2 ). To ensure minimal inter-class distance is larger than intra-class distance, i.e., d 1 = 2α sin( θ 2 ) > d 2 = β -α, which requires β < 2α sin( θ 2 ) + α 3α. Thus 3α is the general upper bound for the feature norm. Especially when K < 2D, according to the Lemma(refer to supplementary), we can ensure that θ ≥ 90 • . Besides, the angle between any two vectors is smaller than 180 • . Then θ 2 ∈ [45 • , 90 • ] and sin( θ 2 ) is a monotonously increasing function within the range [45

• , 90 • ]. Based on sin( θ 2 ) ∈ [ √ 2 2 , 1], the upper bound of L 2 -norm of feature vectors is in the range [(1 + √ 2)α, 3α].

According to the Property 3, the ratio of the upper bound β and the lower bound α is bounded( β α ≤ 3). We can estimate the upper bound of the feature norm based on the original features' L 2 -norm.

In our experiments, we choose the average L 2 -norm as the lower bound to avoid the influence of outliers. For example, if the average L 2 -norm on CIFAR10 is 200, we will choose 483 ≈ (1 + √ 2) × 200 as the threshold to control the feature incay. The feature incay for features with feature norm exceeding 483 will be set as 0.

4. EXPERIMENTS:

In this section, we verify the effectiveness of feature incay through empirical experiments.

4.1. EXPERIMENTAL SETTINGS:

We evaluate feature incay on four datasets, i.e., MNIST, CIFAR10, CIFAR100 and LFW. MNIST consists of 60,000 training images and 10,000 test images from 10 handwritten digits, both CIFAR10 2014)) for training and testing. The weight µ for weight decay is 0.0005 in all experiments. We choose different weight λ in different experiments, i.e., 1.0, 0.1 or 0.01. The momentum is 0.9, and the learning rate starts from 0.1 and is divided by a factor of 10 three times when the training error stops decreasing.

4.2. COMPARISON EXPERIMENTS:

Feature incay is added to Sofmax, L-Softmax and A-Softmax to compare with state-of-the-art approaches, which is represented with RN(Reciprocal Norm loss) plus the baseline method. e.g., RN + Softmax means combining the feature incay with Softmax loss. The reproduced results by Softmax, L-Softmax and A-Softmax following Liu et al. (2016; 2017a) are the same as or slightly better than the referred numbers in general. Here we conduct experiments on CIFAR10 to investigate the influence of hyper-parameter λ. Results are illustrated in Table 3 . Both RN + Softmax and RN + L-Softmax achieve consistent improvement for all different λ except on RN + L-Softmax when λ = 1, which is caused by that the loss item of L-Softmax can be smaller than the loss item of feature incay. To balance the L-Softmax and feature incay for training, λ should be set to a relatively small weight.

4.4. SIMPLY SCALE THE FEATURE:

To verify whether it is possible to improve the performance by simply rescaling the features before computing softmax loss, we conduct extensive experiments on CIFAR10 and present the related results in Table 4 . Simply scaling the features fails to improve the performance, where too large value can cause the network fails to converge. For example, when we scale the features more than 10 times, the softmax loss will explode during training(NAN represents the softmax loss is exploded during training).

4.5. EFFECTS OF WEIGHT DECAY:

Here we also investigate the influence of the weigth decay on the classification accuracy and feature norm, where we conduct experiments considering only Softmax loss for fairness. The results are reported in Table 5 , where we find that it fails to improve neither accuracy nor feature norm by simply increasing the weight decay or decreasing the weight decay. Thus simply changing the weight decay leads to either underfitting or overfitting.

4.6. EFFECTS ON WEIGHTS' DISTRIBUTION:

To avoid overfitting, we consider weight decay in all experiments. However, one main concern is the side effect of feature incay may increase the magnitude of the shallow layers. We plot the weights' distributions of initial state, difference choices of weight decay within Softmax and RN + Softmax in Figure 7 (refer to supplementary for details of the network architectures settings). It is interesting to find that the weights of the first convolution layer(Conv0) becomes even sparser with feature incay. Overall, the influence of feature incay is limited due to the constraint from weight decay. So the feature incay enlarges the feature norm without harming the magnitude of weights from previous layers and will not lead to overfitting. Besides, we can also observe that different weight decay has big impact on the final weight distribution, such as larger weigth decay(e.g., µ = 0.005) results in more weights are constrained to zero, which may lead to underfitting according to their final classification performances. More details are illustrated in Figure 7 .

4.7. RESULT ANALYSIS:

By analyzing the features' L 2 -norm and classification accuracy on CIFAR10 for data of each class, we investigate where the concrete improvements come from. 

6.2. FEATURE NORM WITHIN SOFTMAX LOSS:

Here we mainly want to present you that the feature norm of correctly classifed examples fails to be increased within softmax loss, while feature incay can increase the feature norm consistently.

Property 4 [Feature Norm in Softmax Loss]For any feature vector f i , if P i yi → 1 and P i j → 0(∀j = y i ), then

∂L softmax ∂fi → 0.

Proof. According to definition of softmax loss in Eq.( 3), the gradient of feature vector f i is:

EQUATION

where the P

i j = e w T j f i K k=1 e w T k f i . When P i yi → 1 and P i j → 0(∀j = y i ), ∂Lsoftmax ∂fi → 0. That is

after a training sample is confidently classified correctly, it will have no contribution to its own representation learning. For example, suppose w j = 1, θ wy i ,fi = 0 and θ wj ,fi ≥ 90 • (∀j = y i ) according the Lemma, then w yi f i cos(θ wy i ,fi ) = f i , w j f i cos(θ wj ,fi ) < 0, ∀j = y i .

Putting together, we have P i yi ≥ e f i e f i +K-1 , for a modest number of categories say K = 10, P i yi > 0.999 when f i = 10.

6.3. FUNCTIONAL FORMAT OF FEATURE INCAY:

The choice of the functional format of feature incay is important. Here we mainly analyze three different choices. 

• Linear. F( f 2 ) = -f 2 ∂F ∂f = -2f • Log. F( f 2 ) = -log( f 2 ) ∂F ∂f = -2f f 2 • Reciprocal. F( f 2 ) = 1 f 2 ∂F ∂f = -2f f 2 f 2

Here we mainly analyze the differences of the above three functions by investigating the relationship between the gradients and the original feature norm. Assuming that we have two features f 1 and f 2 satisfying f 2 > f 1 . For the linear function case, we have ∂F ∂f2 > ∂F ∂f1

. Then, the intraclass variance will be increased as the distance between f 1 and f 2 increases with each gradient update. Besides, the gradients of the linear function have the same magnitude with the feature itself, such large gradients update can lead to explosion during the training. For the log function case, we have ∂F ∂f = 2f f 2 = 2u f , where u is the unit vector with feature norm equals 1. Thus

∂F ∂f1:

> ∂F ∂f2

. The gradients within reciprocal function is also that ∂F ∂f1 > ∂F ∂f2 always holds once f 2 > f 1 . Although both log function and reciprocal function increase the features with small feature norm faster than the features with larger feature norm, we find the performance of reciprocal function is better. In summary, reciprocal function can increase the overall feature norm and increase the intra-class similarity simultaneously, where the intra-class similarity along the direction of the weight vectors can be decreased with the log function. We choose the reciprocal function in all of our experiments.

Table 8 reports the classification accuracies adopting the three different considered feature incay. The superiority of Softmax + RN over Softmax + LN and Softmax + LogN is well illustrated, where Softmax + RN can achieve better intra-class similarity according to the above analysis.

6.4. CNN ARCHITECTURES SETTINGS:

For LFW, we adopt 20-layer/64-layer SphereNet following the same settings in Liu et al. (2017a) . We modify the network settings for MNIST/CIFAR10/CIFAR100 based on the previous work (Liu et al. (2016) ) and list them in Table 7 .

6.5. EXPERIMENTAL ANALYSIS:

Figure 8 (a), 8(b) and 8(c) show the accuracy, average feature norm and softmax loss during training by using λ = 1, 0.1, 0.01 respectively on CIFAR10. Feature incay achieves better or comparable accuracy compared with softmax loss under a wide range of λ, and results in larger feature norm on both training and test set. All methods achieve close to zero softmax loss on training set, while feature incay ensures lower softmax loss on test set. Figure 8(d) shows the accuracy vs iteration number on CIFAR100, which is similar to the results on CIFAR10. Chen-Yu Lee, Patrick W Gallagher, and Zhuowen Tu. Generalizing pooling functions in convolutional neural networks: Mixed, gated, and tree. In International conference on artificial intelligence and statistics, 2016.

:

We also plot the histograms of average L 2 -norm and accuracy for Softmax and Softmax + RN. The details are illustrated in Figure6. With feature incay, the feature norm is enlarged and the accuracy is on all ten classes.5 CONCLUSIONSIn this paper, we propose the feature incay implemented as Reciprocal Norm Loss to increase the feature norm. Based on the theoretical analysis of the feature norm, the Reciprocal Norm Loss enlarges the probability margin among different categories and up-weight the contribution from hard examples by managing the feature norm of both the correctly-classified and mis-classified feature vectors. Extensive experiments on MNIST, CIFAR10, CIFAR100 and LFW verify the effectiveness of our method.

Figures and Tables:

Figure 1: Test accuracy and number of samples vs Features' L2-norm on CIFAR10 test set, and the model is trained with softmax loss. The test accuracy increases monotonically with the feature norm. e.g., the test accuracy reaches 100% for samples with feature norm exceeding 400.

MATTERSHere we will illustrate how feature norm influences the softmax loss from two aspects: (1) increasing the feature norm of correctly-classified examples enlarges the probability margin among different categories. (2) increasing the feature norm of mis-classified examples can up-weight the gradients of weight vectors, especially the weight vectors of the correct category and mis-classified category. The first property is similar to the proposition proved by Wang et al. (2017), which states that softmax loss always encourages features of the correctly-classified examples to have larger magnitudes. Property 1 [Feature Norm Matters for Correctly-classified Examples] Suppose weight vectors and directions of the feature vectors are fixed, increasing of the feature norm of correctly-classified examples can enlarge the probability margin among different categories.

Figure 3: Distribution of features' norm over correctly-classified / mis-classified examples.

Figure 4: The original data distribution is on the left of the black dashed line and the data distribution updated according to the Reciprocal Norm Loss is on the right. The numbers 1-6 represent that the points are of increasing feature norm. The black point represents the original point. The lengths of the green bidirectional arrows represent the maximal distance in the direction of the weight vectors within all the points of one class. The purple bidirectional arrow means the minimal distance to origin, which is equal to the minimal feature norm. The red arrows represent the gradients update along the directions of the weight vectors computed with the Reciprocal Norm Loss, while the lengths represent the magnitude of the gradients.

Figure 6: Histograms of Average L2-norm and Accuracy on CIFAR10, (a) the feature norm is increased over all the classes. e.g., the feature norm increases from 230 to 282 for class airplane. (b) the test accuracy is boosted over all the classes. e.g., the test accuracy increases from 95.4 to 96.0 for class ship.

Figure 7: Histograms for Weights' distribution of different layers from model trained on CIFAR10. Here we consider five methods: (1) Weights' distribution after Initialization. (2) Weights' distribution after trained with Softmax loss where the weight decay chooses µ 0.005(classification accuracy is 91.16%). (3) Weights' distribution after trained with Softmax loss where the weight decay chooses µ = 0.00005(classification accuracy is 89.35%). (4) Weights' distribution after trained with Softmax loss where the weight decay chooses µ = 0.0005(classification accuracy is 91.40%). (5) Weights' distribution after trained with RN + Softmax(classification accuracy is 92.16%). The magnitude of the weight parameters is only slightly influenced by the feature incay. Besides, Softmax µ = 0.005 represents larger weigth decay while Softmax µ = 0.00005 represents smaller weight decay compared with the standard settings.(e.g., µ = 0.0005). The weights parameters are very sparse within Softmax µ = 0.005 while very dense within Softmax µ = 0.00005, which induce either underfitting or overfitting.

Figure 8: (a) Accuracy versus iterations with different choices of λ value on the test set of CIFAR10. The RN Softmax achieves 92.04% when λ = 0.1 (b) The training/testing sets' L2 norm vs iterations with different choices of the λ value on CIFAR10. The RN Softmax with different λ all achieve larger L2-norm. (c) The training/testing sets' loss vs iterations with different choices of the λ value on CIFAR10. The RN Softmax achieves notable smaller loss value 0.1432 than the Softmax with 0.1498. The training loss is very small for all the methods, but RN + Softmax has significantly smaller testing loss.(It is best viewed by zooming the figure.) (d) Accuracy vs iterations with Softmax/RN Softmax/L-Softmax/RN L-Softmax on CIFAR100. Both RN Softmax and RN + L-Softmax achieve better performance compared with baseline, where the best method is RN + L-Softmax.

Error Rates (%) on MNIST/CIFAR10/CIFAR100. GEOMETRIC INTERPRETATION OF RECIPROCAL NORM LOSS As stated in the previous subsection, Reciprocal Norm Loss can increase feature norm adaptively to decrease the intra-class variance. Here we prove that there exists an upper bound for the ratio between the feature norm's upper bound and the feature norm's lower bound.

Table content: <table><tr><td>Method</td><td colspan="3">MNIST CIFAR10 CIFAR100</td></tr><tr><td>CNN(Jarrett et al. (2009))</td><td>0.53</td><td>N/A</td><td>N/A</td></tr><tr><td>DropConnect(Wan et al. (2013))</td><td>0.57</td><td>9.41</td><td>N/A</td></tr><tr><td>FitNet(Romero et al. (2014))</td><td>0.51</td><td>N/A</td><td>35.04</td></tr><tr><td>NiN(Lin et al. (2013))</td><td>0.47</td><td>10.47</td><td>35.68</td></tr><tr><td>Maxout(Goodfellow et al. (2013))</td><td>0.45</td><td>11.68</td><td>38.57</td></tr><tr><td>DSN(Lee et al. (2015))</td><td>0.39</td><td>9.69</td><td>34.57</td></tr><tr><td>R-CNN(Liang &amp; Hu (2015))</td><td>0.31</td><td>8.69</td><td>31.75</td></tr><tr><td>GenPool(Lee et al. (2016))</td><td>0.31</td><td>7.62</td><td>32.37</td></tr><tr><td>Hinge Loss(Liu et al. (2016))</td><td>0.47</td><td>9.91</td><td>33.10</td></tr><tr><td>Softmax(Liu et al. (2016))</td><td>0.40</td><td>9.05</td><td>32.74</td></tr><tr><td>L-Softmax(Liu et al. (2016))</td><td>0.31</td><td>7.58</td><td>29.53</td></tr><tr><td>Softmax</td><td>0.35</td><td>8.59</td><td>32.36</td></tr><tr><td>RN + Softmax</td><td>0.31</td><td>7.84</td><td>31.76</td></tr><tr><td>L-Softmax</td><td>0.25</td><td>7.56</td><td>29.95</td></tr><tr><td>RN + L-Softmax</td><td>0.29</td><td>7.22</td><td>29.18</td></tr><tr><td>3.4</td><td/><td/><td/></tr></table>

Face verification accuracy (%) on LFW.

Table content: <table><tr><td>Method</td><td/><td>Data</td><td colspan="2">Network</td><td>mAcc</td></tr><tr><td colspan="2">FaceNet(Schroff et al. (2015))</td><td>200M</td><td colspan="2">N/A</td><td>99.65</td></tr><tr><td>DeepID2(Sun et al. (2015))</td><td/><td>300K</td><td colspan="2">N/A</td><td>99.47</td></tr><tr><td colspan="2">CenterFace(Wen et al. (2016))</td><td>700K</td><td colspan="2">N/A</td><td>99.28</td></tr><tr><td>L-Softmax(Liu et al. (2016))</td><td colspan="5">CASIA-WebFace SphereNet-64 99.10</td></tr><tr><td colspan="6">A-Softmax(Liu et al. (2017a)) CASIA-WebFace SphereNet-20 99.26</td></tr><tr><td colspan="6">A-Softmax(Liu et al. (2017a)) CASIA-WebFace SphereNet-64 99.42</td></tr><tr><td>COCO(Liu et al. (2017c))</td><td/><td>MS-1M</td><td colspan="2">ResNet-101</td><td>99.78</td></tr><tr><td>COCO</td><td colspan="5">CASIA-WebFace SphereNet-20 98.90</td></tr><tr><td>RN + COCO</td><td colspan="5">CASIA-WebFace SphereNet-20 99.02</td></tr><tr><td>L-Softmax</td><td colspan="5">CASIA-WebFace SphereNet-20 99.03</td></tr><tr><td>RN + L-Softmax</td><td colspan="5">CASIA-WebFace SphereNet-20 99.18</td></tr><tr><td>A-Softmax</td><td colspan="5">CASIA-WebFace SphereNet-64 99.42</td></tr><tr><td>RN + A-Softmax</td><td colspan="5">CASIA-WebFace SphereNet-64 99.47</td></tr><tr><td colspan="4">Table 3: Accuracy(%) Comparison of different λ.</td><td/></tr><tr><td>Method</td><td colspan="4">λ = 1 λ = 0.1 λ = 0.01 λ = 0</td></tr><tr><td>RN + Softmax</td><td>91.68</td><td>92.16</td><td>91.96</td><td>91.41</td></tr><tr><td colspan="2">RN + L-Softmax 92.40</td><td>92.78</td><td>92.65</td><td>92.44</td></tr><tr><td colspan="6">and CIFAR100 contain 50,000 training images and 10,000 test images from 10 object categories</td></tr></table>

Table1reports the error rates of compared approaches and our method on MNIST, CIFAR10 and CIFAR100. It can be concluded that feature incay can consistently improve over Softmax and L-Softmax on CIFAR10/CIFAR100. For example, RN decreases the error rate of Softmax from 8.59% to 7.84% while L-Softmax achieves 7.56% on CIFAR10, which demonstrates both of them are better than Softmax. By combining RN and L-Sofmax, we achieve better result 7.22%, which means that they are reciprocal. However, RN + L-Softmax is slightly worse than L-Softmax on MNIST, our hypothesis is that the performance on MNIST is already saturate and difficult to improve further.To further verify our method's effectiveness on more challenging datasets, we test RN + COCO, RN + L-Softmax and RN + A-Softmax on LFW and achieve competitive performance with SphereNet-20 or SphereNet-64. The results are illustrated in Table2. The reproduced results with L-Softmax and A-Softmax are comparable while the reproduced COCO result is not as good due to both the training dataset and network structure are set different. With feature incay, RN + L-Softmax im-Accuracy(%) Comparison by Scaling the Feature.

Table content: <table><tr><td>Scale</td><td>1</td><td>2</td><td>4</td><td>6</td><td>8</td><td>&gt; 10</td></tr><tr><td colspan="7">Softmax 91.41 90.90 90.18 90.91 90.97 NAN</td></tr></table>

Accuracy(%) Comparison of different µ on CIFAR10. µ = 0.0005 is the best choice among all of them, thus we choose this setting in all the other experiments.

Table content: <table><tr><td>Method</td><td colspan="4">µ = 0.00001 µ = 0.00005 µ = 0.0005 µ = 0.005</td></tr><tr><td>Accuracy</td><td>89.21</td><td>89.35</td><td>91.41</td><td>91.16</td></tr><tr><td>Average Feature Norm</td><td>185.8</td><td>202.2</td><td>246.2</td><td>231.3</td></tr><tr><td>4.3 EFFECTS OF λ</td><td/><td/><td/><td/></tr></table>

Average L2-norm and the corresponding number of examples on test set of CIFAR10, e.g., 253.2 / 9141 represents 9141 examples are correctly classified and their average L2-norm is 253.2. Error-fixed represents the examples mis-classified by Softmax but correctly-classified by RN + Softmax. Error-added represents the examples correctly-classified by Softmax but mis-classified by RN + Softmax.

Table content: <table><tr><td>Method</td><td colspan="4">Accuracy correctly-classified Mis-classified Error-fixed Error-added</td></tr><tr><td>Softmax</td><td>91.41</td><td>253.2 / 9141</td><td>167.5 / 859</td><td>169.3 / 336 161.6 / 261</td></tr><tr><td>RN + Softmax</td><td>92.16</td><td>308.3 / 9216</td><td>187.2 / 784</td><td>195.9 / 336 187.3 / 261</td></tr></table>

Table 6 reports the related details and we find 336 examples that are mis-classified by Softmax but correctly classified by RN + Softmax. However, 261 mis-classified examples are further introduced by RN + Softmax, which limits the final performance improvement.

Table content: <table><tr><td/><td>0.14 0.16 0.18 0.20</td><td/><td colspan="5">Conv0 (accuracy) Softmax =0.005 (91.16) Initialization Softmax =0.00005 (89.35) Softmax (91.40) RN + Softmax (92.16)</td><td/><td/><td/><td/><td>0.14 0.16 0.18 0.20</td><td colspan="2">Conv1 (accuracy) Softmax =0.005 (91.16) Initialization Softmax =0.00005 (89.35) Softmax (91.40) RN + Softmax (92.16)</td><td>0.08 0.10</td><td>Conv2 (accuracy) Softmax =0.005 (91.16) Initialization Softmax =0.00005 (89.35) Softmax (91.40) RN + Softmax (92.16)</td></tr><tr><td>probability</td><td>0.08 0.10 0.12</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>probability</td><td>0.08 0.10 0.12</td><td/><td>probability</td><td>0.04 0.06</td></tr><tr><td/><td>0.06</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.06</td><td/></tr><tr><td/><td>0.04</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.04</td><td/><td>0.02</td></tr><tr><td/><td>0.02</td><td/><td/><td/><td/><td/><td/><td/><td/><td/><td/><td>0.02</td><td/></tr><tr><td/><td>0</td><td>0.4</td><td>0.3</td><td>0.2</td><td>0.1</td><td colspan="2">0.0 weight</td><td>0.1</td><td>0.2</td><td>0.3</td><td>0.4</td><td>0</td><td colspan="2">0.08 0.06 0.04 0.02 0.00 0.02 0.04 0.06 0.08 weight</td><td>0.00</td><td>weight 0.04 0.03 0.02 0.01 0.00 0.01 0.02 0.03 0.04</td></tr><tr><td/><td/><td/><td/><td colspan="5">(a) Conv0.x</td><td/><td/><td/><td/><td colspan="2">(b) Conv1.x</td><td>(c) Conv2.x</td></tr><tr><td/><td/><td/><td/><td/><td>probability</td><td>0.04 0.06 0.08 0.10</td><td/><td colspan="4">Conv3 (accuracy) (92.16) (91.40) Softmax =0.00005 (89.35) Initialization Softmax RN + Softmax Softmax =0.005 (91.16)</td><td/><td>probability</td><td>0.04 0.06 0.08 0.10</td><td>Initialization Softmax =0.005 (91.16) (accuracy) Softmax =0.00005 (89.35) Softmax (91.40) RN + Softmax (92.16)</td><td>FC</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.02</td><td/><td/><td/><td/><td/><td/><td/><td>0.02</td></tr><tr><td/><td/><td/><td/><td/><td/><td>0.00</td><td colspan="7">0.04 0.03 0.02 0.01 0.00 0.01 0.02 0.03 0.04 weight</td><td>0.00</td><td>weight 0.04 0.03 0.02 0.01 0.00 0.01 0.02 0.03 0.04</td></tr><tr><td/><td/><td/><td/><td/><td/><td/><td/><td/><td colspan="3">(d) Conv3.x</td><td/><td/><td>(e) FC</td></tr></table>

Results of comparison experiments for three feature incay on CIFAR10, where LN represents the linear form, LogN represents the log form and RN represents the reciprocal form.

Table content: <table><tr><td>Method</td><td colspan="4">Softmax Softmax + LN Softmax + LogN Softmax + RN</td></tr><tr><td>Accuracy</td><td>91.41</td><td>91.86</td><td>91.83</td><td>92.16</td></tr></table>

Kevin Jarrett, Koray Kavukcuoglu, Yann LeCun, et al. What is the best multi-stage architecture for object recognition? In Computer Vision, 2009 IEEE 12th International Conference on, pp. 2146-2153. IEEE, 2009. Yangqing Jia, Evan Shelhamer, Jeff Donahue, Sergey Karayev, Jonathan Long, Ross Girshick, Sergio Guadarrama, and Trevor Darrell. Caffe: Convolutional architecture for fast feature embedding. In Proceedings of the 22nd ACM international conference on Multimedia, pp. 675-678. ACM, 2014. Chen-Yu Lee, Saining Xie, Patrick Gallagher, Zhengyou Zhang, and Zhuowen Tu. Deeplysupervised nets. In Artificial Intelligence and Statistics, pp. 562-570, 2015.

Table content: <table/>

Peer review (iclr2018):
 The manuscript proposes to increase the norm of the last hidden layer to promote better classification accuracy. However, the motivation is a bit less convincing. Here are a few motivations that are mentioned. (1) Increasing the feature norm of correctly classified examples helps cross entropy, which is of course correct. However, it only decreases the training loss. How do we know it will not lead to overfitting? (2) Increasing the feature norm of mis-classified examples will make gradient larger for self-correction. And the manuscript proves it in property 2. However, the proof seems not complete. In Eq (7), increasing the feature norm would also affect the value of the term in parenthesis. As an example, if a negative example is already mis-classified as a positive, and its current probability is very close to 1, then further increasing feature norm would make the probability even closer to 1, leading to saturation and smaller gradient. (3) Figure 1 shows that examples with larger feature norm tend to be predicted well. However, it is not very convincing since it is only a correlation rather than causality. Let*s use simple linear softmax regression as a sanity check, where features to softmax are real features rather than hidden units. Increasing the feature norm seems to be against the best practice of feature normalization in which each feature after normalization is of variance 1. The manuscript states that the feature norm won*t be infinitely increased since there is an upper bound. However, the proof of property 3 seems to only apply to the certain cases where K<2D. In addition, alpha is in the formula of upper bound, but what is the upper bound of alpha? The manuscript does comprehensive experiments to test the proposed method. The results are good, since the proposed method outperforms other baselines in most datasets. But the results are not impressively strong. Minor issues: (1) For proof of property 3, it seems that alpha and beta are used before defined. Are they the radius of two circles?