Cross-Entropy Loss Leads To Poor Margins

1 INTRODUCTION . Training neural networks is challenging and involves making several design choices . Among these are the architecture of the network , the training loss function , the optimization algorithm used for training , and their hyperparameters , such as the learning rate and the batch size . Most of these design choices influence the solution obtained by the training procedure and have been studied in detail ( Kingma & Ba , 2014 ; Hardt et al. , 2015 ; He et al. , 2016 ; Wilson et al. , 2017 ; Nar & Sastry , 2018 ; Smith et al. , 2018 ) . Nevertheless , one choice has been mostly taken for granted when the network is trained for a classification task : the training loss function . Cross-entropy loss function is almost the sole choice for classification tasks in practice . Its prevalent use is backed theoretically by its association with the minimization of the Kullback-Leibler divergence between the empirical distribution of a dataset and the confidence of the classifier for that dataset . Given the particular success of neural networks for classification tasks ( Krizhevsky et al. , 2012 ; Simonyan & Zisserman , 2014 ; He et al. , 2016 ) , there seems to be little motivation to search for alternatives for this loss function , and most of the software developed for neural networks incorporates an efficient implementation for it , thereby facilitating its use . Recently there has been a line of work analyzing the dynamics of training a linear classifier with the cross-entropy loss function ( Soudry et al. , 2018 ; Nacson et al. , 2018a ; b ; Ji & Telgarsky , 2018 ) . They specified the decision boundary that the gradient descent algorithm yields on linearly separable datasets and claimed that this solution achieves the maximum margin.1 However , these claims were observed not to hold in the simple experiments we ran . For example , Figure 1 displays a case where the cross-entropy minimization for a linear classifier leads to a decision boundary which attains an extremely poor margin and is nearly orthogonal to the solution given by the hard-margin support vector machine ( SVM ) . We set out to understand this discrepancy between the claims of the previous works and our observations on the simple experiments . We can summarize our contributions as follows . 1The term “ maximum margin ” is used for ` 2 norm throughout the paper . SVM decision boundary ( largest margin ) Cross-entropy min . decision boundary ( poor margin ) 2 CLASSIFICATION OF TWO POINTS MANIFESTS POOR MARGIN . We start with a simple binary classification problem . Given two points x ∈ Rd and −y ∈ Rd from two different classes , we can find a linear classifier by minimizing the cross-entropy loss function min w∈Rd , b∈R { − log ( 1 e−w > x−b + 1 ) − log ( ew > y−b ew > y−b + 1 ) } , or equivalently , by solving min w̃∈Rd+1 { log ( e−w̃ > x̃ + 1 ) + log ( e−w̃ > ỹ + 1 ) } , ( 1 ) where x̃ = [ x > 1 ] > , −ỹ = [ −y > 1 ] > and w̃ = [ w > b ] > . Unless the two points x and −y are equal , the function ( 1 ) does not attain its minimum at a finite value of w̃ . Consequently , if the gradient descent algorithm is used to minimize ( 1 ) , the iterate at time k , w̃ [ k ] , diverges as k increases . The following theorem characterizes the growth rate of w̃ [ k ] and its direction in the limit by using a continuous-time approximation to the gradient descent algorithm . Theorem 1 . Given two points x ∈ Rd and −y ∈ Rd , let x̃ and −ỹ denote [ x > 1 ] > and [ −y > 1 ] , respectively . Without loss of generality , assume ‖x‖ ≤ ‖y‖ . If the two points are in different classes and we minimize the cross-entropy loss min w̃∈Rd+1 log ( 1 + e−w̃ > x̃ ) + log ( 1 + e−w̃ > ỹ ) by using the continuous-time approximation to the gradient descent algorithm dw̃ dt = x̃ δe−w̃ > x̃ 1 + e−w̃ > x̃ + ỹ δe−w̃ > ỹ 1 + e−w̃ > ỹ with the initialization w̃ ( 0 ) = 0 and the learning rate δ , then lim t→∞ w̃ ( t ) log ( t ) = { σy−σxy σxσy−σ2xy x̃+ σx−σxy σxσy−σ2xy ỹ if σxy < σx , 1 σx x̃ if σxy ≥ σx , ( 2 ) where σx = ‖x̃‖2 , σxy = x̃ > ỹ and σy = ‖ỹ‖2 . Note that first d coordinates of ( 2 ) represent the normal vector of the decision boundary obtained by minimizing the cross-entropy loss ( 1 ) . This vector is different from x + y , which is the direction of the maximum-margin solution given by the SVM . In fact , the direction in ( 2 ) could be almost orthogonal to the SVM solution in certain cases , which implies that the margin between the points and the decision boundary could be much smaller than the optimal value . Corollary 1 describes a subset of these cases . Corollary 1 . Given two points x and −y in Rd , let ψ denote the angle between the solution given by ( 2 ) and the solution given by the SVM , i.e. , ( x+ y ) . If x > y = 1 , then cos2 ψ ≤ 4 2 + σy σx ( 1− 1σx ) , where σx = ‖x‖2+1 and σy = ‖y‖2+1 . Consequently , as ‖x‖/‖y‖ approaches 0 while maintaining the condition x > y = 1 , the angle ψ converges to π/2 . Remark 1 . Corollary 1 shows that if x and−y have disparate norms , the minimization of the crossentropy loss with gradient descent algorithm could lead to a direction which is almost orthogonal to the maximum-margin solution . It may seem like this problem could be avoided with preprocessing the data so as to normalize the data points . However , this approach will not be effective for neural networks : if we consider an L-layer neural network , w > φL−1 ( x ) , and regard the first L− 1 layers , φL−1 ( · ) , as a feature mapping , preprocessing a dataset { xi } i∈I will not produce a normalized set of features { φL−1 ( xi ) } i∈I . Note that we could not normalize { φL−1 ( xi ) } i∈I directly either , since the mapping φL−1 ( · ) evolves during training . Remark 2 . Theorem 1 shows that the norm of w keeps growing unboundedly as the training continues . The same behavior will be observed for larger datasets in the next sections as well . Since the “ confidence ” of the classifier for its prediction at a point x is given by max ( 1 e−w > x−b + 1 , e−w > x−b e−w > x−b + 1 ) , this unbounded growth of ‖w‖ drives the confidence of the classifier to 100 % at every point in the input space , except at the points on the decision boundary , if the algorithm is run for long . Given the lack of effective regularization for neural networks , a similar unbounded growth is expected to be observed in neural network training as well , which is mentioned in ( Bartlett et al. , 2017 ) . As a result , the confidence of a neural network might be highly correlated with the training duration , and whether a neural network gives 99 % or 51 % confidence for a prediction might be of little importance as long as it is above 50 % . In other words , regarding this confidence value as a measure of similarity between an input and the training dataset from the most-likely class should be reconsidered . 3 DATA WITH LOW-DIMENSIONAL FEATURES ATTAIN POOR MARGIN . In this section , we examine the binary classification of a linearly separable dataset by minimizing the cross-entropy loss function . Recently , this problem has also been studied in ( Soudry et al. , 2018 ; Nacson et al. , 2018b ; a ; Ji & Telgarsky , 2018 ) . We restate an edited version of the main theorem of ( Soudry et al. , 2018 ) , followed by the reason of the edition . Theorem 2 ( Adapted from Theorem 3 of ( Soudry et al. , 2018 ) ) . Given two sets of points { xi } i∈I and { −yj } j∈J that are linearly separable in Rd , let x̃i and −ỹj denote [ x > i 1 ] > and [ −y > j 1 ] > , respectively , for all i ∈ I , j ∈ J . Then the iterate of the gradient descent algorithm , w̃ ( t ) , on the cross-entropy loss function min w̃∈Rd+1 ∑ i∈I log ( 1 + e−w̃ > x̃i ) + ∑ j∈J log ( 1 + e−w̃ > ỹj ) ( 3 ) with a sufficiently small step size will converge in direction : lim t→∞ w̃ ( t ) ‖w̃ ( t ) ‖ = w ‖w‖ , where w is the solution to w = argmin u∈Rd+1 ‖u‖2 s.t . 〈u , x̃i〉 ≥ 1 , 〈u , ỹj〉 ≥ 1 ∀i ∈ I , ∀j ∈ J . ( 4 ) The solution ( 4 ) given in Theorem 2 was referred in ( Soudry et al. , 2018 ) , and consequently in the other works , as the maximum-margin solution . However , due to the misleading absence of the bias term in the notation , this is incorrect . Given the linearly separable sets of points { xi } i∈I and { −yj } j∈J , the maximum-margin solution given by the SVM solves minimize w , b ‖w‖22 subject to 〈w , xi〉+ b ≥ 1 ∀i ∈ I , 〈w , −yj〉+ b ≤ −1 ∀j ∈ J . ( P1 ) On the other hand , the solution given by Theorem 2 corresponds to minimize w , b ‖w‖22 + b2 subject to 〈w , xi〉+ b = 〈w̃ , x̃i〉 ≥ 1 ∀i ∈ I , 〈w , −yj〉+ b = 〈w̃ , −ỹj〉 ≤ −1 ∀j ∈ J , ( P2 ) where we define w̃ = [ w > b ] > , x̃i = [ x > i 1 ] > and ỹj = [ y > j − 1 ] > for all i ∈ I , j ∈ J . Even though the sets of constraints for both problems are identical , their objective functions are different , and consequently , the solutions are different . As a result , the decision boundary obtained by crossentropy minimization does not necessarily attain the maximum hard margin . In fact , as the following theorem shows , its margin could be arbitrarily worse than the maximum margin . Theorem 3 . Assume that the points { xi } i∈I and { −yj } j∈J are linearly separable and lie in an affine subspace ; that is , there exist a set of orthonormal vectors { rk } k∈K and a set of scalars { ∆k } k∈K such that 〈rk , xi〉 = 〈rk , −yj〉 = ∆k ∀i ∈ I , ∀j ∈ J , ∀k ∈ K. Let 〈w , ·〉+B = 0 denote the decision boundary obtained by minimizing the cross-entropy loss , i.e . the pair ( w , B ) solves min w , b ‖w‖2 + b2 s.t . 〈w , xi〉+ b ≥ 1 , 〈w , −yj〉+ b ≤ −1 ∀i ∈ I , ∀j ∈ J . Then the minimization of the cross-entropy loss ( 3 ) yields a margin smaller than or equal to 1√ 1 γ2 +B 2 ∑ k∈K ∆ 2 k where γ denotes the optimal hard margin given by the SVM solution . Remark 3 . Theorem 3 shows that if the training points lie in an affine subspace , the margin obtained by the cross-entropy minimization will be smaller than the optimal margin value . As the dimension of this affine subspace decreases , the cardinality of the setK increases and the term ∑ k∈K ∆ 2 k could become much larger than 1/γ2 . Therefore , as the dimension of the subspace containing the training points gets smaller compared to the dimension of the input space , cross-entropy minimization with a gradient method becomes more likely to yield a poor margin . Note that this argument also holds for classifiers of the form w > φ ( x ) with the fixed feature mapping φ ( · ) . The next theorem relaxes the condition of Theorem 3 and allows the training points to be near an affine subspace instead of being exactly on it . Note that the ability to compare the margin obtained by cross-entropy minimization with the optimal value is lost . Nevertheless , it highlights the fact that same set of points could be assigned a different margin by cross-entropy minimization if all of them are shifted away from the origin by the same amount in the same direction . Theorem 4 . Assume that the points { xi } i∈I and { −yj } j∈J in Rd are linearly separable and there exist a set of orthonormal vectors { rk } k∈K and a set of scalars { ∆k } k∈K such that 〈rk , xi〉 ≥ ∆k , 〈rk , −yj〉 ≤ ∆k ∀i ∈ I , ∀j ∈ J , ∀k ∈ K. Let 〈w , ·〉+B = 0 denote the decision boundary obtained by minimizing the cross-entropy loss , i.e . the pair ( w , B ) solves min w , b ‖w‖2 + b2 s.t . 〈w , xi〉+ b ≥ 1 , 〈w , −yj〉+ b ≤ −1 ∀i ∈ I , ∀j ∈ J . Then the minimization of the cross-entropy loss ( 3 ) yields a margin smaller than or equal to 1√ B2 ∑ k∈K ∆ 2 k Remark 4 . Both Theorem 3 and Theorem 4 consider linearly separable datasets . If the dataset is not linearly separable , ( Ji & Telgarsky , 2018 ) predicts that the normal vector of the decision boundary , w , will have two components , one of which converges to a finite vector and the other diverges . The diverging component still has the potential to drive the decision boundary to a direction with a poor margin . In fact , the margin is expected to be small especially if the points intruding into the opposite class lie in the same subspace as the optimal normal vector for the decision boundary . In this work , we focus on the case of separable datasets as this case provides critical insight into the issues of state-of-the-art neural networks , given they can easily attain zero training error even on randomly generated datasets , which indicates the linear separability of the features obtained at their penultimate layers ( Zhang et al. , 2017 ) .

Summary: This paper investigates the properties of minimizing cross-entropy of linear functions over separable data (looks like logistic loss). The authors show a simple example where the minimizer of the cross-entropy loss leads to maximum margin hyperplane where the bias term is regarded as an extra dimension, which is different from the standard max. margin solution of  SVMs with bias not regarded as an extra dimension. The authors then propose a method to obtain the latter solution by minimizing the cross-entropy loss.