Redesigning the Classification Layer by Randomizing the Class Representation Vectors

1 INTRODUCTION . Deep learning models achieved breakthroughs in classification tasks , allowing setting state-of-theart results in various fields such as speech recognition ( Chiu et al. , 2018 ) , natural language processing ( Vaswani et al. , 2017 ) , and computer vision ( Huang et al. , 2017 ) . In image classification task , the most common approach of training the models is as follows : first , a convolutional neural network ( CNN ) is used to extract a representative vector , denoted here as image representation vector ( also known as the feature vector ) . Then , at the classification layer , this vector is projected onto a set of weight vectors of the different target classes to create the class scores , as depicted in Fig . 1 . Last , a softmax function is applied to normalize the class scores . During training , the parameters of both the CNN and the classification layer are updated to minimize the cross-entropy loss . We refer to this procedure as the dot-product maximization approach since such training ends up maximizing the dot-product between the image representation vector and the target weight vector . Recently , it was demonstrated that despite the excellent performance of the dot-product maximization approach , it does not necessarily encourage discriminative learning of features , nor does it enforce the intra-class compactness and inter-class separability ( Liu et al. , 2016 ; Wang et al. , 2017 ; Liu et al. , 2017 ) . The intra-class compactness indicates how close image representations from the same class relate to each other , whereas the inter-class separability indicates how far away image representations from different classes are . Several works have proposed different approaches to address these caveats ( Liu et al. , 2016 ; 2017 ; Wang et al. , 2017 ; 2018b ; a ) . One of the most effective yet most straightforward solutions that were proposed is NormFace ( Wang et al. , 2017 ) , where it was suggested to maximize the cosine-similarity between vectors by normalizing both the image and class vectors . However , the authors found when minimizing the cosine-similarity directly , the models fail to converge , and hypothesized that the cause is due to the bounded range of the logits vector . To allow convergence , the authors added a scaling factor to multiply the logits vector . This approach has been widely adopted by multiple works ( Wang et al. , 2018b ; Wojke & Bewley , 2018 ; Deng et al. , 2019 ; Wang et al. , 2018a ; Fan et al. , 2019 ) . Here we will refer to this approach as the cosine-similarity maximization approach . This paper is focused on redesigning the classification layer , and the its role while kept fixed during training . We show that the visual similarity between classes is implicitly captured by the class vectors when they are learned by maximizing either the dot-product or the cosine-similarity between the image representation vector and the class vectors . Then we show that the class vectors of visually similar categories are close in their angle in the space . We investigate the effects of excluding the class vectors from training and simply drawing them randomly distributed over a hypersphere . We demonstrate that this process , which eliminates the visual similarities from the classification layer , boosts accuracy , and improves the inter-class separability ( using either dot-product maximization or cosine-similarity maximization ) . Moreover , we show that fixing the class representation vectors can solve the issues preventing from some cases to converge ( under the cosine-similarity maximization approach ) , and can further increase the intra-class compactness . Last , we show that the generalization to the learned concepts and robustness to noise are both not influenced by ignoring the visual similarities encoded in the class vectors . Recent work by Hoffer et al . ( 2018 ) , suggested to fix the classification layer to allow increased computational and memory efficiencies . The authors showed that the performance of models with fixed classification layer are on par or slightly drop ( up to 0.5 % in absolute accuracy ) when compared to models with non-fixed classification layer . However , this technique allows substantial reduction in the number of learned parameters . In the paper , the authors compared the performance of dot-product maximization models with a non-fixed classification layer against the performance of cosine-similarity maximization models with a fixed classification layer and integrated scaling factor . Such comparison might not indicate the benefits of fixing the classification layer , since the dotproduct maximization is linear with respect to the image representation while the cosine-similarity maximization is not . On the other hand , in our paper , we compare fixed and non-fixed dot-product maximization models as well as fixed and non-fixed cosine-maximization models , and show that by fixing the classification layer the accuracy might boost by up to 4 % in absolute accuracy . Moreover , while cosine-maximization models were suggested to improve the intra-class compactness , we reveal that by integrating a scaling factor to multiply the logits , the intra-class compactness is decreased . We demonstrate that by fixing the classification layer in cosine-maximization models , the models can converge and achieve a high performance without the scaling factor , and significantly improve their intra-class compactness . The outline of this paper is as follows . In Sections 2 and 3 , we formulate dot-product and cosinesimilarity maximization models , respectively , and analyze the effects of fixing the class vectors . In Section 4 , we describe the training procedure , compare the learning dynamics , and asses the generalization and robustness to corruptions of the evaluated models . We conclude the paper in Section 5 . 2 FIXED DOT-PRODUCT MAXIMIZATION . Assume an image classification task with m possible classes . Denote the training set of N examples by S = { ( xi , yi ) } Ni=1 , where xi ∈ X is the i-th instance , and yi is the corresponding class such that yi ∈ { 1 , ... , m } . In image classification a dot-product maximization model consists of two parts . The first is the image encoder , denoted as fθ : X → Rd , which is responsible for representing the input image as a d-dimensional vector , fθ ( x ) ∈ Rd , where θ is a set of learnable parameters . The second part of the model is the classification layer , which is composed of learnable parameters denoted as W ∈ Rm×d . Matrix W can be viewed as m vectors , w1 , . . . , wm , where each vector wi ∈ Rd can be considered as the representation vector associated with the i-th class . For simplicity , we omitted the bias terms and assumed they can be included in W . A consideration that is taken when designing the classification layer is choosing the operation applied between the matrix W and the image representation vector fθ ( x ) . Most commonly , a dotproduct operation is used , and the resulting vector is referred to as the logits vector . For training the models , a softmax operation is applied over the logits vector , and the result is given to a crossentropy loss which should be minimized . That is , argmin w1 , ... , wm , θ N∑ i=0 − log e wyi ·fθ ( xi ) ∑m j=1 e wj ·fθ ( xi ) = argmin w1 , ... , wm , θ N∑ i=0 − log e ‖wyi‖ ‖fθ ( xi ) ‖ cos ( αyi ) ∑m j=1 e ‖wj‖ ‖fθ ( xi ) ‖ cos ( αj ) . ( 1 ) The equality holds since wyi ·fθ ( xi ) = ‖wyi‖‖fθ ( xi ) ‖ cos ( αyi ) , where αk is the angle between the vectors wk and fθ ( xi ) . We trained three dot-product maximization models with different known CNN architectures over four datasets , varying in image size and number of classes , as described in detail in Section 4.1 . Since these models optimize the dot-product between the image vector and its corresponding learnable class vectors , we refer to these models as non-fixed dot-product maximization models . Inspecting the matrix W of the trained models reveals that visually similar classes have their corresponding class vectors close in space . On the left panel of Fig . 2 , we plot the cosine-similarity between the class vectors that were learned by the non-fixed model which was trained on the STL10 dataset . It can be seen that the vectors representing vehicles are relatively close to each other , and far away from vectors representing animals . Furthermore , when we inspect the class vectors of non-fixed models trained on CIFAR-100 ( 100 classes ) and Tiny ImageNet ( 200 classes ) , we find even larger similarities between vectors due to the high visual similarities between classes , such as boy and girl or apple and orange . By placing the vectors of visually similar classes close to each other , the inter-class separability is decreased . Moreover , we find a strong spearman correlation between the distance of class vectors and the number of misclassified examples . On the right panel of Fig . 2 , we plot the cosine-similarity between two class vectors , wi and wj , against the number of examples from category i that were wrongly classified as category j . As shown in the figure , as the class vectors are closer in space , the number of misclassifications increases . In STL-10 , CIFAR-10 , CIFAR-100 , and Tiny ImageNet , we find a correlation of 0.82 , 0.77 , 0.61 , and 0.79 , respectively ( note that all possible class pairs were considered in the computation of the correlation ) . These findings reveal that as two class vectors are closer in space , the confusion between the two corresponding classes increases . We examined whether the models benefit from the high angular similarities between the vectors . We trained the same models , but instead of learning the class vectors , we drew them randomly , normalized them ( ‖wj‖ = 1 ) , and kept them fixed during training . We refer to these models as the fixed dot-product maximization models . Since the target vectors are initialized randomly , the cosine-similarity between vectors is low even for visually similar classes . See the middle panel of Fig . 2 . Notice that by fixing the class vectors and bias term during training , the model can minimize the loss in Eq . 1 only by optimizing the vector fθ ( xi ) . It can be seen that by fixing the class vectors , the prediction is influenced mainly by the angle between fθ and the fixed wyi since the magnitude of fθ ( xi ) is multiplied with all classes and the magnitude of each class vectors is equal and set to 1 . Thus , the model is forced to optimize the angle of the image vector towards its randomized class vector . Table 1 compares the classification accuracy of models with a fixed and non-fixed classification layer . Results suggest that learning the matrix W during training is not necessarily beneficial , and might reduce accuracy when the number of classes is high , or when the classes are visually close . Additionally , we empirically found that models with fixed class vectors can be trained with higher learning rate , due to space limitation we bring the results in the appendix ( Table 7 , Table 8 , Table 9 ) . By randomly drawing the class vectors , we ignore possible visual similarities between classes and force the models to minimize the loss by increasing the inter-class separability and encoding images from visually similar classes into vectors far in space , see Fig . 3 .

Summary: This paper introduces a new approach to learn a multi-class image classification model by fixing the weights of the classification layer. The authors propose to draw the class vectors randomly and set them as fixed during training instead of training them. They analyze this approach when a model is trained with a categorical cross-entropy and or softmax-cosine loss. The proposed approach is tested on 4 datasets: STL, CIFAR-10, CIFAR-100, TinyImagenet