Redesigning the Classification Layer by Randomizing the Class Representation Vectors
Authors: Gabi Shalev
Gal Lev Shalev
Yossi Keshet
Venue:  (2021)

Abstract: Neural image classification models typically consist of two components. The first is an image encoder, which is responsible for encoding a given raw image into a representative vector. The second is the classification component, which is often implemented by projecting the representative vector onto target class vectors. The target class vectors, along with the rest of the model parameters, are estimated so as to minimize the loss function. 

In this paper, we analyze how simple design choices for the classification layer affect the learning dynamics. We show that the standard cross-entropy training implicitly captures visual similarities between different classes, which might deteriorate accuracy or even prevents some models from converging. We propose to draw the class vectors randomly and set them as fixed during training, thus invalidating the visual similarities encoded in these vectors. We analyze the effects of keeping the class vectors fixed and show that it can increase the inter-class separability, intra-class compactness, and the overall model accuracy, while maintaining the robustness to image corruptions and the generalization of the learned concepts.

Under review as a conference paper at ICLR 2021
REDESIGNING THE CLASSIFICATION LAYER BY RAN- DOMIZING THE CLASS REPRESENTATION VECTORS
Anonymous authors Paper under double-blind review
ABSTRACT
Neural image classification models typically consist of two components. The first is an image encoder, which responsible for encoding a given raw image into a representative vector. The second is the classification component, which is often implemented by projecting the representative vector onto target class vectors. The target class vectors, along with the rest of the model parameters, are estimated as to minimize the loss function.
In this paper, we analyze how simple design choices for the classification layer affect the learning dynamics. We show that the standard cross-entropy training implicitly captures visual similarities between different classes, which might de- teriorate accuracy or even prevents some models from converging. We propose draw the class vectors randomly and set them as fixed during training, thus inval- idating the visual similarities encoded in these vectors. We analyze the effects of keeping the class vectors fixed and show that it can increase the inter-class separa- bility, intra-class compactness, and the overall model accuracy, while maintaining the robustness to image corruptions and the generalization of the learned concepts.
INTRODUCTION
Deep learning models achieved breakthroughs in classification tasks, allowing setting state-of-ther art results in various fields such as speech recognition (Chiu et al. 2018), natural language process- ing (Vaswani et al. 2017), J, and computer vision (Huang et al. 2017). >. image classification task, the most common approach of training the models is as follows: first, convolutional neural network (CNN) used to extract a representative vector, denoted here as image representation vector (also known the feature vector). Then, at the classification layer, this vector is projected onto set of weight vectors of the different target classes to create the class scores, as depicted in Fig. [ Last, softmax function is applied to normalize the class scores. During training, the parameters of both the CNN and the classification layer are updated to minimize the cross-entropy loss. We refer to this procedure the dot-product maximization approach since such training ends up maximizing the dot-product between the image representation vector and the target weight vector.
Image representation
Class representation vectors
Logits vector
CNN encoder
fo (x:)) E R2d
W E R
E R
Input Image
Encoder
#;
Figure 1: A scheme of an image classification model with three target classes. Edges from the same color compose a class representation vector.
Recently, it was demonstrated that despite the excellent performance of the dot-product maximiza- tion approach, it does not necessarily encourage discriminative learning of features, nor does it

---

Under review as a conference paper at ICLR 2021
enforce the intra-class compactness and inter-class separabilityy (Liu et al. 2016 Wang al. 2017 Liu et al., 2017). The intra-class compactness indicates how close image representations from the same class relate to each other, whereas the inter-class separability indicates how far away image representations from different classes are.
Several works have proposed different approaches to address these caveats (Liu et al. 2016 2017
Wang et al.] 2017, (0118)a)))) One of the most effective yet most straightforward solutions that were proposed is NormFace (Wang al. 2017), where it was suggested to maximize the sosila-similarity between vectors by normalizing both the image and class vectors. However, the authors found when minimizing the cosine-similarity directly, the models fail converge, and hypothesized that the cause is due to the bounded range of the logits vector. To allow convergence, the authors added a scaling factor to multiply the logits vector. This approach has been widely adopted by multiple works (Wang et al. 2018b; Wojke & Bewley] 2018 Deng et al. [2019] Wang et al. 201 8a Fan et al., 2019). Here we will refer to this approach as the cosite-similarity maximization approach.
This paper is focused on redesigning the classification layer, and the its role while kept fixed during training. We show that the visual similarity between classes is implicitly captured by the class vec- tors when they are learned by maximizing either the dot-product or the cosine-similarig between the image representation vector and the class vectors. Then we show that the class vectors of visually similar categories are close in their angle in the space. We investigate the effects of excluding the class vectors from training and simply drawing them randomly distributed over a hypersphere. We demonstrate that this process, which eliminates the visual similarities from the classification layer, boosts accuracy, and improves the inter-class separability (using either dot-product maximization or cosine-similarity maximization). Moreover, we show that fixing the class representation vectors can solve the issues preventing from some cases to converge (under the cosine-similarity maximization approach), and can further increase the intra-class compactness. Last, we show that the generaliza- tion to the learned concepts and robustness to noise are both not influenced by ignoring the visual similarities encoded in the class vectors.
Recent work by Hoffer et al. (2018), suggested to fix the classification layer to allow increased com- putational and memory efficiencies. The authors showed that the performance of models with fixed classification layer are par or slightly drop (up to 0.5% in absolute accuracy) when compared to models with non-fixed classification layer. However, this technique allows substantial reduc- tion in the number of learned parameters. In the paper, the authors compared the performance of dot pro-product maximization models with a non non-fixed classification layer against the performance of cosine-similarity maximization models with fixed classification layer and integrated scaling factor. Such comparison might not indicate the benefits of fixing the classification layer, since the dot- product maximization is linear with respect to the image representation while the mosine-similarity maximization is not. On the other hand, in our paper, we compare fixed and non-fixed dot-producct maximization models as well as fixed and non-fixed cosine- maximizaaiiiiiionnnn models, and show that by fixing the classification layer the accuracy might boost by up to 4% in absolute accuracy. More- over, while cosine-maximine maximization models were suggested to improve the intra-class compactness, we reveal that by integrating a scaling factor to multiply the logits, the intra-class compactness is decreased. We demonstrate that by fixing the classification layer in cosine-muximzzaion models, the models can converge and achieve a high performance without the scaling factor, and significantly improve their intra-class compactness.
The outline of this paper is as follows. In Sections 2 and 3 we formulate dot-product and cosine- similarity maximization models, respectively, and analyze the effects of fixing the class vectors. In Section 4. we describe the training procedure, compare the learning dynamics, and asses the generalization and robustness to corruptions of the evaluated models. We conclude the paper in Section 5
2 FIXED DOT-PRODUCT MAXIMIZATION
Assume an image classification task with possible classes. Denote the training set of N examples 1N by S = {(x,, J,N, Ji=1- where x; E X is the i-th instance, and y is the corresponding class such that Ya E {1...., m... In image classification a doot-product maximization model consists of two parts. The first is the image encoder, denoted as fo X + Rd,, which is responsible for representing the input image d-dimensional vector, fo(x) E Rdd where 0 is set of learnable parameters. The second
2

---

Under review as a conference paper at ICLR 2021
arplene
0.31
5.33
Brd
Bid
0,2
0.65
0:37
A1L
064
0.13
0.43
4002
BAX
0.38
0/02
0.0
rorley
malkyy
shlp <0.0 0:04 6:02
6-3 Arali
buct
0:2 9:41 0
6 4:33 0:3 4.04 0.04 0.00 0.04 000 0.70 000
cosine-simiorrty
Figure 2: The matrices show the cosine-similaree between the class vectors of non-fixed (left) and fixed (middle) dot-product maximization models trained on STL-10 dataset. Right is plot showing the number of misclassifications as a function the cosine-similarity between class vectors.
part of the model is the classification layer, which is composed of learnable parameters denoted as W E Ruxd Matrix W can be viewed as m vectors, w ,,,,,, wtm where each vector w' e E can be considered the representation vector associated with the i-th class. For simplicity, we omitted the bias terms and assumed they can be included in W.
A consideration that is taken when designing the classification layer is choosing the operation ap- plied between the matrix W and the image representation vector fa(æ). Most commonly, a dot- product operation is used, and the resulting vector is referred as the logits vector. For training the models, a softmax operation is applied over the logits vector, and the result is given to a cross- entropy loss which should be minimized. That is,
N
N (m()))
N
ellw0ii| [olo))
arg min No. 10
log
arg min
log
ΣØ= fo(x))
T
ΣØ= ellw [z. cos(a,))
2=0
wet wn ,0 2m0
The equality holds since wi:, fo(o;) = -1w0h 11 cos(arp)) where ag is the angle between the vectors wh and fo(x:).
We trained three dot-product maximization models with different known CNN architectures over four datasets, varying in image size and number of classes, as described in detail in Section 4.1 Since these models optimize the dot-product between the image vector and its corresponding learn- able class vectors, we refer to these models as non-fixed dot-product maximization models.
Inspecting the matrix W of the trained models reveals that visually similar classes have their cor- responding class vectors close in space. On the left panel of Fig. 2 we plot the fosine-similarity between the class vectors that were learned by the non-fixed model which was trained on the STL- 10 dataset. It can be seen that the vectors representing vehicles are relatively close to each other, and far away from vectors representing animals. Furthermore, when we inspect the class vectors of non-fixed models trained on CIFAR-100 (100 classes) and Tiny ImageNet (200 classes), even larger similarities between vectors due to the high visual similarities between classes, such as boy and girl or apple and orange. By placing the vectors of visually similar classes close to each other, the inter-class separability is decreased. Moreover, we find a strong spearman correlation between the distance of class vectors and the number of misclassified examples. On the right panel of Fig. 2 we plot the rosine-similarity between two class vectors, w² and w², against the number of examples from category i that were wrongly classified as category j. As shown in the figure, as the class vectors are closer in space, the number of misclassification increases. In STL.110, CIFAR-10, CIFAR- 100, and Tiny ImageNet, we find correlation of 0.82, 0.77, 0.61, and 0.79, respectively (note that all possible class pairs were considered in the computation of the correlation). These find- ings reveal that as two class vectors are closer in space, the confusion between the two corresponding classes increases.
find
We examined whether the models benefit from the high angular similarities between the vectors. We trained the same models, but instead of learning the class vectors, drew them randomly, normalized them ([|we) = 1), and kept them fixed during training. We refer to these models as the fixed dot-product maximization models. Since the target vectors initialized randomly, the
3

---

Under review as a conference paper at ICLR 2021
Table 1: Comparison between the classification accuracy of fixed and non-fixed dot-product maxi-
mization models.
PreActResnet18
ResNet18
MobileNetV2
Dataset
Classes
Fixed
Non-Fixed
Fixed
Non-Fixed
Fixed
Non-Fixed
STL (96x96)
10
79.7% %
76.6%
82.5%
78.1%
81.0%
77.2%
CIFAR-I0 (32x32)
10
94.1%
94.3%
94.2%
93.4%
93.5%
93.1%
CIFAR-100 (32x32)
100
75.2%
75.3%
75.9%
74.9%
74.4%
73.7%
Tiny ImageNet (64x64)
200
59.1 %
55.4%
60.4
58.9%
59.4%
57.3%
sisiie-similarity between vectors is low even for visually similar classes. See the middle panel of Fig. Notice that by fixing the class vectors and bias term during training, the model can minimize the loss in Eq. [nnly by optimizing the vector fo (2z). It can be seen that by fixing the class vectors, the prediction is influenced mainly by the angle between fo and the fixed w since the magnitude of (2x4) is multiplied with all classes and the magnitude of each class vectors is equal and set to 1. Thus, the model is forced to optimize the angle of the image vector towards its randomized class vector.
Table [ compares the classification accuracy of models with a fixed and non-fixed classification layer. Results suggest that learning the matrix W during training is not necessarily beneficial, and might reduce accuracy when the number of classes is high, or when the classes are visually close. Additionally, we empirically found that models with fixed class vectors can be trained with higher learning rate, due to space limitation we bring the results in the appendix (Table Table Table 9 By randomly drawing the class vectors, we ignore possible visual similarities between classes and force the models to minimize the loss by increasing the inter-class separability and encoding images from visually similar classes into vectors far space, see Fig.
Monnttxed product (Resnett BJ
Flad product (Resnett
Figure 3:
Feature distribution visualization of non-fixed and fixed dot-product maximization models trained on CIFAR-10.
FIXED
MAXIMIZATION
Recently, coxine-similarity maximization models were proposed by Wang et al. (0177) for face veri- fication task. The authors maximized the cosine-similarity, rather than the dot-product, between the image vector and its corresponding class vector. That is,
cos
N
arg IIII 10 EDITE
log
arg mii
log
171
cos(a,)
fol=s) (1lll1l
0
Comparing the right-hand side of Eq. 2 with Eq. shows that the sismie-iimilarity maximization model simply requires normalizing (z). and each of the class representation vectors w² ..., w by dividing them with their 12-norm during the forward pass. The main motivation for this reformula- tion is the ability to learn more discriminative features in face verification by encouraging intra-clas clars-class compactness and enlarging the clter-class separability. The authors showed that dot- doo-product max- imization models learn radial feature distribution; thus, the inter-class separability and intra-class
4

---

Under review as a conference paper at ICLR 2021
compactness not optimal (for more details, see the discussion in Wang al. (017)) )) However, the authors found that cosine- similarit maximization models as given in Eq fai to converge and added scaling factor S R to multiply the logits vector follows:
S-cos(ay,)
arg min
log Σド=1 (Scoosaay)
This reformulation achieves improved results for face verification task, and many recent alternations also integrated the scaling factor S for convergences when optimizing the cosine-similarity Wang
et al. (2018b); Wojke& & Bewley
Deng et al. (2019); Wang et al. (2018a) : Fan et al. (2019).
According to Wang al. (2017), cosine-similarit maximization models fail to converge when S = 1 due to the low range of the logits vector, where each cell is bounded between [-1, 1]. This low result,
range prevents the predicted probabilities from getting close to I during training, and the distribution over target classes is close to uniform, thus the loss will be trapped at a very high value on the training set. Intuitively, this may sound reasonable explanation as why directly maximizing the cosine-similarity fails to converge (S = 1). Note that even if an example is correctly classified and well separated, in its best scenario, it will achieve cosine-similarity of with its ground-truth class vector, while for other classes, the cosine-similarie would be (-1). Thus, for classification task with m classes, the predicted probability for the example above would be:
P(Y v:(zx)) el + (m e-
(4)
Notice that if the number of classes m = 200, the predicted probability of the correctly classified example would be at most 0.035, and cannot be further optimized to 1. As result, the loss func- tion would yield a high value for a correctly classified example, even if its image vector is placed precisely in the same direction as its ground-truth class vector.
As in the previous section, we trained the same models over the same datasets, but instead of opti- mizing the dot-product, we optimized the cosine-similarity by normalizing fo(3))) and w ..... th at the forward pass. We denote these models as non-fixed cosine-similarity maximization models. Additionally, we trained the same cosine- similarti maximization models with fixed random class vectors, denoting these models as fixed cosine-similarity maximization. In all models (fixed and non-fixed) we set S = 1 directly maximize the similarim results are shown in Table 2
Surprisingly, we reveal that the low range of the logits vector is not the cause preventing from cosine-similarity maximization models from converging. As can be seen the table, fixed malimm-mximzzation models achieve significantly high accuracy results by up to 53% compared to non-fixed models. Moreover, it can be seen that fixed coxini-maiinization models with S can also outperform dot-product maximization models. This finding demonstrates that while the log- its are bounded between [-1, 1], the models can still learn high-quality representations and decision boundaries.
Table 2: Classification accuracy of fixed and non-fixed cosine-similaati maximization models. In all models S 1.
PreActResnet18
ResNet18
MobileNetV2
Dataset
Classes
Fixed
Non-Fixed
Fixed
Non-Fixed
Fixed
Non-Fixed
STL (96x96)
10
83.7%
58.9%
83.2%
51.6%
77.1%
54.1%
CIFAR- 10 (32x32)
10
93.3%
80.1% 1%
93.2%
70.4%
92.2%
57.6%
CIFAR- 100 (32x32)
100
74.6% %
29.9%
74.4%
27.1%
72.6%
19.7%
Tinylmagenet (64x64)
200
50.4%
19.9%
47.6%
16.7%
41.6%
13.2%
We further investigated the effects of S and train for comparison the same fixed and non-fixed models, but this time we used grid-search for the best performing value. As can be seen in Table 3 increasing the scaling factor 5 allows non-fixed models to achieve higher accuracies over all datasets. Yet, there is no benefit at learning the class representation vectors instead of randomly drawing them and fixing them during training when considering models' accuracies.
To better understand the cause which prevents non-fixed cosine-mann maximization models from converg- ing when S = 1, we compared these models with the same models trained by setting the optimal
5

---

Under review as a conference paper at ICLR 2021
applina
0.40 3.32
033
227
dog
moneey -6.2
0.3
0.10
0.11 4:00
0.45
Figure 4: The matrices show the cosine-similarity between the class vectors of non-fixed cosine- similarity maximization models, trained on STL-10 dataset, with S 1 (left), and 5 20 (middle). Right is a plot showing the relationship between the number of misclassification as a function of the cosine-similarity between class vectors.
Table 3: Comparison between the classification accuracy of fixed and non-fixed cosine-similante maximization models with their optimal S
PreActResnet18
ResNet18
MobileNetV2
Dataset
Classes
Fixed
Non-Fixed
Fixed
Non-Fixed
Fixed
Non-Fixed
STL (96x96)
10
%3.8%
79.0%
83.9%
79.4%
82.4%
81.5%
CIFAR-10 (32x32)
10
93.5%
93.5% %
93.3 %
93.0%
92.6%
92.8%
CIFAR- (32x32)
100
74.6%
74.8%
74.6%
73.4%
73.7
72.6%
Tinylmagenct (64x64)
200
53.8%
54.5%
54.3%
55.6%
53.9%
53.5%
S scalar. For each model we measured the distance between its learned class vectors and compared these distances to demonstrate the effect of S on them. Interestingly, we found that as increased, the cosine-similan vectors are further apart from each other. Compare, for example, the left and middle panels in Fig. 4 which show the cosine-similarity between the class vectors of models trained on STL with S and S = 20, respectively.
between the class vectors decreased. Meaning that by increasing S the class
On the right panel in Fig. 4 we plot the number of misclassification similarity between the class vectors of the non-fixed cosine- saiine-mximization model trained on STL- 10 with S 1. It can be seen that the confusion between classes is high when they have low angular distance between them. As in previous section, we observed strong correlations between the closeness of the class vectors and the number of misclassification. We found correlations of 0.85, 0.87, 0.81, and 0.83 in models trained on STL-10, CIFAR-10, CIFAR-100, and Tiny ImageNet, respectively.
function of the cosine-
By integrating the scaling factor S in Eq.
4 we get
5.1
P(Y = yl|=;) 6-11 + (m 1 e5((11
(5)
Note that by increasing S. the predicted probability in Eq. 5 increases. This is true even when the cosine-similarity between f(z;) and with is less than 1. When S is set to a large value, the gap between the logits increases, and the predicted probability after the softmax is closer to 1. As a result, the model is discouraged from optimizing the cosine-similarity between the image representation and its ground-truth class vector to be close to since the loss is closer to 0. In Table 4 we show that as we increase S, the cosine-similarity between the image vectors and their predicted class vectors decreases.
These observations can provide an explanation as to why non-fixed models with S = 1 fail to converge. By setting S to large scalar, the image vectors are spread around their class vectors with a larger degree, preventing the class vectors from getting close to each other. As result, the inter- class separability increases and the misclassification rate between visually similar classes decreases. In contrast, setting S = allows models to place the class vectors of visually similar classes closer in space and leads to a high number of misclassification. However, a disadvantage of increasing S and setting it to large number is that the intra-class compactness is violated since image

---

Under review as a conference paper at ICLR 2021
vectors from the class spread and encoded relatively far from each other, see Fig. Fixed cosine-maximan maximization models successfully converge when 5 = 1, since the class vectors are
S 10
5
G
(Poweert))
Figure 5: Feature distribution visualization of fixed and non-fixed cosine- cosine-maximizatee maxemmmzaiiiiinn Resnet18
models trained on CIFAR-10.
initially far in space from each other. By randomly drawing the class vectors, models are required to encode images from visually similar classes into vectors, which are far in space; therefore, the improved since models
inter-class separability is high. Additionally, the intra-class compactness encouraged to maximize the cosine-similarig to 1 as S can be set to 1, and place image vectors from the same class close to their class vector. We validated this empirically by measuring the average cosine-similarity between image vectors and their predicted classes' vectors in fixed cosine- maximization models with S 1. We obtained an average cosine-similarity of roughly 0.95 in all experiments, meaning that images from the same class were encoded compactly near their class vectors.
In conclusion, although non-fixed cosine-similarity maximization models were proposed to improve the caveats of dot-produc, maximization by improving the inter-class separability and intra-class compactness, their performance are significantly low without the integration of scaling factor to multiply the logits vector. Integrating the scaling factor and setting it to S > decrease intra-class compactness and introduce a trade-off between accuracy and intra-class compactness. By fixing the class vectors, similiitt maximization models can have both high performance and improved intra-class compactness. Meaning that multiple previous works (Wang et al. (2018b); Wojke & Bewley (2018); Deng al. (2019); Wang al. (2018a); Fan et al. (2019)) that adopted the cosine- maximization method and integrated a scaling factor for convergence, might benefit from improved results by fixing the class vectors.
Table 4: Average similaasim results between image vectors and their predicted class vectors, when is set 1, 20, and 40. Results from non-fixed cosine-similarity maximization models trained on CIFAR-10 (C-10), CIFAR- 100 (C-100), STI,, and Tiny ImageNet (TT).
S=1
S=20
S=40
C-10
C-100
STL
C-10
C-100
STL
T1
C-10
C-100
STL
TI
0.99
0.99
0.97
0.98
0.88
0.63
0.77
0.62
0.36
0.53
0.21
0.37
GENERALIZATION AND ROBUSTNESS TO CORRUPTIONS
In this section we explore the generalization of the evaluated models to the learned concepts and measure their robustness to image corruptions. We do not aim a state-of-the-an -hr- results but rather validate that by fixing the class vectors of a model, the model's generalization ability and robustness to corruptions remain competitive.
4.1 TRAINING PROCEDURE
To evaluate the impact of ignoring the visual similarities in the classification layer we evaluated the models on CIFAR- 10, CIFAR-100 Krizhevkky et al. (2009), STTI et al. (201T), T), and Tiny Ima- geNe (containing 10, 100, 10, and 200 classes, respectively). For each dataset. we trained Resnet18 He et al. (2016a), PreActResmet1 He et al. (2016b), and MobileNetV2 Sandler et al. (2018) models

---

Under review as a conference paper at ICLR 2021
Table 5: Classification accuracy of fixed and non-fixed models for the generalization sets.
Dot-product
Cosine-similarity
Training set
Evaluating set
Fixed
Non-Fixed
Fixed
Non-Fixed
STL
STL Gen
53.7%
50.9%
54.1%
50.4%
CIFAR-10
CIFAR-I8
87.1 %
86.9%
85.7%
85.9%
CIFAR-100
CIFAR-100 Gen
34.8%
32.9%
35.1%
35.4%
with fixed and non-fixed class vectors. All models were trained using stochastic gradient descent with momentum. We used the standard normalization and data augmentation techniques. Due to space limitations, the values of the hyperparameters used for training the models be found un- der our code repository. We normalized the randomly drawn, fixed class representation vectors by dividing them with their nn-momm. All reported results are the averaged results of runs.
4.2 GENERALIZATION
For measuring how well the models were able to generalize to the learned concepts, evaluated them on images containing objects from the same target classes appearing in their training dataset. For evaluating the models trained on STL-10 and CIFAR-100, we manually collected 2000 and 6000 images respectively, from the publicly available dataset Open Images V4 Krasin et al. (2017). For CIFAR-10 used the CIFAR-I0.1 dataset Recht al. (2018). All collected sets contain equal number images for each class. We omitted models trained on Tiny ImageNet from the evaluation since we were not able collect images for all classes appearing this Table [ summarizes the results for all the models. Results suggest that excluding the class representation vectors from training. does not decrease the generalization learned concepts.
4.3 ROBUSTNESS TO CORRUPTIONS
Next, we verified that excluding the class vectors from training did not decrease the model's 's robust- ness to image corruptions. For this we apply three types of algorithmically generated corruptions on the test set and evaluate the accuracy of the models these sets. The corruptions apply are impulse- noise, JPEG compression, and de-focus blur. Corruptions are generated using Jung (2018), and available under repository. Results, shown in Table suggest that randomly drawn fixed class vectors allow models to be highly robust to image corruptions.
Table 6: Classification accuracy of fixed and non-fixed models on corrupted test set's images.
Corruption
Dot-product
Cosine-similarity
Test set
type
Fixed
Non-fixed
Fixed
Non-fixed
STL
52.9%
49.4%
57.2%
44.9%
Balt-and-peppe
CIFAR-10
52.6%
49.5%
49,8%
52.9%
CIFAR-100
36.1%
36.6%
41.3%
40.9%
Tiny ImageNet
30.8%
26.1%
28.6%
27.9%
STL
78.1%
75.8%
76.1%
73.3%
JPEG compression
CIFAR-10
71.8%
71.5%
71.4%
72.4%
CIFAR-100
43.8%
42.3%
44.4%
43.3%
Tiny ImageNet
38.8%
33.0%
35.1%
34.4%
STL
37.7%
35.0%
39.4%
36.1%
Blurring
CIFAR-10
41.3%
41.1%
41.9%
41.1%
CIFAR-100
24.8%
24.4%
22.4%
23.1%
Tiny ImageNct
16.4%
13.4%
12.9%
12.4%
CONCLUSION
In this paper, we propose randomly drawing the parameters of the classification layer and excluding them from training. We showed that by this, the inter-class separability, intra-class compactness, and the overall accuracy of the model can improve when maximizing the dot- dro-product or the cosine sim- ilarity between the image representation and the class vectors. We analyzed the cause that prevents the non-fixed cosine- maximization models from converging. We also presented the generalization abilities of the fixed and not-fixed classification layer.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Chung-Cheng Chiu, Tara N Sainath, Yonghui Wu, Rohit Prabhavalkar, Patrick Nguyen, Zhifeng Chen, Anjuli Kannan, Ron Weiss, Kanishka Rao, Ekaterina Gonina, al. State-of-the-art speech recognition with sequence-to-scquence models. In 2018 IEEE International Conference on Acoustics, Speech and Signal Processing (ICASSP), pp. 4774-4778. IEEE, 2018.
Adam Coates, Andrew Ng. and Honglak Lee. An analysis of single-layer networks in unsupervised feature learning. In Proceedings of the fourteenth international conference on artificial intelli- gence and statisties, pp. 211-22522222 2011.
Jiankang Deng, Jia Guo, Niannan Xue, and Stefanos Zafeiriou. Areface: Additive angular margin loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Patlern Recognition, pp. 4690-4699 2019.
Xing Fan, Wei Jiang. Hao Luo, and Mengjuan Fei. Spherereid: Deep hypersphere manifold embed- ding for person re-identification. Journal of Visual Communication and Image Representation, 60:51-58, 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016a.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Identity mappings in deep residual networks. In European conference on computer vision, pp. 630-645. Springer, 20.6b.
Elad Hoffer, Itay Hubara, and Daniel Soudry. Fix your classifier: the marginal value of training the last weight layer. arXiv preprint arXiv:1801.04440 2018.
Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4700-4708, 2017.
Alexander B. Jung. imgaug. cessed 30-Oct-2018).
2018. [Online; ac-
Ivan Krasin, Tom Duerig, Neil Alldrin, Vittorio Ferrari, Sami Abu-El-Haija, Alina Kuznetsova, Hassan Rom, Jasper Uijlings. Stefan Popov, Andreas Veit, et al. Openimages: A public dataset for large-scale multi-label and multi-class image classification. Dataset available from https://github. com/openimages, 2(3):2-3, 2017.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Weiyang Liu, Yandong Wen, Zhiding Yu, and Meng Yang. Large-margin softmax loss for convolu- tional neural networks. In ICML, volume 2, pp. 2016.
Weiyang Liu, Yandong Wen, Zhiding Yu, Ming Li, Bhiksha Raj, and Le Song. Sphereface: Deep hypersphere embedding for face recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 212-220, 2017.
Benjamin Recht, Rebecca Roelofs, Ludwig Schmidt, and Vaishaal Shankar. Do cifar-10 classifiers generalize to cifar-10? arXiv preprini /80..iv:880...m 00451, 2018
Mark Sandler, Andrew Howard, Menglong Zhu, Andrey Zhmoginov, and Liang-Chied Chen. Mo- bilenetv2: Inverted residuals and linear bottlenecks. Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4510-4520, 2018.
Ashish Vaswani, Noam Shazcer, Niki Parmar, Jakob Uszkoreit, Llion Jones, Aidan N Gomez, łukasz Kaiser, and Illia Polosukhin. Attention is all you need. In Advances in neural information processing systems, pp. 5998-6008, 2017.
Feng Wang, Xiang Xiang, Jian Cheng, and Alan Loddon Yuille. Normface: L2 hypersphere embed- ding for face verification. In Proceedings of the 25th ACM international conference on Multime- dia, pp. 1041-1049, 2017.

---

Under review as a conference paper at ICLR 2021
Feng Wang. Jian Cheng, Weiyang Liu, and Haijun Liu. Additive margin softmax for face verifica- tion. IEEE Signal Processing Letters, 25(7):926 930, 2018a.
Hao Wang, Yitong Wang, Zheng Zhou, Xing Ji, Dihong Gong, Jingchao Zhou, Zhifeng Li, and Wei Liu. Cosface: Large margin cosine loss for deep face recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5265-5274, 2018b.
Nicolai Wojke and Alex Bewley. Deep cosine metric learning for person re-identification. In 2018 IEEE winter conference on applications of computer vision (WACV), pp. 748-756, IEEE, 2018.
A VARIOUS LEARNING RATE INITIALIZATIONS
Table 7: Comparison between the classification accuracy of fixed and non-fixed PreActResnet 18 models using various L.R initializations. Nan values indicate that the model failed converge.
0.01
0.001
Dasaset
Fixed
Non-Fisd
Fised
Non-Fised
Fixed
Non- Son-Fised
Fixed
Non-Fixed
(96x66)
78.2%
CIBAR- 100 (32x32)
72.2%
NaN
75.2%
75.3%
74.5%
72.5%
Tinylmagenet (6Ha64)
59.1%
NaN
55.6%
55.4%
52.9%
52.1%
Table 8: Comparison between the classification accuracy of fixed and non-fixed Resnet18 models using various LR initializations.
LR 0.1
LR 0.01
LR = 0.001
LR = 0,0001
Dataset
Noa-Fixed
Fixed
Non-Fixed
Fixed
Non-Fixed
Fixed
Non-Fised
STL (96x96)
82.5%
75.9%
$1.2%
78.1%
79.1%
76.5%
77.9%
75.8%
CIFAR 100 (32x32)
74.8%
73.1%
75.9%
74.4%
74.5%
72.6%t
Tinylmagenct (64x64)
60.1%
59.0%
54.4%
53.9%
Table 9: Comparison between the classification accuracy of fixed and non-fixed MobileNetV2 mod- els using various LR initializations.
0.01
LRR
Fixed
Non Fixed
Fixed
Noe: Fixed
Fixed
Non Fined
Fixed
NiseF
75.9%
76
74.4%
73.9%
CIFAR-100
70.4%
Tiny Imagenettnt (646644)
56.8%
55.1%
39.3%
57.1%
51.2%
49.9%
10


Reviews:
[{"review_id": "_4X2KRBAW2n", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper claims that in the context of multi-class image classification using neural networks, in the final classification layer, if we use randomly initialized parameters (with normalization) without any training, we can achieve better performance than if we train those parameters. This is an intriguing claim that can potentially have a very broad impact. The authors provide some motivations based on the error-similarity plots, but no theoretical backing. Without convincing theoretical support, such a claim can only be established through extensive and rigorous experimentation, and I find the experiment description in this paper is short on delivering strong evidence. For example, how many runs to achieve the results in Tables 1-3? What are confidence intervals on the results?  Any statistical significance test done? How were hyperparameters selected?  What about the performance on the ImageNet dataset, which has more classes than the datasets reported in the paper? What distribution was used to initialize the random weights in the classification layer? Is the performance sensitive to the distribution? Is the performance sensitive to the complexity of the model used to learn the representation? How does this compare to other ways of improve multi-class classification such as softmax temperature annealing, label smoothing, adding regularization, etc.? Or as a stretch, does this claim generalize to problems with categorical features?\n\nDetails:\n1. page 2, line 9: do you mean \"maximizing the cosine-similarity\"?", "extra_scores": {}, "extra_text": {}}, {"review_id": "VSBrq1TLl_S", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposed a classification layer by randomizing the class representation vectors. This paper first analyses the class vector distributions between different training strategies, and then proposed the randomized class vector to improve the representation learning performance. The proposed model is further extended and analyzed for the fixed cosine-similarity maximization setting. The experiments demonstrate the effectiveness of the proposed method compared with the basic/vanilla baselines.\n\nPros:\nThe motivation of this paper is comprehensive. Some quantitative and visual experimental results introduced the motivation of the proposed model. The randomization weights also provide a novel view for solving more machine learning problems. This is a good point.\n\nCons:\nMy main concern is the experimental results. The experiments are mainly done for evaluating fixed and non-fixed models without any other state-of-the-art methods, and the exact performance is considerably low compared with other state-of-the-art methods. \nTo this end, it is hard to confirm the effectiveness of the model. For example, 1) even though the visualization results are good (Figure 3), it does not mean the final performance is better. 2) the original model could be over-fitting, and a random and fixed weight layer could be considered as a regularizer. There are some experiments should be done: 1) Compare this method with relevant methods such as NormFace and ArcFace to proof the effectiveness of this approach. 2) Compared with exact performance in face relevant datasets and compared with other SOTA methods.\n", "extra_scores": {}, "extra_text": {}}, {"review_id": "MyLlwmcQ5Hk", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "The paper explores deeper into the specific classification layer of a standard supervised learning system. The core idea of the paper is to randomly initialize and then fix the classification layer weights and train the network leading improved discrimination.\nThe writing is satisfactory and the paper develops the ideas sufficiently well to help any reader who is a beginner in this area.\n\nOne of the major concerns regarding the work is that it seems to have is the relatively limited amount of contribution given the context of the current venue. This is no doubt an interesting phenomenon, however, previous works investigating cosine similarity losses have tested their approaches on much larger problems such as large scale face recognition and full Imagenet. The paper currently derives its intuitions from object recognition problems which have very different behavior than problems like face recognition where the number of classes is large yet the number of samples per class is much lower.  \n\nThat said, given the limited scale of the experiments, the paper does offer a wider variety of results supporting its claims. Nonetheless, given the simplicity of the idea, the paper fails to push envelope of results on any of these datasets. Lastly, the performance gains in Table 3 seem limited, given that only one run was performed for each dataset. ", "extra_scores": {}, "extra_text": {}}, {"review_id": "4bvhmpEWirY", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\n\nThis paper introduces a new approach to learn a multi-class image classification model by fixing the weights of the classification layer. The authors propose to draw the class vectors randomly and set them as fixed during training instead of training them. They analyze this approach when a model is trained with a categorical cross-entropy and or softmax-cosine loss. The proposed approach is tested on 4 datasets: STL, CIFAR-10, CIFAR-100, TinyImagenet\n\n**Reasons for score:** \n\nI do not think the technical contribution is strong enough for ICLR. The idea is interesting but the empirical validation of the idea should be improved and some claims should be proved.\n\n\n**Pros:**\n\n- The idea of using fixed-representation is interesting. It can help to reduce the training time.\n- The authors explain why cosine-similarity maximization models cannot converge to 0.\n\n**Cons:**\n\nThe title looks very interesting: “Redesigning the Classification Layer by Randomizing the Class Representation Vectors”. But after reading the paper, it is only about mullti-class image classification. There is no study about other types of data or the multi-label setting. The authors should use a title more accurate about the content of their paper.\n\nOverall, the structure of the paper should be improved. It is quite difficult to read because several sections are a mix of model contributions and experimental results. Maybe using subsections can help to separate the model contributions and experimental results. Also, some information is not at the right place and some sections should be reorganized. For example, the datasets and models are presented in section 4.1 but some results are presented in section 2. The authors should also add a related work section to clearly state the motivations and explain the difference with other approaches.  \n\nThe authors proposed to randomly initialize the weights of the classification layer but they do not clearly explain how the weights are initialized. There are several standard approaches to initialize weights like uniform, normal, Xavier uniform, Xavier normal, Kaiming uniform, Kaiming normal. It can improve the paper if the authors compare these initialization mechanisms. Similarly, the authors should analyze the results for several runs to see how the fixed weights approach is sensitive to the random initialization. \n\nI have a conceptual problem with fixing the bias. The bias is sampled so it means it can have a large or small value. Let’s take an example with 2 classes. The class A can have a large bias (e.g. 0.5) but other class B can have a small value (e.g. -0.5). It means that the class B has a negative bias and will usually have lower scores than A just because there is a difference of 1 between these biases. I am not sure that it is a good idea and there is no motivation about that in the paper. The authors should analyze the bias initialization because it is important.  \n\nIt is important to show the variance when the model is evaluated on several runs (section 4). It can help to understand how the model is sensible to the initialization.  \n\nIt is well known that the SGD is sensible to its hyper-parameter and in particular the learning rate. The model will not converge if the learning rate is too large or too small. The authors should explain how they choose the hyper-parameters. I also wonder how the results are specific to the optimizer. Are the conclusions of the analysis the same for other popular optimizers like Adam.\n\n“These observations can provide an explanation as to why non-fixed models with S = 1 fail to converge.” (page 6): For me it explains why the model cannot converge to 0 but it does not explain why the model fails to converge. They are two different problems.\n\nIn the abstract and in some other parts of the paper the authors claim they improve the compactness of the model. But they never show it. They did not define how they measure the compactness of a model. They should clearly present the definition of compactness, and what approach they used to compute it. Based on my knowledge, measuring the compactness of a model is not easy.\n\nThe authors should results on low resolution dataset (less than 100*100). I wonder if the results can be generalized to larger resolution dataset. For example, does it also work on ImageNet that has more images, larger resolution images and more classes (1000). I also wonder if it works on other type of datasets like fine-grained datasets (e.g. CUB-200, Stanford Cars, FGVC Aircraft). Also, how does it adapt to new domains like medical images and natural scenes.\n\nI am not convinced that ignoring the visual similarities between classes is a good idea. I think it is important to build spaces that encode some semantic structure. For example, I think it is important to encode that two bird species are more semantically similar than a bird and a car.\n\nIt is not clear why the authors decided to focus on the cosine-similarity maximization models. They should motivate this decision more because these models are not so popular.\n\nThe authors claimed that “the low range of the logits vector is not the cause preventing from cosine-similarity maximization models from converging” (page 5) but they did not show results to prove it. The authors should analyze the range of the logits. The current analysis does not allow us to understand if it is because of the range of value, or the normalization of the weights or a bad tuning of some hyper-parameters. \n\n**Minor comments:**\n\nThe authors should give more information on how they generated the figures 3 and 5.\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The reviewers are in consensus that this paper is not ready for publication: cited concerns include simple (interesting) ideas but need to be carefully analyzed empirically, contextualized (other similar studies exist), identifying convincing empirical evidences,. etc.


The AC recommends Reject.
Decision: Reject