Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 4: Ok but not good enough - rejection
review_content: This paper claims that in the context of multi-class image classification using neural networks, in the final classification layer, if we use randomly initialized parameters (with normalization) without any training, we can achieve better performance than if we train those parameters. This is an intriguing claim that can potentially have a very broad impact. The authors provide some motivations based on the error-similarity plots, but no theoretical backing. Without convincing theoretical support, such a claim can only be established through extensive and rigorous experimentation, and I find the experiment description in this paper is short on delivering strong evidence. For example, how many runs to achieve the results in Tables 1-3? What are confidence intervals on the results?  Any statistical significance test done? How were hyperparameters selected?  What about the performance on the ImageNet dataset, which has more classes than the datasets reported in the paper? What distribution was used to initialize the random weights in the classification layer? Is the performance sensitive to the distribution? Is the performance sensitive to the complexity of the model used to learn the representation? How does this compare to other ways of improve multi-class classification such as softmax temperature annealing, label smoothing, adding regularization, etc.? Or as a stretch, does this claim generalize to problems with categorical features?

Details:
1. page 2, line 9: do you mean "maximizing the cosine-similarity"?
review_title: A provocative idea begging for more evidence

final_score: rating: 5: Marginally below acceptance threshold
review_content: The paper explores deeper into the specific classification layer of a standard supervised learning system. The core idea of the paper is to randomly initialize and then fix the classification layer weights and train the network leading improved discrimination.
The writing is satisfactory and the paper develops the ideas sufficiently well to help any reader who is a beginner in this area.

One of the major concerns regarding the work is that it seems to have is the relatively limited amount of contribution given the context of the current venue. This is no doubt an interesting phenomenon, however, previous works investigating cosine similarity losses have tested their approaches on much larger problems such as large scale face recognition and full Imagenet. The paper currently derives its intuitions from object recognition problems which have very different behavior than problems like face recognition where the number of classes is large yet the number of samples per class is much lower.  

That said, given the limited scale of the experiments, the paper does offer a wider variety of results supporting its claims. Nonetheless, given the simplicity of the idea, the paper fails to push envelope of results on any of these datasets. Lastly, the performance gains in Table 3 seem limited, given that only one run was performed for each dataset. 
review_title: Interesting observation but limited demonstrated scope

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary:**

This paper introduces a new approach to learn a multi-class image classification model by fixing the weights of the classification layer. The authors propose to draw the class vectors randomly and set them as fixed during training instead of training them. They analyze this approach when a model is trained with a categorical cross-entropy and or softmax-cosine loss. The proposed approach is tested on 4 datasets: STL, CIFAR-10, CIFAR-100, TinyImagenet

**Reasons for score:** 

I do not think the technical contribution is strong enough for ICLR. The idea is interesting but the empirical validation of the idea should be improved and some claims should be proved.


**Pros:**

- The idea of using fixed-representation is interesting. It can help to reduce the training time.
- The authors explain why cosine-similarity maximization models cannot converge to 0.

**Cons:**

The title looks very interesting: “Redesigning the Classification Layer by Randomizing the Class Representation Vectors”. But after reading the paper, it is only about mullti-class image classification. There is no study about other types of data or the multi-label setting. The authors should use a title more accurate about the content of their paper.

Overall, the structure of the paper should be improved. It is quite difficult to read because several sections are a mix of model contributions and experimental results. Maybe using subsections can help to separate the model contributions and experimental results. Also, some information is not at the right place and some sections should be reorganized. For example, the datasets and models are presented in section 4.1 but some results are presented in section 2. The authors should also add a related work section to clearly state the motivations and explain the difference with other approaches.  

The authors proposed to randomly initialize the weights of the classification layer but they do not clearly explain how the weights are initialized. There are several standard approaches to initialize weights like uniform, normal, Xavier uniform, Xavier normal, Kaiming uniform, Kaiming normal. It can improve the paper if the authors compare these initialization mechanisms. Similarly, the authors should analyze the results for several runs to see how the fixed weights approach is sensitive to the random initialization. 

I have a conceptual problem with fixing the bias. The bias is sampled so it means it can have a large or small value. Let’s take an example with 2 classes. The class A can have a large bias (e.g. 0.5) but other class B can have a small value (e.g. -0.5). It means that the class B has a negative bias and will usually have lower scores than A just because there is a difference of 1 between these biases. I am not sure that it is a good idea and there is no motivation about that in the paper. The authors should analyze the bias initialization because it is important.  

It is important to show the variance when the model is evaluated on several runs (section 4). It can help to understand how the model is sensible to the initialization.  

It is well known that the SGD is sensible to its hyper-parameter and in particular the learning rate. The model will not converge if the learning rate is too large or too small. The authors should explain how they choose the hyper-parameters. I also wonder how the results are specific to the optimizer. Are the conclusions of the analysis the same for other popular optimizers like Adam.

“These observations can provide an explanation as to why non-fixed models with S = 1 fail to converge.” (page 6): For me it explains why the model cannot converge to 0 but it does not explain why the model fails to converge. They are two different problems.

In the abstract and in some other parts of the paper the authors claim they improve the compactness of the model. But they never show it. They did not define how they measure the compactness of a model. They should clearly present the definition of compactness, and what approach they used to compute it. Based on my knowledge, measuring the compactness of a model is not easy.

The authors should results on low resolution dataset (less than 100*100). I wonder if the results can be generalized to larger resolution dataset. For example, does it also work on ImageNet that has more images, larger resolution images and more classes (1000). I also wonder if it works on other type of datasets like fine-grained datasets (e.g. CUB-200, Stanford Cars, FGVC Aircraft). Also, how does it adapt to new domains like medical images and natural scenes.

I am not convinced that ignoring the visual similarities between classes is a good idea. I think it is important to build spaces that encode some semantic structure. For example, I think it is important to encode that two bird species are more semantically similar than a bird and a car.

It is not clear why the authors decided to focus on the cosine-similarity maximization models. They should motivate this decision more because these models are not so popular.

The authors claimed that “the low range of the logits vector is not the cause preventing from cosine-similarity maximization models from converging” (page 5) but they did not show results to prove it. The authors should analyze the range of the logits. The current analysis does not allow us to understand if it is because of the range of value, or the normalization of the weights or a bad tuning of some hyper-parameters. 

**Minor comments:**

The authors should give more information on how they generated the figures 3 and 5.

review_title: Training a multi-class image classification model by fixing the weights of the classification layer

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposed a classification layer by randomizing the class representation vectors. This paper first analyses the class vector distributions between different training strategies, and then proposed the randomized class vector to improve the representation learning performance. The proposed model is further extended and analyzed for the fixed cosine-similarity maximization setting. The experiments demonstrate the effectiveness of the proposed method compared with the basic/vanilla baselines.

Pros:
The motivation of this paper is comprehensive. Some quantitative and visual experimental results introduced the motivation of the proposed model. The randomization weights also provide a novel view for solving more machine learning problems. This is a good point.

Cons:
My main concern is the experimental results. The experiments are mainly done for evaluating fixed and non-fixed models without any other state-of-the-art methods, and the exact performance is considerably low compared with other state-of-the-art methods. 
To this end, it is hard to confirm the effectiveness of the model. For example, 1) even though the visualization results are good (Figure 3), it does not mean the final performance is better. 2) the original model could be over-fitting, and a random and fixed weight layer could be considered as a regularizer. There are some experiments should be done: 1) Compare this method with relevant methods such as NormFace and ArcFace to proof the effectiveness of this approach. 2) Compared with exact performance in face relevant datasets and compared with other SOTA methods.

review_title: The model is reasonable while the experiments are not convincing

Meta-review: The reviewers are in consensus that this paper is not ready for publication: cited concerns include simple (interesting) ideas but need to be carefully analyzed empirically, contextualized (other similar studies exist), identifying convincing empirical evidences,. etc.


The AC recommends Reject.
Decision: Reject