Venue: ICLR 2023 Conference

Reviews:

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The authors present a method that tries to rectify feature collapse for classification, arguing that this is an undesirable property for the case where coarse labels are used for training, but we are (also) interested in a finer-grained downstream task. They advocate to fix the classifiers and use a hinge loss. They experiment on coarse-to-fine as well as long-tail classification.
strength_and_weaknesses: ### Strengths

* The paper dealing a very interesting property of learning with cross-entropy, that of class collapse

* the paper is  experimenting on interesting tasks like long-tail recognition and coarse-to-fine learning.

### Weaknesses

W1) Novelty of the paper is questionable: There are some missing references and comparisons that are very related to the proposed. [A] refers to the same issue tackled here, class collapse for transfer learning. Feng et al in [A] do in some sense provide an answer to the question asked in the intro and propose a loss that avoids collapse. Grafit from Touvron et al [B] is even closer to the task and formulation of this paper: it presents an approach for learning with coarse labels and the loss proposed in [B] is a kNN loss therefore also avoiding collapse. The authors should not only discuss but also compare to such methods.  Another missing reference that should be discussed and compared to is wrf fixed classifiers is [C]. 


W2) Experimental validation is lacking in many regards: 

*  It is unclear how this supervised learning method affect the training task of ImageNet in 4.1 The authors compare against SimCLR, a self-supervised method, but not supervised methods like [A,D] in Table 1. They also do not report performance on the training task of imagenet (one more column should be added). As [Kornblith et al 2021], [E] and other papers have mentioned, there exists a tradeoff between training and transfer accuracy.  What is the performance of all methods at the end of training? 

* For long-tail, the method is not outpuerforming multiple methods of Table 2 on the larger and less toy-scale ImageNet-LT. Also Fig 5c shows that setting the main hyperparameter \tau is far from trivial. Moreover it is unclear what model is used in each dataset and method in Table2 - are all methods using the same backbone?

### References
[A] Feng, Yutong, et al. "Rethinking supervised pre-training for better downstream transferring." ICLR 2022

[B] Touvron, Hugo, et al. "Grafit: Learning fine-grained image representations with coarse labels."CVPR 2021

[C] Hoffer, Elad,et al  "Fix your classifier: the marginal value of training the last weight layer." ICLR 2018.

[D] Khosla, Prannay, et al. "Supervised contrastive learning." NeurIPS 2020

[E] Sariyildiz et al "Improving the Generalization of Supervised Models." arXiv 2022.
clarity,_quality,_novelty_and_reproducibility: The paper is generally written in a very confusing way; the "core" of the method contains unclear parts while it is unclear what Theorem 1 offers.The analysys of NC comes from Papyan et al., 2020 in 2.2, and I dont see how this paper "examines the Neural Collapse phenomenon" as stated in the conclusion. There are also implementation details that are missing in many places and therefore reproducibility is questionable.

### Notes 

* Paragraph "Hinge Loss" in 3.2 is unclear in many places, while Eq9 is written in a very confusing way.

* the first paragraph of the intro is in practice merely advocating for the use of learned vs handcrafted features for computer vision, yet, this is common practice in the AI classification since AlexNet won the Imagenet challenge. It can therefore be significantly trimmed. Same with 2.1 that merely describes classification with  cross entropy loss, again the default loss

* The question asked in the introduction is ill-posed as expressed there. "the" desired property implies that it would be the only, while "deep features" have a multitude of uses- which use does this paper care about is not mentioned till there in the intro. The authors right after mention how they are only looking at a specific task, that of coarse-to-fine classification. The intro needs revision. 
 


summary_of_the_review: The paper has a number of weaknesses detailed above. I encourage the authors to provide answers to the concerns and questions.
review_title: null

final_score: rating: 8: accept, good paper
review_content: summary_of_the_paper: This paper studies the undesirable Neural Collapse (NC) phenomenon for specific classification tasks, including fine-grained classification and long-tailed classification. The authors then propose to learn deep features that have the maximal-separating-cone (MSC) property by optimizing towards their designed hinge loss function. Empirical results verified their motivation and the effects of the proposed approach.
strength_and_weaknesses: Pros:

+ Well-motivated approach: Investigates an intriguing phenomenon of NC, and its undesirable properties for fine-grained classification adaptation.
+ Technically sound solutions with a novel hinge function design.

Cons:
- It is unclear to me why learning MSC features can also benefit long-tailed classification tasks. Experimental results in Table 2 also seem to show weaker performance gain on long-tailed datasets.

clarity,_quality,_novelty_and_reproducibility: The paper is clearly written and technically sound, which proposed a novel feature property named maximal-separating-cone.  
summary_of_the_review: This paper is decent research towards learning deep features with MSC property, and hence tackles certain specific classification problems where NC is undesirable.
review_title: null

final_score: rating: 5: marginally below the acceptance threshold
review_content: summary_of_the_paper: The paper discusses NC - intra-class features collapse to the class mean and different class means are maximally separated. The authors argue that NC is an undesirable property for fine-grained classification. Thus they constrain the features of the same class should lie in a cone instead of collapsing to the mean. A feature extractor pre-trained by this method improves fine-grained classification on multiple datasets.
strength_and_weaknesses: Strengths:
- The paper presents a simple idea and is easy to follow. 
- The method is appropriately designed following the main idea. 

Weaknesses:
- My main concern here is that the paper doesn't offer an explanation of why NC is undesirable for fine-grained classification, i.e., the main idea isn't well justified. I think NC should be characterized carefully in order to support the main idea.  In particular, I disagree that "collapsing to the class mean" is a good enough reason to say NC is undesirable for fine-grained classification. The authors can convince me by quantifying the collapse and demonstrating how it correlates with the fine-grained classification performance. As of now, I need to believe that it's the case and without any proof of concept. Personally, I think "collapsing to the class mean" is quite vague and unclear. For example, one possible interpretation is that the distances between features of the same class are small w.r.t the distance between classes. In this case, it doesn't mean that the feature doesn't encode enough fine-grained information for each instance since intra-class variance can still be high. 

- Likewise, since the method is based on a weak motivation (to me), I am not sure how rearranging features into a cone helps with fine-grained classification. There are methods that propose feature space re-arrangement before (Mag-face, arc-face) and they are very clear about why their geometrical constraints boost their performance. I don't see a similar explanation here. Is there any relation between the proposed  MSC and intra-class variance [1]? If there is a connection here that would add depth to the paper and offer an explanation for what is happening.
[1] Variational Feature Disentangling for Fine-Grained Few-Shot Classification - ICCV21

- The paper introduces some hyperparameters that need to be tuned. 

- The paper doesn't improve the performance on the most challenging case of the imagenet-LT.
clarity,_quality,_novelty_and_reproducibility: Clarity: Good
Quality: Good
Novelty: Generally a new technique but it's not a surprising one. Similar ideas have been proposed for faces [mag face, arc Face]. I would like the authors to discuss them and clarify their novelty.
Reproducibility: Good.
summary_of_the_review: Overall, I am on the fence. The paper is simple and the method seems to work in some cases. The numbers are not substantially strong. The explanation and motivation aren't clear. I am not an expert in this area so please fell free to correct me.
review_title: null

Meta-review: metareview:_summary,_strengths_and_weaknesses: The paper studies a method that tries to prevent model representations of instances with the same class from collapsing when the model is trained using a classification objective. The paper argues that such collapse is undesired when class labels are coarse-grained but we also wish to transfer them to finer-grained classification tasks.

Initially, two reviewers were lukewarm about the paper. The reviewers express concerns about novelty, the motivation for the proposed approach, the clarity of presentation, and the quality of the experimental validation of the work. Some of these concerns were addressed in the discussion, but these reviewers find that there are still missing comparisons. Also, they find the clarity of presentation lacking even in the revised manuscript.

One reviewer (6L41) provided a more positive overall assessment of the paper, but did not provide much context on what they liked about the paper. The AC connected with reviewer 6L41 offline to get more context on their assessment. Reviewer 6L41 let the AC know that, on second thought, they may have overrated the paper a bit and that they share some of the concerns by the other reviewers around novelty and motivation.
justification_for_why_not_higher_score: Multiple reviewers express concerns about novelty, the motivation for the proposed approach, the clarity of presentation, and the quality of the experimental validation of the work.
Decision: Reject