Venue: ICLR 2021 Conference

Reviews:

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper proposes the batch confusion norm (BCN) for dealing with both fine-grained recognition and long-tailed recognition simultaneously. Specifically, BCN considers the confusion regularization within each training batch and an adaptive matrix term is designed for handling the long-tailed problem. Experiments are conducted on fine-grained benchmark datasets (e.g., CUB, CAR, AIR) as well as long-tailed recognition datasets (e.g., iNat18). 

Paper strengths:
- The paper is well organized and easy to follow.
- The problems studied in this paper, i.e., fine-grained recognition and long-tailed visual recognition, are both important, challenging and practical in computer vision, which deserves further studies.
- The proposed method sounds reasonable.

Paper weaknesses:
- Although the proposed method is reasonable, some specific model designs are not quite clear. 1) Regarding Eq. (2), the reason why it requires to optimize the ranking should be further explained and its motivation needs to state. 2) Regarding Eq. (5), what the intuition of the adaptive matrix (i.e., (log_{\mu+1} (N_i+1))^{\delta^{\tau}}) when i = j should be provided to the authors.
- The major issue of this paper is the experimental evaluations. 1) The classification accuracy on these fine-grained benchmark datasets and iNat18 are not significantly better than the accuracy of previous work. Thus, the effectiveness of the proposed method is problematic. 2) Some state-of-the-art methods are not involved in the experimental comparisons, such as [ref1-ref5]. Moreover, the accuracy of the proposed method cannot outperform these methods.

Minor issues:
- There are several typos and writing problems in this paper. For example, on Page 3, "Dubey et al.Dubey et al. (2018)", and "Chen et al.Chen et al. (2019)". On Page 4, "PC Dubey et al. (2018)". On Page 8, "And also solves the long-tailed problem by an adaptive matrix term."

[ref1] Weakly Supervised Fine-grained Image Classification via Guassian Mixture Model Oriented Discriminative Learning, CVPR 2020.

[ref2] Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up, CVPR 2019.

[ref3] Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches, ECCV 2020.

[ref4] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.

[ref5] BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition, CVPR 2020.
review_title: Official Blind Review #2

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Overview:** The paper presents an extension to the Batch Confusion Norm (BCN) regularization technique so that it can account for imbalanced datasets. The extension to BCN implies adding a matrix that determines its values as a function of class imbalanced statistics contained in a batch. The paper presents experiments showing the benefits of the proposed extension on Fine-Grained Visual Classification (FGVC) and long-tailed (LT) tasks. The experiments show modest improvements over the baselines.

**Pros:**
*Clarity of the paper is good.* The clarity of the paper is very good. The motivation behind the FGVC and LT learning problems as well as the proposed method is clear. Thanks to the clarity of the paper I believe the reproducibility should be good. Also, I believe that the paper addresses an important problem with practical value.

**Cons:**
*Novelty.* The novelty of the paper IMHO falls short. This is because the submission mainly extends the BCN paper by adding a matrix A whose entries are a function of the statistics of the statistics of the dataset.  The proposed extension lacks a more rigorous derivation and justification; IMHO, the proposed extension seems to be ad hoc.

*Not convinced about BCN and proposed extension.* 
1) I am not sure if BCN is a proper regularizer. While I understand the geometry behind minimizing the rank of the matrix P, I don't think this is a proper way of processing the columns of matrix P which are *posterior distributions*. I think the paper lacks a clear justification about using BCN as a way to treat posterior distributions without using statistical or probabilistic methods. 
2) BCN is conditioned to operate if the classes in the batch are unique. However, satisfying this condition in practice can be challenging. This becomes challenging when dealing with long-tailed datasets. What is the optimal/efficient way to guarantee a batch that satisfies this constraint? If the constraint is not satisfied, can BCN provide bad gradients because repeated classes with different posteriors in different columns can double its contribution?
3) The extension of adding a matrix A is simple and feels a bit ad hoc. What is the intuition behind the matrix A in terms of the norm? Can the matrix A be considered a way to ` *weight* a posterior as a function of the statistics?

*Insufficient experiments and baselines.* 
1) While I am pleased to see that the paper uses datasets depicting real scenes (e.g., iNaturalist, CUB, CAR, etc.), the experiments only focus on using one family of networks: ResNet. Does this method operate well on other more modern architectures, e.g., EfficientNet or MobileNetV2? 
2) In terms of baselines, I think the paper is missing recent long-tail methods:
  a) Liu, et al. Large-Scale Long-Tailed Recognition in an Open World. CVPR 2019. 
  b) Cao, et al. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. NeurIPS 2019.
3) The results for the long-tail methods are lacking more details. As it is common in various long-tail recognition papers, the paper is not showing performance on head or tail classes. It only shows an overall classification performance. This is misleading as the method can helping more head classes and thus improving the overall classification performance.

*Minor concern*: Most of the figures and diagrams look nearly identical to those presented in the BCM paper.
review_title: Interesting Idea but Not Convinced About the Method nor by the Experiments

final_score: rating: 5: Marginally below acceptance threshold
review_content: This paper presents a novel technique for fine-grained visual classification. This technique addresses the classic issues in this task of inter-class similarity (coupled with intra-class variation) and the “long tailed” dataset problem, prevalent in datasets such as iNaturalist2018. The technique is an extension of Dubey et. al which extends their technique by incentivising the predictions for all samples in a mini-batch to be similar. This is in contrast to Dubey et. al which splits the mini-batch into two halves and incentivizes the aggregate predictions of the two halves to be similar. 

I find this to be of minimal novelty, and I question the practicality of the method. Unless I have misunderstood, this sounds like it would be slower to train and I did not see any runtime analysis comparisons in the experiments section. Experiments showing how the new loss function effects training time would help alleviate this concern. The method does however improve performance on the tested benchmarks. 

Finally, I found that there were numerous grammatical and stylistic mistakes. The writing improves during the discussion of the mathematics of the technique, but the introduction, experiments, and discussion need work. I would like to see the writing improved for a publication. 

review_title: Minimal Novelty but Good Results

final_score: rating: 4: Ok but not good enough - rejection
review_content: **Summary:**
This paper is about fine-grained visual classification, which is challenging due to high inter-class similarity, high intra-class variation and potentially also class imbalance. The authors propose a regularization term that is added to a standard cross-entropy loss of a neural network trained in a supervised fashion. This regularization shares the motivation with prior art to confuse the network and reduce over-fitting by making all predictions within a mini-batch similar to each other. Different to prior art, they authors propose an approximation of a minimum rank objective. To also handle class imbalance, the authors extend the regularization with a learnable matrix that can automatically balance the importance of individual classes.

**Pros:**
- Overall, I think the contributions of the paper is interesting and useful, particularly the extension of the confusion-based regularization for class-imbalanced data sets. Unfortunately, the presentation of the idea needs significant improvement, see below.
- The analysis and discussion in Sections 4.4 and 4.5 are great

**Cons:**
- The writing made this paper really hard to understand. The formulations, particularly in abstract and introduction, are inaccurate and vague. They should rather be specific and concrete. For instance, the statement "When inter-class similarity prevails in a batch, the BCN term can alleviate possible overfitting due to exploring image features of fine details" is hard to understand since it was totally unclear what the BCN term actually is at that point of reading. In both abstract and introduction, I had a very hard time imagining what certain statements mean in terms of who the method would look like, without having read the full paper.
- The results barely improve over SOTA, particularly for the three FGVC data sets. So the bigger advantage I see was for class imbalanced data sets, like iNaturalist. Although the performance is also not significantly better than prior works, the direct competitor (PC) seems to under-perform clearly. The proposed solution alleviates the problem, which is good. In light of this, I think it would make sense to build such data sets synthetically from the existing ones (like Cub, Car, Air) by removing samples to increase class imbalance. This would allow additional experiments on class imbalance in a controlled setup.


**Minor notes:**
- The author names can be dropped with the ICLR citation format: "... Dubey et al.Dubey et al. (2018) construct a Siamese" on page 3.
- Statements like "The confusion-related formulation for dealing with intra-class variations and inter-class similarity in FGVC have two main implications" on page 3 require the reader to be very familiar with these concepts. It would be better to re-formulate it with a brief introduction of the concept (just a sentence or two).

review_title: Interesting idea, poor presentation

Meta-review: The authors address the problem of fine-grained image classification. They propose a batch based regularizer, called the batch confusion norm (BCN), to encourage less over confident predictions. They also tackle the problem of class imbalance during training by adaptively weighting the BCN loss at the class level to take the imbalances in the underlying label distributions into account. Results are presented on four different fine-grained datasets.  

Overall, while the reviewers had some positive comments, there was not broad support for the paper. There are questions that need to be resolved related to the evaluation e.g. the best performing model uses GASPP, however there is no reported GASPP variant for the PC baseline. Similarly, it would be valuable to know how much PC would benefit from an additional class imbalance term in the iNaturalist2018 results. Given that the proposed regularizer builds on PC (Dubey et al.), it is very important that the authors provide a like-for-like comparison so that readers can better understand the merits of the proposed method.  

There were also concerns with the presentation of the paper e.g. several typos (which can be easily fixed), issues with the clarity of the text (which require more work), and uninformative figures (e.g. Fig 2 should be revised to more clearly illustrate the differences between the three methods shown). The authors are encouraged to revise the text to resolve these problems. 

While the paper has some strengths (e.g. the empirical performance on some of the tasks is promising and the method is conceptually simple), there are still a number of concerns from the reviewers e.g. a lack of a clear motivation as to why the proposed method works, and why it is conceptually better than existing alternatives (e.g. PC). Given this lack of support, it is not possible to recommend the paper in its current form. 

Decision: Reject