Natural World Distribution via Adaptive Confusion Energy Regularization
Authors: Yen-Chi Hsu
Cheng-Yao Hong
Wan-Cyuan Fan
Ding-Jie Chen
Ming-Sui Lee
davi geiger
Tyng-Luh Liu
Venue:  (2021)
Keywords: Fine-Grained Visual Classification
long-tailed distribution
confusion energy

Abstract: We introduce a novel and adaptive batch-wise regularization based on the proposed Batch Confusion Norm (BCN) to flexibly address the natural world distribution which usually involves fine-grained and long-tailed properties at the same time. The Fine-Grained Visual Classification (FGVC) problem is notably characterized by two intriguing properties, significant inter-class similarity and intra-class variations, which cause learning an effective FGVC classifier a challenging task. Existing techniques attempt to capture the discriminative parts by their modified attention mechanism. The long-tailed distribution of visual classification poses a great challenge for handling the class imbalance problem. Most of existing solutions usually focus on the class-balancing strategies, classifier normalization, or alleviating the negative gradient of tailed categories. Depart from the conventional approaches, we propose to tackle both problems simultaneously with the adaptive confusion concept. When inter-class similarity prevails in a batch, the BCN term can alleviate possible overfitting due to exploring image features of fine details. On the other hand, when inter-class similarity is not an issue, the class predictions from different samples would unavoidably yield a substantial BCN loss, and prompt the network learning to further reduce the cross-entropy loss. More importantly, extending the existing confusion energy-based framework to account for long-tailed scenario, BCN can learn to exert proper distribution of confusion strength over tailed and head categories to improve classification performance. While the resulting FGVC model by the BCN technique is effective, the performance can be consistently boosted by incorporating extra attention mechanism. In our experiments, we have obtained state-of-the-art results on several benchmark FGVC datasets, and also demonstrated that our approach is competitive on the popular natural world distribution dataset, iNaturalist2018. 

Under review as a conference paper at ICLR 2021
NATURAL WORLD DISTRIBUTION VIA ADAPTIVE
CONFUSION ENERGY REGULARIZATION
Paper ID: 1442
Paper under double-blind review
ABSTRACT
We introduce novel and adaptive batch- wise regularization based on the proposed Batch Confusion Norm (BCN) to flexibly address the natural world distribution which usually involves fine-grained and long-tailed properties at the same time. The Fine-Grained Visual Classification (FGVC) problem is notably characterized by two intriguing properties, significant inter-class similarity and intra-class variations, which cause learning effective FGVC classifier challenging task. Existing techniques attempt to capture the discriminative parts by their modified attention mechanism. The long-tailed distribution of visual classification poses great chal- lenge for handling the class imbalance problem. Most of existing solutions usually focus the laas-balanccing strategies, classifier normalization, alleviating the negative gradient of tailed categories. Depart from the conventional approaches, we propose to tackle both problems simultaneously with the adaptive confusion concept. When inter-class similarity prevails in a batch, the BCN term can alleviate possible overfitting due to exploring image features of fine details. On the other hand, when inter-class similarity is not an issue, the class predictions from different samples would unavoidably yield substantial BCN loss, and prompt the network learning to further reduce the cross-entropy loss. More importantly, extending the existing confusion energy-based framework to account for long-tailed scenario, BCN can learn to exert proper distribution of confusion strength over tailed and head categories to improve classification performance. While the resulting FGVC model by the BCN technique is effective, the performance boosted by incorporating extra attention mechanism. In our experiments, we have obtained state-of-the-art results on several benchmark FGVC datasets, and also demonstrated that our approach is competitive on the popular natural world distribution dataset, iNaturalist208.
be consistently
INTRODUCTION
Fine-grained visual classification (FGVC) is an active and challenging problem in computer vision. Such recognition task differs from the classical problem of large-scale visual classification (LSVC) by focusing on differentiating similar sub-categories of the same meta-category. In FGVC, the inter-class similarity among the object categories is often pervasive, while the intra-class variations further impose ambiguities in learning a unified and discriminative representation for each category. Long-tailed distribution brings in another aspect of challenge that the head categories tend dominate the training procedure. The learned classification model thus performs better on these categories, while yielding significantly poor performance for the tail categories. The performance distribution somewhat resembles the data distribution. As the natural world distribution often assumes both fine-grained and long-tailed properties, how to satisfactorily address the recognition problem under such general setting raises practical and challenging problem.
From the existing literature, there are only few attempts to solving these two problems at the same time. Relevant efforts mostly focus on tackling either task. In FGVC, most of the recent research efforts have converged to learn pivotal local/part details relevant to distinguishing fine- grained categories e.g., (Fu al. 2017: Yang et al.) 20188] Zheng et al.) 2019), and typically require the fusion several sophisticated computer vision techniques to accomplish the task such as (Ge et al. 2019). In resolving the long-tailed issue, previous approaches have looked into data balanced

---

Under review as a conference paper at ICLR 2021
1000
Fiatarn In 2012
Intra-class variation
800
Quarskor
Herring Gull
California Gull
Ring billed Gull
600
Inter-class similarity
0
1002
2000
2000
Catepery
4200
5100
6100
7200
0200
(b)
Figure I: (a) Inter-class similarity vs. intra-class variation: Each column includes two instances of specific "Gull" category from the CUB-200-201 dataset Wah et al. (201T). (b) The natural world distribution dataset Naturallst20018 Van Horn et 2018).
sampling (Huang al. 2016a Wang al.) 2017) and the recent development such Kang et al. (2020) learns the representation at the first stage and refines the classifier by balanced data sampling.
Figure Th illustrates the two aspects of paradoxes in FGVC where the inter-class similarity and the intra-class variations are subtly intertwined, yielding a daunting classification task. For humans, the example convincingly suggests that expert knowledge is needed to differentiate from the other two categories. Alternatively, it also exhibits the challenges of formulating universal criteria in developing machine learning frameworks satisfactorily solve the FGVC problem even for modest case involving just three object categories. Figure presents extreme data distribution that some head categories have 1,000 images but only 2 images are included in tailed category. Hence, a model by conventional training is expected to yield classification performance, displaying the long-tailed distribution on balanced test/val set.
It goes without saying that techniques based on deep neural networks have been the focal point of the recent development in tackling FGVC. Characterized by powerful model capacity and end-to-end feature learning, these state-of-the-aar approaches are craftily designed to extract discriminative local details and consistent global structure, and shown to achieve significant improvements over conventional Don-DNN approaches, e.g., Duan et al. (2012) However, the improvement for solving FGVC by exploring visual features of different levels and resolutions from relevant regions seems to be saturated and also does properly address the long- tailed issue. The concern is reflected by that most FGVC methods do not include experimental results on the natural world distribution dataset INtuuralist2018 (Van Horn al. 2018).
almost all FGVC benchmark datasets.
Motivated by these developments, we propose a flexible and effective regularization design that aims guiding the resulting DNN learning improve model efficiency on tackling the FGVC and long-tailed issues at the same time. Our method is relevant to the pairwise confusion regularization (Dubey al. 2018); however, the proposed formulation goes beyond the restriction of working on pairs of data and develops a batch norm-based framework with sufficient model capacity to simultaneously deal with FGVC and long-tailed issues. We first assume samples/images within batch are of different classes. The targeted confusion energy is then modeled by a batch-wise matrix norm, termed as Batch Confusion Norm (BCN). The matrix is constructed by including prediction results from all images within batch, as well The former used to handle the FGVC task and the latter is for resolving the long-tailes distribution. To achieve efficient DNN learning. we provide approximation scheme BCN that gradient backpropagation can be readily carried out. The promising experimental results support that BCN has good potential to function generic regularizer for solving wide range of classification tasks.
adaptive matrix to adjust class-specifie weights.
2

---

Under review as a conference paper at ICLR 2021
variation
Inter-class variation + Inter-class similarity
LPC
LCE
CCB
Features extraction
Features extraction poraction
Pair of images
(b)
Inter-class and Intra-class variation Inter-class similarity + class Imbalanced
LpCN
LeR
Features extraction
Fregitei
Caiman
B Batch images
Figure 2: Learning to FGVC with (a) conventional Cross Entropy (CE) loss, (b) + Pairwise Confusion (PC) energy and (c) + the proposed Batch Confusion Norm (BCN).
2 RELATED WORK
Researches in fine- ane-grained and long-tailed visual classification going in two different branches. Most articles focus on just one of these issues. We will introduce recent studies on both sides, and then briefly explain approach.
FGVC. In the early works, the training data are annotated with additional information such as part labels. Along this lirr, et al.] (2014) explore the labeled part locations eliminate highly similar object categories for improving the learned classifiers. The approach in Huang et al. (2016b) established based on a two-stream classification network to explicitly capture both object-level and part-level information. However, owing to the rapid research advances in visual classification, the majority of recent FGVC approaches are designed to complete the model learning solely based on
the information of category labels (Sun et al.] 2019; Dubey et al. 2018 Wang et al. 2018 Li et al.
2018
Yang et al. 2018 Zheng et al. 2019: Chen et al. (019)).
Long-Tail. To alleviate the impact of the unbalanced data, the two common basic methods are re-sampling and re-weighting. In recently, most common strategy is called class-balanced sampling (Shen al. 2016). Different from instance-balanced sampling, every image has the same probability of being selected, class- balasceblanedd is to weight the sampling frequency of each image according to the number of samples of different categories. Furthermore, Guptt al. (2019) proposed repeat dlynamicssappiigg mechanism, to balance the instances. In general, re- sampling means that in the case of unbalanced existing data, the training samples that model artificially exposed to during learning category balanced,
factor sampling (RFS),
to reduce the overfitting of the head data to certain extent. Recently, Cao et al. (2019) introduces a label distribution aware margin loss that expands the decision boundaries of few-shot classes. Kang et al. (020)) decouples the learning procedure into two-stage, representations learning and classifier. And gives the conclusion that sittance-balanee sampling gives more generalizable representations which improve the performance after refining the classifiers by re-sampling.
Confusion energy. The conlusion-related formulation for dealing with intra-class variations and inter-class similarity in FGVC have two main implications. First, it can be applied to alleviate the overfitting problem in training a FGVC model. Dubey et al Dubey et al. 2018) construct a Siamese neural network, trained with loss function including pairwise confusion (PC). The reasoning behind the design is that bringing the class probability distributions closer to each other could prevent the learned FGVC model from overfitting sample-specific artifacts. Second, the confusion tactic can be used to boost the FGVC performance by focusing on local evidence. Chen et al Chen et al. (2019) partition each training image into several local regions and then shuffle them by region confusion

---

Under review as a conference paper at ICLR 2021
mechanism (RCM). It implicitly excludes the information about the global object structure and forces the model to predict the category label based on local information. In other words, the ability of identifying the object category from local details is expected to be enhanced through shape confusion.
Our approach to FGVC and long-tail is most relevant to the above ceafuuiion-baseee approaches. We retain the advantages of confusion energy and exploit the potential in the long-tailed distribution. And then propose a novel confusion energy term called Batch Confusion Norm (BCN) which can flexibly adjust the confusion strength corresponding the data distribution.
METHOD
The core our method centers around the proposed BCN to not only alleviate the overfitting problem in training an FGVC model but also boost the classification performance. We begin by discussing the pairwise confusion energy (PC) (Dubey et al.] 2018) and then elaborate the essential components of the proposed framework. We conclude the section with detailed explanations how BCN is used to handle the long-tailed distribution dataset, namely, iNaturalist0018 (Van Horn et al. 2018).
3.1 BATCH CONFUSION NORM
Let T be the FGVC model as illustrated in Figure 2 and D be the training set over totally C' fine- grained categories. An arbitrary sample from D is denoted (x, y) where our case x is an image and y E .I...., C} is the corresponding class label. In learning 4, we follow the standard batch training and set the batch size include M images.
...,.,C'
For each training sample in a batch B, the forward propagation through & would yield class probability (i.e., softmax) distribution Pt E R²C We can then define the batch-wise class prediction matrix by
P |P P2 ... PM] RCXM
(1) the C fine-grained categories. Notice that the
where each p, is the softmax class prediction formulation assumes M S C and all images within a batch B are randomly sampled. In contrast, the confusion regularization of Dubey al. (2018) only affects the paired images with distinct labels.
An explicit purpose of BCN is to infuse slight classification confusions into the FGVC training procedure and drive the learning to work harder for making as many correct predictions in each training batch as possible. To this end, it is reasonable to minimize the rank of the batch prediction matrix P that all individual predictions are similar:
arg min rank(P... +
However, the rank-related minimization problems are often Na-hard, and convex relaxations are instead introduced to approximate the solutions. In our formulation, minimizing the rank of P is reduced to minimizing its nuclear norm. We define the batch confusion norm of as
||
||P||
(3)
where || - is the nuclear norm that computes the sum of the singular values of the underlying tensor/matrix.
Stability. In order to make the matrix decomposition of P being stable and to prevent the negative singular values affecting the training loss, we could replace the right-hand side of (3) with | P|l- since it is known that
rank(P) = rank(P
(4)
Adaptability. The BCN defined in (1) confuses each category evenly. However, the strategy does not take account of realistic data distribution such as long-tailed well as fine-grained, and it could yield performance drop. Pertaining to those tailed classes of a few samples, infusing even small amount of confusion energy could easily degrade their classification outcomes. To resolve this issue, we include an adaptive matrix e RCC into the model so that BCN can adjust the strength of confusion energy for each category. Here couple of criteria for initializing an appropriate A:
4

---

Under review as a conference paper at ICLR 2021
When the data distribution is long-tailed, A should alleviate the confusion energy the tailed categories to prevent the model from getting excessive confusions over these classes.
When the data distribution is balanced, A should be almost the same as the identity matrix.
Follow these guidelines, design the adaptive matrix A
Aij { 0,
{ {
11 (Ng (1og
+ 1))²
i= i x
(5)
where N,, E d VV category, size mean, size standard deviation, and hyper-parameter, respectively. Note that when N2 + μ., we have An + 1. Furthermore, if' o - 0 then A& + 1. This means that A will reduce to the identity matrix when the data distribution is balanced.
11)². and are the size of each
Hence, to incorporate the batch confusion energy with the adaptive matrix A into the total loss function for training, we have
LENN = ||PT A AP||BCN,
(6)
where the batch confusion loss LECN is computed based on the eigenvalues of P AP.
Learnability, In practice, there is no feasible way to ensure that the parameters of A given in (5) assume the optimal confusion. We instead use it good initialization and go for learnable adaptive matrix, denoted Á. Consequently, we modify the LCCN into
ÂBCx = |P ÄT APl[BON + 7 Al22,
where 7) the away from A. Empirically, we initialize Å with and set = to gain improvement over simply using the hand-crafted A as the adaptive matrix.
to regulate that the learnable adaptive matrix À should not be too far
3.2
LOSS FUNCTION
From 7 and the network architecture in Figure 2 the refined feature maps are followed by a fully- connected softmax layer to output the class prediction vector p. The overall loss function can now be readily expressed by
C = LcE + LBON
(8)
where LCe is the cross-entropy loss which is usually applied in classification task and À is a hyper- parameter adjust the influence of the BCN loss to learning the model.
EXPERIMENTAL RESULTS
We conduct extensive experiments to evaluate our approach on three balanced benchmark FGVC datasets and the natural world distribution dataset. We then describe comparisons to prior work as well as the implementation details. We also provide an insightful ablation study for assessing the performance gains of using adaptive confusion energy BCN. Finally, a number of visualization examples are demonstrated for further discussions.
4.1 DATASETS
We first evaluate the effectiveness of the proposed approach on three standard fine-grained visual classification datasets, namely. and FGVC-Aireraft (Maji et al. 2013). Table I shows the detailed statistics with the numbers of training and testing splits along with the category numbers. The data ratio between the training and the testing is about 1 for CUB-200-201 and Stanford Cars, and is about 2 ; 1 in FGVC- Aircraft. The class distribution of the three datasets is nearly balanced which can be used to measure the performance of the proposed method only in the fine-grained scenario with adaptive matrix Á
(Wah et al. 2011), Stanford Cars (Krause et al. 2013)
5

---

Under review as a conference paper at ICLR 2021
Table 1: Statistics of 3 balanced FGVC datasets and the natural world dataset iNaturalist2018.
Dataset
CUB
CAR
AIR
iNaturlist2018
# Train
5,994
8, 144
6,667
437513
# Val/Test
5,794
8,041
3,333
24426
Category
200
196
100
8142
approximating identity matrix. Compared with other datasets for the large-scale visual classification task, these three FGVC datasets have obviously fewer training data for each category.
We then focus on the natural world distribution dataset iNaturalist2018 Van Horn et al. 2018) which has the properties of both fine-grained and long- tailed distribution. Besides, it is also large-scale dataset. Judging from the recent literature (Cao et al.] 2019 Kang et al.j (020). this fairly challenging dataset that the performance can our method. Finally, we remark that the proposed model does not require any additional annotations in the training process but merely the image-level class annotations.
an objective measure about the usefulness of
4.2 IMPLEMENTATION DETAILS
We describe the implementation details with FGVC and iNtuuralist2018. All our inference results are obtained from end-to-end training. We implement method by using Pytorch framewrok (Pazzke ct al. 2017), and the source code will be made available online.
FGVC. Following relevant work (Yang et al. 2018 Chen et al., 2019 Zheng et al. 2019), we evaluate our method on the widely-used classification backbone ResNet-50 (He et al. (016)) which pre-trained on the ImageNet dataset. For the sake of fair comparison in FGVC training, we use the data augmentation setting in Chen al. (2019) that the input size is set as 448 x 448, and horizontal flipping is randomly performed. The initial learning rate and the hyper-parameter are 0.008 and 10, respectively. The training batch size usually is 16 if the GPU memory enough and the training optimizer is Momentum SGD, which accompanies with cosine annealing (Loshchilov & Hutter 2017) as the learning rate decay.
INaturalist2018. We further evaluate the proposed BCN on the iNaturalist2018 dataset. In addition to using similar augmentation schemes, we set up the training conditions as in Kang et al. (2020); Cao al. (2019) that the backbones, input size. training epochs, optimization method, and learning rate schedule are ResNet-50/ResNNr1522 224, 90, SGD, and cosine annealing, respectively.
Evaluation. After training on the FGVC and natural world datasets, we evaluate the models on the corresponding balanced test/validation datasets and report the top-1 accuracy which is used commonly. The value of accuracy is reported in the format of percentage.
4.3 PERFORMANCE OVERVIEW
The proposed BCN not only preserves the benefits of confusion energy in FGVC task, but also addresses the downside of the confusion energy in long-tailed challenge. Table the overall results on three benchmark FGVC datasets and the natural world distribution dataset. In FGVC, we
have developed general way to explore confusion energy (compared with PC). In Table 2h. we compare the results with recent state-of oftee--f-t-htrr-ree approaches on ResNet-50 backbone (Sun et al. 2019
Dubey et al. 2018 Wang et al. 2018 Li et al., 2018 Yang et al.] 2018 Zheng et al. 2019 Chen et al. 2019). 9.. We further boost the performance with a simple attention mechanism GASPP to make BCN more competitive to the state-of-the-art approaches. The details of GASPP can be found in Appendix B.] Table 2p includes the results on iNaturalist0118, We see that when adaptive matrix set to the identity matrix I, the of confusion energy drops the performance in both PC and BCN. The performance drops indicate that the adaptive matrix Á plays pivotal role in solving the long-tailed problem. Indeed, À enables BCN to just focus on the head categories but alleviate the

---

Under review as a conference paper at ICLR 2021
Table 2: Accuracy (%) on the test sets of three FGVC datasets and the validation set of NNturallis22018. The superseript means that the models are re-implemented by the same training setting to ours.
(a)
Method
CUB
CAR
AIR
Method
ResNet-50
ResNet152
PCT
87.0
93.5
92.4
CB-Focal
61.1
DB
87.7
94.3
92.1
LADM (SGD)
64.6
DFL-CNN
87.4
93.1
91.7
Baseline
61.7
65.0
NTS-Net
87.5
93.9
91.4
cRT
65.2
68.5
DCL
87.8
94.5
93.0
LWS
65.9
69.1
TASN
87.9
93.8
PC
61.4
64.1
iSQRT-COV
88.1
92.8
90.0
Ours (A = I)
61.6
64.7
Ours
87.8
94.3
93.2
Ours
66.1
69.5
Ours + GASPP
88.4
94.7
93.5
Dassine
60
Accurar
Frequent
Common
Bam
Sorted catagory Index
Frequent
(a)
(b)
Figure 3: Baseline versus confusion energy models. (a) The noom each category corresponds to the weight w; in the classifier. (b) The classification accuracy of Frequent, Common, and Rare from the baseline and the other two confusion energy methods, PC and BCN.
confusion energy effect on the tailed categories. Note that, our models are trained not only with the common way of data sampling ristance-balanced sampling but also end-to-end. In contrast,
Kang et al. (2020) trains the model in two stages and requires the use of class-balanced sampling.
4.4 ANALYSIS
Dubey et al. (2018) has shown that confusion energy alleviates the overfitting problem and improves the FGVC performance. However, we observe that if the baseline model coupled with the confusion energy directly, the overall performance will drop on the natural world dataset. It suggests that the long-tailed issue needs further investigations beyond the conventional model confusion energy. Consider next the magnitude of each category corresponding to the classifier weight w; in Figure 3h. The scale of [||[w;]] ||]| distribution on the baseline method is very similar to the data distribution. In addition, although PC has alleviated the scale of the head categories, but the distribution does not change significantly. Note that BCN (Á = has the similar phenomenon PC. Nevertheless, the adaptive confusion energy BCN, makes the scale of the head to become smoother. This means that the prediction of the classification will not be dominated by the weights of head categories. We
7

---

Under review as a conference paper at ICLR 2021
(estlee-50)
S0-PPC
(ResNNet-SHHB CNi
(Res Nee Scy
(Resh Net S0HPC)
(Resteet (ResNet-SHHCM S0HBCNI
RRas 50)
RResNet S0:PC]
FResNet S0HBCN)
|Reshlen 50g
(Reshket S0MPCI
iRestict 50HBCN)
(Resh Net
(BesNee-50)
MassNis-ooHPP
(R (ResNet-S0--
Classifictioo result
Classification result
Figure 4: visualizzainnn of testing images by Grad-CAM (Selvaraju et al. 2017). The spatial heatmaps show the responses of the network to different images. For each image the first column shows the input images; the remained three columns show the corresponding heatmap of each model.
further split the categories into three groups, Frequent (N; >= 100). Common (Nę >= 10), and Rare (Nr < 10), and evaluate the performance of each group. The results presented in Figure Bp. where the conventional confusion energy still behaves like the baseline, but BCN yields a more uniformly distributed performance.
4.5 DISCUSSIONS
In summary, BCN provides several benefits. First, it alleviates the overfitting problem of the cross- entropy loss. While training with the cross-entropy loss concerning the ground truth label in the manner of the one-hot vector, the inter-class similarity information is usually significantly suppressed. Consequently, it is able capture the fine-grained essence by one single cross-entropy loss while handling the overfitting issue. The proposed BCN successfully alleviates this issue. Second, BCN forces the model learn the inter-class similarity so that the classifier is more focused on the discriminative parts. This phenomenon can be found by using the class activation mapping (Grad-CAMe (Selvaraju et al. (017)) presented in Figure Without any attention mechanism and additional annotation, we successfully make the attention region smaller and accurately attend on the discriminative parts only by BCN. Third, BCN does not require additional processing of inputs and outputs during training and there is no extra cost at inference time, which makes it flexible and applicable to real applications. Final, BCN solves the confusion energy problem while meets the long-tailed distribution. BCN coupled with the adaptive matrix Å not only preserves the benefits of confusion energy in FGVC task, but also addresses its downside in long-tailed scenario.
CONCLUSIONS
We have developed a general regularization technique specifically designed for addressing the fine- grained visual classification and the long-tailed data distribution problems simultaneously. The proposed Batch Confusion Norm (BCN), together with the standard cross entropy loss can be used to account for the inherent classification difficulties due to inter-class similarity and intra-class variations. And also solves the long-tailes problem by an adaptive matrix term. The proposed BCN considers the confusion regularization within each training batch and thus is more general than the relevant formulation of pairwise confusion energy. The resulting model is shown to be capable learning discriminative features within regions of interest and alleviating the overfitting problem in training. The provided experimental results nearly achieve state-of-the-ar over the three mainstream FGVC datasets and competitive leading long-tailed approaches the natural world distribution dataset. Our future work will focus on generalizing the adaptive BCN concept to tensors and also on extending its applications other challenging computer vision problems.
8

---

Under review as a conference paper at ICLR 2021
REFERENCES
Thomas Berg, Jiongxin Liu, Seung Woo Lee, Michelle I. Alexander, David W Jacobs, and Peter N Belhumeur: Birdsnap: Large-scale fine-gra fine-grained visual categorization of birds. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 201 1-2018, 2014.
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with label-distriuttionaaaar margin loss. In Advances in Neural Information Processing Systems, pp. 1567-1578, 2019.
Liang-Chich Chen, George Papandreou, Iasonas Kokkinos, Kevin Murphy, and Alan L Yuille. Deeplab: Semantic image segmentation with deep convolutional nets, atrous convolution, and fully connected erfs. IEEE transactions pattern analysis and machine intelligence, 40(4):834 848, 2017.
Yue Chen, Yalong Bai, Wei Zhang, and Tao Mei. Destruction and construction learning for fine- grained image recognition. In Proceedings of the IEEE Conference on Computer Vision and Puttern Recognition, pp. 5157-5166, 2019.
Kun Duan, Devi Parikh, David Crandall, and Kristen Grauman. Discovering localized attributes for fine-grained recognition. In 2012 IEEE Conference on Computer Vision and Pattern Recognition, pp. 3474-3481. IERE, 2012.
Abhimanyu Dubey, Otkrist Gupta, Pei Guo, Ramesh Raskar, Ryan Farrell, and Nikhil Naik. Pairwise confusion for fine-grained visual classification. In Proceedings of the European Conference on Computer Vision (ECCV), pp. 70 70-86, 2018.
Jianlong Fu, Heliang Zheng, and Tao Mei. Look closer to see hetter: Recurrent attention convolutional neural network for fine-grate fine-grained image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 4438-4446, 2017.
Weifeng Ge, Xiangru Lin, and Yizhou Yu. Weakly supervised complementary parts models for fine-grained image classification from the bottom up. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 3034-3043, 2019.
Agrim Gupta, Piotr Dollar, and Ross Girshick. segmentation. In CVPR, 2019.
LVIS: A dataset for large vocabulary instance
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 3375-53384 2016a.
Shaoli Huang, Zhe Xu, Dacheng Tao, and Ya Zhang. Part-stacked can for fine tne-grained visual categorization. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 1173-1182, 2016b.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. In International Conference on Learning Representations, 2020. URL httpt:tpp/.. http://oopeeeeeeeeeeeeeee ffrrmm?
id=r1gRIcVPv2
Jonathan Krause, Michael Stark, Jia Deng, and Li Fei-Fei. 3d object representations for fine-grained categorization. In 4th International IEEE Workshop 3D Representation and Recognition (3dRR-13), Sydney, Australia, 2013.
Peihua Li, Jiangtao Xie, Qilong Wang, and Zilin Gao. Towards faster training of global covariance pooling networks by iterative matrix square root normalization. In Proceedings of the IEEE Conference Computer Vision and Pattern Recognition, pp. 947-955, 2018

---

Under review as a conference paper at ICLR 2021
Iya Loshchilov and Frank Hutter. SGDR: stochastic gradient descent with warm restarts. In 5th International Conference on Learning Representations. ICLR 2017, Toulon, France, April 24-26, 2017, Conference Track Proceedings, 2017.
Subhransu Maji, Esa Rahtu, Juho Kannala, Matthew B. Blaschko, and Andrea Vedaldi. Fine-grained visual classification of aircraft. CoRR, abs/1306.5151, 2013. URL
1306. 5151
Adam Paszke, Sam Gross, Soumith Chintala, Gregory Chanan, Edward Yang, Zachary DeVito, Zeming Lin, Alban Desmaison, Luca Antiga, and Adam Lerer. Automatic differentiation in pytorch. 2017.
Ramprasaath R Selvaraju, Michael Cogswell, Abhishek Das, Ramakrishna Vedantam, Devi Parikh, and Dhruv Batra. Grad-cam: Visual explanations from deep networks via gradient-based local- ization. In Proceedings of the IEEE International Conference on Computer Vision, pp. 618-626, 2017.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In Computer Vision ECCV 2016 14th European Conference, Amstendam, The Netherlands, October 11-14, 2016, Proceedings, Part VII, pp. 467-482, 2016. doi: 1111077978-33331-444777
Guolei Sun, Hisham Cholakkal, Salman Khan, Fahad Shahbaz Khan, and Ling Shao. Fine- grained recognition: Accounting for subtle differences between similar classes. arXiv preprint arXiv:1912.06842. 2019.
Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEÉE conference on computer vision and pattern recognition, pp. 8769 87788 2018.
C. Wah, S. Branson, P. Welinder, P. Perona, and S. Belongie. The Caltech-UCSD Birds-200-2011 Dataset. Technical Report CNS-TR-2011 1-001, California Institute of Technology, 2011.
Yaming Wang, Vlad Morariu, and Larry $ Davis. Learning a discriminative filter bank within a cnn for fine-grained recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 4 4148-4157, 2018.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. In Advances in Neural Information Processing Systems, pp. 7029-7039, 2017.
Ze Yang, Tiange Luo, Dong Wang. Zhiqiang Hu, Jun Gao, and Liwei Wang. Learning to navigate for fine-grat mine-grined classification. In Proceedings of the European Conference Computer Vision (ECCV), pp. 420-435, 2018.
Heliang Zheng, Jianlong Fu, Zheng-Jun Zha, and Jiebo Luo. Looking for the devil in the details: Learning trilinear attention sampling network for fine-grate finegrrined image recognition. In Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition, pp. 5012-5021, 2019.
10

---

Under review as a conference paper at ICLR 2021
APPENDIX
In this section, we provide some supplementary information to explore how confusion energy regularization term works FGVC task. And then present some interesting phenomena that the conventional methods does not have.
A ABLATION STUDY
Table 3: FGVC accuracy comparisons on the standard FGVC datasets CUB-200-2011 (CUB),
Stanford Cars (Cars), and FGVC-Aircrate (Aircraft).
ResNet-50
ResNeXt-50
ResNeXt-101
Model
CUB
CAR
AIR
CUB
CAR
AIR
CUB
CAR
AIR
baseline
85.5
92.7
90.3
86.3
93.1
90.9
87.3
93.5
91.6
baseline + PC
87.0
92.4
90.1
87.5
93.2
91.2
88.2
93.7
92.4
baseline BCN
87.8
94.3
93.2
88.1
94.4
93.3
88.6
94.5
93.5
To investigate the performance of different confusion energies between the different backbones, make an ablation study on the ResNet-50, ResNcXt-50, and ResNeXt101. Table 3 shows the ablation between PC and BCN. The confusion energies both obviously improve the baseline performance the CUB. Since CUB the most difficult in the three benchmark datasets. But while the datasets are more easier, although BCN gains improvement, the confusion energies provide little help. And take a look at the FGVC researches recently, it seems to have reached the limitation so far. Hence, reasonable go through the more challenge task, fine-grained and long-tailed.
B ARCHITECTURE DETAILS
Let F' E Re be the feature maps obtained by the last convolutional layer of the ResNet backbone as shown in Figure We extend the network with two streams: one for learning discriminative features and the other for uncovering the proper attention responses. In both streams, we use the atrous spatial pyramid pooling (ASPP) technique to simultaneously extract features/attentions from different field-of-views.
ResNet-50 backbone
GASPP
MLP
LBN
10000
1000
L
Batch h of images
coRv2
Figure 5: The proposed neural network architecture for tackling the task.
B. ATTENTION GATED ASPP
In addition, our method is implemented with an attention-gated network, boosted with the use of Atrous Spatial Pyramid Pooling (ASPP) technique. As shown Figure in the feature stream, per- forming ASPP would covert F into ASPP, (F)) while in the attention stream, the similar procedure would yield attention feature maps, ASPP, (F). Note that parameters in the two streams are not shared but learned jointly. After regulating with respect the respective activation function, carry
11

---

Under review as a conference paper at ICLR 2021
Atrous Spatial Pyramid Pooling for Features
Attention Gated ASPP
GASPP
of ReLU max(0,
1xl Conv, 3x3 Conv with rata-1, 3, Spatial Pyramid Pooling Attentions
GAP
Elementwise multiplication
Ja Sigmaid o(s)
1x1 Conv, 3x3 Conv with rate=1, 3,5,7
Figure 6: The attention gated ASPP architecture
the gated element-wise product to output the adjusted feature maps F, weighted by the predicted attentions. That is, the attention caattiionggedddddd ASPP feature maps F are derived by
F σy (ASSPP)(()) o o. (ASPP(((F))
where F E Rexhxw remains the same dimensions, @ denotes the element-wise product, øy in our implementation is ReLU and a is the sigmoid function for gating. The ASPP operation is similar to that Chen et al. (2017) except that we use different dilated rates. In summary, ASPP, would learn the image features across the whole spatial domain and ASPPa (-) instead predicts the attention heatmaps. The gated fusion between the two streams leads to the output of discriminative feature maps for FGVC.
C BCN REASONING.
To justify the design of BCN, we use Figure to illustrate the underlying mechanism. As we have mentioned before, the complexity of FGVC originates from the dilemma of simultaneously addressing the inter-class similarity and intra-class variations. Our formulation requires the samples in each training batch to be of different labels, and thus leads to two extreme cases to be considered. When the inter-class similarity in a batch significant, as shown in Figure 77, the BCN loss can be considered typical regularization term to avoid overfitting. On the other hand, when inter-class similarity is not of concern as in Figure pb, the distributions of class prediction from the samples in a batch could vary significantly and induce substantial BCN loss. Thus, the training would explore discriminative features to reduce the cross-entropy loss and consequently boost the FGVC performance.
D THE ADVANTAGE OF BCN
To investigate the advantage of the proposed BCN, we show the prediction layer with softmax activation Figure 8 In standard deep neural networks for classification, the loss function for prediction layer is cross-entropy. Other methods such as NTS-Net Yang et al. (2018), DCL Chen al. (2019) and Zheng et (2019) adopt many loss functions, but their loss function for the prediction layer remains the same. If only the cross-entropy is used to learn the prediction layer, the output probability tends be very close one-hot vector even though the prediction is correct. We highlight the observation in Figure 8 This outcome conflicts the real scenarios in FGVC problems. In FGVC, the categories sharing high similarities should have similar classification probabilities to recognize them but not concentrating on a specific category as one-hot vector. With the help of BCN, which already learned the characteristics of FGVC, the satisfactory results are expected.
Figure 9p shows the influence of confusion regularization in respect of the training loss. This experiment shows that training our model various value of has similar convergence speeds but has noticeable saturated training losses. The higher weight of confusion regularization brings the larger training loss, and the larger training loss means the model is harder to over-fit fit the training data. Notice that the standard FGVC datasets often have small number of training images for each category, hence, it is easy for a deep neural network to over-fit fit the training data. Figure 99 presents that with the confusion regularization term, the entropy of prediction p, would be larger than the one
12

---

Under review as a conference paper at ICLR 2021
Dual-effect of BCN loss: C LCe + XLBON
(a) Heavy inter-class similarity Batch: BCN alleviates overfitting
Herring Guil
Ring billed Guil
Californi Gull
Glaucous Winged Gull
(b) Moderate inter-class similarity Batch: BCN improves FGVC in tail classes
Painted Bunting
Rose Breasted Grosbeak
Horned Grebe
Winter Ween
All images in a training batch are of different class labels. (a): When inter-class similarity prevails in a batch, the BCN loss functions as regularization to avoid overfitting. (b): When inter-class similarity is not obvious, the batch would yield a significant BCN loss and the optimization would turn to further reduce the cross-entropy loss and thus is expect to boost the FGVC performance in inference.
Figure 7:
Testing Image
Testing Image
Eickes
07
00 Biad Gal--
0.7-
Prediction with BCN
Predictions
0.6
400
Probability
with and w/o BCN
90
Similer image
Similar image
033
BOCHe Maged
03
Calferra
Prediction
Gf
50
Category
Catagory
Figure 8: Prediction viibaiiiaiissaaliiationn of FGVC. The X-axis denotes the category index, and the Y-axis denotes the classification confidence. This figure shows the predictions of ResNet-50 with/without coupling BCN. Blue lines indicate coupling correctly classifies the two testing images. Green lines indicate the predictions of the vanilla ResNet- 50, which only correctly classify the right testing image. One similar image is attached under each test image. With the aids of BCN, the classification model try to exert regularization effect from other images in the same batch, hence shows hesitation among similar categories. However, the model trained without confusion loss seems very confident though making wrong prediction.
proposed BCN with the vanilla ResNet-50, which
without in inference. The model has learnt the inter-class similarity and needs to make the decision from the discriminative points. For The black line in Figure D shows that it is easy to over-fit the CUB-200-2011 Standord Cars and FGVC-Aireraf datasets using ResNet-50 without applying BCN or PC. Therefore, the proposed BCN and previous PC can be used to reduce the over-fitting issue in the FGVC task.
13

---

Under review as a conference paper at ICLR 2021
CUB-200
BGN/ 122
PCI
0.7
Cross entropy towa
PGI
0.4
CUB-200
@nteltes33 BCH (mooe)
0
Confusion Energy
Dieetee-13
ORecken-15 Gaaciee:0) yreas]
20
60
80
100
opoch
(a)
Figure 9: The impacts by BCN and PC on CUB-200-2011. (a): The regularization terms promote the performance in the FGVC task by confusion energy. Each point represents class in the task. The BCN more stable than PC, giving each class similar confusion energy, which eventually leads to more classes with higher accuracy. (b): The training CE loss with different À values. The hyper-parametts À used control the weight BCN and PC in the loss function. According to these figures, we observe that the phenomenon is similar in each data set.
Take attention to the situation with À = 100. Figure 9p. while BCN provides valid predictions, the PC destroys the entire training process. This indicates that BCN is insensitive and stable the influence of hyper-parameter A. At the same time, there is wide range between 10 and 100 but the performance very similar. So don't have to perform manual tuning. In addition, when the hyper- aperr-paaaettr À is greater than 1.0, the PC may provide better results, but it may also cause the model be completely untrainable.
14


Reviews:
[{"review_id": "evX4kdIRf8r", "reviewer": "AnonReviewer1", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Overview:** The paper presents an extension to the Batch Confusion Norm (BCN) regularization technique so that it can account for imbalanced datasets. The extension to BCN implies adding a matrix that determines its values as a function of class imbalanced statistics contained in a batch. The paper presents experiments showing the benefits of the proposed extension on Fine-Grained Visual Classification (FGVC) and long-tailed (LT) tasks. The experiments show modest improvements over the baselines.\n\n**Pros:**\n*Clarity of the paper is good.* The clarity of the paper is very good. The motivation behind the FGVC and LT learning problems as well as the proposed method is clear. Thanks to the clarity of the paper I believe the reproducibility should be good. Also, I believe that the paper addresses an important problem with practical value.\n\n**Cons:**\n*Novelty.* The novelty of the paper IMHO falls short. This is because the submission mainly extends the BCN paper by adding a matrix A whose entries are a function of the statistics of the statistics of the dataset.  The proposed extension lacks a more rigorous derivation and justification; IMHO, the proposed extension seems to be ad hoc.\n\n*Not convinced about BCN and proposed extension.* \n1) I am not sure if BCN is a proper regularizer. While I understand the geometry behind minimizing the rank of the matrix P, I don't think this is a proper way of processing the columns of matrix P which are *posterior distributions*. I think the paper lacks a clear justification about using BCN as a way to treat posterior distributions without using statistical or probabilistic methods. \n2) BCN is conditioned to operate if the classes in the batch are unique. However, satisfying this condition in practice can be challenging. This becomes challenging when dealing with long-tailed datasets. What is the optimal/efficient way to guarantee a batch that satisfies this constraint? If the constraint is not satisfied, can BCN provide bad gradients because repeated classes with different posteriors in different columns can double its contribution?\n3) The extension of adding a matrix A is simple and feels a bit ad hoc. What is the intuition behind the matrix A in terms of the norm? Can the matrix A be considered a way to ` *weight* a posterior as a function of the statistics?\n\n*Insufficient experiments and baselines.* \n1) While I am pleased to see that the paper uses datasets depicting real scenes (e.g., iNaturalist, CUB, CAR, etc.), the experiments only focus on using one family of networks: ResNet. Does this method operate well on other more modern architectures, e.g., EfficientNet or MobileNetV2? \n2) In terms of baselines, I think the paper is missing recent long-tail methods:\n  a) Liu, et al. Large-Scale Long-Tailed Recognition in an Open World. CVPR 2019. \n  b) Cao, et al. Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss. NeurIPS 2019.\n3) The results for the long-tail methods are lacking more details. As it is common in various long-tail recognition papers, the paper is not showing performance on head or tail classes. It only shows an overall classification performance. This is misleading as the method can helping more head classes and thus improving the overall classification performance.\n\n*Minor concern*: Most of the figures and diagrams look nearly identical to those presented in the BCM paper.", "extra_scores": {}, "extra_text": {}}, {"review_id": "NUSZc-uBoOC", "reviewer": "AnonReviewer2", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper proposes the batch confusion norm (BCN) for dealing with both fine-grained recognition and long-tailed recognition simultaneously. Specifically, BCN considers the confusion regularization within each training batch and an adaptive matrix term is designed for handling the long-tailed problem. Experiments are conducted on fine-grained benchmark datasets (e.g., CUB, CAR, AIR) as well as long-tailed recognition datasets (e.g., iNat18). \n\nPaper strengths:\n- The paper is well organized and easy to follow.\n- The problems studied in this paper, i.e., fine-grained recognition and long-tailed visual recognition, are both important, challenging and practical in computer vision, which deserves further studies.\n- The proposed method sounds reasonable.\n\nPaper weaknesses:\n- Although the proposed method is reasonable, some specific model designs are not quite clear. 1) Regarding Eq. (2), the reason why it requires to optimize the ranking should be further explained and its motivation needs to state. 2) Regarding Eq. (5), what the intuition of the adaptive matrix (i.e., (log_{\\mu+1} (N_i+1))^{\\delta^{\\tau}}) when i = j should be provided to the authors.\n- The major issue of this paper is the experimental evaluations. 1) The classification accuracy on these fine-grained benchmark datasets and iNat18 are not significantly better than the accuracy of previous work. Thus, the effectiveness of the proposed method is problematic. 2) Some state-of-the-art methods are not involved in the experimental comparisons, such as [ref1-ref5]. Moreover, the accuracy of the proposed method cannot outperform these methods.\n\nMinor issues:\n- There are several typos and writing problems in this paper. For example, on Page 3, \"Dubey et al.Dubey et al. (2018)\", and \"Chen et al.Chen et al. (2019)\". On Page 4, \"PC Dubey et al. (2018)\". On Page 8, \"And also solves the long-tailed problem by an adaptive matrix term.\"\n\n[ref1] Weakly Supervised Fine-grained Image Classification via Guassian Mixture Model Oriented Discriminative Learning, CVPR 2020.\n\n[ref2] Weakly Supervised Complementary Parts Models for Fine-Grained Image Classification from the Bottom Up, CVPR 2019.\n\n[ref3] Fine-Grained Visual Classification via Progressive Multi-Granularity Training of Jigsaw Patches, ECCV 2020.\n\n[ref4] Learning Imbalanced Datasets with Label-Distribution-Aware Margin Loss, NeurIPS 2019.\n\n[ref5] BBN: Bilateral-Branch Network with Cumulative Learning for Long-Tailed Visual Recognition, CVPR 2020.", "extra_scores": {}, "extra_text": {}}, {"review_id": "GkhzrehohLy", "reviewer": "AnonReviewer4", "summary": "", "questions": "", "limitations": "", "rating": 5, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "This paper presents a novel technique for fine-grained visual classification. This technique addresses the classic issues in this task of inter-class similarity (coupled with intra-class variation) and the “long tailed” dataset problem, prevalent in datasets such as iNaturalist2018. The technique is an extension of Dubey et. al which extends their technique by incentivising the predictions for all samples in a mini-batch to be similar. This is in contrast to Dubey et. al which splits the mini-batch into two halves and incentivizes the aggregate predictions of the two halves to be similar. \n\nI find this to be of minimal novelty, and I question the practicality of the method. Unless I have misunderstood, this sounds like it would be slower to train and I did not see any runtime analysis comparisons in the experiments section. Experiments showing how the new loss function effects training time would help alleviate this concern. The method does however improve performance on the tested benchmarks. \n\nFinally, I found that there were numerous grammatical and stylistic mistakes. The writing improves during the discussion of the mathematics of the technique, but the introduction, experiments, and discussion need work. I would like to see the writing improved for a publication. \n", "extra_scores": {}, "extra_text": {}}, {"review_id": "7O5zRWYL6rD", "reviewer": "AnonReviewer3", "summary": "", "questions": "", "limitations": "", "rating": 4, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "", "summary_of_the_review": "", "strength_and_weaknesses": "", "limitations_and_societal_impact": "", "review_text": "**Summary:**\nThis paper is about fine-grained visual classification, which is challenging due to high inter-class similarity, high intra-class variation and potentially also class imbalance. The authors propose a regularization term that is added to a standard cross-entropy loss of a neural network trained in a supervised fashion. This regularization shares the motivation with prior art to confuse the network and reduce over-fitting by making all predictions within a mini-batch similar to each other. Different to prior art, they authors propose an approximation of a minimum rank objective. To also handle class imbalance, the authors extend the regularization with a learnable matrix that can automatically balance the importance of individual classes.\n\n**Pros:**\n- Overall, I think the contributions of the paper is interesting and useful, particularly the extension of the confusion-based regularization for class-imbalanced data sets. Unfortunately, the presentation of the idea needs significant improvement, see below.\n- The analysis and discussion in Sections 4.4 and 4.5 are great\n\n**Cons:**\n- The writing made this paper really hard to understand. The formulations, particularly in abstract and introduction, are inaccurate and vague. They should rather be specific and concrete. For instance, the statement \"When inter-class similarity prevails in a batch, the BCN term can alleviate possible overfitting due to exploring image features of fine details\" is hard to understand since it was totally unclear what the BCN term actually is at that point of reading. In both abstract and introduction, I had a very hard time imagining what certain statements mean in terms of who the method would look like, without having read the full paper.\n- The results barely improve over SOTA, particularly for the three FGVC data sets. So the bigger advantage I see was for class imbalanced data sets, like iNaturalist. Although the performance is also not significantly better than prior works, the direct competitor (PC) seems to under-perform clearly. The proposed solution alleviates the problem, which is good. In light of this, I think it would make sense to build such data sets synthetically from the existing ones (like Cub, Car, Air) by removing samples to increase class imbalance. This would allow additional experiments on class imbalance in a controlled setup.\n\n\n**Minor notes:**\n- The author names can be dropped with the ICLR citation format: \"... Dubey et al.Dubey et al. (2018) construct a Siamese\" on page 3.\n- Statements like \"The confusion-related formulation for dealing with intra-class variations and inter-class similarity in FGVC have two main implications\" on page 3 require the reader to be very familiar with these concepts. It would be better to re-formulate it with a brief introduction of the concept (just a sentence or two).\n", "extra_scores": {}, "extra_text": {}}]

Author rebuttal:

Meta-review: The authors address the problem of fine-grained image classification. They propose a batch based regularizer, called the batch confusion norm (BCN), to encourage less over confident predictions. They also tackle the problem of class imbalance during training by adaptively weighting the BCN loss at the class level to take the imbalances in the underlying label distributions into account. Results are presented on four different fine-grained datasets.  

Overall, while the reviewers had some positive comments, there was not broad support for the paper. There are questions that need to be resolved related to the evaluation e.g. the best performing model uses GASPP, however there is no reported GASPP variant for the PC baseline. Similarly, it would be valuable to know how much PC would benefit from an additional class imbalance term in the iNaturalist2018 results. Given that the proposed regularizer builds on PC (Dubey et al.), it is very important that the authors provide a like-for-like comparison so that readers can better understand the merits of the proposed method.  

There were also concerns with the presentation of the paper e.g. several typos (which can be easily fixed), issues with the clarity of the text (which require more work), and uninformative figures (e.g. Fig 2 should be revised to more clearly illustrate the differences between the three methods shown). The authors are encouraged to revise the text to resolve these problems. 

While the paper has some strengths (e.g. the empirical performance on some of the tasks is promising and the method is conceptually simple), there are still a number of concerns from the reviewers e.g. a lack of a clear motivation as to why the proposed method works, and why it is conceptually better than existing alternatives (e.g. PC). Given this lack of support, it is not possible to recommend the paper in its current form. 

Decision: Reject