Delving into Semantic Scale Imbalance
Authors: Yanbiao Ma
Licheng Jiao
Fang Liu
Yuxin Li
Shuyuan Yang
Xu Liu
Venue: ICLR 2023 poster (2023)
Keywords: Imbalanced Learning
Model bias
Long-tailed distribution

Abstract: Model bias triggered by long-tailed data has been widely studied. However, measure based on the number of samples cannot explicate three phenomena simultaneously: (1) Given enough data, the classification performance gain is marginal with additional samples. (2) Classification performance decays precipitously as the number of training samples decreases when there is insufficient data. (3) Model trained on sample-balanced datasets still has different biases for different classes. In this work, we define and quantify the semantic scale of classes, which is equivalent to the feature diversity of classes. It is exciting to find experimentally that there is a marginal effect of semantic scale, which perfectly describes the first two phenomena. Further, the quantitative measurement of semantic scale imbalance is proposed, which can accurately reflect model bias on multiple datasets, even on sample-balanced data, revealing a novel perspective for the study of class imbalance. Due to the prevalence of semantic scale imbalance, we propose semantic-scale-balanced learning, including a general loss improvement scheme and a dynamic re-weighting training framework that overcomes the challenge of calculating semantic scales in real-time during iterations. Comprehensive experiments show that dynamic semantic-scale-balanced learning consistently enables the model to perform superiorly on large-scale long-tailed and non-long-tailed datasets, which is a good starting point for mitigating the prevalent but unnoticed model bias. 

Published as a conference paper at ICLR 2023
DELVING INTO SEMANTIC SCALE IMBALANCE
Yanbiao Ma & Licheng Jiao & Fang Liu & Yuxin Li & Shuyuan Yang & Xu Liu
Key Laboratory of Intelligent Perception and Image Understanding of the Ministry of Education School Artificial Intelligence, Xidian University
Xi'an 710071, China
(ybmamail, 12 | @stu. cestu.xidian. xidian.ddii.. edu en, lchjiao@mail. wwt.mmaaaliiiiiiiiiii F631iuu163 com, syyangexidian. edu. en, xuliu3610163.co com
eidxiiiin anniannee edu, en,
ABSTRACT
Model bias triggered by long-tailed data has been widely studied. However, mea- sure based on the number of samples canmot explicate three phenomena simulta- neously: (1) Given enough data, the classification performance gain is marginal with additional samples. (2) Classification performance decays precipitously as the number of training samples decreases when there is insufficient data. (3) Model trained on sample-balancee datasets still has different biases for different classes. In this work, we define and quantify the semantic scale of classes, which is used to measure the feature diversity of classes. It is exciting to find experimentally that there is marginal effect of semantic scale, which perfectly describes the first phenomena. Further, the quantitative measurement of semantic scale imbalance is proposed, which can accurately reflect model bias on multiple datasets, even on sample-balancee data, revealing novel perspective for the study of class imbalance. Due to the prevalence of semantic scale imbalance, we propose semantic-scae balanced learning, including a general loss improvement scheme and a dynamic re-weighting training framework that overcomes the challenge of calculating se- mantic scales in real-time during iterations. Comprehensive experiments show that dynamic mmmantic-calle-allneee learning consistently enables the model to perform superiorly on large-scale long-tailed and non-long-tailed natural and medical datasets, which a good starting point for mitigatinn nitigating the prevalent but unnoticed model bias. Furthermore, we look ahead future challenges in the field.
INTRODUCE
In practical tasks, long-tailed class imbalance is common problem, and the imbalance in number makes the trained model easily biased towards the dominant head classes and perform poorly on the tail classes (25 78). However, what is overlooked is that, in addition to long- long-tailed data, our study finds that the model trained on This model bias is not taken into account by the study for class imbalance problem, and it cannot be ameliorated by the current methods proposed for long-tailed data [ 26 34 69). For natural datasets, classes artificially divided by different semantic concepts correspond different semantic scales, which can lead to different degrees of optimization when the deep metric is a single scale [52]. In this study, attempt to uncover more information from the data itself and introduce and quantify the semantic scale imbalance for representing more general model bias. The semantic-scall callebbllanced learning is further proposed, which used improve loss to mitigate model bias.
data still shows different biases for different classes.
The classes corresponding to different semantic concepts have different feature diversity, and we equate the diversity to the semantic scale. Usually, the finer the semantic concept of a class label, the less rich the feature diversity, the less information a model can extract, and the worse the model
performs on that class M 52 75 9 72 The manifold distribution hypothesis (57] states that a specific class of natural data is concentrated on a low-dimensional manifold. The larger the range of value variation along a specific dimension of the manifold, such as illumination and angle, the richer the feature and the larger the volume of the manifold. For example, in Figure [ since "Swan" is subclass of "Bird". its semantic concept is finer, so the feature diversity of "Bird" is richer than that of "Swan", and the corresponding volume of manifold is larger.

---

Published as a conference paper at ICLR 2023
Obviously, the semantic scale can be measured by the volume of manifold. We present a reliable and numerically stable quantitative measurement of the semantic scale and further define the semantic scale imbalance. To avoid confusion, we refer to the volume of the manifold calculated the sample space as the sample volume and the feature space as the feature volume. In addition, our innovative study for semantic scale imbalance can simultaneously and naturally explain the following two phenomena that cannot be explicated by existing studies of class imbalance:
(1) As the number of samples increases linearly, the model performance shows trend of rapid improvement in the early stages and leveling off in the later stages [61]
(2) Even models trained on the dataset with balanced sample numbers still suffer from class bias.
The experiments demonstrate that semantic scale im- balance is more widely present in natural datasets than sample number imbalance, allowing the study scope for class imbalance to be extended arbitrary datasets. Then, how to mitigate the adverse effects of semantic scale imbalance? We are inspired by classification meth- ods for long-tailed data, and current solutions for class imbalance problem usually adopt re-sampling strate- gies [59] 80 6) and cost-sensitive learning (24 74 (6).
Bind
remoppakk
However, re-sampling may either introduce a large num- ber of duplicate samples, making the model susceptible to overfitting when oversampling. or discard valuable samples when undersampling. Therefore by drawing on the classical re-weighting strategy (54), we propose caanttii-ssal--bbllanee learning. Its core idea is to dynamically rather than invariably measure the degree of imbalance between semantic scales in the feature space, to achieve dynamic evaluation of the weaker classes and assign greater weights to their corresponding losses.
122
Figure 1: The features from "Bird" mapped by CNNs are concentrated on a low- dimensional manifold, and the three-color point sets represent the three sub-classes of "Bird". Among them, the orange point set represents "Swan", whose feature vol- ume is obviously smaller than that of "Bird". The classification experiments on sample- balanced datasets show that the models are biased towards the classes with larger se- mantic scales, such as the decision surface shown by the green line. In this case, the re-weighting strategy based on the number of samples does NOT work, while our pro- posed re- weighting approach based on the semantic scale biases the decision surface toward the class with larger feature vol- ume (red line).
the dynamic
In this work, our key contributions are summarized as:
(1) We propose the novel idea of leveraging the volume of manifold to measure the semantic scale (Sec B.2 is also innovative to find that the semantic scale has the marginal effect (Sec 3.3), and that the semantic scale of the dataset is highly consistent with model performance in terms of trends.
(2) We introduce and define semantic scale imbalance, aiming to measure the degree of class imbalance by se- mantic scale rather than sample number, which reveals a new perspective for the study of class imbalance problem. Experiments show that semantic scale imbalance is prevalent in the dataset and can more accurately reflect model bias that affects model performance (Sec)) 3.4).
(3) Semantic-scale-baie balancee learning is proposed to mitigate model bias, which includes general loss improvement scheme (Sec/4.1) and a dynamic re- re-weighting training framework (See 4.2) that overcomes the challenge of calculating semantic scales in real-time during iterations. Comprehensive experiments demonstrate that semantic-scale. salane-alannedd learning is applicable to variety of datasets and achieves significant performance gains on multiple vision tasks (Sec
2 SLOW DRIFT PHENOMENON OF FEATURES AND MARGINAL EFFECT
Since the model parameters are changing during training, 666 studies the drifting speed of embeddings by measuring the difference in features of the same instance across training iterations. Experiments on the Stanford Online Products (SOP) dataset [41] show that the features change drastically in the early stage of training, become relatively stable after traversing the dataset twice, and drift gets
2

---

Published as a conference paper at ICLR 2023
extremely slowly when the learning rate decreases. This ensures that it is reasonable to leverage historical features to calculate semantic scales.
Marginal effect [1]] describes that in the early stages of model training, the network is able to learn features quickly, but since there is information overlap among samples, as the number of samples increases, the information of data will gradually saturate and the improvement in model performance from newly added samples will diminish. The effective number samples is proposed to represent the information of data, but its limitation is that it does work when the number of samples for each class is balanced.
Assume that the volume of each sample is unit volume 11 and define the set of all samples for class as S with volume N and N21. A new sample may overlap with a previous sample such that the probability of overlap is P and that of non-overlap is I-P. As the number of samples increases. the probability P will be higher.
where denotes the number of samples, hyperparameter 3 N-1 E [0, 1) controls how fast E,, grows increases. When = L, B and En = 1, mcaning that all samples can be represented by single prototype via data augmentation. When N + B - implying that there is no overlapping, then lim -lim (1-8") lim -
Define the effective number of samples as En, and E. 1+ B-B------ I-3
[--1 1-3 /-1 (1-3y 3-1 -1 nn, which indicates that the effective number of samples does not increase faster than the number of samples, but this is not consistent with our observation.
The effective number of samples E,, an exponential function of the number of samples n. The hyperparameters 3 corresponding to classes of different grain should be different. However, the selection requires more information from the data itself, but this problem not addressed by [Л which is forced to assume that / is the same for all classes. In this case, compared to the number of samples, E, simply uses the exponential function to obtain smoother weights. Furthermore, when the number of samples is balanced, En is the same for each class and cannot be used to mitigate model bias, so we attempt to mine the data for information (or feature diversity) of each class to facilitate the study of imbalance problem.
3 SEMANTIC SCALE IMBALANCE
In this section, first, sample volume, feature volume, and semantic scale imbalance are defined. Next, we derive quantitative measurement of feature volume to measure the semantic scale from the perspective of singular value decomposition of the data matrix and information theory. Then, the marginal effect semantic scale is investigated. Finally, we discuss the relationship between semantic scale imbalance and model bias.
3.1 DEFINITIONS
Different semantic concepts correspond to different semantic scales, for example, the scale of "Bird" is larger than that of "Swan". For each class, we equate its feature diversity to its semantic scale and measure the semantic scale by the volume of subspace spanned by samples or features. Deep neural networks can be viewed as combination of feature mapping function [(x, 0) and a trained downstream classifier g(=), i.e., I +
f(x:0) + g(z). Let the samples of a class be X [1, T2, ... F. and the embeddings learned by deep neural networks represented as 7 2 f (x, 0) 0) Rd 1, .., m).
Definition 3.1. (Sample volume) The volume of the subspace spanned by sample set X.
Definition 3.2. (Feature volume) The volume of the subspace spanned by feature vectors Z. Definition 3.3. (Semantic scale imbalance) A phenomenon of imbalance in the size semantic scales measured by sample volume or feature volume.
3.2 QUANTIFICATION OF SEMANTIC SCALE
Given the data X [x] [21. .2,...... [mm and the learned embeddings Z = [2 ...... 2md] e Rdxm s(x,,0 0) E Rdd 1,2,..., m. the volume of subspace spanned by the ran-
Fm]]
3

---

Published as a conference paper at ICLR 2023
dom vector 2; (i.e., the feature volume) is derived below, and the sample volume can be caleu- lated in the same way (Appendix E. The covariance matrix of random vector 2 is estimated
Σ E
2
LZZ m
E Rd X 2 A2 2
Aa are the eigenvalues
of real symmetric matrix. The singular value decomposition (SVD) Z yields Z UEVT and the singular values are dg Vλj.i 1, ........ d. The volume of the space spanned by the vector 2j is proportional to the product of all singular values of the feature matrix Z, i.e.,
d
Vol(Z) x 03
II >g. After the determinant expansion, the characteristic polynomial of 5=
==1
the matrix E is (2)) = -et((A - x (a11 + 022 + ... -.--aa)) + +... +............................. (-1)4 detE. and λ2 PA E. Therefore, the volume of the space spanned by the vector 2; is proportional to the square root of the determinant of the covariance matrix of Z:
Zol & V det( 1 277). m
The same result can be derived from the volume of a parallel hexahedron defined by vectors (Appendix G). Considering that real-world metric tools typically have dynamic range, for example, ruler always has multiple scales (Imm, lcm, or even 10cm) to measure objects of different scales, we expect the quantitative measurement of feature volume to have multi-scale metric and therefore use the sphere packing method g 377, which is normally adopted information theory, to implement
There is an error at the boundary when filling with hyperspheres because all spheres cannot be exactly tangent to the edges the manifold. The error of the finite feature vectors is assumed to be independent additive Gaussian noise 8 24' = + Wi, where w; NN,,, =" /) (n is the space dimension). The estimate of the number of spheres of radius E needed pack the space spanned by all vectors is Voolol //ool (Ball). The adjustment of the metric scale can be achieved by tuning the radius E of the spheres, thus controlling the measurement result of feature volume. Then the covariance matrix of the vector z is E'
_ + - zz" E Rddd such that (
Vol(Z') x (det(= + 1 ZZT) and Vol( to Ne:
ddet(= T). The feature volume is proportional
m
Vol (Z')
det (#1 + 1ZZT)
Vol (Z) x No
Vol (Ball)
det (1
Vdet (I+
n ZZT) ms?
The dimension of feature 2, is d, let n d. In order to increase the numerical stability, perform a logarithmic transformation of the above equation, which does not affect the monotonicity of the
function, and we can obtain Vol (Z) x log, det (I+ - ZZT) = 1/ log det (I+ ms? ZZT). In practical training, it essential normalize the feature vectors so that their mean value is O. In this work, we set 1000, and the value of & does not affect the relative size of the space spanned by feature vectors of each class. The volume of the space spanned by can be can be written as:
1 logg det(1 + - -(Z Z Zmeon (Z (2- Zmean)
Vol(Z) = 2
(3)
where Zeen is the mean value of Z, Vol(2) > 0 when the number of samples m >> 1. We measure the semantic scale S' by the feature volume, i.e., S' Vol and the larger S', the richer the feature diversity, which we verify in Appendix C usiin multiple Stanford point cloud manifolds.
3.3 MARGINAL EFFECT OF SEMANTIC SCALE
The marginal effect describes that the feature richness will gradually saturate as the number of samples increases, so the change of semantic scale should also conform to the marginal effect. Figure 2 illlssrrates that as the number of samples increases, the semantic scale S" measured by sample volume gradually saturates, which indicates that the quantitative measurement of semantic scale is as
4

---

Published as a conference paper at ICLR 2023
expected. In addition, the growth rate of the semantic scale varies across classes, which is determined by the grain size of the class itself. It leads to different semantic scales even if all classes have the same number of samples.
To investigate why adding samples has a marginal improvement in model performance when the training samples are sufficient,
the following method to generate new training datasets for classification experiments: assume that the total number of classes in the original dataset is C, and m samples are randomly selected from each class form a sub-dataset with a total number of samples of C' x m. The sub-datasets generated based on CIFAR-10, CIFAR-100 30 and Mini-ImageNet 62 are shown in Appendix D.I]
We train ResNet-18 and ResNet-34 17 on each of the 31 training sets in Table and the sum of the semantic scales for all classes and the corresponding top-1 accuracy are shown Figure We are pleasantly surprised to find that when the semantic scale increases rapidly, the model performance improves swiftly with it, and when the semantic scale becomes saturated, the improvement is small.
3.4
SEMANTIC SCALE IMBALANCE AND MODEL BIAS
3.4.1
QUANTIFICATION OF SEMANTIC SCALE IMBALANCE
Table I: Pearson correlation coefficients between the accuracy of classes and the semantic scales S with different a. N denotes the number of samples, and S' represents the semantic scale without considering inter-class interference. E, denotes the number of effective samples.
S
Dataset
Model
N
Fnp
W
S'
a=1
a=1.5
o=2
a=2.5
a=3
ResNet-18
0.8346
0.8664
0.2957
0.8688
0.8456
0.9603
0.9553
0,9398
0.9269
CIFAR-1O-LT
ResNet-34
0.7938
0.8476
0.3186
0.9426
0.7950
0.9678
0.9884
0.9854
0.9796
ResNct-18
0.0950
0.0950
0.1743
0.5433
0.7850
0.7250
0.6644
0.6060
0.5607
CIFAR-10
ResNet-34
0.1502
0.1502
0.2075
0.5750
0.8056
0.7442
0.6870
0.6465
0.5906
The previous subsection shows that the sum of semantic scales for all classes in the dataset is highly correlated with the model performance, and further investigate the relationship between semantic scale and model bias for different classes. When class is closer to other classes, the model performs worse on that class [40; I. Therefore, inter-class interference is additionally considered when quantifying the degree of imbalance between semantic scales of classes. When class i is closer to other classes, smaller weight w; is applied to the semantic scale of class i.
Sance Fesceii
Specifically, the semantic scale of m classes af- ter maximum normalization is assumed to be S' S S'ml' " and the centers of all classes are O [ot, o2...., om]" De- fine the distance between the centers of class and class j as dr, I1,, 2 the weight w; = - - ||o: - ozlly- The weights of m classes are written as W" [w1, ....... Wm] T After the maximum normalization and loga- rithmic transformation of W", we can obtain W=log, (a + W'), 2 1, where is used to control the smoothing degree of W. After consid- ering the inter-class distance, the semantic scale S= o W. and the role of S' in dominating the degree of imbalance is greater when is larger.
Un
Uri Irogeter
selanc 100 1077
Left column: curves of semantic scales with increasing number of samples for the first ten classes from different datasets. Right column: for different sub-datasets, curves of the sum of semantic scales for all classes and top-1 accuracy curves of trained ResNet-18 and ResNet-34. All models are trained using the Adam optimizer (28 with an initial learning rate of 0.01 and then decayed by 0.98 at each epoch.
Figure 2:
To obtain the most appropriate a, we calculate the Pearson correlation coefficients between the semantic scale and the accuracy of ResNet- 18 and ResNet-34 trained on CIFAR-10-LA and CIFAR-
5

---

Published as a conference paper at ICLR 2023
10, shown in Table I The experimental settings in Appendix D.2 It can be found that S' is more dominant on long-tailed data than on non- non-long-ten loog--tiiieedd taig-tailed data, and the improved S is better than S' and far better than the number of samples in reflecting model bias. In the following experiments, let a be 2 on the long-tailed data and on the non-tong-taile data.
3.4.2 SEMANTIC SCALE IMBALANCE ON
DATA
Previous studies have roughly attributed model bias to the imbalance in the number of samples. The experimental results in the first row of Figure (3 show that even though the number of MNIST-LI-1 is similar to that of MNIST-LT-2, the wlsss-wise aceuracy on MNIST-LT-I 1 is closer to that MNIST, just as their semantic scales are also more similar.
X003
2000
0" on actes meattr?
53
ACOBA
Con
Oun
wete to- Adcho
PoaIe IN aps ACOIN
Tex am 20
In addition, Figure 3 indicate that models on certain classes with fewer samples out- perform those classes with more samples, and that the semantic scale S reflects model bias more accurately. Further, also ob- serve that the accuracy of the CIFAR-I00-LT does not show significant decreasing trend, which can be explained by the marginal effect of the semantic scale (Sec 3.3
Lorela
Figure 3: Correlation study of accuracy with the num- ber of samples and semantic scale on MNIST, MNIST- LT (Appendix D.3). CIFAR-10-LT and CIFAR- 100- LT datasets.
3.4.3 SEMANTIC SCALE IMBALANCE ON NON-LONG-TAILDS DATA
Figure 4 demonstrates the model bias not only on long-tailed data but also on sample- balanced data. Usually, the classes with smaller semantic scales have lower accura- cies. Depending on the size of the semantic scale, it can make it possible for the weaker and dominant classes to be well differentiated. It should be noted that the weaker classes are not random, and experiments in Figure show that the models always perform worse on the same classes. More semantic scale im- balance of the datasets is shown in Appendix D.4 summary, semantic scale imbalance represent model bias more generally and appropriately, and further, we expect to im- prove the overall performance of the model when facing the semantic scale imbalance problem. Therefore, we propose the dynamic semantic-scale-balee balanced learning by drawing on the re-weighting strategy.
sore in My ANCITOI Easteen MOCIII
32343 PL0LEISAAN TUROOO
Kaster-- 10 VOC.16
2200
5.
Casa CFOD aut-criostt
Figure 4: curacy and semantic scale the sample-balanced dataset. Bottom row: performance of different mod- els (48) trained on the CIFAR-10 dataset and perfor- mance of ResNet- 18 trained on different sub-datasets of CIFAR-10 from of Appendix D.]
Top row: correlation study between ac-
4 DYNAMIC SEMANTIC-SCALES D LEARNING
Deep neural networks can be viewed combination of the feature mapping function and the classifier, and several studies have shown that model bias is mainly caused by classifier bias [80 (7) 3) so we are more concerned with semantic scale imbalance in the feature space, i.e., semantic scale measured by the feature volume. In this section, we propose general semantic-scale-bat based loss improvement scheme and design a training framework for the successful application of the scheme.
4.1 DYNAMIC SINANTICC-CAALE SAA---AALLLLLLEEED Loss
During training, the feature vectors corresponding to the samples vary with the model parameters, and thus the semantic scale per class is constantly changing. Compared with the traditional re-weighting

---

Published as a conference paper at ICLR 2023
strategy, propose to calculate the degree of imbalance between semantic scales in real-time at each iteration in order to dynamically evaluate the weaker classes and assign greater weights to their corresponding losses. Specifically, for class i at each iteration, normalized re-weighting
a terms 0; S.
inversely proportional to the semantic scales that take into account
:=1
inter-class interference, are introduced, and C' is the total number of classes. Given the embedding seatiic--cal--balaaeeee
of sample and label 8/., the dynamic (2, 31) = S. Lzz, 34)) 2, C, where yi is the label of the sample from class i. How to combine general loss to generate DSB loss is described in Appendix F.] Our approach has great potential improve the methods of re-balancing loss and adjusting sampling rate based the number of samples, because both the semantic scale and the number of samples are natural measures and they are not nedee-ddpendent.
(DSB) loss can be expressed as
However, the number of samples used at each iteration is limited, and it is not possible to obtain the features of all samples for calculating the semantic scales. Therefore, we propose dynamic re- re-weighting training framework that enables DSB loss to be successfully applied.
4.2 DYNAMIC RE- EEIGIIIG TRAINING FRAMEWORK
we design a storage pool Q to store and update historical features and propose a three-stae stage training framework. mini-batch of features can be dynamically updated at each iteration, and the semantic scale of each class is calculated using all the features in the storage pool. The three-stage training framework shown in Figure T6 and Algorithm 2 (More details are in Appendix F2), with the following textual description.
Inspired by the slow drift phenomenon of features 66 35
(1) In the first stage, all the features and labels generated by the 1st epoch are stored in Q, but they cannot be used directly to calculate semantic scales due to the large drift of historical features from current features in the early stage.
(2) The second stage corresponds to epoch 2 to epoch n. At each iteration, the oldest mini-batch features and labels in Q removed and those generated by the current iteration are stored. The goal is to continuously update the features in Q until the feature drift is small enough.
Imagehees TUP1 ACC1%)
55
TOP)
70
45
DSB-CE Imagenet-LT ResNext50
We set n to 5 in our experiments, and the original loss function is used in the first two stages. Figure shows the
DSB. RIDE imagenet- LT ResNext50
40
DSB-CE CIFAR100 ResNext18
effect of n on the model performance. A larger n does not hurt the model performance, but only takes a little more
DSB-CE CIFAR100 VGG16
5
10
11
time. Experience suggests that setting to 5 is sufficient.
Figure 5: The performance of different models with different losses on different datasets for different values of n.
(3) The third stage corresponds to epoch > n. At each iteration, the semantic scales are calculated using the fea- tures in Q after updating Q. and the original loss is re-whighted.
The comparison and analysis of the video memory and training speed are in Appendix F.2 We answer possible questions about the methods section in detail Appendix B
5 EXPERIMENTS
To validate the superiority and generality of the proposed dynamic semantic-scale-balaneee learning. we design four experiments. The first experiment is conducted on large-scale long-tailed datasets, ImageNet-LT and INaturalist2018 8 [60], to confirm the superior performance of our approach on long-tailed data. The second experiment uses large-scale ImageNet (4) and sample-balanced CIFAR- and the third experiment selects CIFAR-100-L and benchmark datasets commonly used in deep metric learning (CUB-2011 I [63) and Cars 196 (2]). The fourth experiment is performed on MSCOCO-GLT (56) to demonstrate the effectiveness of our approach in generalized long- tailed classification. The comprehensive experiments demonstrate the generality and superiority of our proposed method. More experimental results are provided in Appendix [D.5] Appendix (D6
(Results on the fundus dataset OIA-ODIR (3I)) and Appendix D.7 (Remote sensing image scene classification). The ablation experiments and additional analyses are in Appendix H
7

---

Published as conference paper at ICLR 2023
Table 2: Top-1 1 (6) on ImageNet-LT and INaturalis2218.. We use ResNext-50 [70] on ImageNet- LT and ResNet-50 [77 INaturalist0018 the network backbone for all methods. And conduct model training with the SGD optimizer based on batch size 256 (for ImageNet-LT) 512 (for
iNaturalist), momentum 0.9, weight decay factor 0.0005, and learning rate 0.1 (linear LR decay).
ImageNet-L (ReeN
iNaturalist 2018(R)sNet50)
Methods
Head
Middle
Tail
Overall
Head
Middle
Tail
Overall
BBN 80
43.3
45.9
43.7
44.7
49.4
70.8
65.3
66.3
DIVE 18
64.1
50.4
31.5
53.1
70.6
70
67.6
69.1
CE
63.9
37.5
7.70
44.4
67.2
63.0
56.2
61.7
CB-CE
39.6
32.7
16.8
33.2
54.8
53.2
54.0
DSB-CE
42.5
21.4(+13.7)
49.2(+4..8
63.4
62.7(+6.5)
64.3(+2.6)
DSB-CE+1FL [56]
43.4
22.5(+14.88)
50.1(+5.7)
64.3
63.4(+7.2)
65.0(+3.3)
Focal
41.0
13.1
47.2
61.1
CB-Focal M|
61.2
DSB-Focal
68.1
44.2
23.7(+10.6)
50.6(+3.4)
70.6
62.8
58.4
63.5(+2.4)
LDAM 4
60.0
49.2
31.9
51.1
64.6
DSB-LDAM
60.7
50.5
33.4(+1.5)
52.3(+1.2)
69.4
66.5
61.9
65.7(+1.1)
BS [3]
62.4
47.7
32.1
$1.2
60.1
51.4
46.7
53.2
DSB-BS
63.2
48,9
35,4(43.)
52.8(+1.6)
61.4
52.8
49.4(+2.7)
55.1(+1.9)
LADE [99
62.3
49.3
31.2
51.9
69.7
DSB-LADE
62.6
50.4
33.6(+2.4)
53.2(+1.3)
72.3
70.7
65.8
70.5(+0.8)
PaCo [10]
63.2
51.6
39.2
54.4
69.3
72.3
73.1
72.3
DSB-PaCo
64.1
52.9
41.5(+2.3)
55.9(+1.5)
70.2
73.4
74.6
73.4(+1.1)
MBJ[35]
61.6
48.4
39.0
52.1
70.0
DSB+MBJ
63.2
49.6
40.7(+1.77
53.3(+1.2)
73.6
70.2
66.2
70.9(+0.9)
RIDE 6S
67.9
52.3
36.0
56.1
70.9
72.4
73.1
72.6
MBJ+RIDE 35
68.4
54.1
37.7
$7.7
73.0
DSB+RIDE
68.6
54.5
38.5(+2.5)
58.2(+2.1)
70.7
74.0
74.2(+1.1)
73.4(+0.8)
5.1 RESULTS IMAGENET-LT INATURALIST2015
ImageNet-LTT is long-tailed version of ImageNet containing 1,000 classes with between 1,280 and samples per class. iNaturalist2018 is real-world, extremely unbalanced dataset containing 437,513 images from 8,142 classes. We adopt the official training and validation splits (I].
Table shows that when CE, Focal 33 and RIDE are combined with our approach (Appendix D.1), the model overall performance is significantly improved. For example, the overall accuracy of DSB-CE is 4.8% and 2.6% higher than CE on ImageNet-LT and iNaturalist2118 respectively. We three subsets (Head: more than 100 images, Middle: 20-100 images, Tail: less than 20 images) of these two datasets. It can observed that our proposed method has the largest improvement for the tail subset without compromising the performance of the head subset, where DSB-CE and DSB- -oc-l improve 13.7% % and 10.6%. respectively, over the original method in the tail subset of ImageNet-LT, effectively alleviating the model bias. addition, IFL (56] considers the intra-class long-tailed problem, and when we combine DSB-CE with it (i.e., DSB-CE-IFL), the performance of the model further enhanced. Therefore, we encourage researchers focus on the intra-class soig-tilled problem.
also report the performance
5.2 RESULTS ON IMAGENET AND CIFAR-
Table 3: Comparison ImageNet and CIFAR-100. On ImageNet,
random clipping, mixup 7771. and cutmix 766 to augment the training data, and all models are optimized by Adam with batch size of 512, learning rate of 0.05, momentum of 0.9, and weight decay factor of 0.0005. On CIFAR-100, we set the batch size to 64 and augment the training data using random clipping, mixup, and cutmix. An Adam optimizer with learning rate of 0.1 (linear decay), momentum 0.9, and weight decay factor of 0.005 is used to train all networks.
ImageNet Top-1 Acc(%)
CIFAR-100 Top-1 Acc(%)
Methods VGG16
CE
DSB-CE
A
DSB-CE
71.6
72.9
+1.3
71.9
73.4
+1.5
BN-Inception
53
73.5
74.4
+0,9
74.1
75.2
+1.1
ResNet-18
70.1
71.2
+1.1
75.6
76.9
+1.3
ResNet-34
73.5
74.3
+0.8
76,8
77.9
+1.1
ResNet-50
76.0
76.8
+0.8
77.4
78.3
+0.9
DenseNet-201
22
77.2
78.1
+0.9
78.5
79.7
+1.2
SE- 5E-ResNet-50
77.6
78.4
+0,8
78.6
79.3
+0.7
ResNeXt-100
70
78.8
79.7
+0.9
77.8
78.8
+1.0
8

---

Published as a conference paper at ICLR 2023
Table 4: Results on CUB-2011 and Cars 96. We evaluate the model performance with Recall and Normalized Mutual Information (NMI) [46].
Dataset
CUB-2011
Cars
Metric
R@1
R@2
NMI
R@
R@2
NMI
NormSoftmax
57.8
70.0
65.3
76.8
85.6
66.7
DSB-NSM
59.2(+1.4)
70.7(+0.7)
66.5 (+1.2)
77.9(+1.1)
86.4(+0.8)
67.8(+1.1)
dim
SoftTriee
60.1
11.9
66.2
78.6
86.6
67.U
64
DSB-ST
61.3 (+1.2)
72.7(+0.8)
67.3(+1.1)
79.8(+1.2)
87.5(+0.9)
68.3 (+1.3)
Circle
66.7
77.4
83,4
89.8
NormSottmax
63.9
13.5
68.3
83.2
89.3
69.7
dim
DSB-NSM
65.1(+1.2)
76.3(+0.8)
69.2(+0.9)
84.0(+0.8)
90.2(+0.7)
(0.9(+1.2)
512
SoftTriple
65.4
76.4
69.3
84.5
90.7
70.1
DSB-ST
66.4(+1.0)
77.0(+0.6)
70.6(+1.3)
85.6(+1.1)
91.3(+0..)
71.1(+1.0)
We use the ILSVRC2012 split contains 1,281,165 167 training and 50,000 validation images. Each class of CIFAR-100 contains 500 images for training and 100 images for testing. The results in Table
3 indicate that our approach is able to achieve performance gains greater than 1 % for a variety of networks both datasets. In particular, it enables VGGI6 to improve 1.3% and 1.5%
ImageNet
and CIFAR-100, respectively, compared to the original method. This implies that there is a semantic scale imbalance in non-long-taie tailed datasets and it affects the model performance.
5.3 RESULTS ON CUB-2011, CARS AND CIFAR-100-LY
Since also improve the classical losses (NormSoftmax and SoftTriple 42 in the field of deep metric learning, we abide by the widely adopted backhone network, experimental parameters, and the division of datasets in this field (Appendix (.5). The two improved loss functions are denoted as DSB-NSM and DSB-ST, respectively, and their formulas are given in Appendix F.I
Results on CUB-2011 and Cars196 Table summarizes the performance of our method with 64 and 512 embeddings, respectively. The experiments show that DSB loss is able to consistently improve by more than 1 % on RI and NMI. DSB-ST with 512 embeddings performs superiorly on Cars196, where R@ and R @2 exceed the Cirele loss [51 by 2.2% and 1.5 %, respectively.
Table 5: Results CIFAR- 100-LT. The imbalance factor of dataset is defined the value of the number of training samples in the largest class divided by that in the smallest class.
Dataset
CIFAR-
Imbalance factor
10
50
200
Metric
RØL
R@2
NMI
R@
R@2 2
NMI
R@
R@2
NMI
NormSoftmax
54.6
65.2
62.4
49.6
60.5
58.0
43.4
54.5
52.9
CB-NSM
55.7
66.1
63.3
50.5
61.1
58.7
45.5
55.3
53.8
DSB-NSM
56.3
66.7
63.5
51.3
61.4
59.1
46.0
56.1
54.4
dim 64
SoftTriple
67.6
63.9
49.5
61.0
46.6
57.8
55.4
CB-ST
58.1
68.4
65.1
51.1
62.8
59.4
48.2
59.5
56.6
DSB-ST
58.8
69.0
65.8
51.5
62.5
59.7
49.3
60.6
57.3
Results on CIFAR-100-LTT Classebalanced loss (CB loss) that performs well on long-tailed data and is also based on the ce-weighting strategy is selected for comparison with DSB loss. Analyzing the results in Table 5 the DSB loss outperforms the CB loss overall. Among them, when the imbalance factor of long-tailed CIFAR-100 200, DSB-ST performs significantly better than CB-ST, with higher performance than SoftTriple R 1, @@2 and NMI by 2.7 % .288 % and 1.9%.
Extensive experiments show that the semantic scale imbalance needs to be paid extensive attention.
DISCUSSION
In this work, we pioneer the concept and quantitative measurement of semantic scale imbalance, and make two important discoveries: (1) semantic scale has marginal effects, and (2) semantic scale imbalance can accurately describe model bias. It is important to note that our proposed semantic scale, like the number of samples, is natural measure class imbalance and does not depend on the model's predictions (See Related Work in Appendix A). Semantic scale can guide data augmentation, e.g., semantic scale imbalance can evaluate which classes are the weaker classes that need to be augmented, and marginal effects assist to select samples. We expect that our work will bring more attention to the more prevalent model bias, improve the robustness of models and promote the development of fairer AI.
appropriate number of

---

Published as a conference paper at ICLR 2023
7 ACKNOWLEDGEMENTS
This work was supported in part by the Key Scientific Technological Innovation Research Project by Ministry of Education, the State Key Program and the Foundation for Innovative Research Groups of the National Natural Science Foundation of China (61836009), the Major Research Plan of the National Natural Science Foundation of China (91438201, 91438103, and 91838303), the National Natural Science Foundation of China (U22B2054, U1701267, 62076192, 62006177, 61902298, 61573267, 61906150, and 62276199). the 111 Project, the Program for Cheung Kong Scholars and Innovative Research Team in University (IRT 15R53), the ST Innovation Project from the Chinese Ministry of Education, the Key Research and Development Program Shaanxi Province of China 201 9ZDLGG03-06) the National Science Basic Research Plan in Shaanxi Province of China(202IIQ-607)) the China Postdoctoral (0055)0)2))000066 the Scientific Research Project of Education Department In Shaanxi Province of China (No.20JY023), the National Natural Science Foundation of China (No. 61977052).
REFERENCES
[1] Elie Aljalbout, Vladimir Golkov, Yawar Siddiqui, Maximilian Strobel, and Daniel Cremers. Clustering with deep learning: Taxonomy and new methods. arXiv preprint 1eXXiv:801..700 2018.
[2] Dong Cao, Xiangyu Zhu, Xingyu Huang, Jianzhu Guo, and Zhen Lei. Domain balancing: Face recognition on long-tailed domains. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 5671 1-779, 2020.
[3] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-distributionawwwae margin loss. Advances in neural information processing systems, 32, 2019.
[4] Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with abbel-distribution-awar margin loss. Advances in neural information processing systems, 32, 2019.
[5] Nitesh V Chawla, Kevin W Bowyer, Lawrence O Hall, and W Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321- 357, 2002.
[6] Hsin-Ping Chou, Shih-Chieh Chang, Jua-Yu Pan, Wei Wei, and Da-Cheng Juan, Remix: Rebalanced mixup. In European Conference on Computer Vision, pages 95-110. Springer, 2020.
[7] Peng Chu, Xiao Bian, Shaopeng Liu, and Haibin Ling. Feature space augmentation for long- tailed data, In Computer Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23 28, 2020, Proceedings, Part XXIX 16, pages 694-710. Springer, 2020.
[8] Thomas M Cover. Elements of information theory. John Wiley & Sons, 1999.
[9] Ekin Dogus Cubuk, Ethan S Dyer, Rapha Gontijo Lopes, and Sylvia Smullin. Tradeoffs in data augmentation: An empirical study. 2021.
[10] Jiequan Cui, Zhisheng Zhong, Shu Liu, Bei Yu, and Jiaya Jia. Parametric contrastive learning. In Proceedings of the IEEE/CVF International Conference on Computer Vision pages 715-724, 2021.
[11] Yin Cui, Menglin Jia, Tsung-Yi Yi Lin, Yang Song, and Serge Belongie. Class-balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 9268 9277, 2019.
[12] Charles Elkan. The foundations of cost-sensitive learning. In International joint conference on artificial intelligence, volume 17, pages 973-978. Lawrence Erlbaum Associates Ltd, 2001.
[13] Andrew Estabrooks. Tacho Jo, and Nathalic Japkowicz. A multiple resampling method for learning from imbalanced data sets. Computational intelligence, 20(1):18-36, 2004.
10

---

Published as a conference paper at ICLR 2023
[14] Chengjian Feng, Yujie Zhong, and Weilin Huang. Exploring classification equilibrium in long-tailed object detection. Proceedings of the IEEE/CVP International Conference on Computer Vision, pages 3417-3426, 2021.
[15] Yuanfan Guo, Minghao Xu, Jiawen Li, Bingbing Ni, Xuanyu Zhu, Zhenbang Sun, and Yi Xu. Hese: Hierarchical contrastive selective coding. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 9706-9715, 2022.
[16] Hui Han, Wen-Yuan Wang, and Bing-Huan Mao. Borderline-smote: new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing, pages 878 887. Springer, 2005.
[17] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition, In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 770-778, 2016.
[18] Yin- Yin He, Jianxin Wu, and Xiu-Shen Wei. Distilling virtual examples for long-tailed recogni- tion. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 235-244, 2021.
[19] Youngkyu Hong. Seungju Han, Kwanghee Choi, Seokjun Seo, Beomsu Kim, and Buru Chang. Disentangling label distribution for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 6626-6636, 2021.
[20] Ting-I Hsich, Esther Robb, Hwann-Tzong Chen, and Jia-Bin Huang. Droploss for long-tail instance segmentation. In AAAI, volume 3, page 15, 2021.
[21] Jie Hu, Li Shen, and Gang Sun. Squeezze-an--exciiation networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 7132-7141, 2018.
[22] Gao Huang, Zhuang Liu, Laurens Van Der Maaten, and Kilian Q Weinberger. Densely connected convolutional networks. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 4700-4708, 2017.
[23] Sergey loffe and Christian Szegedy. Batch normalization: Accelerating deep network training by reducing internal covariate shift. International conference on machine learning, pages 448-456. PMLR, 2015.
[24] Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsuan Yang, Liqiang Wang, and Boqing Gong. Rethinking class-balanced methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 7610-7619, 2020.
[25] Nathalic Japkowicz and Shaju Stephen. The class imbalance problem: A systematic study. Intelligent data analysis, 6(5):429-449) 2002.
[26) Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 2019.
[27] Bingyi Kang, Saining Xie, Mareus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv: 1xXx:19110009117 2019.
[28] Diederik P Kingma and Jimmy Ba. Adam: A method for stochastic optimization. arXiv preprins arXiv:1412.6980, 2014.
[29] Jonathan Krause, Michael Stark, Jia Deng. and Li Fei-Fei. 3d object representations for fine- grained categorization. In Proceedings of the IEEE international conference on computer vision workshops, pages 554-561, 2013.
[30] Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers features from tiny images. 2009.
11

---

Published as a conference paper at ICLR 2023
[31] Ning Li, Tao Li, Chunyu Hu, Kai Wang, and Hong Kang. A benchmark of ocular disease intelligent recognition: One shot for multi-disease detection. In International Symposium on Benchmarking, Measuring and Optimization, pages 177-193. Springer, 2020.
[32] Xiaotong Li, Yongxing Dai, Yixiao Ge, Jun Liu, Ying Shan, and Ling- Yu Duan. Uncertainty modeling for out distribbtti generalization. arXiv preprint arXiv:2202.03955. 2022.
[33] Tsung-Yi Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pages
2017.
[34] Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2979,9979, 2020.
[35] Jialun Liu, Jingwei Zhang, Wenhui Li, Chi Zhang, Yifan Sun, al. Memory- based jitter: Improving visual recognition on long-tailed data with diversity in memory. arXiv preprint arXiv:2008.0960 09800, 2020.
[36] Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale long-tailed recognition in an open world. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 2537-2546, 2019.
[37] Yi Ma, Harm Derksen, Wei Hong, and John Wright. Segmentation of multivariate mixed data via lossy data coding and compression. IEEE transactions on pattern analysis and machine intelligence, 29(9): 154--156,, 2007.
[38] Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pages 181-196, 2018
[39] Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed repre- sentations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
[40] Erxue Min, Xifeng Guo, Qiang Liu, Gen Zhang, Jianjing Cui, and Jun Long. A survey of clustering with deep learning: From the perspective of network architecture. IEEE Access, 6:39501-39514 2018.
[41] Hyun Oh Song, Yu Xiang, Stefanic Jegelka, and Silvio Savarese. Deep metric learning via lifted structured feature embedding. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 4004 4012, 2016.
[42] Qi Qian, Lei Shang. Baigui Sun, Juhua Hu, Hao Li, and Rong Jin. Softtriple loss: Deep metric learning without triplet sampling. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 6450-6458, 2019.
[43] Jiawei Ren, Cunjun Yu, Xiao Ma. Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances neural information processing systems, 33:4175-411 2020.
[44] Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in Neural Information Processing Systems, 33:4175-4186, 2020.
[45] Olga Russakovsky, Jia Deng. Hao Su, Jonathan Krause, Sanjeev Satheesh, Sean Ma. Zhiheng Huang, Andrej Karpathy, Aditya Khosla, Michael Bernstein, et al. Imagenet large scale visual recognition challenge. International jonnaaatinanouunnaa journal of computer vision, 1 115()):1112522 2015.
[46] Hinrich Schütze, Christopher D Manning, and Prabhakar Raghavan. Introduction to information retrieval, volume 39. Cambridge University Press Cambridge, 2008
12

---

Published as a conference paper at ICLR 2023
[47] Abhinay Shrivastava, Abhinav Gupta, and Ross Girshick. Training region- region-based object detectors with online hard example mining. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 761-769, 2016.
[48] Karen Simonyan and Andrew Zisserman. Very deep convolutional networks for large-scale scale image recognition. arXiv preprint 1409. 10099.556. 1556, 2014.
[49] Adarsh Subbaswamy, Roy Adams, and Suchi Saria. Evaluating model robustness and stability to dataset shift. In International Conference on Artificial Intelligence and Statistics, pages 2611-2619. PMLR, 2021.
[50] Yanmin Sun, Mohamed Kamel, Andrew KC Wong, and Yang Wang. Cost-sensitive boosting for classification of imbalanced data. Pattern recognition, 40(12):3358-3372 2007.
[51] Yifan Sun, Changmao Cheng, Yuhan Zhang, Chi Zhang, Liang Zheng, Zhongdao Wang, and Yichen Wei. Circle loss: A unified perspective of pair similarity optimization. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6398 6407, 2020.
[52] Yifan Sun, Yuke Zhu, Yuhan Zhang, Pengkun Zheng, Xi Qiu, Chi Zhang, and Yichen Wei. Dynamic metric learning: Towards a scalable metric space to accommodate multiple seman- tic scales. In Proceedings of the IEEE/CVF Conference on Computer Vision and Putlern Recognition, pages 5393-5402, 2021.
[53] Christian Szegedy, Wei Liu, Yangging Jia, Pierre Sermanet, Scott Reed, Dragomir Anguelov, Dumitru Erhan, Vincent Vanhoucke, and Andrew Rabinovich. Going deeper with convolutions. In Proceedings of the IEEE conference computer vision and pattern recognition, pages 1-9, 2015.
[54] Jingru Tan, Changbao Wang, Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-taine tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pages 11662-11671, 2020.
[55] Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33:1513-1522 2020.
[56] Kaihua Tang. Mingyuan Tao, Jiaxin Qi, Zhenguang Liu, and Hanwang Zhang. Invariant feature tailed classification. In European Conference Computer Vision,
learning for generalized long-tailed pages 709-726. Springer, 2022.
[57] Joshua B Tenenbaum, Vin De Silva, and John C Langford. A global geometric framework for nonlinear dimensionality reduction. science, 290(5500):2319) 2223, 2000.
[58] Junjiao Tian, Niluthpol Chowdhury Mithun, Zachary Seymour, Han-Pang Chiu, and Zsolt Kira. Striking the right balance: Recall loss for semantic segmentation. In 2022 International Conference on Robotics and Automation (ICRA), pages 5063-5069. IEEE, 2022.
[59] Ivan Tomek et al. Two modifications of cnn. 1976.
[60] Grant Van Horn, Oisin Mac Aodha, Yang Song, Yin Cui, Chen Sun, Alex Shepard, Hartwig Adam, Pietro Perona, and Serge Belongie. The inaturalist species classification and detection dataset. In Proceedings of the IEEE conference on computer vision and pattern recognition. pages 8769-8778, 2018.
[61] Grant Van Horn and Pietro Perona. The devil is in the tails: Fine-grained classification in the wild. arXiv preprint arXiv:/709.011 1709.014500 2017.
[62] Oriol Vinyals, Charles Blundell, Timothy Lillicrap, Daan Wierstra, et al. Matching networks for one shot learning. Advances in neural information processing systems, 29:3630-3638, 2016.
[63] Catherine Wah, Steve Branson, Peter Welinder, Pietro Perona, and Serge Belongie. The caltech-uesd birds-200-2011 dataset. 2011.
13

---

Published as a conference paper at ICLR 2023
[64] Tong Wang, Yousong Zhu, Chaoyang Zhao, Wei Zeng, Jinqiao Wang, and Ming Tang. Adaptive class suppression loss for long-tail object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 3103-3112 2021.
[65] Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution-aware experts. arXiv preprini arXiv:2010.01 809, 2020.
[66] Xun Wang. Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embedding learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pages 6388-6397, 2020.
[67] Yiru Wang, Weihao Gan, Jie Yang, Wei Wu, and Junjie Yan. Dynamic curriculum learning for imbalanced data classification. In Proceedings of the IEEE/CVF International Conference Computer Vision, pages 5017-5026, 2019.
[68] Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin. Distribution-balaneed loss for lulti-label classification in long-tailed datasets. In European Conference on Computer Vision, pages 162-178. Springer, 2020.
[69] Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for Vision, pages 247-263. Springer, 2020.
classification. In European Conference on Computer
[70] Saining Xie, Ross Girshick, Piotr Dollár, Zhuowen Tu, and Kaiming He. Aggregated residual transformations for deep neural networks. In Proceedings of the IEEE conference on computer vision and pattern recognition, pages 1492-1500, 2017.
[71] Honggang Yang. Jiejie Chen, Rong Luan, Mengfei Xu, Lin Ma, and Xiaoqi Zhou. Base on megapixel color fundus photos for multi-label disease classification. In 2022 14th International Conference on Advanced Computational Intelligence (ICACI), pages 29-35. IEEE, 2022.
[72] Shuo Yang, Lu Liu, and Min Xu. Free lunch for few-shot learning: Distribution calibration. arXiv preprint arXiv: 100::2011.06995 2021.
[73] Yuzhe Yang, Hao Wang, and Dina Katabi. On multi-domain long-tailed recognition, imbalanced domain generalization and beyond. European Conference on Computer Vision, pages 57-75. Springer, 2022.
[74] Yuzhe Yang and Zhi Xu. Rethinking the value of labels for improving class-imbalanced learning. arXiv preprint arXiv:2006.07529 2020.
[75] Xi Yin, Xiang Yu, Kihyuk Sohn, Xiaoming Liu, and Manmohan Chandraker. Feature transfer learning for face recognition with under-represented data. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 5704-5713, 2019.
[76] Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. In Proceedings of the IEEE/CVF international conference on computer vision, pages 6023-6032, 2019.
[77] Hongyi Zhang, Moustapha Cisse, Yann N Dauphin, and David Lopez-Paz. mixup: Beyond empirical risk minimization. arXiv preprint arXiv:1710.09412, 2017.
[78] Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596, 2021.
[79] Zizhao Zhang and Tomas Pfister. Learning fast sample re-weighting without reward data. In Proceedings of the IEEE/CVF International Conference on Computer Vision, pages 725-734, 2021.
[80] Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-brande network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition, pages 9719-9728, 2020.
14

---

Published as a conference paper at ICLR 2023
[81] Zhi-Hua Zhou and Xu- Xu-Ying Liu. Training cost-sensitive neural networks with methods ad- dressing the class imbalance problem. IEEE Transactions on knowledge and data engineering, 18(1):63-77, 2005.
15

---

Published as a conference paper at ICLR 2023
A RELATED WORK
world datasets tend to toiget The extreme imbalance in the number of samples for long- tailed data prevents the classification model from learning the distribution of the tail classes adequately, which leads to poor performance on the tail classes. Therefore, the methods of re-balancing the
number of samples (26) T6: 13 79 and balancing the loss incurred per class [12 81 50 i.e.,
re-sampling and cost-sensitive learning, are proposed. Among them cost-sensitive learning is most
relevant to our work.
44 proposes to use the frequency of labels to adjust the loss during training to mitigate class bias.
33] assigns weights the loss for each class, and the hard samples are given higher weights. Recent
studies have shown that re-weighting the loss strictly by the inverse of the number of samples is moderate [38 39]. Some methods that generate weights for re-weighting in more "smooth" manner perform better, such as taking the square root of the number of samples (38 as weights. CB loss [II] attributes the better performance of this more "smooth" approach to the presence of marginal effects, while other studies (2) (44] attribute it to the negative gradient over-suppression. Distribution- balanced loss [68] proposes negative-tolerant regularization to mitigate gradient differences, and recalculates the loss by calculating the ratio of the expected to the actual sampling frequency for each class. Seesaw Loss [64] leverages mitigation and compensation factors to dynamically suppress excessive negative sample gradients the tail classes while complementing the penalty for misclassified samples. Furthermore, with the proposal of decoupled training [80]. (26 adopts decoupled training, using cross-entropy loss learn features in the first stage and re-balancing loss to learn the classifier in the second stage.
Unlike the above studies of re-balancing loss, which all re-balance loss based on the number of samples or the ratio of positive and negative gradients, our work proposes novel measure, called semantic scale. Compared to the number of samples, the semantic scale also considers the sample distribution scope. In contrast to gradient- gradient-based measures, the semantic scale does not depend on the model output and gradient back- propagationn and is a natural measure similar to the number of samples. Work on model robustness has focused on the out-of-domain generalization performance of models, an area known as "out-of-distributtin generalization of models". For example, [49] aims to maintain the good performance of the model when the test distribution deviates from the training distribution. Similarly, [32) aims to allow the model to learn more information outside the domain. Unlike them, makes the model perform poorly in certain classes.
concerned with the problem that model bias introduced by unbalanced data
B EXPLANATION OF FEW KEY POINTS
B.1 HOw DOES THE SECTION "SLOW DRIFT PHENOMENON AND MARGINAL EFFECTS OF CHARACTERISTICS" RELATE TO THE REST OF THE PAPER?
(1) Why do we have to introduce marginal effects?
The effective number of samples discusses the relationship between the sample number and feature diversity in a class, and it argues that feature diversity has marginal effects. However, the effective number of samples has many major drawbacks (introduced in Section P such as it does not work on sample-balanee datasets and does not give a quantitative measure of feature diversity. Therefore, we extend the mechanism of the efficient number of samples and propose the "semantic scale" that can effectively measure feature diversity in sample-balanced dataset. On the one hand, our approach is more general compared to the effective number of samples. On the other hand, the marginal effect proves that our extension is reasonable and appropriate. In addition, our approach simultaneously explains three phenomena that cannot be explicated by other methods, which indicates the reliability of our proposed method. In brief, the logic of our paper is as follows.
CB loss introduced the concept of the effective number of samples based on marginal effects.
We extend the effective number of samples and propose the concept of semantic scale.
Experiments show that the semantic scale still has marginal effects (Section 3.3
16

---

Published as a conference paper at ICLR 2023
According to the properties mentioned in step 3, explore many applications based on semantic scale that require marginal effects as theoretical support (c.g.. the selection method of representative samples supplemented in Appendix
In summary, we would like to explain that the birth of semantic scale measurement (or semantic scale imbalance based on semantic scale) was inspired by the effective number of samples with marginal effects. If the marginal effect is discarded, then our early motivation and the later practical applications are theoretically weak and unconvincing.
(2) Association with feature slow drift
Since experiments show a very high correlation between semantic scale and model bias, we propose re- re-weight the loss function with the inverse of the semantic scale. The features are dynamically changing during training, and all the feature vectors are needed to calculate the semantic scale of each class. Obviously, the data in one batch is not enough, so we propose to dynamically update and store the historical features to calculate the semantic scale, and the feature slow drift phenomenon ensures the feasibility of this operation.
Section 2 is indispensable for the whole paper, it ensures the coherence of the paper.
B.2
WHY FOCUS ON THE RELATIONSHIP BETWEEN SEMANTIC SCALE AND ACCURACY?
is important to note that we are concerned with the model bias introduced by unbalanced data, which causes models to perform poorly some classes. In the past, researchers believed that models performed poorly on classes with fewer samples and therefore defined classes with fewer samples as tail classes and classes with more samples as head classes, proposing a long-tailed identification task. However, we observe that the model does not necessarily perform poorly on classes with fewer samples, which explains why some of the tail classes are "overbalanced" in many long-tailed identification methods. The higher similarity between our proposed semantic scale and model performance allows us to redefine the imbalance problem by replacing the number of samples with semantic scale. The superior performance achieved on the bample-balanced datasets shows that our proposed semantic scale imbalance is reliable. The semantic scale measure does not depend on the model and is calculated directly from the data. Even more surprising is that the semantic scale of the class can predict the performance of the class, which can lead to further understanding of what the model learns from the data. It can facilitate the development of data-driven artificial intelligence.
B.3 WHY SHOULD THE LOSS FUNCTION BE DYNAMICALLY WEIGHTED?
The working process of DSB loss: in each iteration, the semantic scale of each class is calculated in the feature space, and the loss function is re-weighted by the inverse of the semantic scale.
Why is the loss "dynamically" weighted? Because the semantic scales change continuously as the features change during training, and we need to update the features in each iteration and re-calculate the semantic scales to we-weight the loss function. The term "dynamic" refers to the dynamic update of the semantic scale in each iteration.
However, there is difficulty in implementing dynamic weighting, i.c., all features needed to calculate the semantic scale, and we cannot extract the features of all samples in each iteration, which would be time consuming. Therefore, we analyze the "feature slow drift" phenomenon in Section 2 and propose to calculate the semantic scale by dynamically storing and updating the historical features (i.e., dynamic re- re-weighting training framework). Comparative experiments on the training speed and memory consumption of the above training framework are presented in Appendix F22 and the results show that our approach is efficient.
B.4
WHAT IS THE POINT OF PROPOSING A VARIETY OF COST-SENSITIVE LEARNING METHODS? WHY NOT DIRECTLY USE THE ACCURACY OF EACH CLASS TO WEIGHT THE LOSS?
What is the point of proposing a variety of sent-stns learning methods? For example, using the inverse of the number of samples the effective number of samples to reweight the loss, rather than directly using the accuracy of each class to weight. After careful consideration, we believe there are several reasons.
17

---

Published as a conference paper at ICLR 2023
(I) Reweighting loss with class accuracy may cause the model over-focus weak classes, so that other classes are ignored. Recent studies have shown that reweighting the loss strictly by the inverse of the number of samples has modest effect [38 39]. Some "smoother" methods perform better, such as taking the square root of the number of samples 38 as the weight. [1]] argues that the reason why the "smoother" method performs better is due the existence of marginal effects. Our approach can be understood as a smoothed version of class accuracy because our proposed semantic scale has marginal effects and a high correlation with class accuracy. We note a recent work (CDB loss) published in LJCV that measures class difficulty. In addition, domain balancing also measures class-level difficulty, so we compare semantic-scale scale-balanced learning with them. The introduction and comparison experiments of the above two methods are shown in Appendix H.2
(2) The method of weighting with model performance cannot bring us new cognition. Why do models perform poorly on some data and well on others? For example, face recognition models usually do not perform well in dark environments. When we encounter such a problem, the first thing to think about is whether the lack of data in the dark environment causes the model to not be fully learned. Since there is a lot of data available, this problem is not caused by the few samples, so is there any other explanation? We argue that the pattern of faces in the dark environment is not rich enough, which leads to a large number of samples clustered around the manifold with smaller volumes, making it difficult to distinguish between faces. Our approach is not only address the performance imbalance, but also to advance researchers' understanding of deep neural networks. Advances in science are usually accompanied by the establishment of new cognition.
(3) Our proposed semantic scale has great potential for application. In engineering applications, how many samples should be collected for each class is the most appropriate? When too few samples are collected, the class is under-representte, while too many will consume huge costs. Our approach can effectively solve this problem by stopping the collection when the semantic scales tend to be saturated. When we communicate with technology companies, we find that they have 100 million data, but there no proper way to select representative data. So we design an idea to select representative data using semantic scales, the details of which are added in Appendix
C EXPERIMENTS ON STANFORD POINT CLOUD MANIFOLDS
Since (Z Zmean) (Z Zmeon) T is real symmetric matrix, it is semi comi-positive definite. Further,
I+ P (Z Zmean] (Z - Zmean)" (T is positive definite matrix and therefore det(I + R (Z
Zmean) (Z Zmean) )²) > O. The semantic scale measure is derived from the singular value decomposition of the data matrix, which is jointly determined by most of the samples. Therefore, our method is insensitive to noisy samples, i.c., the semantic scale measure is numerically stable. The semantic scales of multiple Stanford point cloud manifolds with different sizes are calculated and plotted in Figure Let the center point of bunny be Chunny- We increase the volume of bunny by performing w * (bunny Chunngy). ). and the other point clouds are scaled up in this manner. As the object manifold is scaled up, the calculated volume then increases slowly and monotonically, indicating that our method can accurately measure the relative size of the manifold volume and is numerically stable, an advantage that will help mitigate the effects of noisy samples.
D EXPERIMENTAL DETAILS
D. MARGINAL EFFECT OF SEMANTIC SCALE
We use the following method to generate new training datasets for classification experiments: assume that the total number of classes in the original dataset is C, and m samples are randomly selected from each class to form a sub-dataset with a total number of samples of C x m. The details of the sub-datasets generated based on CIFAR-10. 10, CIFAR- 100 and Mini-ImageNet are in Table
D.2 QUANTIFICATION OF SEMANTIC SCALE IMBALANCE
We train ResNet- 18 and ResNet-34 CIFAR-1O-LT and CIFAR-10 with imbalance factor of 200, respectively, and the test set of CIFAR- 10-LT is consistent with CIFAR- 10. During training, the batch size is fixed to 64, and the optimizer adopts Adam. The learning rate is initially 0.01 and becomes
18

---

Published as a conference paper at ICLR 2023
0.2
03
0.0
0.0
00
0.2
0.8
0.6
-0.30.0
0.4
0.0
0.4
0.3 0.4 0.5 0.6
0.2
0.2
0.3
0.4
0.5
0.6
0.2
Duntyy dragon hapdy
Volume of the ma manifold
C.1
0.0
0.2
0.0 0.1 2
0.01
0.0 0.1 0.2
0.3
2
10
W
Figure 6: Increase three Stanford point cloud manifolds, and calculate their semantic scales.
Table 6: The sample-balanced sub-datasets with a total of 31. Among them, 13 sub-datasets are from CIFAR-10 10, sub-datasets are from CIFAR-100 and the rest from Mini- Mani-ImageNet. The test set remains the original test set. denotes the total number of classes in the original dataset and m is the number of samples per class in the sub-dataset.
Dataset
Number
Sub-datasets
50
200
300
1,000
2,000
3,000
4,000
5,000
O
1,000
2,000
3,000
4,000
5,000
6,000
8,000
10,000
20,000
30,000
40,000
50,000
10
20
30
50
100
200
300
400
500
CIFAR-IO0
100
C'x
1,000
2,000
3,000
5,000
10,000
20,000
30,000
40,000
50,000
20
30
50
200
300
400
500
Mini Nei
100
3,000
0.98xthe previous learning rate after each epoch. We do not employ other additional tricks and data augmentation strategies.
D.3 SEMANTIC SCALE IMBALANCE ON LONG-TAILED DATA
We artificially produce two long-tailed versions of the MNIST dataset, called MNIST-LT-1 and MNIST-LT-2. The number of samples per class is listed in Table
Figure 3 shows the class-wise accuracies of ResNet-18 18 and ResNet-34 trained CIFAR- and CIFAR-100-LT with the same training settings as in Appendix [.2 Taking CIFAR-I0 as an example, labels to 10 correspond to: airplane, automobile, bird, cat, deer, dog.frog horse, ship, truck. The prediction scores of classification experiments on CIFAR-10-L3 and CIFAR-10 10 find that cat (label 4) most easily confused with dog (label 6). as shown by their lowest accuracy CIFAR-10 (Figure However, the accuracy cat and dog on CIFAR- ---------LTTT--- is higher than that of deer (label 5), which due to the dominant role of semantic scale S' in S for long-tailed data.
19

---

Published as a conference paper at ICLR 2023
Table 7: The two long-tailed MNIST datasets resampled from MNIST.
Dataset
MNIST-LT-I
Class label
0
4
5
8
9
Number
5.923
3,590
2,940
2,518
2,256
1,972
1,700
1,300
1,100
900
Dataset
MNIST-LI--2
Class label
0
4
8
9
Number
5,923
3,090
2,540
1,818
1,356
972
484
272
122
74
D.4 SEMANTIC SCALE IMBALANCE FOR MORE DATASETS
have demonstrated the semantic scale imbalance on MNIST, MNIST-LT, CIFAR-IO, CIFAR- 10-LT, CIFAR-100 and CIFAR-I00 in Section 3.4. Figure additionally shows the degree of semantic scale imbalance on CUB-2011 19rs and Mini Mini-ImageNet indicating that the semantic scale imbalance is indeed prevalent in all kinds of datasets.
Mini-lmageNNet
Figure 7: Semantic scale and number of samples per class. Different angles of the radar plot represent different classes, and the number of samples in the largest class normalized to 0.5 for ease of observation.
D.5 EXPERIMENTAL SETTINGS FOR SECTION 5.3 AND MORE EXPERIMENTS
D.5.1 EXPERIMENTAL SETTINGS FOR SECTION 5.3
Backhone Network and Experimental Parameters. Since we improve on the classical loss in the field of deep metric learning, abide by the widely adopted backbone network, experimental parameters, and the division of datasets in this field. The BN-Inception [23 53] pre-trained on ImageNet is adopted horizontal flipping and random cropping. All images are cropped to 224 224 as the input of the network. The output of the network after global average pooling is fed into single fully connected layer to obtain 64- or 51 2-2dimmesional feature embeddings, and then all embeddings are clustered by K-means. The model is optimized by Adam with the batch size as 32 and the number of epochs as 50. We evaluate the performance of the learned embeddings with Recall @ K and Normalized Mutual Information (NMI). The remaining experimental parameters used in the training are consistent with those reported in NormSoftmax and SoftTriple
the backbone network, and the training is augmented by using random
42
Dataset Introduction (CUB-2011, Cars196 and CIFAR-100-LT), The CUB-2011 dataset has 5,864 images in the first 100 classes for training and 5,924 images in the second 100 classes for testing. The Cars 196 dataset consists of 196 classes totaling 16,185 images, with the first 98 classes for training and the remaining classes for testing. The CIFAR-100 100 has 100 classes, each containing 600 images.
185
20

---

Published as a conference paper at ICLR 2023
We create three long-tailed CIFAR-100 with the first 60 classes for training (See Figure 8 and test on the remaining classes [42]
600
100
μ=110
==
Number of closs
-00
μ=50
μ=
400
μ=200
μ=
1000
40
200
20
20
CIFAR-100 class
Cans196 index
Figure 8: Long-tailed CIFARIO0 and C9rs196 with different imbalance factors. We use the exponen- tial function NutA01M M)) to yield the number of training samples for each class, where i is the class index (0-indexed), the number training samples in the largest class, jr the imbalance factor, and M is the total number of classes.
D.5.2 MORE EXPERIMENTS
The long-tailed Cars196 is created using the first 98 classes for training (See Figure (D) and the test set is the remaining classes. Both Mini-ImageNet and CIFAR-100 datasets contain 100 classes, each containing 600 samples. For the Mini-ImageNet dataset, the first 64 classes are the training set and the last 36 classes are the test set, and for the CIFAR-100 dataset, the first 60 classes are the training set and the remaining classes are the test set. Note that the experiments on CIFAR-100 in this section are different from the classification experiments of this experiments is to complement the effectiveness of our proposed method on both long-tailed and cample-balanced datasets for the field of deep metric learning.
CIFAR-100 in 5.33 The purpose
Table 8: Comparison on long-tailed Cars 96.
Dataset
Long-taile tailed Cars
Imbalance factor
10
20
50
Metric
R@1
@#2
NMI
R@1
R@2
NMI
R@1
R@2
NMI
NormSoftmax
66.4
76.9
58.9
63.1
74.2
54.9
59.5
71.1
52.9
CB-NSM
68.9
78.0
60.1
64.9
75.2
56.1
61.7
72.5
53.3
DSB-NSM
69.5
78.6
60.7
65.4
75.7
56.6
62.3
73.0
54.1
dim
64
SoftTriple
70.2
80.5
61.4
64.7
75.8
57.5
62.9
74.1
55.2
CB-ST
71.9
81.3
62.9
66.5
76.9
58.6
64.8
75.4
56.0
DSB-ST
72.3
81.8
63.4
66.8
77.5
59.7
65.4
75.3
56.6
Table [ shows the performance comparison DSB-NSM and DSB-ST on the long Car196. When the imbalance factor is 10, DSB-NSM outperforms NSM (NormSoftmax) by 3.1 % on R @ and DSB-ST outperforms ST (SoftTriple) by 2.1% When the imbalance factor is 50, DSB-NSM and TT improve 2.8 % and 2.5%. respectively, on R compared to the original method. In addition, DSB loss performs better than CB loss on all metrics.
Table shows that compared with the original losses, DSB-NSM and DSB-ST are able consistently improve R@ by 1.3% on average and NMI by 1-2% for both sample-balanced balanced datasets, with all other metrics outperforming the original. The results in Tables and 9 further confirm that proposed dynamic semantic-scue waa----aalllleeeeeee learning is applicable to long-tailed and sample-balane balanced datasets in the field of deep metric learning, and has broad application prospects.
21

---

Published as a conference paper at ICLR 2023
Table 9: Comparison Mini-ImageNer and CIFAR-100.
Dataset
Mini- Mini-lmageNet
CIFAR-100
Metric
R@D
R@2
NMI
R@1
R@2
NMI
NormSoftmax
85.7
91.2
74.1
60.1
71.5
49.4
DSB-NSM
87.1(+1.4)
92.0(+0.8)
75.5(+1.4)
61.4(+1.3)
72.2(+0.7)
50.6(+1.2)
Soft
86.9
92.0
77.3
62.1
73.3
52.0
DSB-ST
88.0(+1.1)
92.8(+0.8)
78.8(+1.5)
63.5(+1.4)
73.9(+0.6)
53.0(+1.0).
D.6
RESULTS ON THE FUNDUS DATASETS OIA-ODIR AND OIA-ODIR-B
D.6.1 DATASET INTRODUCTION
The OIA-ODIR dataset 31 was made public in 2019, and it contains total of 10,000 fundus images classes. As shown in Figure the eight classes are: Normal(N), hypertensive retinopathy(D), glaucoma(G), cataract(C), agerelated macular degeneration(A) hypertension complication (H), pathologic myopia (M), other disease/ abnormality(OO) Considering that usually appears together with other discases, to reduce ambiguity, adopt the data splitting scheme of (71), using only the data of the first 7 classes, and the number training samples and test samples for each class shown in Figure 10
N
A
H
M
Figure 9: Eight fundus images in the OIA-ODIR dataset.
The OIA-ODIR dataset suffers from unbalanced number of samples. To fully validate method, we produced a balanced version of the OIA-ODIR dataset, OIA-ODIR-B, by using the class with the least number of samples the benchmark. As shown in Figure 10 each class of OIA-ODIR-B contains 103 training samples and 46 test samples.
In addition the number of samples, plot the degree of semantic scale imbalance for the training sets of OIA-ODIR and OIA-ODIR-BS in Figure 10
D.6.2 BACKBONE NETWORK EXPERIMENTAL PARAMETERS
We used ResNet-50, pre-trained on ImageNet, as the backbone network. An adam optimizer with learning rate of 0.1 (linear decay), momentum of 0.9, and weight decay factor of 0.005 was adopted to train networks. In keeping with 77., average precision (AP) was used as the performance metric of the model.
D.6.3 RESULTS ON OIA-ODIR
We improved the advanced class rebalancing method (BS [43. Focal loss [T]] LDAM 4 and the classification results plotted in Figure The experimental findings summarized follows.
22

---

Published as a conference paper at ICLR 2023
0.95
0.95
Train
Trein
Number of samples
Test
0.90
Semantic scale per dlss
Number of samples
Test
0.90
scale per class
Semantic scale
Semantic scale
800
0.80
80
0.85
600
0.75
70
0.80
400
60
0.75
200
0.65
50
0.60
0.70
N
D
OIA-ODIR
OIA-ODIR-B
Figure 10: The number training and test samples for each category and the degree semantic scale imbalance in OIA-ODIR and OIA-ODIR-B.
LAV
158 Rook
ProdecT
Figure 11: The enhancement effect our method for CE, BS, Focal, and LDAM on the OIA-ODIR.
Although the sample from class H is the smallest, all methods outperform on class H than on class C, class M, and class A. This again shows that the number of samples is not the best measure of class imbalance.
Methods based on sample numbers usually result larger boosts for the classes with the smallest sample numbers and thus fail to give more attention to class C, class M, and class A. Our method has the most significant boosts for these three classes, indicating that semantic scale imbalance more accurately reflect the difficulty of the classes.
D.6.4 RESULTS ON OIA-ODIR-B
Since the class rebalancing method based on the number of samples cannot be applied to the dataset with balanced number of samples, additionally adopted VGG-16, 18 and SE-ResNet-50 as the backbone network to test the effect DSB-CE on CE enhancement, and the experimental results shown in Figure 12 The experimental findings summarized follows.
15
250
OSS-CE
050 CC
CDB CE-CC
300-0 (VSC-50
CU--20EL (taalte-321
01A-2E5 1hallee-301
Figure 12: Performance gains from our approach for multiple backbone networks on OIA-ODIR.
With a balanced number samples, the model still performs poorly on class C, class M and class A. Figure I0 shows that the semantic scales of these three classes are significantly smaller than the other classes.
Our approach results in significant performance gains for all models on class C. class M, and class and promotes more balanced model performance on all classes, which is important in medical Al.
Experiment Summary. We validated the effectiveness of mrmantic-ssale-balee on a dataset of fundus images with balanced sample numbers and on a long- tailed dataset fundus images. Experimental results show that semantic scale imbalance exists in medical image datasets
learning both
23

---

Published as a conference paper at ICLR 2023
and significantly limits the performance of deep neural networks, so it is necessary to introduce semantic-scale-- scale-balanced learning in medical image classification.
D.7 REMOTE SENSING IMAGE SCENE CLASSIFICATION
In this section, we validate the effectiveness remote sensing image classification task, which demonstrates the necessity of introducing semantic scale imbalance into the field of remote sensing image recognition.
paaaanti--scal-ballaeeee learning in a sample-balance
D.7.1 DATASET INTRODUCTION
RSSCN7 dataset contains 800 remote sensing images which are classified into 7 typical scene categories: grassland. forest, farmland, parking lot, residential region, industrial region, and river and lake. Figure shows the images of the seven scenarios. Following the official split, the number of images for training and testing is 50% of the total number each.
NWPU-RESISC45 dataset contains a total of 31, 500 images with pixel size of 256 x 256, covering 45 scene classes with 700 images in each class. This dataset has large intra-class variability and inter-class similarity due to the large differences in image spatial resolution, untitled pose, and illumination. Following the official split, 20% of the images are used for training and 80% for testing.
grassland
farmland
industry
riverlake
forest
resident
parking
Figure 13: The seven scenarios are included in the RSSCN7 dataset.
D.7.2 BACKBONE NETWORK AND EXPERIMENTAL PARAMETERS
We select VGG-16, GoogLeNet, and ResNet-34 34 as the backbone networks. The Adam optimizer (default parameter) is adopted to update the model until convergence, and the learning rate decays 10 times every 50 epochs. In addition, the batch size is set to 100 and no data augmentation used throughout the training process.
24

---

Published as a conference paper at ICLR 2023
VSG-16
VGG-16-058
0.99
Grass
Reld
0.08
0.87
0.01
0.00
0.03
0.01
0.00
Red
0.06
0.69
0.01
0.00
0.03
0.01
0.00
0.01
0.03
0.84
0.03
0.02
0.06
0.01
0.02
0.02
0.01
0.02
0.04
River Lake
3.03
0.89
0.00
River Lake
0.00
0.89
Forest
3.02
0.02
0.00
0.03
0.91
0.01
0.01
0.1
Forest
0.02
0.02
0.00
0.03
0.91
0.01
0.01
-94
0.01
0.05
0.02
0.90
02
0.02
0.02
-02
Parking
3.00
0.02
0.07
0.01
0.03
0.01
0.86
Parking
0.00
0.02
0.05
0.01
0.03
0.00
0.5
000
Grara
Fiedd
badostry
River Lakso
Farest
Residlont
Parking
Field
Indlustry
River Lake
Foreat
Resident
Parkiing
Gooplarlee
Gaoglarleet-055 155
0.87
0.01
0.04
0.00
Grass
0.03
0.03
0.03
0.02
Held
3.58
0.88
0.07
0.01
0.03
0.01
0.01
Rei
0.06
0.05
0.02
0.01
0.03
0.02
0.01
Ind.stry
3.31
0.03
0.82
0.04
0.01
0.02
0.07
0.01
0.03
0.85
0.03
0.01
0.02
0.05
River
Forest
0.03
0.02
0.00
0.03
0.90
0.01
0.01
Forest
0:03
0.02
0.00
0.03
0.89
0.02
0.01
Resident
0.01
0.02
0.06
0.02
0.01
0.88
0.00
Resident
0.01
0.02
0.05
0.02
0.01
0.89
0.00
-02
Parking
3.30
0.07
0.02
0.03
0.01
Parking
0.01
0.02
0.03
0.01
0.87
002
-00
Grass
Field
Indkstrry
River Lake
Forest
Residemi
Parking
Grees
Felld
Ieduatry
Rivor Lake
Forest
Residdent
Parking
Aos 34
1.2
AesMet 34 CSS
1.0
0.33
0.00
0.01
0.00
0.02
0.01
0.00
0.03
Reld
3.05
0.91
0.00
Red
0.92
0.00
Ind bolustry
3.00
0.03
0.88
0.02
0.00
0.02
0.05
Industry
0.00
0.02
0.90
0.01
0.00
0.02
0.05
3.01
0.02
0.01
River
0.00
0.92
0.00
0.01
Forest
3.92
0.01
0.00
0.02
0.94
0.01
0.00
Forest
0.02
0.01
0.00
0.02
0.94
0.01
0.00
Resident
3.30
0.01
0.04
0.02
0.01
0.92
0.00
Resident
0.00
0.01
0.04
0.02
0.01
0.92
0.00
Parking
0.30
0.06
0.01
Barking
0.00
0.2
-00
Grass
Reid
Induttry
River Laie
Forest
Residentt
Parking
Grass
Fiild
Industr
Riser Lake
Forest
Resideent
Parking
confusion matrix of VGG-16, GoogLeNet, and ResNet-34 on the RSSCN7 dataset. Right column: confusion matrix of VGG-I6-DSB, GoogLeNet-DSB, and ResNet-34-DSB on the RSSCN7 dataset.
Figure 14: Left column:
D.7.3 RESULTS ON RSSCN7
We trained all backbone networks with dynamic semantic-scale balanced learning. The experimental results are shown in Figure [4] It can be found that our method improves the performance of all models. When our method is not employed, all backbone networks are significantly weaker in recognizing industrial regions than other scenes. Our method makes the recognition ability of the model for different scenes more balanced, thus improving the overall performance of the model.
Specifically, dynamic semantic-scale-balame industrial regions and parking lots by 4% and 3%, respectively, significantly reducing the bias of the model. Dynamic sewtaatii--cal--balnnceee learning also performs well on GoogLeNet and ResNet-34, where it improves the overall accuracy of GoogLe and ResNet by 1% and 0.6%, respectively.
learning improves VGG-16'ss recognition accuracy for
D.7.4 RESULTS ON NWPU-RESISC45
We significantly improved the performance of multiple backbone networks by employing dynamic caaantic-caal-balaaae
learning on the SWUU-RRSSSC45 dataset, and the experimental results are illustrated in Figure [5] It can be observed that the overall performance of VGG-16-DSB is 1.8% higher than that of VGG-16. Mcanwhile, dynamic semantic-scale-bal overall performance of GoogLeNet and ResNet-34 by 1.6% and 1.2%.
learning improves the
25

---

Published as a conference paper at ICLR 2023
Backbone Network
TOP-1 ACC(%) ((%)
DSB Backbone Network
82
80
AlexNet
VGG-16
GoogLeNet
ResNet-34
Figure 15: Comparison of four backbone networks before and after combining with dynamic semantic scale-balanced learning on dataset NWPU-RESISC45.
Experiment Summary. On two remote sensing image datasets with balanced sample numbers, our method shows significant improvements on common backbone networks. The experimental results show that semantic scale imbalance exists in the remote sensing image dataset and affects the performance of deep neural networks to some extent. Remote sensing images hold great promise for applications in agriculture, industry, and the military, so it is crucial promote the fairness of deep neural networks on remote sensing images.
E PSEUDO CODE FOR SAMPLE VOLUME
An image can be considered as a point in the sample space, and the dimension of the sample space is the the number of image pixel points. The manifold distribution law considers that multiple images from a class are distributed around low-dimensional manifold in the sample space. We calculate for each class the volume of its corresponding manifold and call the sample volume. We provide the pseudo code for the calculation of the sample volume in Algorithm | resize the image (16, 16, and then calculate the sample volume after flattening.
In this work, we
Algorithm 1 Calculation of Sample Volume
Input: Training set D ((r,, M with the total number C' of classes Onput: Sample volumes for all classes
for j 1 to do
Select the sample set D, = ()) for class j from D, my is the number of samples for class j
Resize the image to (imagesize, imagesize, 3)
Flatten the image into a vector length = imagesize x imagesize x and store it in
2, 21, 22.... ... 2m, E RdXm;
Z, Z, NumPy. mean (Zj. 1)
Calculate the covariance matrix Σj = 1 2
Calculate the sample volume Vol (E,) glogz det (I + +dE)) for class j end for
F DYNAMIC SEMANTIC-SCALE-BAAAAAA
LEARNING
F.1 DSB-NSM, DSB-ST AND DSB-FOCAL LOSs
Given the embedding z of a sample and label yt. the dynamic cemantic-scale-balacee (DSB) loss can be expressed as:
1 M(z, v:), 1,2, 1,2,...,C, S.
DSB (,)
where yi is the label of the sample from class i. To show how to combine the general loss to generate the dynamic cemantio-scale-balanee entropy loss and SoftTriple (ST) loss. NormSoftmax removes the bias term in the last linear layer and an 1.2 normalization module is added the inputs and weights before the SoftMax loss.
loss, improve the NormSoftmax (NSM) cross-
26

---

Published as a conference paper at ICLR 2023
[W1, W2. wel RF is the last fully connected layer, then the DSB-NSM with temperature generated by embedding & can be written as:
1 S;
exp(w 2/03)
DSB NSB-NSM(2,B8) -S log
(5)
وز= exp( (wf 2
The SoftTriple loss combined with the semantic-ssall--allneee term expressed as
exp(A(D' 2.44 -)) exp(A(D's.. 3.3. - -5) - + exD()
DSB - ST(, =
S: log(
(6)
where À is scaling factor and is a hyperparameter. The relaxed similarity between embedding expl #= =
and class is defined as D' D'z,c- = Σ E, expl x" s"wf) w where & number of centers for each class.
The purpose of Focal loss is to apply small loss weights to samples with high classification confidence, thus increasing the proportion of loss of hard samples with low classification confidence to the overall loss. The a- e-balanced variant of Focal loss regulates the proportion of loss among samples while assigning different weights each class, which denoted as FL (pt) -ay pt) log (Pt). where is the probability that the sample belongs to the true class. When a, = 3. Focal loss is transformed into DSB-Focal loss.
F.2 DYNAMIC RE- WEIGHTING TRAINING FRAMEWORK
Given the training samples X I2..., TTN] containing C' classes and corresponding la- bels Y = W1, 3/...... MN]. the number of samples per class is N; (í = 1, 2...... tal number of samples is N. The d-dimensional features extracted by the CNNs are denoted as this work, we conduct experiments for two types of tasks, image
......,C C), and the to-
Z = [21, 22. .,, E dx classification and deep metric learning. In the field of deep metric learning, 64 64-dimensional features are generally adopted, while the features extracted by the network in image classification tasks tend to be of high dimensionality. For example, the feature dimension extracted by ResNet-50 is 2048, which will occupy more video memory. Therefore, when saving historical features in the classification task, one-dimensional average pooling is performed on all features reduce the feature dimension 64, which is consistent with the common feature dimension in the field of deep metric learning while preserving the geometry of the distribution (because the pooling operation is translation invariant, rotation invariant, and scale invariant).
In the following, we describe the stree-stage training framework in detail.
Storage Pool
enqueeee
First Stuge
epoch-1
dequeue
enqueue
cepochsi
Second Stage
epoch>n
mini batch
dequeue
enqueue
Loss
re-weighting
Third Stage
Sementic Scale
Figure 16: The three-stage training framework. The features in the storage pool are continuously updated during training and semantic scales calculated using all the latest features.
27

---

Published as a conference paper at ICLR 2023
The stree-stage training framework is shown in Figure 16
(1) In the first stage, all the features and labels generated by the 1st epoch are stored in Q. which is denoted as:
Zn
ZI
Z Y
:
E {R(
Q
Za
ZaN
1/1
YN
Q contains the features and labels of all samples, but in the early stage of training, the historical features have large drift from the current features and cannot be used directly calculate the semantic scale.
(2) The second stage corresponds epoch 2 to epoch n. each iteration, the oldest mini-bater features and labels in Q are removed and those generated by the current iteration are stored. The goal to continuously update the features Q until the feature drift is small enough. We set experiments, and the original loss function is used in the first two stages. Figure shows the effect of n on the model performance. A larger n does not hurt the model performance, but only takes a little more time. Experience suggests that setting to 5 is sufficient.
in our
(3) The third stage corresponds to epoch > n. At each iteration, the semantic scales are calculated using the features in Q after updating Q, and the original loss is re- weighted.
Algorithm 2 shows how to apply the dynamic re-weighting training framework by taking DSB-ST loss example.
Algorithm 2 Dynamic Re- Reightngg Training Framework
Input: Training set D {(x;, m)) total training epochs N Nepoch, defined encoder is model() Initialize the queue Q for epoch = to N Nepoch do
M for iteration 0 to batchsize do
Sample mini-batch {(Ii, y;) [i] botcheise from D
Calculate features F = -/f,,..,Tie, Jo...., /batcheisel St = model (xi) = 1,..., batchsize Store F and label into Q: enqueue (Q, (1. if epoch < n then
if epoch > 1 then Dequeue the oldest mini-batch features from Q
end if
Calculate loss L = SoftTriplesoss (F,y)
else
Dequeue the oldest maii--bachh features from Q Calculate loss L DSB. USBBSoffTwee roppllilessss (F,
end if
Perform back propagation: I.backward() optimizer.step()
end for end for
The proposed three-stage training framework overcomes the difficulty of not being able to calculate class-wit wise semantic scales during training due to the limited number of samples per batch. In fact, there is simple and brute-force method to achieve the goal of calculating class-wise semantic scales in real time, which is to extract features of all samples using the current model after each iteration. However, this would take lot of time, for example, when training ImageNet with batch size of 512, one epoch contains about 2,500 iterations. This also means that all features need to be extracted 2,500 times, which is unacceptable.
Our method causes almost no reduction in training speed. In terms of video memory consumption, even the extracted features of million-level dataset like ImageNet can all be placed on graphics card (about 6,000 MB of video memory is needed). In this work, we use 4 NVIDIA 2080Ti GPUs to train all the models. comparison of the video memory consumption and training speed for some
28

---

Published as a conference paper at ICLR 2023
Table 10:
Comparison of DSB-ST and SoftTriple in terms of memory consumption and training speed. The speed is measured by the average number of iterations per second. The additional video memory consumption due to our method is almost negligible.
GPU Memory
Training speed
Dataset
SoftTriple
DSB-ST
SoftTriple
DSB-ST
ImageNet-LT
24.29 GB
24.75 GB
6.01 it/s
5.56 it/s
iNaturalist2018
45.13 GB
46.88 GB
3.32 it/s
3.03 it/s
Cars
3491 MB
4097 MB
20.21 it/s
18.95 it/s
CUB-2011
3225 MB
3647 MB
21.95 it/s
18.58 it/s
Mini-ImageNet
3491 MB
3713 MB
23.34 it/s
20.36 it/s
experiments is shown in Table 10 It can be noticed that the consumption of our method is negligible for the video memory, and the training speed is about 90% of the original method.
G VOLUME FORMULA FOR THE LOW-DIMENSIONAL PARALLEL HEXAHEDRON IN THE HIGH-DIMENSIONAL SPACE
In Section 3.2 of the main content, we deduce from the singular value decomposition of the matrix 7 (21, 22.-.. 2mm] RR composed of features that the volume (ol(Z) of the subspace spanned
by 2; is proportional V (let(_17777 Here, assume that in Rdd given d-dimensional vectors, these vectors will define parallel hexahedron in R" The problem is how to calculate the parallel hexahedron. For example, consider two vectors
21
2
, 22
2
The parallel hexahedron defined by these two vectors is parallelogram in R3, We want to find formula to calculate the area of the parallelogram. (Note that the true three-dimensional volume of the planar parallelogram is 0, just as the length of the point is 0 and the area of the line is 0. Here, are trying to measure the two-dimensional "volume" of the parallelogram.)
Next will introduce two special cases of parallel hexahedral volume, for a single vector
01
R
whose parallel hexahedral is itself. Here "volume" means the length the vector, and according to the Pythagorean theorem its volume is
af
(7)
Another case is to give vectors R., Suppose that these n vectors are
[ai]
aia
21
2n
am
we can know that the volume of the resulting parallel hexahedron is
d11
ain
det
(8)
an
Onn
29

---

Published as a conference paper at ICLR 2023
In the non-special case. the formula for the volume of low-dimensional parallel hexahedron in -igh-dimensional space will contain Results (7) and (8). Here, we first present the final formula and then discuss why it is reasonable. Write the & vectors . in R column vectors. Let
= ..,,.... =k] E R
and the volume of the parallel hexahedron derived from the vectors 21....,
/et[
We now discuss why /det[Z must be the volume in the general case.
Lemma For a matrix Z = [21y..., #ke, we can get
121 2
21 22
21 2/
zTz
2k 21
2k 22
|zx|
where 2ÿ denotes the dot product of the vectors and 2j and |2s| of the vector.
V2z denotes the length
The proof of Lemma I needs to focus only on
2T
z
2k
If apply any linear transformation that preserves angularity and length in R" (in other words, we perform rotation operation on 12",, the numbers |2 and 2, 2j do not change. The multiple sets of all linear transformations that preserve angle and length IR" form a group, called the orthogonal group and denoted as O(n). This allows us to reduce the problem to that of finding the volume of parallel hexahedron in R
Proof. It is known that
121 2
21 22
21 2k
det ZTTZ
det
:
2x 21
2k 22
12x
To prove that the above equation must be the formula for the volume, first consider set of standard basis of R":
0
61
62
On
According to Lemma we are able find a rotation of R", which able to maintain both length and angle, and also rotate our vectors 21,.... such that they can be fully represented linearly by the first standard vectors e (which geometrically reasonable). After the rotation, the latter - dimensions of each vector zi are 0. Therefore we can think of our parallel hexahedron as consisting of vectors & and we already know how to calculate it, which
121
2
21
22
21 2
det
26
21
2k
22
2k
30

---

Published as a conference paper at ICLR 2023
H MORE ANALYSIS
In this section, we will add the experimental results of the following three questions.
(1) The effectiveness of dynamic semantic-scale-balatee learning without considering inter- class interference.
Comparison with other methods of measuring class-level difficulty.
Dividing ImageNet into three subsets based on semantic scale, and showing the performance of dynamic semantic-scale seaetti--ccall--a--eeeedde learning on the three subsets.
H.1 THE EFFECTIVENESS OF DYNAMIC SEMANTIC SEMANTIC-SCALE CONSIDERING INTER-CLASS INTERFERENCE
LEARNING WITHOUT
We have weakened the effect of inter-class interference when designing the measurement of semantic scale imbalance. The semantic scale of m classes after maximum normalization is assumed to be S' [$1, .,,........ S _-_ and the centers of all classes are (01, .2..... Omn] Define the distance be- tween the centers of class and class
di.j = ||o; 0j 11l- the weight
m-1 Ej=1 =1 10, 9|||
The weights of m classes are written as W' = [w1 W2..... ................................ Wmm After the maximum normalization and logarithmic transformation of W', we can obtain Wllogg (a W a 1, where used control the smoothing degree of W. After considering the inter-class distance, the semantic scale @ W, and the role of in dominating the degree of imbalance is greater when a is larger. The second-order derivative of the function Wllog (a + W') is less than 0, so the increment of W decreases as a increases. When the value of a is taken to large, W" hardly works.
The Pearson correlation coefficients between class accuracy and inter-class interference, semantic scale, and semantic scale considering inter-class interference, respectively, are shown in Table [ It can be seen that the Pearson correlation coefficient between semantic scale and class accuracy CIFAR-IO-LT without considering inter-class interference still reaches 0.8688, while the correlation coefficient between inter-class distance W and class accuracy is only 0.2957, which illustrates the importance semantic scale. In addition, we have added the correlation coefficients between the effective sample numbers and class accuracies in Table [ It be observed that the correlation between effective sample number and class accuracy is almost the same as the correlation between sample number and class accuracy, which due the fact that effective sample number monotonic function of sample number. To demonstrate the performance of dynamic semantic- scale-balanced learning without considering inter-class interference more detail, we conducted experiments ImageNet-LT.. The experimental settings are the same as those in Table
03.55
.67.567.3
DSB
57.0
661.568..1
DS9: Focal
60
DSB CE-1
DS9: Focal
60
Top-1 ACC (96)
50
Top-1 ACC (96)
42.212.5
5.0
40
43.944.2
11.0
40
30
20
23.323.7
10
7.7
20
13.1
0
10
Head
Middle
Tail
Overall
Head
Middle
Tail
Overall
Figure 17: Accuracy comparison ImageNet-LT..
The dynamic semantic-scale-balann loss without considering inter-class interference is denoted as DSB-CE-L and DSB-Fooal-L.. The experimental results are shown in Figure [7] can be observed that DSB-CE-1 and DSB-Focal- have almost no performance degradation compared to DSB-CE and DSB-Focal. The above observation is as expected, since our recent study shows that the correlation between the separation degree of feature manifolds and the accuracy of the corresponding class decreases during training and that the existing model can eliminate the main effect of separation degree between feature manifolds model bias.
31

---

Published as a conference paper at ICLR 2023
H.2 COMPARISON WITH OTHER METHODS OF MEASURING CLASS-LEVEL DIFFICULTY
Difficult example mining 47 58
is an instance-level approach, while we focus on class-level work measuring class difficulty (CDB loss (37) published
difficulty. We note that in UJC, which can be compared with our work. In addition, LOCE [1] and domain balancing 2 also measure class-level difficulty, but LOCE is designed for object detection tasks, so we compare semantic-scale sel----aalebbbbceeddddeeeee--eed learning with CDB loss and domain balancing. The description of CDB loss, LOCE and domain balancing as follows.
The imbalance in class performance is referred the "bias" of the model, and 37 defines
the model bias as
max N c= Ac min Ag +
bios max
1, 1,0),
where Ag denotes the accuracy of the e-th class. When the accuracy of each class is identical,
bias = 0. [37] computes the difficulty of class e using 1 - A. and calculates the weights of the loss function using nonlinear function of class difficulty.
LOCE [14] uses the mean classification prediction score to monitor the learning status for different classes and apply to guide class-level margin adjustment for enhancing tail-class performance (78).
Domain balancing 2 studied a long-tailed domain problem, where small number of domains (containing multiple classes) frequently appear while other domains exist less. To address this task, this work introduced novel domain frequency indicator based on the inter-class compactness of features, and uses this indicator re-margin the feature space of tail domains 78
DS3-CE
DSB-CE
Domain balancing CE
OJB-AN-C
58.7
62.6
CUB-WICE
Domain balanciingCC
Top-3 ACC (96)
Top-1 ACC (96)
50-
50
16.845.5
41.542.5
42. 40.7700
38.5y7.4
30-
30
21.4 11.619.3987
20
20
10=
IF=100
55=50
IF=10
Head
Middle
Tail
Overall
CIFAR-100-LT
ImageNet-LT
Figure 18: Accuracy comparison with other methods of measuring class-level difficulty.
We implemented CDB loss (37. LOCE [1], and domain balancing 2 on ImageNet-LT. Observing Figure T8, our proposed semantic scale balanced learning outperforms these three approaches. In addition the comparison with the class-level difficulty weighting method, we add the results of the improvements to PaCO (10] in Table Balanced softmax is included in PaCO, and although we have shown Table 2 that our method significantly improves Balanced softmax, we still improved PaCO and conducted experiments to allay researchers' concerns. All experiments adopted the same training strategy and parameters as in Table 2
H.3 THE PERFORMANCE OF DYNAMIC SEMANTIC-SCALE. BALL-CE LEARNING IN GENERALIZED LONG-TAILED LEARNING
Invariant feature learning (IFL 566 considers both inter-class long tail and intra-class long tail and further defines the generalized long-tailed classification. The intra-class long tail has not been considered before, and invariant feature learning takes it into account the long-tailed classification problem for the first time, which is remarkable progress in solving the long-tailed problem. IFL decomposes the probabilistic model of the classification problem as P(y x) defaults the class with few samples to be the weak class. should be noted that our study found that the geometric properties of the manifolds corresponding to different class distributions P(x) will
P(zly) P(y) and
P(2)
32

---

Published as a conference paper at ICLR 2023
affect the classification difficulty, which breaks the previous perception, the inter-class long-tail problem still has huge research potential. The existence of data manifolds is already a consensus, and data classification be regarded as the unwinding and separation of manifolds. Typically, deep neural network consists of a feature extractor and classifier. Feature learning can be considered as manifold unwinding, and well-learned feature extractor is often able unwind multiple manifolds for the classifier to decode. In this view, all factors about the manifold complexity may affect the model's classification performance. Therefore, we suggest that future work can explore the inter-class long-tailed problem from geometric perspective. Also, both the inter-class long tail and the intra-class long tail need to be considered, which will greatly alleviate the long-tailed problem.
Invariant feature learning estimates relatively unbiased feature centers by constructing the resampling strategy and uses center loss for unbiased feature learning. We applied IFI. to dynamic semantic- scale-balanced learning to consider both the inter-class long tail and the intra-class long tail, and validated it on ImageNet-LT and NNaturalist2018. Experiments show that DSB-CE combined with IFL achieves further performance improvement, and we have supplemented the results and analysis Table
Table 11: Evaluation on MSCOCO-GLT.
Protocols
CLT
GLT
ALT
Accuracy Precision >
Overall
Overall
Overall
cRT
73.64
75.84
64.69
68.33
49.97
50.37
LWS
27
72.60
75,66
63.60
68.81
50.14
50.61
Deconfound-
55
73.79
74.90
66.07
68.20
50.76
51.68
BLSoftmax
43
72.64
75.25
64.07
68.59
49.72
50.65
Re-balance
BBN 80
73.69
77.35
64.48
70.20
51.83
51.77
LDAM
4
75.57
77.70
67.26
70.70
55.52
56.21
DSB-LDAM
76.63
78.95
68.15
71.87
56.16
56.87
BLSoftmax IFL 156
73.72
77.08
64.76
70.00
52.97
53.52
DSB-BL BSB-BLLofttaxx
73.96
77.37
65.03
70.15
50.24
51.36
DSB BLSSfttaxx + IFL
74.64
78.06
65.47
70.83
53.08
53.75
cRT + IFL [56]
76.21
79.11
66.90
71.34
52.07
52.85
DSB-cKt
76.82
79.95
67.26
71.73
51.41
51.94
LWS + IFL [56]
75.98
79.18
66.55
71.49
52.07
52.90
DSB-LWS
76.55
80.06
67.03
72.15
51.64
51.16
We note that IFL. proposes two datasets ImageNet-GLT and MSCOCO-GLT and three testing protocols. Since our paper already contains a large number of experiments, we selected to conduct experiments on MSCOCO- Table On the CLT and GLT protocols, we significantly improve the performance BL-softmax, LDAM. and BL-softmax+IFL. Also, approach promotes the performance of the above three methods on the ALT protocol, which may be caused by the additional gain from stronger inter-class discriminability. Due page limitations, the experiment is tentatively supplemented in the appendix, and we will include this experiment the main text if the paper is accepted. The experiments show that alleviating both inter-class long tail and intra-class long tail can significantly improve the model performance, so we encourage researchers to pay attention to the intra-class long-tailed problem.
with the same experimental settings as IFL. The results are shown in
H.4 PERFORMANCE OF DYNAMIC CIHANTIC-SCALES BALLE-BALANCED LEARNING ON THREE SUBSETS OF IMAGENET
We divided ImageNet into Head, Middle, and Tail subsets based on semantic scale, which contain 333, 333, and 334 classes, respectively. The performance of DSB-CE and CE on the three subsets when the backbone networks are VGG-16 and ResNet-18 shown Figure 19
The experimental results show that semantic-scale-aalare performance of CE on Tail subset. Meanwhile, DSB-CE also outperforms CE on Head and Middle, which may be caused by the performance gain from better feature learning. In addition to the classification problem, we hope to introduce semantic scale imbalance the fields of object detection, semantic segmentation, etc. promote the fairness of the model.
learning significantly improves the
33

---

Published as a conference paper at ICLR 2023
100
100
(VGG-16)
(HesNet-18)
DSB-CE (VGG-16)
DSB-CE (ResNet-18)
90
90
Top-1 ACC (%)
Top-1 ACC (%)
80.5
80.2
70.2 69.5
71.6
70.8
70.1
63.4
62.5
60
59.8
60
59.4
Head
Middle
Tail
Overall
Head
Middle
Tail
Overall
Figure 19: Comparison of CE and DSB-CE on ImageNet, where the backbone networks are VGG-16 and ResNet-34, respectively. It can be observed that dynamic semantic-scale- balancedd learning significantly improves the tail class performance.
1 APPLY SEMANTIC SCALE TO SOLVE OTHER PROBLEMS
1.1 SELECT RELLEEEERESENTED DATA
Downsampling the head class is one of the methods to alleviate the long tail problem. which balances the number of samples but leads to the loss of head class information. Therefore, it is important to develop a downsampling method that preserves the head information. We propose an idea to select well-represented data based the geometric meaning of semantic scale.
The existence of data manifolds is a consensus that the same class of data is usually distributed around low-dimensional manifold. Different dimensions of the manifold represent different physical characteristics, and samples located at the edges of the manifold often tend to overlap with other manifolds. Therefore, we believe that the following two principles should be obeyed when downsampling:
Uniform sampling inside the manifold. It ensures that the volume of the manifold does not shrink significantly after downsampling.
Increase the sampling rate of samples at the edges of the manifold. makes the sampled dis- tribution with significant bounds, which helps to improve the robustness of the classification model.
As shown in Figure 20 we refer to the strategy that obeys the above sampling principles as "pizza" sampling.
Pre-trained CNN
Feature
dog
Figure 20: Schematic diagram of pizza sampling. Yellow samples indicate the selected samples and green samples indicate the discarded samples.
Uniform sampling is easy to do, but how do we sample as many samples as possible from the edges of the manifold? We propose randomly sample & subsets in the original sample set and calculate the semantic scales of the subsets. Then repeat the above operation several times and select the subsets with the largest semantic scales as the final samples.
34

---

Published as a conference paper at ICLR 2023
1.2 GUIDE DATA COLLECTION
When collecting data that has never been studied before, we do not know how many samples to collect represent their corresponding class well because of the lack of prior knowledge. When too few samples are collected, the class is under-represented. And sampling too many samples will consume huge costs. The marginal effect of the semantic scale can help us judge whether the currently collected samples have enough feature diversity, and we can stop collecting samples when the feature diversity tends to be saturated. Specifically, the data collection process is as follows.
(1) For class c, m samples are collected each time.
(2) After the (n 1 )th collection of samples, there are (n 1) m samples, and the semantic scales of these samples are calculated.
(3) After the nith collection of samples, there are n x m samples, and the semantic scales of these samples are calculated.
(4) Calculate the increment of the semantic scale for the nth time relative to the (n 1)th time.
(5) Calculate (S.-S.-1) the increment of semantic scale is less than a% of S,, it means S, that the feature diversity of class has not changed significantly and the sample collection can be stopped. The parameter can be adjusted according to the needs of the task.
Geometric analysis of data manifolds can bring new perspectives to data science. We will open source the toolkit for measuring the information geometry of data, which includes the application of semantic scale in various scenarios, such as data collection, and representative data selection.
J MORE EXPLANATION OF FIGURE 2
To see it more clearly, we zoomed in on Figure 2 and plotted it in Figure 24 Previous studies have observed that (1) given sufficient data, the classification performance gain is marginal with additional samples. (2) When the data is insufficient, the classification performance drops sharply as the number of training samples decreases. We speculate that phenomenon 1 may be caused by the marginal effect of feature diversity. It should be noted that CB loss considers marginal effects, but it only qualitatively describes the gradual flattening of feature diversity with the increasing number of samples. Taking example, we first select few samples for each class, train the model and test the accuracy. Then new samples are continuously added to the original samples instead of re-selecting more samples train the model. The experiments corresponding to each point in Figure are trained from scratch. While increasing the data we find that there are marginal effects of semantic scale, which indicates that our proposed measurement is as expected. The marginal effects of feature diversity explain phenomenon 1.
CIFAR-10
However, phenomenon 2 not explained by the marginal effects, and the effective number of samples from CB loss does not predict phenomenon 2 at all, because the effective number of samples does not grow faster than the number of samples (which we have analyzed in Section 2]. We experimentally find that when the samples are few, the feature diversity measured by the semantic scale increases rapidly with the number samples, and this increase is faster than the linear increase. The rapid increase of feature diversity measured by the semantic scale explains phenomenon 2.
K CAN THE SEMANTIC SCALE CAPTURE THE HIERARCHICAL STRUCTURE?
HCSC 15 constructs the hierarchical structure of classes by bottom-up k-means, and we use the
example shown by HCSC to validate our approach. Given the following seven classes: Poodles, Samoyeds, Labradors, Persian, Siamese, Chimpanzee, and Gorilla, each class contains 1, 000 samples, and the hierarchical structure of the seven classes is shown in Figure 25
We collect 1, 000 images for each of the three parent classes (Dogs, Cats, and Monkeys), which can adequately represent the three parent classes, i.e., the feature richness is sufficient. Then can the semantic scale be used to match the correct parent classes for the seven classes? According to our theory, the manifolds of the child classes should be the manifold of the corresponding parent class, and they have inclusion relationship. Therefore, when the data of the child classes are mixed
35

---

Published as a conference paper at ICLR 2023
CIFAR-10
CIFAR-10
250
2000
200
Semartic dateeet
1800
150
class 0
1600
class
Semrnnc
200
class
class
1400
60
class
class
-50
class
scale per
class 5
class
Reshitt Top-1 ACC
40
class
Top-1
1000
10000
20000
30000
40000
50000
Number
Number of images per detaset.
CIFAR-100 100
CIFAR-100
18000
17000
1.0
16000
445
14000
50
class
cless
per
clecs
12000
Reslitt- lop-1 ACC15)
clece
Restiat Tap-1 ACCIN)
11000
300
200
mages
Number images per dataset
Mini-ImageNet
Mini-ImageNett
300
Semantic datazet
230
20000
200
ctess
diece
350
diecs
dles
diss
100
0ioc
10000
dlss
per
dios
0b35
clacs
5000
Restlet- Tap-1 ACC
Resillet Tap-1 ACCIN)
200
480
520
Number mages per class
Number mages per dateset
Figure 21: Left column: curves semantic scales with increasing number samples for the first ten classes from different datasets. Right column: for different sub-datasets, curves of the of semantic scales for all classes and top-1 accuracy curves of trained ResNet-18 and ResNet-34 All models trained using the Adam optimizer (28) with initial learning rate of 0.01 and then decayed by 0.98 at each epoch.
into the data of the parent class, the manifold volume of the parent class will not change significantly. We propose the matching method of semantic hierarchy based on this property. The specific steps are follows.
train ResNet-18 classification model on seven child classes. We the batch size to 64 and adopt the adam optimizer with a learning rate of 0.01 (linear decay), momentum of 0.9, and a weight decay factor of 0.005.
Extract the features of all samples from seven child classes and three parent classes.
(3) Calculate the semantic scales of the three parent classes.
(4) Select child class e from the seven child classes.
(5) Mix the data of child class into the data each parent class and calculate the semantic scale of the mixed data, we can get three values.
36

---

Published as a conference paper at ICLR 2023
Cats
Dogs
Persian
Poodles
Slamese
Labradors
Samoyads,
Monkeys
Gorila
Chimpanzee
Figure 22: Image datasets typically contain multiple semantic hierarchies.
Table 12: Results of matching parent class for each child class, where the Ratio of semantic scales denotes the ratio of the semantic scales of the parent class after mixing to before mixing. Predicted parent class means the parent class we matched for the child class, and Real parent class denotes the real parent class corresponding the child class.
child class
Poodles
Samoyeds
Labraders
Persian
purent class
Dogs
Cats
Monkeys
Dogs
Cats
Monkeys
Dogs
Cats
Monkeys
Dogs
Cats
Monkeys
Ratio semantic scales
1.68
1.75
Predicted parent class
V
V
V
Real parent class
child class
Siamese
Chimpanzee
Gorilla
parent class
Dogs
Cats
Monkeys
Dogs
Cats
Monkeys
Dogs
Cats
Monkeys
Ratio semantic
1.69
1.02
1.83
Predicted parent class
Real parent class
(6) Calculate the changes in the semantic scales of the three parent classes and them.
Match the parent class with the smallest change in semantic scale for child class c.
Perform steps (3) (7) for the remaining six child elasses.
We summarize the ratio of the semantic scales of the parent classes after mixing to before mixing in Table [2] If the change in the semantic scale of parent class small after child class is mixed into that parent class, they are considered to have nested relationship. Based on the above method, we successfully match each child class the parent class. Experimental results show that our proposed measure of semantic scales can capture the semantic hierarchy of classes. Our study can inspire hierarchical feature learning as well facilitate its performance in downstream tasks.
L FUTURE WORK AND CHALLENGES
L. MODEL-INDEPENDENT MEASURE OF DATA DIFFICULTY
The performance of the models varies across classes. In the past, was believed that model bias was caused by an imbalance in sample numbers, but growing body of research suggests that sample numbers are not the only factor affecting model bias. Of course, model bias is also introduced not by the model structure, but by the characteristics of the data itself that affect model performance. Therefore. it is very important to propose model-ing moodel-independers measurements to represent the data itself, and this work will greatly contribute to understanding of deep neural networks. In this
37

---

Published as a conference paper at ICLR 2023
paper, the effeet of the volume of the data manifold on the model bias is explored from geometric perspective. provides new direction for future work, namely the geometric analysis of deep neural networks. The geometric characteristics of the data manifold will help further reveal how neural networks learn and inspire the design of neural network structures.
X1
X X1
Ximage size
X class number
Figure 23: Changes in the geometry of data manifolds as they are transformed in deep neural network. The classification process of the data includes untangling the manifolds from each other and separating the different manifolds.
L.2 A GEOMETRIC PERSPECTIVE ON DATA CLASSIFICATION
Natural datasets have intrinsic patterns that can be generalized to the manifold distribution principle: the distribution of class of data is close low-dimensional manifold. As shown in Figure 23 data classification can be regarded as the unwinding and separation of manifolds. When a data manifold entangled with other perceptual manifolds, the difficulty of classifying that manifold increases. Typically, deep neural network consists of a feature extractor and classifier. Feature learning can be considered as manifold unwinding, and well-learned feature extractor often able unwind multiple manifolds for the classifier to decode. In this view, all factors about the manifold complexity may affect the model's classification performance. Therefore, we suggest that future work can explore the inter-class long-tailed problem from geometric perspective.
L.3 INTRODUCE SEMANTIC SCALE IMBALANCE IN OBJECT DETECTION
Long-tailed distribution is one of the main difficulties faced by object detection algorithms in real- world scenarios. The classical object detection algorithms generally trained designed datasets with relatively balanced data distribution. contrast, the accuracy of these algorithms tends to suffer significantly on long-tailed distributed datasets. So far, methods for breeground-backroummg imbalance and class imbalance have been proposed extensively, but these methods based on the number of objects define the degree of imbalance and cannot explain more phenomena. We will give examples below.
manually
In the field of object detection, is often encountered that although class does not appear frequently. the model can always detect such instances efficiently. It is easy to observe that classes with simple patterns are usually easier learn, even the frequency with low frequency in object detection are not necessarily always harder to learn. We believe that it a valuable research direction to analyze the richness of the instances contained in each class, and then pay more attention to the hard classes. The dimensionality of all images or feature embeddings in the image classification task is the same, which facilitates the application of the semantic scale proposed in this paper. However, the non non-fixed dimensionality of each instance in the field of object detection brings new challenges, so we have to consider the effect dimensionality on the semantic scale, which is direction worthy of further study.
such classes low. Therefore, classes
38

---

Published as a conference paper at ICLR 2023
L.4 CHALLENGES OF CLASS IMBALANCE IN DEEP LEARNING
Class imbalance remains a major challenge in the field of deep learning. Data imbalance classification, although widely studied, still lacks effective and clear methods and guidelines. The problem of object detection for class imbalance is still in its infancy and requires greater investment of attention. In the following, we summarize the important future challenges and research directions this field.
(1)
The more precise measure of class difficulty. An increasing number of studies have shown that the sample number does accurately reflect the accuracy of the model in recognizing classes. Therefore, more extensive measures should be proposed to redefine the long-tail distribution facilitate classification and object detection tasks and further expand the scope of research on long tailed recognition. For example, dataset with perfectly balanced sample numbers may not balanced under other measures.
(2) Long-tailed distribution of properties in classes. As shown Figure 24 [56. previous studies have focused the imbalance between classes and ignored the imbalance of properties within each class. For example, most pandas have black and white fur, and only small proportion of pandas are brown. In visual recognition tasks, should not only pursue the overall accuracy of the class but also pay attention to whether samples with sparse properties in class be classified accurately. In medical image classification, the above point is particularly important. For example, pulmonary diseases contain many different types of diseases, and generally the more severe the disease tends to have smaller sample number, suggesting that there is an imbalance of properties under the label of pulmonary disease. We hope to be able to recognize more severe diseases more accurately so that patients do not miss the best time to treat them.
Long-tail distribution of attribute levels
Long-tail distribution attiibuee levels
Mansinccss
Brown
Blue
Black & White
Brown
Frequent
Class index
Rare
Figure 24: Class-level long-tailed distribution and intra-class attribute long-tailed distribution.
(3) Generalization performance of the model outside the training domain of the tail class. As shown Figure 25 tail classes often have very few samples, so these samples do not well represent the true distribution of the tail classes, which results in the model consistently failing to learn and adapt to the tail classes correctly. Obviously, recovering the underlying distribution of the tail classes helps the generalization performance of the model outside the training domain of the tail classes. It is currently shown that similar classes have similar distribution statistics (variance), which can lead researchers to recover the underlying distribution of tail classes. However, the current research is still in its infancy, and it is not sufficiently stringent assumption that similar classes have similar variances. Therefore, we hope that in the future researchers will be able to help recover the true distribution of tail classes by more means.
(4) How to choose the appropriate long-tailed recognition method in the task. Up to now, a large number of visual recognition methods long-tai tail distribution have been proposed. While individual methods have positive performance long-tailed recognition tasks, some combinations of methods may have negative effects. Few studies have focused the
39

---

Published as a conference paper at ICLR 2023
(a)
(b)
(c)
The Distribution
Train samples
Test samples
Coneet Prediction Incorrect Prediction
vix
Decision houndary
Figure 25: (a) When the samples uniformly cover the true data distribution, the model can learn the correct decision boundaries and can correctly classify unfamiliar samples be tested. (b) When the samples cover only portion of the true distribution, unfamiliar samples to be tested are highly likely to misclassified due the error in the decision boundary. (c) The direction in which the arrow points is the best direction expand the sample.
selection and combination of different training techniques and methods. In the future, is possible to explore how to select existing methods on specific tasks, and further, effective combinations of different methods are important.
(5) Multi-domain deep long-tailes learning. Past research has typically focused the problem long-tailed distribution over a single domain, which has limited the research ideas. As shown in Figure 26 data from multiple domains can complement each other alleviate the long-tailed distribution of classes 73]. For example, in plant and animal classification, cameras are placed in different places capture animals, but some animals only appear in a fixed area, which leads to different label distributions for animals captured by different cameras. But by combining the data from all cameras, more balanced class can be obtained. Similarly, a similar situation occurs in other practical applications. For example, in visual recognition problem, the few classes from "photo" images can be complemented by potentially rich sample from "sketch" images. In autonomous driving, a few classes of "real" life accidents can enriched by accidents generated "simulations" medical diagnosis, data from different populations can be mutually augmented, c.g., small sample from one institution can combined with the majority of possible instances from other institutions. In these examples, different data types act different domains, and such multi-domale data can also be utilized effectively to address data imbalances.
addition, in
#Instances
#Instances
#Instances
dog
horse
dog
horse
dog
horse
Multi-domain deep long-tailed learning
Figure 26: The frequency of the same class appearing in different domains may differ significantly, assuming smaller sample of horses in the real world and larger number of horses in cartoon form. Images from different domains can complement each other to form dataset with a balanced sample number. The purpose of multi-domain deep long-tailed learning is to train unbiased models using data from multiple domains and generalize over all domains.
(6) Recognition of unbalanced data streams. Continuous learning aims to process new data that is continuously generated in order to dynamically update and adapt the model to the latest data domain. Challengingly, as new data is generated, the degree of imbalance between classes changes and what used to be a tail class may become a head class. The long-taile distribution of properties within classes also affect the performance of the model if
40

---

Published as a conference paper at ICLR 2023
concept drift occurs. Thus the key to handling unbalanced data streams is to evaluate the class-level difficulty and the long-tailed distribution properties within classes in real time, which is huge challenge.
Augmentation methods for other modally unbalanced data. Methods for multi-sample synthesis are widely used image data augmentation, such as Mixup and Cutout, but there is still lack of data augmentation methods for other modal data (e.g., speech and table). Researchers can design more general method to generate samples of any type of data.
(8) Other long-tailed visual recognition tasks. Current research focuses on long-tail image classi- fication, while less attention has been paid to long-tail object detection, image segmentation, and regression tasks. In object detection, there are multiple imbalances, such as foreground- background imbalance and imbalance between classes belonging to the foreground, which are unresolved challenges. With further applications of deep learning, research on imbalance learning in various fields will be of great benefit for weal-world applications.
This study suggests some future avenues of inquiry further deepen and expand the study of unbalanced learning. Of course, the scope of future inquiry into unbalanced learning is not limited to the eight challenges mentioned above, and we believe that new questions will arise the course of inquiry into these eight challenges, but that researchers will eventually address them over time.
Science is endless, an eternal mystery.
Albert Einstein
41


Reviews:
[{"review_id": "yGItNI61cCd", "reviewer": "Reviewer_kLsB", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes a new concept semantic scale, which conceptually captures the richness or diversity of the feature space, e.g., “Bird” features should be richer than “Swan” features. This quantity is quantified by the volume of the learned feature space, proportional to the determinant of feature covariance matrix. Using this new concept, the paper attempts to explain 1) the change of performance improvement w.r.t the number of training data and 2) implicit semantic bias in classification when classes are balanced. Finally, based on the discrepancy in semantic scales, the paper proposes a dynamically weighted loss function to show improvement on classification under balanced and imbalanced settings. ", "summary_of_the_review": "The paper presents a novel idea to quantify implicit bias in classification. However, questions regarding claims and some less well explained components in the method make the overall message less convincing. ", "strength_and_weaknesses": "Pros:\n\n1, The concept of semantic scale is novel and well-motivated mathematically. \n\n2, The usage of sphere packing method to control the scale of semantic scale is a very neat idea.  \n\nCons: \n\n**1, Strong correlation with model performance weakens the contribution of the proposed loss function.**  The paper highlights the correlation between semantic scale and model performance. There exits method such as hard-class mining [1] and recall loss [2] that rely on the model performance to re-weight losses. This strong correlation makes one wonder why a loss function weighted by semantic scale would be better than one weighted by performance. \n\n**2, Adding inter-class interference seems ad-hoc and obscures the contribution of semantic scale.** In sec 3.4.1, an inter-class interference term is added to the semantic scale metric. This combination seems ad-hoc. Even though the performance of the combined metric seems to work well, it becomes less clear how much contribution the vanilla semantic scale has. \n\n**3, Marginal effects of semantic scale and model performance w.r.t the amount of data represent correlation not causation.**  In the sec.3.3, the authors observe a strong correlation between the marginal effects of semantic scale and model performance w.r.t the amount of data. This is an interesting correlation, however, which cannot be used to explain why classification gain is marginal when data are sufficient or why performance drops quickly when data are scarce, as claimed in the abstract. Because the observation only establishes a correlation not a causation between the two events. \n\n**4, Semantic scale is model-dependent.** Even though the authors claim that semantic scale is a natural measure that is model-dependent, it is calculated from the output (feature space) of a trained model and therefore should be model-dependent. The same dataset, with the same class statistics, will lead to different semantic scales if models are different. \n\n[1] A. Shrivastava, A. Gupta, and R. Girshick, “Training region-based object detectors with online hard example mining,” in Proceedings\nof the IEEE conference on computer vision and pattern recognition, 2016, pp. 761–769.\n\n[2] Tian, Junjiao, et al. \"Striking the Right Balance: Recall Loss for Semantic Segmentation.\" 2022 International Conference on Robotics and Automation (ICRA). IEEE, 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "wu7teKUhB3", "reviewer": "Reviewer_DeDZ", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This work studies long-tailed learning in the semantic level, which goes deeper beyond the conventional quantity bias. Specially, the authors explore to explain multiple phenomena by defining the semantic level imbalance and propose a measure by volume of manifold to reweight the learning objective, which has been demonstrate to improve the performance effectively.", "summary_of_the_review": "In summary, the current work has both pros and cons. I advise the authors to carefully consider the advices in the review and I can consider to raise the score if the concerns have been well solved.", "strength_and_weaknesses": "In summary, there are some positive points regarding this work, summarized as follows,\n\n(1) The topic is relatively novel and meaningful in the perspective of long-tailed learning, since the quantity bias is not the essential factor to decide the performance of imbalance learning. \n\n(2) The proposed measure for imbalance degree is effective, which is demonstrated by a range of validation on uniform datasets and long-tailed datasets. Specially, the trend of the curve is approximately similar to the final performance, which is heuristic to future extensions.\n\n(3) The proposed semantic-scale-based balanced loss has been demonstrated useful to a range of previous methods to improve the performance in both class-imbalanced and class-balanced cases.\n\nHowever, there are also some clear flaws in terms of the current submission, which needs to be further improved.\n\n(1) It is confusing that the authors discuss both the marginal effect and semantic-scale imbalance. Especially, the marginal effect is not the unique characteristic of imbalance learning, and also exists in the absolutely balanced learning. When the samples are sufficient to make the model reach the optimal, more samples might not bring gains. \n\n(2) In terms of the semantic-scale-based imbalance learning, it is the approximately same topic with the recent work termed as the generalized long-tailed learning [1], which takes into the fine-grained pattern imbalancedness into account. The authors missed this work and did not discuss their difference as well as the corresponding comparison in the experimental part. Especially, they have released a benchmark in the github repository, which can be used in this work.\n\n(3) The evaluation in the quantity-uniform datasets like CIFAR10 and CIFAR100 is ill-posed, since both the training and the test are IID. The improvement is not convincing to attribute to the imbalance learning, and conversely it is more possible to be explained as the effect of the hard negative mining by the re-weighting. Here, the authors need to carefully discuss the improvement in the IID instead of the Non-IID case about the training and the test. \n\n(4) There are some concerns about the experiments, since it seems that the authors do not demonstrate the model can achieve the better improvement on the basis of the state-of-the-art method like PaCO and LA, which have adjusted the model according to the quantity imbalance. Note that, RIDE is a multi-branch ensemble model, and it is meaningless to show the improvement on RIDE achieves the state-of-the-art, since PaCO can also integrate this mechanism to achieve better performance. It will be better to conduct some experiments to compared LA or PaCO and your method in the same backbone, which can show the gap of the improvement between the explicit quantity bias and the implicit semantic-scale bias.  Again, it is better to have some comparison in the benchmark of [1].\n\n[1] Invariant Feature Learning for Generalized Long-Tailed Classification. ECCV 2022.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}, {"review_id": "Q-onsSes0_", "reviewer": "Reviewer_MKbH", "summary": "", "questions": "", "limitations": "", "rating": 8, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the problem of imbalance in classification problems. To be specific, they propose a measure to quantify the `semantic scale' of classes and use this measure as a means to analyze the imbalance among classes. Moreover, they use this measure to mitigate the effect of imbalance in various classification datasets. They report significant improvements over the baseline methods.\n\nI have reviewed an earlier version of the paper for ICML2022 and I see that the authors have significantly improved their paper since then (e.g. experiments on ImageNet, iNaturalist, MNIST & MNIST-LT). Thank you for taking into account my earlier comments.", "summary_of_the_review": "Fond of the paper. Novel approach, good analyses & insights, strong results.", "strength_and_weaknesses": "Strengths:\n- I am very fond of the idea of measuring imbalance and using such measures instead of counting the number of samples in classes. This is a very promising research direction.\n\n- I am very fond of the idea of extending the mechanism of Cui et al. [10] to develop a measure that better reflects imbalance among classes.\n\n- I am very fond of the analysis and the insight provided by the paper, before comparing it with other methods.\n\nWeaknesses:\n\nAlthough I am very fond of the approach and the ideas, I believe that the paper has significant limitations and is not ready for publication yet. Here are the major issues that I've identified, the minor problems are listed further below:\n\n1. First of all, the overall story of the paper can be improved: \n\n1.1. The paper starts with a very convincing example of semantic scale (birds and swans), motivating the overall approach of introducing a measure for it, then it does not provide any insight or analysis about whether or not the proposed measure is really capturing it: The analyses in Section 3 do not provide any information about the class labels nor a discussion about them.\n\n1.2 Although the motivation and the story of the paper is toward imbalance in classification problems, in the experimental analysis section, the paper adopts a deep metric learning setup and compares the proposed approach against the SOTA method of that domain. Although an analysis in a deep metric learning setup is not irrelevant, the story of the paper necessitates performing an extensive comparison with the SOTA of imbalance mitigation strategies.\n\n2. I recommend comparing the proposed scale against other measures of hardness / difficulty. I find the lack of such an analysis to be a significant limitation of the paper.\n\n\nTypos and Unclear Bits:\n- \"As the information of data increases\" => What's meant is unclear.\n- Section 3.2, 5th line: Error in notation. The covariance matrix for vector z_i is being defined, yet the summation runs over index i. I recommend using j in the summation just to avoid confusion.\n- Eq. 1: Did you want to write \"Vol(z_i)\" instead, as implied by the previous sentence? Or you should update the sentence.\n- \"Considering that real-world metrics ... scales. We\" => \"scales, we\". Otherwise, the sentence \"Considering that...\" is incomplete.\n- Figure 2 is way too small and not readable. \n- \"which indicates that the quantitative measurement of semantic scale is reasonable. In addition, the growth rate of the semantic scale varies across classes, which is determined by the grain size of the class itself. It leads to different semantic scales even if all classes have the same number of samples.\" => From Figure 2, what we see is that (i) semantic scale saturates and (ii) it is different for different classes. I would describe these results as \"as expected\" and consider \"reasonable\" a strong word.\n- Figure 2 right column: Changing dataset size requires re-tuning the hyper-parameters. Without re-tuning the hyper-parameters, it is expected that the performance will drop when data is subsampled. It is not clear whether this has been performed and whether the results in Figure 2 are reliable.\n- \"the centers of all classes are C\" => It is not clear in what space these centers are.\n- \"C\" is used in Section 3.3 and 3.4 for two different things. It is better to use different symbols. There is a similar problem with N (there might be other symbols).\n- Table 1: It would have been nicer to add E_n to the table.\n- Section 3.4.1: It would be nice to visualize or provide insight about the calculated weights.\n- Section 5.1 and \"We create long-tailed CIFAR-100 and long-tailed Cars196 for training using the first 60 and 98 classes of CIFAR-100 and Cars196, respectively, and the test sets are the remaining classes.\" => The authors should cite the literature for the setup.", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works.", "empirical_novelty_and_significance": "4: The contributions are significant, and do not exist in prior works."}, "extra_text": {}}, {"review_id": "MkR2_iN4swk", "reviewer": "Reviewer_bNtG", "summary": "", "questions": "", "limitations": "", "rating": 6, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies class imbalance and proposes that we should model the imbalance based on the semantic scale per class instead of the number of samples per class. Following this, this paper conducts a series of empirical studies to verify this idea and proposes a new method to handle class imbalance. Empirical results demonstrate the effectiveness of the proposed method.", "summary_of_the_review": "This paper is overall good, but there are several problems that need to be resolved. \n\n*******************\nPost-rebuttal: Thanks for the response and modifications. The quality of this paper is further improved now. However,  I still think the proposed method is a mix of different ideas, considering that inter-class interference is not that important. Empirical analysis and suggestions regarding inter-class interference are good, but it is not that elegant when designing them into one method. Even so, I appreciate the efforts of the authors and increase my rate to 6.", "strength_and_weaknesses": "Strengths:\n1. The problem of class imbalance is important for practical applications.\n2. It is interesting and reasonable to rethink class imbalance beyond just the sample number per class.\n3. The formulated semantic scale is new to me.\n\nWeaknesses:\n1. The designed DSB loss relies not only on the semantic scale but also the inter-class interference. Therefore, it is unclear how much performance gains are from the semantic scale and how much is from the inter-class interference. Since the main innovation is the semantic scale, it is better to show its empirical effectiveness.\n2. In Sec 4.1, this paper argues that \"all methods that rebalance the loss function or adjust the sampling rate based on the sample number can be improved with the semantic scale, since both are natural measures and they are not model-dependent.\" This argument is not clear enough. First of all, the evaluation of the semantic scale and the inter-class interference are based on model features, so they are model-dependent. Moreover, in my view, combining the proposed reweighting loss and existing class re-balancing is not that easy, since they are competing with each other in reversing imbalance to some degree. Therefore, the combination pipeline should be carefully designed and the hyper-parameters should be carefully tuned; otherwise, the model performance may not be improved and even become worse.\n3. There are many re-weighting long-tailed approaches that do not rely on the number of samples, like Uncertainty-based margin learning [A], LOCE [B] and Domain balancing [C]. Please discuss some of them if you have not done so and also compare the proposed method with them to show superiority.\n[A] Striking the right balance with uncertainty. In CVPR 2019.\n[B] Exploring classification equilibrium in long-tailed object detection. In ICCV 2021.\n[C] Domain balancing: Face recognition on long-tailed domains. In CVPR 2020.\n4. Can the proposed semantic scale be used to guide re-sampling for long-tailed data? Whether is it better guidance than the sample number per class for class-balanced re-sampling?\n5. The performance of BS [39] on ImageNet-LT is not that bad. Many recent studies have verified this. Please recheck your implementation in the experiment.\n6. In Section 5.2, can you order the semantic scale of different classes, and group them into three groups? I wonder how many accuracies of different groups are improved. Whether are the classes with small semantic scales improved more?", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work.", "empirical_novelty_and_significance": "3: The contributions are significant and somewhat new. Aspects of the contributions exist in prior work."}, "extra_text": {}}]

Author rebuttal:

Decision: Accept: poster