HOW SAMPLING AFFECTS TRAINING: AN EFFECTIVE SAMPLING THEORY STUDY FOR LONG-TAILED IMAGE CLASSIFICATION
Authors: Gong Zhang
Yongqiang Gao
Haijing Liu
Venue: Submitted to ICLR 2023 (2023)

Abstract: The long-tailed image classification problem has been very challenging for a longtime. Suffered from the unbalanced distribution of categories, many deep vision classification methods perform well in the head classes while poor in the tail ones. This paper proposes an effective sampling theory, attempting to provide a theoretical explanation for the decoupling representation and classifier for long-tailed image classification. To apply the above sampling theory in practice, a general jitter sampling strategy is proposed. Experiments show that variety of long-tailed distribution algorithms exhibit better performance based on the effective sampling theory. The code will be released soon later.

Under review as a conference paper at ICLR 2023
HOW SAMPLING AFFECTS TRAINING: AN EFFECTIVE SAMPLING THEORY STUDY FOR LONG-TAILED IMAGE CLASSIFICATION
Anonymous authors Paper under double-blind review
ABSTRACT
The long-tailed image classification problem has been very challenging for long time. Suffered from the unbalanced distribution of categories, many deep vision classification methods perform well in the head classes while poor in the tail ones. This paper proposes an effective sampling theory, attempting to provide a theoretical explanation for the decoupling representation and classifier for long- tailed image classification. To apply the above sampling theory in practice, a general jitter sampling strategy is proposed. Experiments show that variety of long- tailed distribution algorithms exhibit better performance based on the effective sampling theory. The code will be released soon later.
INTRODUCTION
The image classification problems are fundamental tasks in computer vision, and many methods based on deep learning have achieved gratifying results on artificially constructed datasets so far. However, due the large discrepancy between distributions for different classes, the classification model performs very well for head categories, but usually gives an inaccurate prediction for the tail ones at the same time. This phenomena dose not only occurs in image classification, but also in other common vision tasks such as semantic segmentation He et al. (2021); Wang et al.] (2020a). object detection Ouyang et al. (2016); Li et al. (2020) and so on.
Researches on long-tail classification problems mainly focus on the following research perspectives including loss function re-weighting Cao et al. (2019), training data re-sampling MMaajan et al.) (2018), and transfer learning strategies in embedding level Liu et al. (2020). The main idea solving the imbalanced classification problem is to enhance the training proportion for the tail categories so as to alleviate the overfitting for the head ones. Kang et al. (2019) points out the strong dependence between the representation learning for backbone network and classifier learning for the the last fully connected layer, and concludes that the optimal gradient for training the backbone network and classifier are obtained from the original sampling distribution and re-sampling distribution such class-balanced sampling respectively, from which the mainstream of two-stage optimization strategy is gradually accepted by more researchers. Xiang et al. (2020) further alleviates the strong dependence of the single-expert model with specific training distribution, leading to an improvement of classification accuracy both for head and tail categories.
Kang (2019; Zhou et al. (2020) mentions that the mainstream methods for long-tailed distribu- tion requires two stages learning. Sampling process need be conducted within the original distribution learn the first step stage for representation, without an ample theoretical explanation for this
phenomena however. Inspired by CCui et al. (2019), we realised that the growth between the actual effective samples and the actual number samples does not change synchronously stage, where the effective sample growth formula is given by Cui et al. (2019). Based the concept of effective sample, our expanded effective sampling theory is proposed. Here we give two important findings. The total number of effective samples is the primary factor affecting the training for loigetailed distribution, and the second one is the effective sample utilization. The improvement of accuracy on the long-tailed distribution can be achieved through the process of maximizing the total number of effective samples and balancing the effective samples utilization among categories.
the first training
The main contributions of this paper follows:

---

Under review as a conference paper at ICLR 2023
1. We build complete theory on effective sampling, which could be used for studying the properties of sampling with/without replacement, through which the optimal sampling methods are proposed.
2. A general jitter sampling strategy is proposed for the piratical application, and experiments on various public datasets have been carried out. The experimental results reach the competitive performance which further verify the core factor of our theory, that is, the total number of effective samples is the core factor affecting the first learning stage and the process of effective samples equalization among classes is beneficial for model training.
2 RELATED WORK
Re-sampling Redesigned sampling frequencies for different classes are used in re-sampling based strategies. Early ideas mainly focus on under-sampling the head classes and over-sampling the tail classes. [Drmmmond al. (2003) argues that over- over-samp is better than under-sampling because the latter process may loss important samples, while over-sampling the tail classes may lead to the over-fitting problem at the same time. Chawla et al. (2002); HHan et al. (2005); ); He et al. (2008) By introducing the generated new data for tail through interpolation, the above problem could be solved. However, the imprecise interpolation may also introduce new noises. The process of representation
learning and classifier learning should be decoupled with their suitable distributior et al. (2019); Zhou et al. (2020).
Re-weighting Re-weighting refers to assigning different weights to loss computation denoted by the corresponding classes The reciprocal of sample frequency adopted to correlate with weight in early studiesHHun et al. (2016); Wang et al. Re
by the number of effective samples of each class is utilized in Mahajan et al. (2018); Mikolov et al. (2013). LDAM Cao et al. (2019) adopted the loss determined by the classification decision boundary distance, where categories with larger magnitude are closer to the decision boundary. Meta-learning based methodJamal et al. (2020) also is used for better weights estimating Zhang et al. (021a). considers the difficulty and total number of the data to determine loss weights. In addition, some methods based on difficult Zahnn et al. (2021a) and logits adjustment& Menon et al. (2020) also belongs to re-weighting.
Transfer learning Transfer learning attempts to transfer knowledge from source domain to enhance performance on the target domain Zhang et al. 2021b). BBN Zhou et al. (2020) is trained on the origin distribution in the early steps. which transfer to tlasses-balancee distribution later for the optimization of classifier. LEAP|Liu al.) (2020) constructs a "feature cloud" of tail classes transferred from head ones features to better support the classification boundaries. LFME Xiang al. (2200) trains multi-expert models separately on multiple sub-datasets, and produce student model through knowledge distillation. RIDE Wang et al. (2020b) uses dynamic routing to control the number experts involved.
Research for long-tailed classification mainly focuses on the above aspects. In addition, there are some theoretical studies on training strategies for long-tailed distribution. Kang et al. (2019) and Zhou et al. (2020) show an empirical law of long-tail tail classification research, that is, the process of representation learning and classifier learning is uncoupled. Menon et al. (2000) points out using Adam-type optimizers may not be conducive the training for long-tailed datasets. Cui et al. (2019) introduces the concept of the effective number of samples because of the finding that the total number of non repeated samples actually participating in the training may not be as large expected.
3 EFFECTIVE SAMPLING THEORY
Inspired by the concept of the effective samples Cui et al. (2019), this paper proposes a hypothesis to explain the effective sampling in training processes. We believe that the total number effective samples is the primary core factor in the representation learning, and then the next one is the utilization of effective samples between categories. The performance of the representation learning can be improved by the increasing the total number of effective samples and equalizing the effective sample utilization.
2

---

Under review as a conference paper at ICLR 2023
3.1 CONCEPT DESCRIPTION
The image information redundancy occurs during model training when objects have similar features. As the number of instances of a certain category increases, the probability of redundant samples usually increases, which is due to the inconsistent difficulty of data collection from source. For example of the data collection on the cat category, the total samples of hairless cats Probably much smaller than any hairy type.In addition, multiple repeated sampling from the same source for different angles suffers lower generalization performance than the separate sampling from different source in the same category. Redundancy causes the asynchronous growth of category frequency and information content.
For the sample xı, 22 and the encoder Lencoder if the image substructure E and 82 E 22 and fencoderr (81) ) Sencoder(82)] = ||||| - =2ll < 6, then these samples are redundant. Then the updated conception of effective sampling is proposed, which refers to those sampling processes that do not generate new redundancy in the existing data sets. Specifically, for the specific structure o and category k, if one sampling is performed, and the category of this sample belongs to ks which producing no redundancy with a, then an effective sampling processes happens, during which the total number of effective samples of the lo category +1. The above sampling process is called effective sample sampling. The ratio of the number of valid samples to the total number of this category is defined as the effective sample proportion.
Based on the concept of effective sampling and effective sample proportion, the effective sample theory is established. The effective sample theory studies how the category sampling distribution affects the actual training efficiency. In this process, we define the two concepts of the total number of effective samples and the utilization rate of effective samples, and then give the quantitative analyze for those two concepts in different sampling methods.
Suppose there are N samples in dataset with m class labels. The number of each category is (mu, 102, .... )m The effective sample proportion of each category is (a1, 02, .... Um), and the actual sampling frequency is set to (U1, U2 ..., um).
3.1.1 SAMPLING WITH REPLACEMENT
Sampling with replacement means that the data of each category once sampled still has a certain probability to be sampled in next iteration.
Total number of effective samples
Let Ei,. be the expected number of effective samples of the category sampled by times,which which satisfies the following equation:
E = =4 max(a,n, - -)) (Fi,-1+ + 1)+ (1- 11 max(a,n; - Ein-1.0)
Fiin-
Simplify the above formula (see Appendix Effective Sampling Theory). p. and Ein ny * aj (1 2e )m). We note that total number of effective samples of the overall dataset after sampling times is Sn, and we have: S, = Σx= Σド=ıann(( (1- wf') where wj ny"
", When is large enough,
the analytical solution of in satisfies the following equation:
- Eit, (1 Aijn) * + 01
Aijn
Σi
Asgn*n
n4
This formula shows that the optimal sampling frequency is approximately equal to the class frequency ratio of the original distribution when the sampling times is large enough, that u, n.
Effective sample utilization
The effective sample proportion is defined as R.,. F... It describes the proportional of the total number of effective samples in the total number of samples of category i after sampling n times. On the condition of sampling with replacement, this proportional expression is simplified as follows:
a,n,
w!
Rin
0,77
3

---

Under review as a conference paper at ICLR 2023
Consider the ratio of the effective sampling proportions Qr, for any two classes and
Rín a, * n4 * (1 w * Ui
Qi.i.n
Rjn aj nj (1 w²)
when n large enough,Qu... =
Rzn ayenyowe" For sampling with replacement, the optimal sampling frequency needs to be approximately proportional to the product of the number of class and its effective sample proportion, which is: u, n, * a;
Ra.. d
3.1.2 SAMPLING WITHOUT REPLACEMENT
Sampling without replacement means that the data of each category is sampled completely according to the preset sampling frequency, which once sampled will not return to their original category set until the next epoch comes.
Total number of effective samples
On the condition of sampling without replacement, Ei,. satisfies the following equation:
Ein- (mmin((,,,n-- + a, a,,),,))) + (1-w;, U.
After Simplifying the above formula (see Appendix Effective Sampling Theory). Ein = n ag. and S,, Σラ= min(a, (nn ayny). Sort the tuple (a,, nis u;) with descending order by a, and we obtain the following sequence:
(a 1121 Uny ), (dagg Pang Uzz)s .... (arz.. Thrm Mer.
a When satisfies Σj= Ray In < Ej=1 Pay. S., to reach its maximum value,on which condition the sampling frequency satisfies the following equation
Th
Ej-1
Rzy
May
Uh Ur.+2 0, Ur.. 0
72
n
n
Obviously, Sn. just obtains the maximum value when and only when ΣØ=1 n1; VA N. On the condition of sampling without replacement, the growth rate the total number of effective samples is much greater than that on the condition of sampling with replacement. Theoretically, it is the optimal sampling strategy for increasing the total number of effective samples.
N, where
Effective sample utilization
The effective sample utilization R2 and the ratio of the effective sampling proportions Qi can be expressed as follows:
min(a,u,n, a,m)
R
unn
ats
in
azti u,7
otherwise
Ran
R; 0g Uy
Qi.
1; us x n4 *
R
nj aj
On the condition of sampling without replacement, balanced utilization of effective samples among classes needs the sampling frequency и, be proportional to he product of the number of class and its effective sample proportion.
3.1.3 SOLUTION
Firstly, we found that the primary core factor affecting the training under the long-tailed distributions is the total number of effective samples. By maximizing the total number of effective samples, the encoder can obtain those gradient generated from samples with fewer redundancy information, which benefits for increasing training efficiency for the first stage of representation learning. Secondly,
4

---

Under review as a conference paper at ICLR 2023
as the training progresses, the actual utilization rate of single effective sample becomes different, which lead to efficiency discrepancy of learning for different structures. The difference of those two key factors between Sampling methods will further affect the final classification accuracy.
In the previous studies, it was found that using the original sampling distribution for the first stage of training is more effective than class balanced sampling distributions KKnn et al. (2019), partly because the total number of effective samples reaches near maximum by simply setting the sampling frequency to be proportional to the sample frequency, which well supported by our theory. However, according to the formula of maximizing the total number of effective samples and the effective sample utilization between classes, the maximization of the total number of effective samples and the total balance of effective sample utilization between categories can never be achieved theoretically the same time, due to the existence of sample redundancy. reasonable trade-off is to ensure the total number of effective samples close to the maximization primarily and balancing the effective samples utilization between categories.
3.2 JITTER SAMPLING STRATEGY
The effective sampling theory reveals the contradiction between optimizing the total number of effec- tive samples and optimizing the utilization of effective samples between categories, and Estimating the accurate redundancy of real-world category samples directly can be also difficult. Fortunately, the effective sample theory suggests that the optimal sampling frequency is actually close to the original sampling distribution, which implies that the total number effective sample can be approached as long as the distance between sampling frequency and originally distribution is in a controlled range.and the deviation from the original distribution gives a possibility to balance the utilization of effective samples. Another reasonable assumption is that, for certain category, more sample frequency usually brings less effective sample proportion, which will be well explained in aappendix (4)
Based the above analysis, the jitter sampling strategy is proposed. We design sampling schedule in which the sampling frequency fluctuates around the original sampling distribution, exploring to maximizing the total number of effective samples and balancing the sample utilization through random walks. For the case of sampling with replacement, we build meta-dataloader, which contains multiple sub-dataloaders. During each iteration, it randomly selects one from meta-dataloader with preset probability. and samples a data-batch at sampling frequency that approximates the original distribution. For sampling without replacement, single dataloader is used to sample a data-batch with a preset sample frequency close to the original distribution. In this process, we dynamically adjust the actual sampling distribution by introducing a control factor related to training time.
Ug x f(n,r))r :
In the early stage of training, our strategy is relatively conservative which adopts sampling strategy almost same as the original distribution, and gradually explore from multiple sub-distributions as the training progresses. In the appendix, we prove that if the hyperparameters are properly chosen, jitter sampling strategy can perform better than the original strategy by trading off optimizing the above two key points.
3.2.1 SAMPLING WITH REPLACEMENT
On the condition of sampling with replacement, two different strategics are proposed. The first method mainly controls the change range of sampling frequency through temperature (jitter factor), where the selection probability for each sub-dataloaders are fixed. In the second method, we select the fixed original sampling distribution and the reverse sampling distribution for each sub-dataloaders, and complete the actual sampling process by dynamically adjusting the selection probabilities of the two.
(see appendix 1 for more details for the effectiveness proof process)
Method 1
5

---

Under review as a conference paper at ICLR 2023
Three sub- cab-datlloaders initialized with varying sampling frequencies follows:
dataloader : (u1, U2..... , um)
Um 1+6r) dataloaderg (u]+ar, 112
1+8t
dataloaderg- : (u) [u]-ét uz 1
where Mi 74
where of is jitter factor, which is updated at each epoch. The rule to update dt is:
{
epoch
őt = random(0, 1) max
(ppoch epoch
.7
Each sampling selects one of the sub-dataloaders for sampling with a preset probability (Pi, P2. Pal. (seelape apeendixi for more details for the effectiveness proof process) Method 2
Two sub-dataloaders are initialized accordance with:
dataloader; (u1, U2...., Um)
dataloaderg (Mi U42" ..... um!)
where uę ni
where öt is the jitter factor, which is updated at the arrival of each epoch with the following rule:
{
epoch epoch
81 = -max
.7
Each sampling selects one of the dataloaders for sampling with preset probability [1 öt, öt).
The first jitter sampling method is more general, and we will demonstrate its effectiveness in detail in the appendix. The second dithering method cannot theoretically guarantee to maximize the total number valid samples, instances has greater redundancy, then the second jittering method can be considered. For sampling without replacement, we only use a dataloader like:
it is assumed that in the real data set, the category with more sample
dataloaderg (u, 222
++t 1+
Um
1+81), OX
where öt varies as follows:
8t candom(-0..) -0.5,0.5) a max { (Leooc
epoch
epoch
For the actual sampling, queue is maintained separately for each category of samples, and the samples of that category are initially filled in random order. uo N samples are drawn from class for each epoch, and when the classes sample queue is emptied, a new round of filling is performed again random order. When a new epoch arrives, the dataloader prioritizes the samples still in the queue from the previous round until all samples have been drawn. The purpose of doing this is to avoid the situation where there is put-back sampling between each epoch.
In theory, sampling without replacement by dithering will lose valid samples, but since the probability of equalizing the utilization of valid samples will increased at the same time, we will prove it in detail in the appendix.
4
EXPERIMENTS
4.1 EXPERIMENTAL SETUP
Dataset The long-tailed benchmark datasets commonly used are selected: CIFARIO-LT, CIFARI00-
LT Cao et al. (2019) and ImageNet-LT Liu et al. (2019) which sampled from CIFAR1O Krizhevsky
et al. (2099) CIFAR 100 Krizhessky et al. (2009) and ImageNe[Deng et al] (2009). respectively. The imbalance ratio of CIFARTO-LT: CIFARTO0 T00-IT and ImageNet-IT are 100. 1e0000026.

---

Under review as a conference paper at ICLR 2023
Methods
topl-acel %)
Lin
FOLTR
Deng ef 2009
43,4
Methods
topl-acc( %
+LDAM-DKW C20 et
2019
44.4
1BBN
Zhoü
20,00
42.6
*Focal loss *OLTR +LDAM-DRW
68.6
fr-norm Cao et 2019
45.4
78.7
feRT Cao
2019
42.6
78.4
*LFME Xning al 6020
43,8
+BBN fr-norm tcRT Logit adjustment
42.6
Logit adjustmicnt
Menon Ca.l.
2020
43.9
79.6
#De-confound
Tang
2020
47.3
79.2
D00-confound-TDee Tang e 21
2020
48.3
81
x KIDDEdAxpetts+reduss
Waiee
2020
49.5
+De-confound
72.5
# RIDE(4experts)
Wang cl 20206
50
iDe-confound-DDE
80.4
[LC(eexperts)) Lreral
49.8
RIDE((experts)
81.7
(TI.C(4experts)
80.4
group. (omm))lnggtrannouurs)))))
# J RIDE(4experts reduce)(urs))
49,5
#J RIDE(eexpets: roduceXours)
81.2
#J+ RIDE (Jexperts)ourr))
50.6
#J + RIDE(dexperts))onns))
82
Table evaluation results cifar100-1
Table 2: evaluation results cifar
Evaluation Metries In long-tailed learning, the overall performance on all classes and the per- formance for head, middle and tail are usually reported. The overall performance all classes is reported in this paper, and we average the class-specific accuracy and use the averaged accuracy as the metric.
The experiments are mainly conducted on two aspects. First, we experiment on major long-tailed classification benchmarks in the evaluation results, which mainly verifies the actual effectiveness of the proposed jitter sampling strategy. Taking the fairness of experiment, we follow the RIDE Wang al. (2020b) ensemble learning framework. Second, we experiment the ablation studies on the effectiveness of each component.
4.2 EVALUATION RESULTS
Experiments for single-experts model
cirfa100-H and cirfal (--It are used as the experimental datasets, where ResNet-32 He et al. (2016) selected as the backbone. In order to improve the classification accuracy without increasing the amount of calculation greatly, to expanse features before the original full connection layer of classifier through L2 normlization module by pre-designed groups. Batchsize is set to 256, and SGD is used as optimizer. The total training epochs are 500, and the learning rate is initialized to 0.5, with a learning rate decays of 0.01 350 epoch and 450 epoch, respectively. The warmup epoch set 5. The jitter strategy setting adopts the second method on the condition of sampling with replacement. In the first stage (0-350), 1.5, and the second stage(350-500), we directly set ôt 0.5. The cross
the a special designed module, which is called group L2norm,
we set 1 and entropy loss is used as our loss function.
The module design motivation of group L2norm increase the richness of features, and ensure output features with a certain controllable norm in different groups, so as to avoid the phenomenon that neural network reduces the loss by simply increasing the data norm, while the classification boundaries may not be well optimized. The disadvantage is that longer training is required when adding the norm module to the existing network module.
Table shows that our proposed method (I-sampling+grooup 1.2nom=+longtrain) surpasses most current methods (slightly lower than De-confound-TDD TDE)) with single backbone Resnet32, with an acceptable computational complexity increase.
Experiments for multi-experts model
Using the jitter sampling strategy combined with RIDE, the current sota ensemble learning methods are compared cifar100-1 and cifar this group of experiments, the jitter sampling strategy on the condition of sampling replacement is adopted. The experimental settings follows: batchsize
"4 denotes results copied from their paper, respectively.
1 denotes the results reported by their public code of sampling with replacement.
2 denotes the results reported by multi-experts
7

---

Under review as a conference paper at ICLR 2023
set to 128, training epochs is set to 200, the learning rate is initially set to 0.1 with the decay rate of 0.01 at 160 epoch and 180 epoch, respectively. The warmup epoch is set to The jitter setting adopts the sampling with replacement one. In the first stage (0- (0-160), and select cross entropy as the loss function. In the second stage, LDAA loss is adopted. Here we delete the feature enhancement method of group 1.2norm, because the 1.2norm operation plays role of repetition with NormFC module of the classifier in the original RIDE.
set 0.05
22 0.01,
be seen from Table | that the cifar00-It dataset, the single-model results better than most of the experimental results (slightly lower than De-confound-TDES and the multi-model results are better than the existing results (among which. J-sampling (oIDS)-RID(4 is 0.6 points higher than RIDE(4experts) It can be seen from Table that on the cifal0-It dataset, J-sampling (ours) + RIDE (4experts) outperforms all existing algorithms.
4.3 ABLATION STUDIES
In this subsection, we experiment the ablation studies on the effectiveness of each component, which including replacement strategy, Jitter strategy and the training time. We further discuss two key factors that affecting the representation learning: the total number of effective samples and the effective sample utilization. The basic experimental setting keeps the same experiments for multi-experts model.
Dataset
Expert
Reduce
with(%)
without(Se) %
CIFAR 100
47.8
49
CIFARION 100
48.4
49.5
Table 3: with / without replacement
Comparison of the replacement strategy When the sampling rate proportional to the number of class samples, the total number of effective samples without replacement is theoretically greater than the total number of effective samples with replacement. As shown in Table it can be seen that the actual accuracy of sampling without replacement is significantly higher than sampling with replacement.
Dataset
Expert
Reduce
with(
without (6)
cifa10
4
0
81.7
82
cifa100
3
1
47.8
48.7
cifal00
3
0
49.7
49.8
cifal00
4
1
48.4
49.5
cifa100
4
O
50
50.6
ImageNet
3
1
54
54.1
ImageNet
3
0
54.6
54.5
ImageNet
4
1
54.6
54.9
ImageNet
4
0
55
55.2
Table 4: with without jitter
Comparison of the Jitter strategy We compare the jitter with and without replacement sampling on cirfa100 and imagent-lt, respectively. We have proved the effect of jitter appendix theory, which is also validated from the experimental, that adding jitter within sampling frequency actually improve the accuracy with a certain probability. For non-replacement sampling, although the total number of effective samples theoretically decreases slightly, at the same time, the utilization of effective samples is also balanced. One possible reason for why there is no obvious accuracy improvement
sirfa100-It.i that the image size of imagenet-lt is larger than the cifar series dataset, so the redundancy between images is not high (the foreground only occupies part of the image, while the background difference between instances is obvious).
mmagenet-lt.compare
8

---

Under review as a conference paper at ICLR 2023
with replacement jätter long train
without replacement + no jitter
Dataset
Expert
Reduce
short train
short train
long train
cifa100
48.7
50
49
cifal00
49.5
50.4
49.5
cifal00
4
0
50.6
51.5
ImageNet
55.2
54
55.2
ImageNet
0
55.2
55
ImageNet
4
1
55.8
54.6
56
Table 5: short train
Comparison of the training time The training epochs is positively correlated with the total number of effective samples. When training epochs normal, is obvious that the total number effective Jitter strategy
samples the condition of sampling with replacement is lower than the without helps increasing the total number of effective samples and balance the effective sample utilization between categories. For longer training, the effective sample has been saturated, leading to limited accuracy improvement.
CONCLUSION
We have established effective sampling theory explain the sampling efficiency gap in differ- ent sampling methods and jitter sampling strategy is developed to improve the actual training effect,based which our proposed methods perform well many long-tailed datasets. If we find a way to eliminate information redundancy precisely, our theory may be further optimized. We will explore in the next experiment.
REFERENCES
Kaidi Cao, Colin Wei, Adrien Gaidon, Nikos Arechiga, and Tengyu Ma. Learning imbalanced datasets with aabel-distribution-awaa margin loss. Advances in neural information processing systems, 32, 2019.
Nitesh V Chawla, Kevin W Bowyer, Lawrence Hall, and Philip Kegelmeyer. Smote: synthetic minority over-sampling technique. Journal of artificial intelligence research, 16:321-357, 2002.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song. and Serge Belongie. Class-balancee balanced loss based on effective number of samples. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9268-9277, 2019.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large- hierarchical image database. In 2009 IEEE conference on computer vision and pattern recognition, pp.
leee, 2009.
Chris Drummond, Robert C Holte, al. C4. 5, class imbalance, and cost sensitivity: why under- sampling beats over-sampling. In Workshop on learming from imbalanced datasets II, volume 11, pp. 1-8. Citeseer, 2003.
Hui Han, Wen- Yuan Wang, and Bing-Huan Mao. Borderline-smote: a new over-sampling method in imbalanced data sets learning. In International conference on intelligent computing, pp. 878-887, Springer, 2005.
Haibo He, Yang Bai, Edwardo Garcia, and Shutao Li. Adasyn: Adaptive synthetic sampling approach for imbalanced learning- In 2008 IEEE international joint conference neural networks (IEEE world congress on computational intelligence), pp. 1322-1328. IEEE, 2008
Kaiming He. Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp. 770-778, 2016.

---

Under review as a conference paper at ICLR 2023
Ruifei He, Jihan Yang, and Xiaojuan Qi. Re-distributing biased pseudo labels for semi- semantic segmentation: A baseline investigation. Proceedings of the IEEE/CVF International Conference Computer Vision, pp. 6930 000--69002220 2021.
Chen Huang, Yining Li, Chen Change Loy, and Xiaoou Tang. Learning deep representation for imbalanced classification. In Proceedings of the IEEE conference computer vision and pattern recognition, pp. 5375-5384, 2016.
Muhammad Abdullah Jamal, Matthew Brown, Ming-Hsure Yang. Liqiang Wang, and Boqing Gong. Rethinking
methods for long-tailed visual recognition from a domain adaptation perspective. In Proceedings of the IEEF/CVE Conference on Computer Vision and Pattern Recognition, pp. 7610-7619, 2020.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint 191.09/91 2019.
Alex Krizhevsky, Geoffrey Hinton, et al. Learning multiple layers of features from tiny images. 2009.
Bolian Li, Zongbo Han, Haining Li, Huazhu Fu, and Changqing Zhang. Trustworthy long-tailed classification. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 9979, 2022.
Yu Li, Tao Wang. Bingyi Kang. Sheng Tang, Chunfeng Wang. Jintao Li, and Jiashi Feng. Overcoming classifier imbalance for long-tail tail object detection with balanced group softmax. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991-11000 2020.
Tsung- Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp. 2980-2988, 2017.
Jialun Liu, Yifan Sun, Chuchu Han, Zhaopeng Dou, and Wenhui Li. Deep representation learning on long-tailed data: A learnable embedding augmentation perspective. In Proceedings of the IEFF/CVF conference on computer vision and pattern recognition, pp. 2970-2979. 2020.
Ziwei Liu, Zhongqi Miao, Xiaohang Zhan, Jiayun Wang, Boqing Gong, and Stella X Yu. Large- scale laigetailed recognition in an open world. In Proceedings of the JEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 2537-2546 2019.
Dhruv Mahajan, Ross Girshick, Vignesh Ramanathan, Kaiming He, Manohar Paluri, Yixuan Li, Ashwin Bharambe, and Laurens Van Der Maaten. Exploring the limits of weakly supervised pretraining. In Proceedings of the European conference on computer vision (ECCV), pp. 181-196, 2018.
Aditya Krishna Menon, Sadeep Jayasumana, Ankit Singh Rawat, Himanshu Jain, Andreas Veit, and Sanjiv Kumar. Long-tail learning via logit adjustment. arXiv preprint arXiv:2007.07314, 2020.
Tomas Mikolov, Ilya Sutskever, Kai Chen, Greg S Corrado, and Jeff Dean. Distributed representations of words and phrases and their compositionality. Advances in neural information processing systems, 26, 2013.
Wanli Ouyang, Xiaogang Wang, Cong Zhang, and Xiaokang Yang. Factors in finetuning deep model for object detection with long-tail distribution. In Proceedings of the IEEE conference on computer vision and patlern recognition, pp. 864-873, 2016.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long- classification by keeping the good and removing the bad momentum causal effect. Advances in Neural Information Processing Systems, 33:1513-1524, 2020.
Tao Wang, Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng. The devil is in classification: A simple framework for tail instance segmentation. In European conference on computer vision, pp. 728-744. Springer, 2020a.
10

---

Under review as a conference paper at ICLR 2023
Xudong Wang, Long Lian, Zhongqi Miao, Ziwei Liu, and Stella X Yu. Long-tailed recognition by routing diverse distribution aware experts. arXiv preprint arXiv:2010.01809, 2020b.
Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Learning to model the tail. Advances in neural information processing systems, 30, 2017.
Liuyu Xiang, Guiguang Ding, and Jungong Han. Learning from multiple experts: Self-paced knowledge distillation for long-tailed classification. In European Conference on Computer Vision, pp. 247-263. Springer, 2020.
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2361-2370, 2021a.
Yifan Zhang, Bingyi Kang, Bryan Hooi, Shuicheng Yan, and Jiashi Feng. Deep long-tailed learning: A survey. arXiv preprint arXiv:2110.04596, 2021b.
Boyan Zhou, Quan Cui, Xiu-Shen Wei, and Zhao-Min Chen. Bbn: Bilateral-brance network with cumulative learning for long-tailed visual recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 9719-97233 2020.
A APPENDIX
A.I APPENDIX
Jittering method Sampling with replacement is effective.
Proof 1: In sampling with replacement, the total number of effective samples obtained using jittering method is greater than the that of the original sampling with replacement, and the effective sample utilization is equalized between classes.
Let the expectation of the total number of effective samples sampled by the jittering method one be Jn whose values are:
Jn E(S,)
m
(1-(1-55)
3
(1)
Σ
Pi
ny
1-
n2
PIG(1) P2G 8t) + p3G(1 8t)
Where (6) is the sampling rate of the i-th dataloader for the j-th class. Let uy (x) is a function that calculates the sampling rate of the j-th class in dataloaders. The sampling rate of dataloader
1 is uf (1). The sampling rate of dataloader is (2) = u1 + 8t). The sampling rate of dataloader is u u;(1 (1) uz(x) = Σ
m k= T2
And G(x:) is function with:
m
(z(x))
G(x) = Σ
ajni
1-
j=l
= Const
Σ
a,ns
1
uz((x)
ng
==1
11

---

Under review as a conference paper at ICLR 2023
Consider the derivative of G with respect
aG
116
(1 1- 11(x)
n-1
@uz(z)
ajni
71
dx
Nj
7ij
dr
==1
a (1 Ug uy(x)
n-1
n
In n > m
k=1 ni
n
-k=1 (ng Im
m
A= nx 2
n
nj
/=1
When 1:
aG
n-1
ny Iu Σk= 12k - n1 =
(ng In ns)
(
= n
N
C m²)
2=1
==1
If
8G 0t
there is:
m
aj4i Inn
Thk
(ne In
This usually doesn't 't hold. is worth mentioning that when d1 02 redundancy of each class of the dataset the same, the above equation holds. Therefor, normally:
am that is, when the
aG
OG
0 or
0
dx a==1
Oz l==1
Then there must exist 5t such that G(1 + 81) > G(1) or G(1 8t) > G(1). So, a probability combination (pI, P2, Pa) must exist that let G(L) + p2G öt) psG(1 8t) G(I..
In summary, we prove that there must exist parameters ôt and (21, P3) that allow the the total number of effective samples obtained using jittering method one of sampling with replacement is greater than the total number of effective samples.
When is sufficiently large, the effective sample utilization between classes is the same as the formula when there is sampling without replacement. The sample utilization between classes is balanced at this time, as we will prove Appendix 2.
Proof 2: In sampling with replacement, the utilization of the effective samples between classes obtained by jittering method is more balanced than that of the original sampling with replacement.
The effective sample utilization for the ith category is Ri In method 2 of sampling with replacement, when n is large enough, it is:
a174(1 w
R = z
+ (1 x)
U,n
0g79
0,74
RS x
+ (1 x)
unn
U.M
Where w; =
Σ == 2.
The derivative of R.,. is:
ÖR
a,M,
a;M;
dr
A more balanced effective sample utilization corresponds to a distribution of category effective
sample utilization P = Ex Ny. ..... Σx My
Ri
Rm. and uniform distribution Q
1
with
m
12

---

Under review as a conference paper at ICLR 2023
less KL divergence.
Qi P:
(L(QIIP)
Σ 1=1
Qi log
1
-log m
log Ri + log Σ Ri,a
Consider the derivative of the KL divergence.
RR.
0R
ØKL dz
1
8x
m
Rin
m
i=
Ri,n
When x = 1, Ri
0:R1
0KL
4:32
0x
771
1
jjm1
aj bj
Σ m
2j=1
a-jR1
uyn
Σ
Uk
m
f=1
From Appendix 4, the proportion of effective samples is low for categories with large sample size, which means that the lager u, is, the smaller 0; is. Therefore,
1
0g
0
j=1 ag
That means
ØKI.
>0
Or
===
Therefore, there exists E (0, 1) such that KL(1 81) < KL.(). other words, the dithering method 2 can make the effective sample utilization more balanced.
A.2 APPENDIX 2
Jittering method Sampling without replacement is effective.
Proof:
In sampling without replacement, the effective sample utilization of jittering method would be more balanced than without jittering.
The effective sample utilization for the ith class is:
0g
R
U1 (x(x)
A more balanced effective sample utilization corresponds to a distribution of category effective
sample utilization P = Ex Ny. ..... Σx My
Ri
Rm. and uniform distribution Q with
m
13

---

Under review as a conference paper at ICLR 2023
less KL divergence.
m
Qi Qi log
(L(Q
Pi
log
m
m.:
log m
Σ 1=1
log P:
Consider the sampling rate function (.(x).
(x(x))
2k=1 T
For = + 5t and öt > 0, the sampling rate increases for the head classes and decreases for the tail classes. We have n u,( and effective sample utilization of tail classes is Rgn
ni tail classes. So, effective sample utilization of tail classes is Ra,n a;
a.2,
Might as well let m-th class has the largest number of samples. There exists öt quite small such that 2m. Ant the utilization of the others a, So the KL divergence is:
Rm,, Mem (Z)
KL(QII)) log m
1
m
log
aj
+ log
am Tl Mays (2)
m
m- k=
0k +
fi Mm (2)
778
km
Ok +
Who (2)
m
m 1
O am
aym 1ym
log m
log ag + log
m log
Cle +
(x)m
Mym (zr)m
The derivative of KL. divergence with respect to
ØKL
1
(x))
am T0m 22 (r)n
Oum (2x)
Plm
Oum (1) Ox
Öx
m
Am
Ox
Σ m
k=
+
am, Pam.
(2))
2 (x))n
Um (x)
m
am
Fum (xr)
m
Ovn Peya
k=1 ag
um (x))
22 (x)m
Or.
Where
Oum(x)
is:
8x
Oum (x)
n mz
Dk Th - -n Σ (nt In n1.)
Fx
(Em m
When 1, we have:
ØKL
m71
am
Ny In 1mm
Σ Ek=1 1i
nm
Σte (ng Inne)
1=1
am
1=
70m
Since is the largest and am is the smallest from Appendix 4, we have
0KL
<
Ox
So when öt > 0 and closer to 0, the jittering of (,(1 + 5t) causes KL(Q))P) to drop, that is, the effective sample utilization between classes is more balanced.
Similarly, it can be shown that when öt and closer to 0, the jittering of u, - 6t) causes the effective sample utilization between classes be more balanced.
14

---

Under review as a conference paper at ICLR 2023
A.3
APPENDIX 3
A.3.1
SAMPLING WITH REPLACEMENT
Total number of efficient samples
Define the effective number of samples obtained for the i-th class after sampling as Ei,,, whose recursive formula is
in = max(a; - E,,n 1. - (1-14- max(a,n, Eim=1:10) Ei,n-
74
When the total number of effective samples reaches a,ml;, no new effective samples are added. Therefore, to simplify the discussion, we consider the case where the upper bound has not yet been reached. Simplifying the above equation yields:
Ein a,u, + (1
Ui n4
Ei
Let WN = 1 1i, it's easy to know: n;"
Ein
a,U4
Ei
w
wh
Ein
0g44
+ Eg
w."
jml
W
Ein
0:404
a;
w.
We note that total number of effective samples of the overall dataset after sampling n times is Sn, and we have:
S Σ (1
1=1
Maximize Sn
max
Sn
s.t.
Uj =1
i=1
M4 >> 0
We introduce Lagrange multipliers and try solve for the conditions satisfied when S, reaches its extreme value:
L(u Umn A) Sn + A
1 Σ
1=1
Calculate its derivative:
ØL
a,n (1
n-1
X
On;
n2;
Let aL 8u,
O, get:
aL
OL
Zug
1/a
a
15

---

Under review as a conference paper at ICLR 2023
When is large enough, the analytic solution of satisfies the following equation.
1 1 Ei 125 Ai...))
di 1 1 Eigs my Ai...m// n
04
Aij.. a
This equation shows that in sampling with replacement the optimal sampling frequency is approxi- mately equal to the category frequency ratio of the original distribution when the number of samples is large enough, which also implies that theoretically obtain close the upper limit of the total number of valid samples by using sampling ratio that approximates the original distribution. The optimal sampling rate u, satisfies:
Effective sample utilization
The effective sample proportion defined as follows.
Ein
Rí,
u,2
In sampling with replacement, this expression simplified as follows:
a, (1 - w(f
R
where w;
u,7
m2
Consider the ratio of the effective sampling proportions of any two classes Qij-
Rin
Qij
Rja
acnyy (1 w.²)
a,n,u:(1 -w0
w
is not difficult find that when is sufficiently large, Qi, satisfies:
a,n, Uj
Qi,
ag
Therefore, in sampling with replacement, when the sampling frequency approximates the effective number of the class is proportional, the effective sample utilization is balanced. That is:
C 0,77
A.3.2 SAMPLING WITHOUT REPLACEMENT
Total number of effective samples
In sampling without replacement, E, satisfies the following relation.
F (min((E + a; 1,):))) (1 ,,, Ei,n
Simplify get:
Ein a;u;n
Therefore S, satisfies:
m S, Σ
min(a,u,m,a,m.)
16

---

Under review as a conference paper at ICLR 2023
May wish to consider that d1 S 02 ... am. For any & E [1, when satisfies:
k+1
Σ
n4 S n
n4
The condition for S, to reach its extreme value is
k -1= ny
T2k
01
0k++ = 0, Um
0
n
is easy know that S, just obtains the maximum value of the above equation when and only when N, when it only needs to satisfy:
m 2i=1
74
N
Effective sample utilization
Define after samples the effective sample utilization Ran as follows.
min(a,u,n, a,n))
Rin
14,72
ags
if <
otherwise
When n is large enough, for any i, j. considering the condition of 1, there are:
Rin
Qi.i.
Rjn
0,74 Uj
aj 04
For sampling without replacement, the condition for achieving a balanced utilization of effective samples among classes that the sampling frequency must be proportional to the number effective samples in class.
Us X a;7,
A.4 APPENDIX 4:
Proof:
When the number of classes is sufficiently large, the classes redundancy and the number of classes are negatively correlated.
We assume that class i obeys priori Gaussian distribution, and the actual data of class i is actually obtained from that Gaussian distribution.
Each point the numerical axis represents the sample actually sampled, and the probability density of the location it was actually sampled is
(- (t (t-m))²² o²
f(()
ov2T exp
We define two samples t,, ty as redundant if their sample positions It, tyl < 5. Let the sampling position to, then the probability that the sampling position greater than position to is P P(t ln)-
1 to
P(t > (0) =1 -1
f(t)dt
The expectation of the number the sample location is greater than to is NP for N independent acquisitions times.
17

---

Under review as a conference paper at ICLR 2023
Consider the case when N P 1, whose physical meaning is that after collecting N times expects only once its sampling position is larger than to. to can be considered as the average upper bound of the sample position of N times sampling. Considering symmetry, the lower bound of the sampling position is 2/1; tp. Then the upper level of the effective sample proportion d, satisfies:
2to 2p4
oN
It's easy to know:
lim a. (1(1 u,) = +00
8
Although rigorous derivation of ag cannot be given, show that the upper bound a4 dg decreases monotonically after sampling a certain range and ã, tends to O when the number of samples tends infinity. Since then, we have completed the derivation of the negative correlation between the category redundancy and the number of class samples.
The above is based the assumption that the dimension sample is while the actual datatype we deal with is much more complicated like image-type. In the real case, we think that the an image sample can represented as potential variable X; which can be generated by self encoding tht is:
Sencoder (1: = [Xa, X12, X [is.
Xis] = Xzi ddecoder (Xi) -
In the study of VAE, X,is defined as the potential variable obeying a specific Gaussian distribution:
X; N(u;, o²)
and then the a, of T be expressed follows:
d S ai 0gi 0gi... asi
donates the effetive sample proposition of and when X, maintains statistical independence with each other(Xy)). equality can be established.
18


Reviews:
[{"review_id": "uRkIzxdJNs", "reviewer": "Reviewer_xj1Z", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "The paper proposes an “effective sampling” theory which provides a theoretical explanation for the two-stage training scheme utilized in long-tail learning. The paper claims that both the total number of effective samples and the effective sample utilization affect training and performance for long-tail datasets. A general jitter sampling strategy is proposed based on the theory. Experiments are shown to verify the approach by comparing it to SOTA long-tail approach on common long-tail benchmarks. ", "summary_of_the_review": "Leaning to reject the paper for lack of empirical support, missing important baselines, and not being applicable for large-size datasets.", "strength_and_weaknesses": "Strengths:\n- The paper provides a theoretical explanation for the decoupling representation and classifier approach proposed by Kang et al [1].\n- A novel sampling strategy is proposed.\n- Experiments are conducted on long-tail learning benchmarks to show the effectiveness of the sampling strategy.\n\nWeaknesses:\n- The provided results do not show the effectiveness of the proposed method.\n  \t(1) Results for group_norm + longtrain (without jitter strategy) is missing. While \n                 “J+group_norm + longtrain” achieves 47.1 on cifar-100, it does not show the \n                 effectiveness of the jitter strategy.\n\t(2) In addition, the accuracy for both “J+RIDE [0] (4 experts+reduce)” and “RIDE [2] \n                 (4 experts+reduce)” is 49.5 (no improvement is shown).\n- Many, Med, and Few shot accuracies are missing. They are important to understand the effectiveness of the proposed approach compared to other methods (sampling and non-sampling methods)..\n- The paper proposes a jitter sampling strategy, based on effective theory, which aims to select effective samples to improve training. \n\t(1) There are no comparisons to other sampling approaches like [3] or [4]. \n\t(2) Active learning approaches aim for the same task: Select samples that are most \n                 effective for training. Comparisons to such methods (e.g [5]) would \n                 be sensible.\n- The performance improvement of the proposed approach compared to “RIDE [2]”  is very small.  +0.6% on cifar100, +0.3 on cifar10. Although the improvement can be due to the new sampling approach, it can also be due to variance. Reporting results variance for the proposed method and baselines is necessary.\n- The approach is only effective on small image-size datasets. The paper states that no obvious accuracy can be seen on Imagenet-lt dataset due to large image size. No empirical results on Places-LT and iNatrualist.\n\n\n[1] Decoupling Representation and Classifier for Long-Tailed Recognition, Kang et al.\n[2] Long-tailed Recognition by Routing Diverse Distribution-Aware Experts, Wang et al.\n[3] The Effects of Data Sampling with Deep Learning and Highly Imbalanced Big Data, \n[4] Not All Samples Are Created Equal: Deep Learning with Importance Sampling, Katharopoulos et al.\n[5] Active Learning for Imbalanced Datasets, Aggrawal et al.\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "rkhFLiP2Zt", "reviewer": "Reviewer_vZeb", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 3, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper studies long-tailed image classification problem. The authors identify two key factors that affect the performance of long-tailed image classification: (1) the total number of effective samples  and (2) the effective sample utilization. Based on this finding, the authors proposes an effective sampling theory for theoretical explanation and propose a general jitter sampling strategy for practical implementation. \n", "summary_of_the_review": "In general, I found the efficient sampling idea interesting. However, the motivation for this efficient sampling is not strong enough, which requires either statical or theoretical evidence.\n\nThe Paper is hard to read. There are many types and the paper's organization in Sections 3 and 4 might also need to be improved.\n\nThe method section is not very informative, and details including how effective samples are selected are not carefully introduced.\n\n", "strength_and_weaknesses": "Cons:\n\n1. This paper is based on the observation that \"data redundancy hurts long-tailed image classification performance\" in Section 3. Is there any statistical or theoretical evidence on this point? This point needs to be better justified with evidence, otherwise, the proposed methods will be vacuous.\n\n2. The presentation in Section 3 needs to be improved. The first 3 paragraphs in Section 3.1 is hard to follow and the definitions are mixed together. Maybe consider formally defining \"effective sampling\", \"effective samples\", and \"effective sample proportion\". Besides, how do you select substructures s1 s2 from image x1 x2? How are the effective samplings selected from the full dataset? How a1 ... am are computed? Many details are missing when reading this section.\n\n3. Could you please explain why \"the maximization of the total number of effective samples and the total balance of effective sample utilization between categories can never be achieved theoretically at the same time, due to the existence of sample redundancy.\" above Section 3.2?\n\n4. The authors need to highlight the difference between the effective sampling method with existing re-sampling and re-weighting methods.\n\n5. Experiment section is also hard to follow. The authors are suggested to briefly introduce the experiment setup, highlight the key results of the experiment, then provide a discussion and take-home messages for readers. ", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "Us71AjZ-ab", "reviewer": "Reviewer_y4qK", "summary": "", "questions": "", "limitations": "", "rating": 1, "confidence": 5, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "In this paper, the authors study the problem of long-tailed visual recognition. For this end, they propose a new sampling strategy from the perspective of \"effective samples\" proposed earlier by Cui et al. (2019).", "summary_of_the_review": "Severe issues and limitations regarding clarity, quality, novelty and reproducibility.", "strength_and_weaknesses": "Strengths:\n\n- Effective number of samples is a promising idea to explore for sampling.\n\nWeaknesses:\n\n1. The paper is extremely difficult to read, making it infeasible to comprehend its contributions, its line of reasoning, its methodology, its overlaps / differences to prior work etc.\n\n1.1 The paper has too many grammatical errors, some which are listed below for the first page. I gave up providing recommendations for the rest of the paper. I strongly suggest the authors to submit only polished, proof-read papers to venues, especially top ones such as ICLR. In its current state, the papers feels like it has been rushed and it might be containing different types of errors, not only grammatical ones.\n\n1.2. Many unclear parts or statements, e.g. (only for the first one/two pages): \n\n- \"inconsistent difficulty of data collection from source\" => It is not clear what's meant by inconsistent difficulty.\n- \"For example of the data collection on the cat category, the total samples of hairless cats Probably much smaller than any hairy type\" => This is Probably not a convincing statement.\n- \"multiple repeated sampling from the same source for different angles\" => What is an angle here?\n\n- It is not clear how the formulations in Section 3.1.1 are derived.\n\n- Table 1, 2: It is not clear what the imbalance ratios are.\n\n2. I find it too strong to call an approach a theory. \n\n3. Section 2 is missing approaches using regularization or normalization as a form of regularization, e.g. \"Long-Tailed Recognition via Weight Balancing\", CVPR2022. The paper also missed earlier work focusing on the design of sampling approaches: \"Assessing the Effect of Training Sampling Design on the Performance of Machine Learning Classifiers for Land Cover Mapping Using Multi-Temporal Remote Sensing Data and Google Earth Engine\", 2021.\n\n4. Another important limitation is about evaluation. The paper provides a mixed comparison in Tables 1 & 2, without discussing explicitly the contribution over existing sampling approaches. The paper should dedicate a separate section & analysis for a comparison with existing sampling approaches.\n\n5. The paper provides a SOTA comparison only on CIFAR10 & CIFAR100. Results on more challenging datasets such as ImageNet-LT and iNaturalist are necessary to see how the method fairs in challenging settings.\n\nMinor comments:\n\n- \"This phenomena dose not only occurs in image classification\" => \"This phenomena dose not only occur in image classification\".\n- \"semantic segmentation He et al. (2021); Wang et al. (2020a)\" => I think the correct cite command is \\citep. The correct form should have been: \"semantic segmentation (He et al., 2021; Wang et al., 2020a)\". Similar problems exist in the rest of the paper.\n- \"Kang et al. (2019); Zhou et al. (2020) mentions\" => \"Kang et al. (2019); Zhou et al. (2020) mention\".\n- \"two stages learning\" => \"two-stage learning\".\n- \"Sampling process need be conducted\" => \"Sampling process needs be conducted\".\n- \"*\" means multiplication in a programming language. You should use \\cdot or \\times.\n- Please follow the following guide for writing equations: http://www.ai.mit.edu/courses/6.899/papers/mermin.pdf\n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "2: Several of the paper’s claims are incorrect or not well-supported.", "technical_novelty_and_significance": "2: The contributions are only marginally significant or novel.", "empirical_novelty_and_significance": "2: The contributions are only marginally significant or novel."}, "extra_text": {}}, {"review_id": "JkFPOtkT7Mu", "reviewer": "Reviewer_3mbq", "summary": "", "questions": "", "limitations": "", "rating": 3, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "main_review": "", "summary_of_the_paper": "This paper focuses on classification problems for long-tailed image datasets. Specifically, the authors first establish an effective sampling theory to theoretical study the total number of effective samples and effective sample utilization in sampling with/without replacement. Then two general jitter sampling strategies are proposed for increasing the actual training effect in sampling with replacement. Finally, the experiments on three long-tailed datasets for many baselines indicate the classification performance of the proposed methods.\n", "summary_of_the_review": "This paper notices a critical problem, the long-tail distribution in image classification. New sampling strategies are proposed to address this problem. However, the novelty of the technical contribution is lacking, and the experimental results do not support the claim that the performance is improved.\n", "strength_and_weaknesses": "This paper focuses on a fundamental and valuable problem, the long-tail problem on real-world datasets. As one of the solutions to the long-tailed classification problem, it is significant to study re-sampling, especially the theoretical analysis of sampling strategies on long-tailed distributions.\nSeveral experiments are carried out on three real-world image datasets. The classification performances of the three proposed methods are evaluated by comparing them with thirteen baseline methods (both single-expert and multi-expert). In addition, the authors conduct ablation studies to highlight the importance of each component in the proposed methods.\nHowever, my main concern is that the novelty and significance of the new theory are not well stated and that the experimental results do not support the contributions well. Here are the detailed comments:\n[Presentation] First of all, the writing of the paper is not easy to follow for two reasons. (1) There are many statements and claims without clear meanings. For example, the meaning of the actual sampling frequency may need to explain when first mentioned. (2) Some notations are not clearly defined or have been used before the definition. For example, the definition of the subscript \"m\" is not given or is written in an inconspicuous place.\n[Novelty] I think the proposed new theory is not innovative enough compared to the existing conclusions (Kang et al., Cui et al.), and it does not bring a conceptual breakthrough. According to the statements in the paper, I cannot understand what and how the new theory improves the existing theories on which it is based. Currently in my opinion, it simply expresses an intuitive conclusion in mathematical terminology and may not bring a few new perspectives on how to design sampling methods in the future.\n[Method Details] Method 1 and Method 2 lack important details. I cannot find any description of the parameter settings. For example, what do \\alpha, \\beta, and \\gamma represent? Are they learnable, or are they hyperparameters? These are critical parts of the methods, and the lack of information hampers my understanding of how the methods work.\n[Experiments] In the experimental results of image classification, the methods proposed by the paper show equivalent or even reduced performances to some of the other alternative methods. Based on such experimental results, contributions compared to previous work cannot be significantly proved. In addition, some parts of the manuscript (abstract, conclusion) seem to overstate the performance and should be reformulated. \n", "limitations_and_societal_impact": "", "review_text": "", "extra_scores": {"correctness": "3: Some of the paper’s claims have minor issues. A few statements are not well-supported, or require small changes to be made correct.", "technical_novelty_and_significance": "1: The contributions are neither significant nor novel.", "empirical_novelty_and_significance": "Not applicable"}, "extra_text": {}}]

Author rebuttal:

Decision: Reject