Pursuing Better Decision Boundaries for Long-Tailed Object Detection via Category Information Amount
Authors: Yanbiao Ma
Wei Dai
Jiayi Chen
Venue: ICLR 2025 Poster (2025)
Keywords: Long-tailed recognition
Class Imbalanced
Image processing
Primary area: unsupervised, self-supervised, semi-supervised, and supervised representation learning

Abstract: In object detection, the number of instances is commonly used to determine whether a dataset follows a long-tailed distribution, implicitly assuming that the model will perform poorly on categories with fewer instances. This assumption has led to extensive research on category bias in datasets with imbalanced instance distributions. However, even in datasets with relatively balanced instance counts, models still exhibit bias toward certain categories, indicating that instance count alone cannot explain this phenomenon. In this work, we first introduce the concept and measurement of category informativeness. We observe a significant negative correlation between a category’s informativeness and its accuracy, suggesting that informativeness more accurately reflects the learning difficulty of a category. Based on this observation, we propose the Informativeness-Guided Angular Margin Loss (IGAM Loss), which dynamically adjusts the decision space of categories according to their informativeness, thereby mitigating category bias in long-tailed datasets. IGAM Loss not only achieves superior performance on long-tailed benchmark datasets such as LVIS v1.0 and COCO-LT but also demonstrates significant improvements for underrepresented categories in non-long-tailed datasets like Pascal VOC. Extensive experiments confirm the potential of category informativeness as a tool and the generalizability of our proposed method.

Published as a conference paper at ICLR 2025
PURSUING BETTER DECISION BOUNDARIES FOR LONG- OBJECT DETECTION VIA CATEGORY INFORMATION AMOUNT
Yanbiao Ma, Wei Dai & Jiayi Chen Xidian University, Xi'an an, 710071, China ybmamai sustu. xidian. edu. en
ABSTRACT
In object detection, the instance count is typically used to define whether dataset exhibits a long-tail distribution, implicitly assuming that models will underper- form on categories with fewer instances. This assumption has led to extensive research category bias in datasets with imbalanced instance counts. However, models still exhibit category bias even in datasets where instance counts are rel- atively balanced, clearly indicating that instance count alone cannot explain this phenomenon. In this work, we first introduce the concept and measurement of cat- egory information amount. We observe significant negative correlation between category information amount and accuracy, suggesting that category information amount more accurately reflects the learning difficulty of a category. Based on this observation, we propose Information Amount-Guide Angular Margin (IGAM) Loss. The core idea of IGAM is to dynamically adjust the decision space of each category based on its information amount, thereby reducing category bias in long-tail datasets. IGAM Loss not only performs well on long-tailed benchmark datasets such as LVIS v1.0 and COCO-LT but also shows significant improvement for underrepresented categorics in the non-long-tailed dataset Pascal VOC. Com- prehensive experiments demonstrate the potential of category information amount as a tool and the generality of our proposed method.
INTRODUCTION
In object detection tasks, long-tailed distribution is common phenomenon, where most instances are con- centrated in few categories, while other categories have relatively few instances Jiao et al. (2019); Liu et al. (2020); Zou et al. (2023); Oksuz et (2020). A widely accepted perspective is that the imbalance in the num- ber of instances causes the model to be more biased to- wards frequent categories during training, ignoring the less frequent ones, leading to significant category bias during testing Cho & Krähenbühl (2023); Alshammari al. (2022); Ren et al. (2020); Cui al. (2019); Wang et al. (2020a). However, recent research in image clas- sification suggests that category bias is not only caused by the imbalance in sample numbers but may also be closely related to the complexity of intra-category fea- tures Ma et al. (2023a;c); Kaushik et al. (2024). This is evidenced in datasets with perfectly balanced samples, where models still exhibit bias. Out of curiosity, amined the correlation between category average preci- sion (AP) and the number of instances on Pascal VOC, target detection dataset with relatively balanced num- ber of instances (see Figure 1), and found that the corre- lation between the two was very low. This indicates that object detection, model bias may also originate from the complexity of intra-category features.
Pascal VOC
Number insterces
20000
Staz
8.855
Cua 3.80
15000
0:74
0.7
10000-
Pearsog comelation confficient .0.171
5000
005
0.4
oe
char
bottle
DR
bost
mone disingta0e
fren
Figure 1: The left vertical axis repre- sents the number of instances per class. The right vertical axis represents the per- formance of Faster R-CNN trained with cross-entropy loss using R-50-FPN as the backbone across all classes, trained on the Pascal VOC. The model was trained using the settings described in Section 4.2. The red text box displays the Pearson correlation coefficient between class per- formance and the number of instances.
ex-

---

Published as a conference paper at ICLR 2025
Traditional long-tailed object detection methods mainly al- leviate this issue by re-weighting low-frequency categories Shen et al. (2016); Gupta et al. (2019); Alshammari al. (2022); Cui et al. (2019); adjusting gradients Lin et al. (2017b); Wang et. (2021a); Li al. (2022); Tan el. (2020; 2021). and employing data augmentation techniques Ghiasi al. (2021); Ma al. (2023b); Zang et al. (2021); Ma et al. (2024b). However, these approaches primarily focus on the impact of the number of instances while ignoring the com- plexity within categories. As a result, the model may fail to focus on some disadvantaged categories, limiting its overall performance. To better reveal and mitigate category bias, we introduce the concept of category information amount and its corresponding measurement. Category information amount quantifies the diversity of instances within a category, and a straightforward hy- pothesis is that the greater the information amount of category. the more difficult it is for the model to learn and memorize that category. We calculated the correlation between category information amount and AP on the relatively balanced Pascal VOC dataset and the long-tailed dataset (LVIS v1.0 and COCO-LT), shown in Table and Figure Category information amount, compared to the number of instances, better reflects the difficulty of a category. Therefore, we aim to leverage category information amount to direct the model's 's attention to truly challenging categories rather than simply focusing on categories with fewer instances.
Table 1: Pearson correlation coef- ficient between category informa- tion amount and class average pre- cision on long-taile datasets. The model is Faster R-CNN with R-50- FPN backbone.
Dataset
LVIS v1.0
CE
SeeSaw
Focal
IA Dataset
-0.68
-0.66
-0.70
COCO-LT
IA
-0.66
-0.65
-0.69
Pascal VOC
Peerson Correlation Cooffiiient
-0.8-
Information Amount
Number of Instances
Consider the following scenario: if head category has very little category information amount, the model can more easily learn and abstract the patterns of that cate- gory Cui et al. (2019). From an information compres- sion perspective, the decision space required for that category need not be very large. However, recent stud- ies Wang et al. (2022); Qi et al. (2023) have shown that the severe imbalance in data volume leads to patho- logical decision boundaries, where the decision space for tail categories is significantly compressed, while head categories have disproportionately large decision spaces. This unreasonable allocation of decision space represents a waste of model capacity. Can we directly equalize decision spaces?
-0.67
-0.8-
-.83
0.61
0.50
-0.6
-0.4
025
0.27
-0.2
0.17
0.20
021
0.20
-Gross Franegase Freopy-dacctee
-SeeRow
-Foce
-Fooi
Figure 2: Pearson correlation coefficients between category information amount and category average precision and between category instance count and category av- erage precision, under two backbone net- works and three loss function settings.
Differences in category information amount are inherent Ma et al. (2023a); Li & He (2024). For a recognition task, forcibly equalizing decision spaces means that for categories with higher infor- mation amounts, the model needs to learn stronger invariant representations to compress such cat- egories into a decision space of the same size as those with lower information amounts. However, the model does not receive additional constraints or support to improve its learning for categories with higher information amounts. Therefore, we propose dynamically adjusting the proportion of decision space allocated to each category based its information amount. Specifically, we design a novel loss function-IIformatinn Information Ammoutt-Guiden Angular Margin (IGAM) Loss, which aims to reduce model bias by dynamically adjusting decision spaces according to the information amount of each category. To enable dynamic updates to the information amount, we also designed a low-cost, end-to-end training strategy. Comprehensive experiments on long-tailed benchmark datasets LVIS v1.0, COCO-LT, and Pascal VOC demonstrate that our method surpasses most existing approaches in both overall performance and reducing model bias.
The main contributions of this paper are as follows: (1) We propose the concept of category In- formation amount and define its measurement (Section 3.1). A surprising finding is that in object detection tasks, category informativeness, compared to the number of instances, better reflects the difficulty of learning each category. This provides a useful tool for future research on improving performance on challenging categories. (2) We introduce the Information Amount-Guided Angular Margin (IGAM) Loss (Section 3.2), which adjusts the decision space of categories based on their information amount, encouraging the model to focus more on challenging categories. To dynam- ically update the information amount of categories, we propose an end-to-end training framework for applying IGAM at low cost (Section 3.3). (3) On long- tail benchmark datasets LVIS v1.0 and
2

---

Published as a conference paper at ICLR 2025
COCO-LT, our method achieves the best performance in most cases, particularly in improving the model's 's accuracy on rare categories (Sections 4.4 and 4.5). On the relatively balanced Pascal VOC dataset, our method significantly outperforms other approaches on challenging categories.
RELATED WORK
2.1 LONG-TAILED OBJECT DETECTION
In the research of long-tailed object recognition, the main approaches include data re- sampling, specialized loss function design, architectural improvements, decoupled training, and data augmen- tation. Data re-sampling is common method to address imbalanced datasets by increasing the sampling frequency of tail class samples to balance the data distribution. Common re-sampling strategies include Class-aware sampling Shen et al. (2016) and Repeat factor sampling (RFS) Gupta et al. (2019), These methods can be employed at different stages of training to achieve a multi- stage training process. Specialized loss function design is another technical approach to tackling long-tailed challenges. For instance. EQL Tan et al. (2020) reduces suppression on tail classes by truncating the negative gradients from head classes. The subsequent EQLv2 Tan et al. (2021) further improves this approach through gradient balancing mechanism. Other methods, such as Seesaw Loss Wang et al. (2021a), Equalized Focal Loss Li et al. (2022), ACSI. Wang et al. (2021b), and LOCE Feng et al. (2021), reduce excessive suppression of tail classes by dynamically adjusting classification logits or suppressing overconfident scores. C2AM Wang et al. (2022) observed that the severe imbalance in weight norms across classes leads pathological decision boundaries, and therefore proposes learning fairer decision boundaries by adjusting the ratio of weight norms.
Current research mainly focuses these two directions. In addition, module improvement empha- sizes modifying the structure of detectors to address long-tailed distribution issues. For example, BAGS Li et al. (2020) and Forest R-CNN Wu et al. (2020) mitigate the impact of head classes on tail classes by grouping all classes based on valuable prior knowledge. Decoupled training Kang al. (2019) has found that long-tailed distributions do not significantly affect the learning of high- quality features, thus some methods freeze the feature extractor parameters during the classifier learning phase, adjusting only the classifier Ma al. (2023b); Wang et al. (2020a); Zhang et al. (2021). Data augmentation, as a means of introducing additional sample variability, has been shown to provide further improvements long-tailed detection tasks. Recently proposed methods such as Simple Copy-Paste Ghiasi et al. (2021), FDC Ma et al. (2023b), FASA Zang et al. (2021), and FUR Ma et al. (2024b) supplement the insufficiency of tail-class samples by performing data augmenta- tion in both image and feature spaces. RichSem Meng et al. (2024) and Step- wise Learning Dong et al. (2023) introduce Transformer-bet based object detection architectures, with the former relying on external data and adding new network branches, while the latter incorporates multiple modules and multi-stage training. The core advantage of our proposed IGAM lies in its simplicity and efficiency.
2.2 METHOSS FOR MEASURING CLASS DIFFICULTY
The study of class difficulty is most relevant to our work. Most research addressing class bias has focused on scenarios with sample imbalance, where rebalancing strategies based on sample size can be somewhat effective. However, recent studies have reported that even when sample sizes are per- feetly balanced, classification models still exhibit significant performance disparities across different classes. Investigating the root causes of model bias in scenarios where sample sizes are balanced is crucial for improving model fairness and understanding learning mechanisms. However, research on this issue is still limited. From a geometric perspective, DSB Ma et al. (2023a), CR Ma et al. (2023c), and IDR Ma et al. (2024a) conceptualize the data classification process as the disentangling and separating of different perceptual manifolds. These three studies respectively reveal that the ge- ometric properties of perceptual manifolds monifolds-volumee curvature, and intrinsic dimensionality- are significantly correlated with class performance. Kaushik et al. (2024) discovered that differences in the spectral features of classes could be source of class bias. Unfortunately, in the field of object detection, there has been research exploring the underlying causes of model bias. Our work is the first to directly report on the widespread bias present in object detection models and to attempt to explore the potential mechanisms underlying this bias.
PURSUING BETTER DECISION BOUNDARIES WITH THE HELP OF CATEGORY INFORMATION AMOUNT
In this section, we first define and compute the category information amount (Section 3.1), then gradually derive how dynamically adjust the decision space of categories based on their informa-
3

---

Published as a conference paper at ICLR 2025
tion amount (Section 3.2). Finally, propose low-cost, end-to-end training strategy enable the dynamic update of the category information amount (Section 3.3).
3.1 DEFINITION AND MEASUREMENT OF CATEGORY INFORMATION AMOUNT
Recent studies have shown that the response of deep neural networks images similar human vision, following the manifold distribution hypothesis, where the embeddings of images lie near low-dimensional perceptual manifold embedded in nigh-dimensional space Cohen et al. (2020); Li et al. (2024). Continuous sampling along dimension of this manifold corresponds to continuous changes in physical features. Therefore, the volume of the perceptual manifold mapped by deep neural network can effectively measure the information amount of category. Based on this theory, we define the information amount 14 of category as the volume of its perceptual manifold: = (X,, where X; U2,......... Tmm represents the set of embeddings for instances in category i, m denotes the number instances. Vol(X,) measures the volume of the perceptual manifold, reflecting the information amount of the category. It is important to note that the embeddings used to calculate the information amount should be extracted from the classification module of the object detection model, not the regression module. Below is the method for calculating Vol(X.).
Given the embedding set X; [2,, T2..... Tmm E Rexm where each instance embedding denotes the dimension of the embedding. We first compute the covariance matrix of the embedding set X (X2) = m [ - (z, z)²,
Listing 1: Category Information Amount
RP, .p
import numpy np
Function to compute information Amount,
def http:utteiinrenne_ddiiee
(embeddings): p enbeddings. shape
aeaan_ambedding (sc.aaggg........ammmmmmiiinn (smbeddings, axis=()
where E is the mean vector of the embedding set X3: ē- Ej Zy.
cente nettered_eeee Beteered_ebbbdnngge
embedd: nga
sear_enbbdding
sample_cov np. (centered (centered_enbedddngg
The covariance matrix Σ(X;) captures the distri- bution characteristics of the category in the high- dimensional embedding space, and its determinant can be used to calculate the volume of the per- ceptual manifold. To enhance the accuracy of the covariance matrix estimation, employ the Ledoit-Péché nonlinear shrinkage method Burda & Jarosz (2022), which improves robustness in high-dimensional spaces through eigenvalue trans- formation. The covariance matrix EXX;) recon- structed as follows:
centered_embedddngnn) / m.
sigvals, eigvecs up. np.linslg.eigh sample_cov)
a p / n
lambda
op. (c))
lambda_ plus - (1 + ap. sgrt (=1) d ap. www.maximmmmmmmmmmmmmmmmm (eigval,,, lamidd__minua shrunk_con = no. (aigvecs, rp.dot inp.
diag (d), eigvecs.T))
Information 0.5 .p. (np. iip.linalg.de. ap.eye (p) + atrunk_cov!)
return Information
EXX) = , VT)
where V the matrix of eigenvectors, and X; = max(A,, A_), with X_ = -1- vp/m)² as the nonlinearly transformed minimum eigenvalue. Finally, the information amount of category i is formally defined as:
4 = Vol(X,) = log, det((E(;)) + 1),
where det(E(X,) + I) is the determinant of the matrix Σ(X,) + I, and I is the identity matrix. The determinant reflects the spread of the category's 's embeddings, representing the volume of the perceptual manifold. In this way,
quantify the information amount of category
3.2 INFORMATION AMOUNT- AMOUNT-GUIDED ANGULAR MARGIN (IGAM) LOss
Cross-entropy loss is widely used in deep learning, especially in classification tasks. It measures the difference between the model's predicted probability distribution and the true label distribution. For the classification component in object detection tasks, given feature vector æ and a label i, the cross-entropy loss typically defined as:
WFx
== T=-lop log
(3)
21=1
where W, is the j-th column of the final fully connected layer, corresponding to the weight vector for category j. In long-tailed scenarios, it has been observed that the norm of the weight vectors corresponding to each category is extremely unbalanced, making it more difficult recognize tail
4

---

Published as a conference paper at ICLR 2025
categories. For example, in a binary classification task, when W W 2, we have:
W ( ||W2l22 cos()2), 0 0 02 - 2
(4)
If | 1l2 > ||2ll2 then 01 >> 02 must hold for the Equation (4) to be true. Clearly, when || Wille >> ||W2] 12. the decision space for category 2 is pathologically compressed. To address this, a simple solution is to directly equalize the decision space, ignoring the norm of the weight vectors for each category, resulting in:
L = log
(5)
(-006(9))
where cos(0.) [W. | and s is hyperparameter introduced to stabilize training. The opti- mization goal can be understood as minimizing the angle between W; and z. Although this approach addresses the pathological decision space allocation, recent studies show that absolute equal allo- cation among all categories is not the optimal solution. A straightforward explanation is that with the same number of samples for each category, their learning difficulties vary. Thus, absolute equality restricts the model's 's sensitivity and attention to different categories.
W
From an information compression perspective, if the information amount of a category is signifi- cantly larger than others, but its decision space is required to be compressed to the same extent, this is clearly unreasonable. Therefore, we propose using each category's 's information amount to dynam- ically adjust the decision boundaries, allowing categories with larger information amounts to have larger decision spaces. Specifically, we introduce an angular margin my based on the information amount of each category into Equation (5). The final optimization objective is expressed as:
(=000((f.)
= - log (0.c0((0)) + EC L e esscos(0
(6)
C++,, 1- 4. In this where: my. max (o. 4 log( ))) and: 1. Ej= cn/liivo)
formula, 1/ represents the normalized information amount of category i, with the normalization method adopted from Ma et al. (2023a). The term mij is based on the ratio of the information amounts of the category and j. If the information amount of category i is larger (i.e., 1 > I)), then Maj is positive, meaning the decision space for that category should be expanded. Conversely, if < <T;, then my is negative, and the decision space for class is compressed. By incorporating information amount, IGAM Loss can more accurately reflect the internal complexity of categories, rather than solely relying on instance numbers. This allows the model allocate more decision space to complex categories, improving detection accuracy for these categories.
In practical training, face an engineering challenge: the information amount of categories changes as the model parameters evolve, necessitating dynamic updates. However, calculating the information amount requires the covariance matrix of all instance embeddings. and extracting em- beddings for the entire dataset in each iteration would lead to excessive memory and time costs, interrupting training. We propose a novel training framework to address this issue.
3.3 Low-Cost DYNAMIC UPDATE OF INFORMATION DENSITY
3.3.1 DYNAMIC UPDATE STRATEGY
The phenomenon of feature slow drift Wang et al. (2020b); Ma al. (2023a) indicates that as training progresses, the distance between the embeddings of the same sample at different training stages becomes increasingly smaller, to the extent that the previous version of an embedding can approximate the latest version. Inspired by this, we propose the most straightforward approach: store the embeddings of all instances generated during training equal to the total number of instances in the dataset. After each training epoch, all embeddings in the queue can be updated once. At the end of each epoch, the embeddings the queue are used to calculate and update the information amount for all categories. In the following, we refer to this approach as the original strategy. Although the original strategy avoids repeatedly extracting embeddings for the entire dataset, it increases the demand for storage space. Considering that the essence of calculating the information amount lies in obtaining the covariance matrix of the instance embeddings, we propose
queue, with the queue length
strategy that significantly reduces storage space. The core idea of this new strategy is to calculate the global covariance matrix of the samples using multiple local sample covariance matrices. The specific steps follows:
5

---

Published as a conference paper at ICLR 2025
(1) Initialize queue to store instance embeddings. The length of the queue be adjusted according to the GPU memory size. Suppose the object detection dataset contains C' cate- gories with a total of N instances, and the queue length is d. If d < NN, it means the queue cannot hold all instance embeddings. In this work, we set the queue length to 50,000, which store 50,000 instance embeddings.
(2) At the beginning of training epoch, first store the instance embeddings generated in each batch into the queue until full (i.e., storing d instance embeddings). Then, use the em- beddings in the queue to calculate the local covariance matrix and mean for each category. Continuously update the queue, and once all old embeddings in the queue are updated, calculate the local covariance matrix and mean for each category again. By the end of an epoch, we can calculate [N/d] + local sample covariance matrices 2 and means yu for each category i 1...., C, k .., [N/d] +.
(3) At the end of an epoch, use the stored local covariance matrices to calculate and update the information amount for each category. Taking category i an example, first calculate the global mean:
[N/d] +1
T P
where N, is the total number of instances in category i, and n is the number instances in the local sample. Then, calculate the global covariance matrix:
Nidj+1 +1
[N/d]+1 Σ
Σ, N
Σ
ni n²( - -mi)( - 1A))
(8)
The proof of this formula is provided in Appendix A. By integrating local covariance matrices to obtain the global covariance matrix, we significantly reduce the additional stor- age space required update the information amount. Further, the information amount of category is estimated as Vol, = log, det(I + E.).
3.3.2 STORAGE SPACE COMPARISON
Assume the object detection dataset contains N instances, each with an embedding dimension of p, and there are C' categories. The queue length is set to d. The storage space required by the original strategy is: Soriginal N x p. The storage space required by the new strategy is:
Swew = d x p + C x (LNV/) + + pp.
where C (LN/d) +1) x d² represents the space needed to store the local covariance matrices. To analyze when the new strategy saves more space, we define the storage space ratio R::
Snew
d x x (LN/d) + x x
R=
(10)
Soriginal
When R 1, the new strategy saves more stor- age space. To visually compare the storage space requirements of the new and original strategies, we take the Pascal VOC and MS COCO datasets as examples and plot the function graph of the storage space ratio R as it varies with the queue length in Figure 3. It can be observed that in most cases, the storage space required by the new strategy is less than that of the original strategy. By visualizing our proposed storage space ratio, it becomes easy to choose the optimal queue length d, thereby saving approximately 60% of the stor-
Poscal VOC
1.75
1.50
Threshold R
Optimal 11517
Space Rarto
2.0
1.5
0.75
1.0
0.50
0.3
0.0
20000 40000
500300
Qususe Length
Figure 3:
The function of the storage space ratio R as it varies with the queue length d on the Pascal VOC and MS COCO datasets.
age space. Given two examples {N 55800, p = 128, C = 20} and {N = 605638, p = 128, C' = 80). corresponding to the Pascal VOC and MS COCO datasets respectively, we use Listing 2 to select the optimal queue lengths d of 11517 and 68182. For Example 1, the original strategy re- quires an additional 27. MB of memory, while the new strategy only requires 11.87 MB, saving approximately 56.44% of memory. For Example 2, the new strategy can reduce memory usage from 295.72 MB to 78.29 28 MB, saving approximately 73.52% % of memory.

---

Published as a conference paper at ICLR 2025
Listing 2: Code for selecting the optimal d
The new training framework significantly reduces storage space utilization by merging the local sample covariance matrices, ensuring that the cal- culated value of information density remains un-
import matplot matplottlb.. pyb.pyplod a8 plt
import cопру as np
$ Dafine parameters
N 558C0
$ Total numbez of instances
2 128
Embedding dimension
changed. This innovative strategy not only pro- vides an efficient solution for dynamically updat- ing information density but also offers low-cost
c = 20
Number of categories
# Dafine the range
iengthe
d_values np. ipp.pane (1000, N, 100)
Rvvalues 1ist (d__alue
inp.
storage solution for future research.
d_valuea) + +1) minuR_values min (RLvalues)
/ N)
optims-_d
min [_vaues]
4 EXPERIMENTS
print optimal iss', optima _d]
We conducted a comprehensive evaluation of the effectiveness of IGAM on long-tailed and relatively balanced object detection benchmark datasets. The experiments are divided into two parts: the first part is carried out on the long-tailed large-vocat auge-vocabulary datasets LVIS v1.0 and COCO-LT, while the second part is conducted on the relatively balanced Pascal VOC dataset.
4.1 DATASETS AND EVALUATION METRICS
LVIS v1.0 Gupta al. (2019) contains 1,203 categories, with the training set consisting of 100k images (approximately 1.3M instances) and the validation set containing 19.8k images. Based on the frequency of occurrence in the training set, all categories (1~10 images), common (11~100 images), and frequent (more than 100 images). In line with EFL Li et al. (2022), we report not only the widely used object detection metric A across IOU thresholds (from 0.5 to 0.95) but also the bounding box AP for frequent (AP)), common (AP'), and rare (AP,) categories separately. The COCO-LT Wang et al. (2020a) dataset a long-tailed subset of MS COCO Lin et al. (2014), and they share the same validation set. Consistent with previous work Wang et al. (2021a), we divided the 80 classes in COCO-LT into four groups based on the number of training instances per class: fewer than 20 images, 20~400 images, 400~8000 images, and 8000 or more images. Pascal VOC Everingham et al. (2015) includes two versions, 2007 and 2012, comprising total of 20 classes. Following standard practice Tong & Wu (2023), trained on the train+val sets of VOC2007 and VOC2012 and tested on the test set of VOC2007. We report the Average Precision (AP) for each class Pascal VOC, and on COCO-LT, we report the accuracy of the four groups as AP/, AP/. AP3, and API. The mean average precision is reported as mAPi
divided into three groups: rare
4.2 IMPLEMENTATION DETAILS
Consistent with previous studies Qi et al. (2023), we implemented the Faster R-CNN Ren et al. (2015) detector using the MMDetection Chen et al. (2019) toolbox and adopted ResNet-50 and ResNet- He et al. (2016) with an FPN Lin et al. (2017a) structure as the backbone networks. During training, we set the batch size to 16 and the initial learning rate to 0.02, consistent with EFL Li et al. (2022) and C2AM Wang al. (2022). We trained the model using an SGD optimizer with momentum of 0.9 and weight decay rate of 0.0001 for 24 epochs. The learning rate was reduced to 0.002 and 0.0002 at the end of the 16th and 22nd epochs, respectively. In all experiments, we applied random horizontal flipping and multi-scale jittering for data augmentation. We did not use any test-time augmentations. We did not use any test-time augmentations.
4.3 MAIN RESULTS: EFFECTIVENESS OF IGAM Loss
Table 2: Results of IGAM Loss with different hyper-parameter on LVIS v1.0. The model is Faster R- CNN with R-50-FPN backbone.
Recalling Section 3.2, we introduce the hyperparameter s in the IGAM Loss, which scales the cosine value to ensure it falls within an appropriate range, thereby stabilizing the op- timization process. This is a standard practice in cosine- based classifiers and has been widely adopted. We refer to Wang et al. (2022) for rigorous determination of and tested the model's performance under different settings. The ex- perimental results, summarized in Table 2, show that when
8
mApb
AP,
AP
A AP,
10
16.8
0.8
12.6
27.8
20
26.2
17.6
23.8
31.5
30
26.8
19.0
25.2
31.4
s = 30, the model trained with IGAM Loss achieves opti- mal performance. After determining the hyperparameter,
40
25.9
18.4
23.5
30.9
50
24.6
16.9
22.7
30.5
7

---

Published as a conference paper at ICLR 2025
Table 3: Evaluation results on LVIS v1.0. The mAP" APr, APc. and AP; (%) for each method are reported, with green arrows indicating performance improvements.
Framework
Backbone
Loss
mAP
AP.
AP.
AP/
Cross-Entropp (CE)
19.3
1.1
16.1
30,9
ResNet-50-FPN
IGAM Loss
26,8 17.5
19.0117.9
25.2
31.4
Cross-Entrope (CE)
20.9
1.0
18.2
32.7
Faster R-CNN
ResNet-101-FPN
IGAM Loss
28.0 17.1
20.1 119.1
26.8
32.5
Cross- Entropy (CE)
25.4
6.2
24.5
35.3
Swin-T
IGAM Loss
31.7 16.3
21.4 115.2
30.8
37.1
Cross- Cioss-Entropy (CE)
22.7
1.5
20.6
34.4
ResNet-50-FPN
IGAM Loss
29.1 16.4
21.5 120.0
27.7
33.9
Cross-Entropy (CE-FIN)
24.5
23.1
35.8
Cascade Mask R-CNN
ResNet-101-FPN
IGAM Loss
29.7 15.2
21.9 119.3
34.6
Swin-T
Cross-Entropy (CE)
31.3
6.8
30.2
39.4
IGAM Loss
37.9 16.6
25.2 118.4
35.5
38.7
Cross-Entropy (CE)
21.8
3.3
21.2
30.5
ResNet-50-FPN
IGAM Loss
27.6 15.8
18.5
115.2
27.0
32.7
Cross-Entropy (CE)
23.1
3.7
23.4
32.2
ResNet-101-FPN
DETR
IGAM Loss
30.4 17.3
20.7
17.00
30.0
35.5
Cross-Entrope (CE)
30.2
6.3
28.9
38.2
Swin-T
RichSem Meng et al. (2024)
34.9
26.0
32.6
41.3
IGAM Loss
37.3 17.1
24.8
118.5
34.8
38.3
Table 4: Performance comparison with state-of-the-art methods on LVIS val set. The ResNet-50- FPN and ResNet-101-FPN are adopted backbones for Faster R-CNN. All methods are trained with 2x schedule, i.e., 24 epochs in total. The mAP", AP,, APc, and AP, y (%) for each method are reported. The best and second-best results are shown underlined bold and bold, respectively.
LVIS v1.0
Strategy
Methods
RcsN
ResNet-
I0I-FPN
mAP
AP,
APc
AP;
mAP
AP,
AP.
AP,
RFS Gupta al. (2019)
24.2
14.2
22.3
30.6
25.7
15.9
23.7
32.2
FQI. Tan et al. (2020)
21.8
3.6
21.1
30.5
23.4
4.5
22.9
32.3
DropLoss Hsieh et (2021)
21.8
5.2
21.8
29.1
23.5
5.9
23.9
30.7
RIO Chang et al. (2021)
23.4
15.3
21.2
29.4
25.5
17.2
23.7
31.2
Forest R R-CNN Wu
(2020)
BALMS Ren et al. (2020)
24.1
15.2
23.0
29.4
26.9
18.5
25.2
32.4
De-comfound-TDD: Tang et al. (2020)
23.7
10.0
22.4
31.2
End-to-end
EQLv2 et al. (2021)
25.4
15.8
23.5
31.7
26.8
17.1
24.9
33.1
Scesaw Wang
(2021a)
24.8
14.8
22.7
31.6
26.6
14.9
25.2
33.3
FASA Zang al. (2021)
21.5
7.4
19.2
30.2
22.9
9.0
20.6
31.6
Li al. (2022)
26.0
16.6
25.1
30.8
26.3
18.5
23.9
32.6
C2AM Wang et al. (2022)
25.4
15.6
24.2
30.9
26.5
18.1
25.5
31.2
SimCal Wang al. (2020a)
BAGS Li ct al. (2020)
23.7
14.2
22.2
29.6
25.4
14.9
25.2
31.4
Decoupled
ACSI. Wang et al. (202
22.2
9.9
21.3
28.5
23.7
11.0
23.0
30.2
DisAlign Zhang et (2021)
20.9
3.9
20.4
25.5
24.5
LOCE Feng et al. (2021)
25.1
15.7
24.2
30.1
26.7
18.4
25.5
31.7
BACI. Qi et al. (2023)
26.1
16.0
25.7
30.9
27.8
18.
27.3
32.6
End-to-end
IGAM
26.8
19.0
25.2
31.4
28.0
20.1
26.8
32.5
compare IGAM Loss with the bascline model. Since our method is derived from cross-entropy loss, the baseline model trained using cross-entropy loss.
To validate the effectiveness of IGAM Loss, we employed Faster R-CNN and Cascade Faster R- CNN as detection frameworks, with ResNet-50 and ResNet-101 backbone networks. The baseline models were trained using cross cros-eetrrppy loss. As shown in Table 3, the four baseline models trained with cross-entropy loss achieve an average precision (APr) only 1. 1.1%, 1.0%. 1.5%. and 2.6% on tail classes, making it nearly impossible to recognize tail instances. In contrast, IGAM significantly improves the detection accuracy for tail classes. For example, with Mask R-CNN the framework and ResNet-50-FPP as the backbone, IGAM raises APr by 17.9%. Moreover, IGAM also brings a 9.1% performance gain in the APc metric. Beyond the substantial improvements performance, IGAM surpasses the baseline model in overall performance across the four different detection frameworks and backbone configurations by 7.5 %. 7.1 %, 6.4%, and 5.2% respectively.
tail-class
IGAM demonstrates remarkable generalization capabilities across various configurations. and its performance leap over the baseline models highlights the effectiveness and versatility of our pro- posed method, which refines decision space partitioning dynamically using information amount.
8

---

Published as a conference paper at ICLR 2025
4.4 COMPARISON WITH STATE- OHE
Table 4 presents the experimental results on LVIS v1.0. IGAM outperforms the current state-of- the-art methods on both ResNet-50-FPy and 101-FF--O1-FPN backbones, achieving overall per- formances of 26.8% and 28.0%, respectively. Notably, for rare categories, IGAM suraasess the second-best method by 2.4 % and 1.6% on the two backbones, respectively. It is worth mention- ing that despite BACL incorporating a series of techniques, including sengrouund-allnncee loss, synthetic hallucination samples, and decoupled training, our method still exhibits strong competi- tiveness. For the most frequent categories, although EQLv2 and Seesaw exhibit exceptional perfor- mance, they significantly lack attention to rare categories. In contrast, IGAM demonstrates superior performance
categories while maintaining competitive results on the most frequent cate- gories. On the ResNet-50-FPP backbone, IGAM's AP, is only 0.3% and 0.2% lower than that of EQLv2 and Seesaw, respectively. We attribute IGAM's superior overall performance its accurate measurement of category learning difficulty through information amount, allowing IGAM not to impair the performance of frequent categories by focusing too much on rare categories.
Table 5: Evaluation results on COCO-LT. The m.PP" API. AP APS, and AP (%) for each method are reported. An asterisk (*) indicates results reproduced by our implementation. The best and second-best results are shown in underlined bold and bold, respectively.
COCO-LT
Methods
ResNet-S 50-FPN
ResNet-
mApb
AP
AP1
AP
AP
m.
APP
AP
AP
APr
Cross-Entropy (CE)
24.5
14.6
29.6
32.9
26.0
0
16.4
31.4
34.2
Seesaw Wang et al. (202 la)
23.9
3.0
14.5
28.4
32.3
24.9
3.2
14.5
30.0
33.4
EQLv2 Tan et al. (2021)
25.7
3.8
18.1
29.6
33.0
26.8
3.2
19.4
30,8
34.1
EFL* Li al. (2022)
25.0
3.8
16.3
29.5
32.5
25.4
3.6
16.5
30.2
32.8
C2AM* Wang et al. (2022)
24.7
2.8
15.6
29.4
32.3
25.1
2.9
15.6
30.2
32.7
IGAM
25.8
18.0
29.7
32.5
27.0
6.6
19.0
30.5
33.4
4.5
EVALUATION RESULTS ON COCO-LT
The COCO-LT dataset is not yet a widely recognized benchmark in the field of long-tailed object detection, and thus, there are relatively few studies validating methods on We trained baseline models on COCO-LT using cross-entropy loss and compared Seesaw and EQLv2 following Qi et al. (2023). Additionally, we independently implemented EFL and C2AM. The experimental results summarized in Table 5. It can be observed that IGAM achieves the best overall performance on both backbone networks, with 25.8% and 27.0 %2 respectively. Notably, IGAM surpasses second-best best method in the AP& metric by 2.3% and 3.0% the two backbones, respectively, highlighting the significant advantage of our method on rare categories. While cross-entropy loss and EQLv2 perform well on the most frequent categories, they exhibit clear gap in performance rare categories compared to IGAM. Furthermore, IGAM does not fall behind cross-entropy loss and EQLv2 in the AP metric, demonstrating that our method effectively balances performance across both rare and frequent categories.
4.6 EVALUATION RESULTS ON PASCAL VOC
We trained models using Seesaw, EFL, and C2AM losses on the relatively balanced Pascal VOC dataset for comparison. Table presents the performance of each method across all categories as well as the overall performance. It can be observed that IGAM outperforms the other methods in terms of overall performance on both backbone networks, achieving 77.7% and 78.6 respectively, surpassing the second-bor best method by 0.8% and 1.1%.
More importantly, our method significantly improves the performance of underperforming cate- gories. We selected five representative poorly performing categories for observation: aeroplane, boat, bottle, chair, and pottedplant. With the ResNet-50 backbone, IGAM achieved the best perfor- mance across all five categories, surpassing the second-best method by 2.2%, 1.3%. 3.4%, 0.7%, and 2.1 ,, respectively. Similarly, with the ResNet- 101 backbone, IGAM also achieved the best performance across all five categories, exceeding the second-best method by 1.6%, 2.7% 3.0%

---

Published as a conference paper at ICLR 2025
Table 6: Evaluation results on Pascal VOC. (%) for each class using different methods with ResNet-50 and ResNet-101 the backbone. We selected the five most challenging classes, with the best and second-bess results are shown underlined bold and bold, respectively. Green arrows and values indicate the difference between the best and second-best results.
Class
ResNet-50
ResNet-101
CE
Seesaw
C2AM BACL
IGAM
Seesaw
EFL
C2AM
BACL
IGAM
cat
84.8
89.5
88.3
88.1
89.6
87.8
88.9
91.4
91.2
89.8
91.8
88.0
car
75.7
79.8
78.9
80.8
80.3
81.3
79.2
82.2
81.1
82.0
82.7
83.3
horse
81.6
85.9
86.3
87.0
85.5
88.1
83.6
86.5
86.6
87.6
86.7
87.1
bus
80.2
84.8
82.9
84.5
84.1
85.2
82.4
83.6
84.4
85.3
86.1
86.5
bicycle
82.1
86.1
84.5
85.2
87.8
84.6
84.0
87.0
85.8
85.5
88.1
86.2
dog
81.7
88.4
86.5
88.9
90.8
90.2
87.1
91.3
86.7
person
75.8
79.1
77.8
79.5
81.1
84.9
82.8
78.2
85.6
79.8
train
80.4
84.2
82.9
85.3
85.6
87.0
83.9
86.4
85.4
85.5
87.7
86.1
motorbike
79.4
83.6
80.1
82.0
82.6
83.2
83.2
87.4
84.1
82.6
86.8
83.6
cow
80.3
82.9
79.8
80.7
82.8
82.1
78.4
81.8
80.1
81.0
81.5
81.9
aeroplane
64.4
69.7
72.1
71.8
70.1
74.3
12.2
69.0
73.1
70.0
72.5
71.3
74.7
116
tymonitor
71.3
75.8
76.4
74.1
75.9
75.7
68.0
75.2
69.9
77.0
71.6
78.5
sheep
73.4
76.5
74.8
75.6
78.8
76.5
75.9
80.4
78.2
75.5
79.6
77.4
bird
75.5
79.3
75.2
79.7
77.3
81.0
69.5
74.1
72.3
80.4
72.7
81.2
diningtable
70.8
74.0
71.3
73.4
76.3
75.1
68.7
75.5
73.3
74.3
74.7
76.1
sofa
75.2
78.7
72.7
79.5
70.6
77.2
72.5
79.7
73.5
80.6
boat
61.8
66.4
62.4
64.2
67.7
1..3
61.5
63.3
64.2
64.2
67.9
12.7
bottle
50.6
53.6
50.8
54.8
54.2
58.2
13.4
50.0
52.4
51.5
55.6
53.6
58.6
13.0
chair
61.5
65.8
61.1
62.3
61.7
66.5
10.7
58.9
62.3
61.9
63.5
63.1
66.0
22.5
pottedplant
49.3
52.7
49.1
51.5
51.1
54.8
12.1
48.2
53.9
50.0
54.0
52.2
57.3
3.3
Total
72.8
76.9
74.6
76.2
77.7
10.8
73.5
77.5
75.8
77.0
77.3
78.6
11.
2.5%, and 3.3%, respectively. Notably, for the two worst-performing categories, bottle, and pot- tedplant, IGAM consistently demonstrated the most significant improvements. The experimental results indicate even on relatively balanced datasets, our method can effectively focus on and enhance the performance of the underrepresented categories. This further validates that the category information amount we proposed more accurately measures the learning difficulty of each category.
4.7 EFFECTIVENESS IN REDUCING MODEL BIAS
LVIS v1.0
To more clearly demonstrate the effectiveness of our method in mitigating model bias, we use the variance of class-wise average precision (AP)
Seesow (R FPNI
47.09
measure of model bias. The comparison results on LVIS v1.0 are shown in Figure 4. It can be observed that the models trained with our method exhibit lower bias compared Seesaw, EFL, and C2AM, across two different back- bones. Notably, compared to Seesaw, duces model bias by approximately 50%. These results are attributed to the accurate reflection of learning dif- ficulty through category information amount. We en- courage other researchers to explore additional potential factors influencing model bias, aiming to design more equitable object detection models.
C24M 50 FPN
Methods
Seasan (8-10L-PPM)
58.70
31.16
method re-
CDAN (8-10L-P1M0
28.76
IGAM (R-50-PPU)
25.37
16-LOL-FMM
40
Class Average Preciston (AP)
Figure Model bias from models trained with different methods on LVIS v1.0.
CONCLUSION
This work addresses the issue that instance count fails to explain the generalized bias present in deep learning models for object detection tasks. We propose using information amount to measure the detection difficulty of categories, and experiments reveal significant negative correlation between a category's 's information amount and its accuracy. Based on this finding, we propose dynamically adjusting the decision boundaries of categories using their information amount. Comprehensive empirical studies demonstrate that information amount helps the model focus more on learning challenging categories, both in long-tailed and non-long-tailed datasets.
10

---

Published as a conference paper at ICLR 2025
REFERENCES
Shaden Alshammari, Yu-Xiong Wang, Deva Ramanan, and Shu Kong. Long-tailed recognition via weight balancing. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6897-6907, 2022.
Zdzislaw Burda and Andrzej Jarosz. Cleaning large-dimensional covariance matrices for correlated samples. Physical Review E, 105(3):03413,, 2022.
Nadine Chang, Zhiding Yu, Yu-Xiong Wang, Animashree Anandkumar, Sanja Fidler, and Jose M Alvarez. Image-level or object-level? tale of two resampling strategies for long-tailed detection. In International conference on machine learning, pp. 1463-1472. 1472. PMLR, 2021.
Kai Chen, Jiaqi Wang. Jiangmiao Pang. Yuhang Cao, Yu Xiong, Xiaoxiao Li, Shuyang Sun, Wansen Feng, Ziwei Liu, Jiarui Xu, et al. Mmdetection: Open mmlab detection toolbox and benchmark. arXiv preprini arXiv: 011099900075555 2019.
Jang Hyun Cho and Philipp Krähenbühl. Long-tail detection with effective class-margins. arXiv preprint arXiv:2301.00 09924,, 2023.
Uri Cohen, Sue Yeon Chung, Daniel D Lee, and Haim Sompolinsky. Separability and geometry of object manifolds in deep neural networks. Nature communications, 11(1):746, 2020.
Yin Cui, Menglin Jia, Tsung- Yi Lin, Yang Song, and Serge Belongie. balance loss based on effective number samples. In Proceedings of the IÉEE/CVF conference on computer vision and pattem recognition, pp. 9268 918779977-00 2019.
Na Dong, Yongqiang Zhang, Mingli Ding, and Gim Hee Lee. Boosting long-tailed object detec- tion via step-wise learning on smooth-tail data. In Proceedings of the IEEF/CVF International Conference on Computer Vision, pp. 19440,949, 2023.
M. Everingham, S. M. A. Eslami, L. Van Gool, C. K. I. Williams, J. Winn, and A. Zisserman. The pascal visual object classes challenge: A retrospective. International Journal of Computer Vision, 111(1):98-138 January 2015.
Chengjian Feng, Yujic Zhong, and Weilin Huang. Exploring classification equilibrium long-tailed object detection. In Proceedings of the IEEE/CVF International conference computer vision, pp. 3417-3426, 2021.
Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yr Yi Lin, Ekin D Cubuk, Quoc V Le, and Barret Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 2918-2928, 2021.
Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmen- tation. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 5356-53666 2019.
Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recog- nition. In Proceedings of the IEEE conference on computer vision and pattern recognition, pp- 770-778, 2016.
Ting-I Hsieh, Esther Robb, Hwann-Tzong Chen, and Jia-Bin Huang. Droploss for long-tail instance segmentation. In Proceedings of the AAAI conference on artificial intelligence, volume 35, pp. 1549-1557, 2021.
Licheng Jiao, Fan Zhang, Fang Liu, Shuyuan Yang, Lingling Li, Zhixi Feng, and Rong Qu. A survey of deep learning- based object detection. IEEE access, 7:128837-128868. 2019.
Bingyi Kang, Saining Xie, Marcus Rohrbach, Zhicheng Yan, Albert Gordo, Jiashi Feng, and Yannis Kalantidis. Decoupling representation and classifier for long-tailed recognition. arXiv preprint arXiv:1910.099 2019.
11

---

Published as a conference paper at ICLR 2025
Chiraag Kaushik, Ran Liu, Chi-Heng Lin, Amrit Khera, Matthew Y Jin. Wenrui Ma, Vidya Muthukumar, and Eva L Dyer. Balanced data, imbalanced spectra: Unveiling class disparities with spectral imbalance. arXiv preprint 242.::240..1174,, 2024.
Bo Li, Yonggiang Yao, Jingru Tan, Gang Zhang, Fengwei Yu, Jianwei Lu, and Ye Luo. Equalized focal loss for dense long-tailed object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 6990-6999, 2022.
Qianyi Li, Ben Sorscher, and Haim Sompolinsky. Representations and generalization artificial and brain neural networks. Proceedings of the National Academy of Sciences, 121(27):c211885111 2024.
Xuelong Li and Rubin He. Measuring the information of images (in chinese). SCIENTIA SINICA Informationis, 2024.
Yu Li, Tao Wang, Bingyi Kang, Sheng Tang, Chunfeng Wang, Jintao Li, and Jiashi Feng. Overcom- ing classifier imbalance for long-tail object detection with balanced group softmax. In Proceed- ings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 10991-11000, 2020.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C Lawrence Zitnick. Microsoft coco: Common objects in context. In Computer Vision-ECCV 2014: 13th European Conference, Zurich, Switzerland, September 6-12, 2014, Proceedings, Part V 13, pp. 740-755. Springer, 2014.
Tsung- Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature pyramid networks for object detection. Proceedings of the IEEE conference on com- puter vision and pattern recognition, pp. 2117-2125, 2017a.
Tsung-Y Yi Lin, Priya Goyal, Ross Girshick, Kaiming He, and Piotr Dollár. Focal loss for dense object detection. In Proceedings of the IEEE international conference on computer vision, pp- 2980-2988, 2017b.
Li Liu, Wanli Ouyang, Xiaogang Wang, Paul Fieguth, Jie Chen, Xinwang Liu, and Matti Pietikäinen. Deep learning for generic object detection: A survey. International journal of computer vision, 128:261-318 2020.
Yanbiao Ma, Licheng Jiao, Fang Liu, Yuxin Li, Shuyuan Yang, and Xu Liu. Delving into semantic scale imbalance. The Eleventh International Conference on Learning Representations, 2023a. URL
net /tt/ 2ttt://ooooo1111050tt-1
Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, and Puhua Chen. Feature distri- bution representation learning based knowledge transfer for long-tailed classification. IEEE Transactions on Multimedia, 2023b.
Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang. Xu Liu, and Lingling Li. Curvature-balanced feature manifold learning for cong-tailed classification. In Proceedings of the IEEF/CVE confer- ence on computer vision and pattern recognition, pp. 15824-15835, 2023c.
Yanbiao Ma, Licheng Jiao, Fang Liu, Lingling Li, Wenping Ma, Shuyuan Yang, Xu Liu, and Puhua Chen. Unveiling and mitigating generalized biases of dnns through the intrinsic dimensions of perceptual manifolds, arXiv preprint arXiv:2404.1388 2024a.
Yanbiao Ma, Licheng Jiao, Fang Liu, Shuyuan Yang, Xu Liu, and Puhua Chen. Geometric prior guided feature representation learning for long-tailed classification. International Journal of Com- puler Vision, pp. 1-18, 2024b.
Lingchen Meng. Xiyang Dai, Jianwei Yang, Dongdong Chen, Yinpeng Chen, Mengchen Liu, Yi- Ling Chen, Zuxuan Wu, Lu Yuan, and Yu-G Yu-Gang Jiang. Learning from rich semantics and coarse locations for long-tailed object detection. Advances in Neural Information Processing Systems, 36, 2024.
12

---

Published as a conference paper at ICLR 2025
Kemal Oksuz, Baris Can Cam, Sinan Kalkan, and Emre Akbas. Imbalance problems in object detection: A review. IEEE transactions on pattern analysis and machine intelligence, 43(10); 3388-3415, 2020.
Tianhao Qi. Hongtao Xie, Pandeng Li, Jiannan Ge, and Yongdong Zhang. Balanced classification: A unified framework for long-tailed object detection. JEEE Transactions on Multimedia, 2023.
Jiawei Ren, Cunjun Yu, Xiao Ma, Haiyu Zhao, Shuai Yi, et al. Balanced meta-softmax for long- tailed visual recognition. Advances in neural information processing systems, 33:4175-411 4186, 2020.
Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster r-cnm: Towards real-time object detection with region proposal networks. Advances in neural information processing systems, 28, 2015.
Li Shen, Zhouchen Lin, and Qingming Huang. Relay backpropagation for effective learning of deep convolutional neural networks. In Computer Vision-ECCV 2016: 14th European Confer- ence, Amsterdam, The Netherlands, October 11-14, 2016, Proceedings, Part VII 14, pp. 467-482. Springer, 2016.
Jingru Tan, Changbao Wang. Buyu Li, Quanquan Li, Wanli Ouyang, Changqing Yin, and Junjie Yan. Equalization loss for long-tailed object recognition. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 11662-11671, 2020.
Jingru Tan, Xin Lu, Gang Zhang, Changqing Yin, and Quanquan Li. Equalization loss v2: A
new gradient balance approach for long-tailed object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 1685-1694, 2021.
Kaihua Tang, Jianqiang Huang, and Hanwang Zhang. Long-tailed classification by keeping the good and removing the bad momentum causal effect. NeurIPS, 2020.
Kang Tong and Yiquan Wu. Rethinking pascal-voc and ms-coco dataset for small object detection. Journal of Visual Communication and Image Representation, 93:103830 2023.
Jiaqi Wang, Wenwei Zhang, Yuhang Zang, Yuhang Cao, Jiangmiao Pang, Tao Gong, Kai Chen, Ziwei Liu, Chen Change Loy, and Dahua Lin. Seesaw loss for long-tailed instance segmentation. In Proceedings of the IEEF/CVE conference on computer vision and pattern recognition, pp- 9695-9704, 2021a.
Tao Wang. Yu Li, Bingyi Kang, Junnan Li, Junhao Liew, Sheng Tang, Steven Hoi, and Jiashi Feng- The devil is in classification: A simple framework for long-tail instance segmentation. In Com- puter Vision-ECCV 2020: 16th European Conference, Glasgow, UK, August 23-28, 2020, Pro- ceedings, Part XIV 16, pp. 728-744. Springer, 2020a.
Tong Wang. Yousong Zhu, Chaoyang Zhao, Wei Zeng, Jinqiao Wang, and Ming Tang. Adaptive class suppression loss for long-tail object detection. In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 3103-3112, 2021b.
Tong Wang, Yousong Zhu, Yingying Chen, Chaoyang Zhao Bin Yu. Jinqiao Wang. and Ming Tang. C2am loss: Chasing better decision boundary for long-tai tail object detection. In Proceedings of the IEEE/CVF Conference on computer vision and pattern recognition, pp. 6980-6989, 2022.
Xun Wang, Haozhi Zhang, Weilin Huang, and Matthew R Scott. Cross-batch memory for embed- ding learning. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, pp. 088886637,.2 2020b.
Jialian Wu, Liangchen Song, Tiancai Wang, Qian Zhang, and Junsong Yuan. Forest r-cnn: Large- vocabulary long-tailed object detection and instance segmentation. In Proceedings of the 28th ACM international conference on multimedia, pp. 1570-1578, 2020.
Yuhang Zang, Chen Huang, and Chen Change Loy. Fasa: Feature augmentation and sampling adaptation for long-tailes instance segmentation. Proceedings of the IEEE/CVF international conference on computer vision, pp. 3457-3466, 2021.
13

---

Published as a conference paper at ICLR 2025
Songyang Zhang, Zeming Li, Shipeng Yan, Xuming He, and Jian Sun. Distribution alignment: A unified framework for long-tail visual recognition. In Proceedings of the IEEE/CVE conference computer vision and pattern recognition, pp. 2361-2370. 2021.
Zhengxia Zou, Keyan Chen, Zhenwei Shi, Yuhong Guo, and Jieping Ye. Object detection in 20 years: A survey, Proceedings of the IEEE, 111(3))257-27,, 2023.
14

---

Published as a conference paper at ICLR 2025
A APPENDIX
Proof 1: Integrating Local Covariance Matrices to Obtain the Global Covariance Matrix. Assume have dataset containing N instances, and divide these instances into K batches, each containing instances. For the k-th batch, let the instances be [T21, Tkng J. The mean vector and local covariance matrix for this batch are defined follows:
Rk
Pk
Tkis
Tik
Σg
Tki Ma) px)
Thk
1=1
The global covariance matrix is the covariance matrix of batches, defined as:
K
Tkis
N
K
nk
E
ahss ()( - 1.)T.
First, calculate the global mean Jr:
K
n2
K
nkf4--
N
N
Then, split (Irss μ1) in the global covariance matrix D into (IRE= (4x) and (Mo 14):
Σ
(Ire Me + Mk m)( P4 + 14 11
Expanding this, we get:
Σ
[2ks (ZK) -mx T + (0 pm) 11)²
Am
(ju (u(a - PT) + (ux - (uuu - -1)
According to the properties of the covariance matrix, the first term the local covariance matrix Ex. and the expectation values of the second and third terms are zero. The fourth term can be calculated as:
Σ (pR pmw 1n = () 11)T.
Finally, the expression for the global covariance matrix is:
Σ= N
Σ ngg E
me(u 1)) 107²)
This formula demonstrates that the global covariance matrix can be calculated by taking a weighted sum of the local covariance matrices and adding the difference terms between local means and the global mean. This integration method effecctivlyy utilizes the unbiasedness and independence of the local covariance matrices, ensuring the accuracy of the global covariance matrix.
15

---

Published as a conference paper at ICLR 2025
B SUPPLEMENTARY EXPERIMENTS ON THE PASCAL VOC DATASET
We have also included the improvement brought by IGAM to the baseline methods when using Cascade Mask R-CNN and DETR target detection frameworks. The experimental results shown in the Table 7, and it can be observed that IGAM significantly improves the performance of the baseline methods in all four cases.
Table Evaluation results on Pascal VOC.
Framework
Backbone
Loss
m. pb
Cross-Entropy (CE)
74.1
ResNet-50-FPN
IGAM Loss
78.7
Cascade Mask R-CNN
Cross cooss-Entropy (CE)
75.6
ResNet-101-FPM 101-FPN
IGAM Loss
Cross-Entropy (CE)
75.8
ResNect550FPPN
IGAM Loss
80.5
DETR
Cross Tross-Entropy (CE)
76.5
ResNet-101-FPN
IGAM Loss
81.0
16


Reviews:
[{"review_id": "cBtkXp5bkH", "reviewer": "Reviewer_NFCd", "summary": "The proposed manuscript highlights the presence of category bias even in relatively balanced datasets and introduces the Information Amount-Guided Angular Margin (IGAM) to address this issue. IGAM is designed to dynamically adjust the decision space of each category based on its information amount, thereby mitigating category bias in long-tailed datasets. Additionally, the effectiveness of the proposed method was demonstrated on the LVIS v1.0, COCO-LT, and PASCAL VOC datasets.", "questions": "Overall, the validation of the experiments is insufficient, and the authors need to analyze external factors contributing to category bias, such as scale and occlusion, which are important metrics in the COCO dataset.", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "__1.__ Overall, this paper was easy to understand.\n\n__2.__ The performance presented in Tables 1 to 5 demonstrates the effectiveness of the proposed method, although comparisons with more recent approaches are lacking.\n\n__3.__ The discussion in Section 4.7 of the paper is persuasive.", "weaknesses": "__1.__ The authors raised the problem in the introduction based on observations from the PASCAL VOC dataset, but their main results include LVIS and COCO-LT datasets. Thus, the authors need to conduct experiments across datasets such as LVIS and COCO-LT to verify that the results are consistent with the observations made from PASCAL VOC.\n\n__2.__ The proposed method was evaluated on established CNN-based models (e.g., Faster R-CNN, Cascade Mask R-CNN). However, with the emergence of various ViT-based object detection networks (e.g., DETR, H-DETR), it is crucial to assess whether the observed effects are specific to the tested models or if they generalize across different architectures.\n\n__3.__ More importantly, the authors did not compare their method with recent long-tailed object detection approaches. Notably, [Ref_1] includes experiments on both CNN- and ViT-based models and appears to outperform the proposed method. A comprehensive comparison and analysis distinguishing this work from such methods are necessary.\n\n__4.__ The same issue applies to [Ref_2]. Although the proposed method may seem reasonable at first glance, the experimental validation is not sufficiently convincing. A logical comparison and thorough analysis with recent approaches, including those presented in [Ref_1] and [Ref_2], are essential to substantiate the contributions and effectiveness of the proposed method.\n\n__5.__ Could the category bias in balanced datasets be influenced by factors such as object scale or degree of occlusion?\n\n[Ref_1] L. Meng et al., \"Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection,\" in NeurIPS 2023.\n\n[Ref_2] N. Dong et al., \"Boosting Long-tailed Object Detection via Step-wise Learning on Smooth-tail Data,\" in ICCV 2023", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "f0rT17ZvEa", "reviewer": "Reviewer_7hzz", "summary": "This paper proposed a new measurement to learn the information of different categories and then proposed the corresponding Information Amount-Guided Angular Margin (IGAM) Loss to optimize the decision space of the object detection networks.", "questions": "1. Figure 2 only shows information about two attributes. How is the class-wise average precision reflected?\n2. In formula 6, how does the operation of loss affect the compression and expansion of the decision space? For example, if i is the least informative, then the m_ij of other components j are all negative. Intuitively, it increases the second term in the denominator and thus increases the loss L, but why does this operation compress i's decision space?\n3. In the previous formulas 1 and 2, the author used variance to represent the amount of information. If m_ij is negative, it means that the amount of information (variance) of i is already relatively small, so why do we need to continue to compress its decision space?", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The paper reasonably introduces the current problems of long-tail distribution target detection.\n2. The author uses a new category information measurement method to optimize the network and fully verifies its effectiveness through extensive experiments", "weaknesses": "The paper's observations on the problem are reasonable. However, I do not feel that the method proposed by the paper is strongly related to the proposed scenarios: long-tail distribution and object detection. Here are the corresponding reasons:\n1. In the introduction stage, if the author simply wanted to propose the concept of category information, then there is no need to compare in the long-tailed scenario, and it is more direct to show the performance with information in the general scenario. If the author wants to explain it in the long-tailed scenario in Figure 1, then the performance compared with the category information can be in the previous long-tailed dataset. So I think there is no alignment here.\n2. In the first paragraph of Section 3.1, the authors mention that ‘the amount of information should be extracted from the classification module’. So why not address this issue directly in the classification process, but instead put it in the object detection scenario, and emphasize that this work is the first to directly report on the widespread bias present in object detection models in object detection? How about classification?\n3. In the experimental part, the three methods compared in Table 5 are all long-tailed methods. And they are all between 2021 and 2022. Should they be compared with newer methods? In addition, as mentioned in 1, if the author's intention in Figure 1 holds true, should this method be equally effective in the case of networks such as Faster R-CNN on relatively balanced Pascal VOC datasets?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "bvRQn9NUbl", "reviewer": "Reviewer_TGj3", "summary": "This paper aims to solve the longtail problem in object detection. The authors find that the instance amount cannot be a good metric to balance the training. They introduce the category information amount and design a new loss function with it to solve the problem.", "questions": "1. Do you have any reference to support Line 164-169?\n2. Do you maintain several individual queues for different categories, or a single queue for all categories?", "limitations": "", "rating": 8, "confidence": 3, "soundness": 3, "presentation": 4, "contribution": 4, "strengths": "I like this paper. The motivation is clear and the introduction delivers the idea very well. The novelty of this paper is good. The finding and solution are interesting, and straightforward but effective. The experiments can support their method. If all the content is original, then it deserves to be published.", "weaknesses": "Some suggestions:\n1. Fig.1 please align the class names and the axis ticks.\n2. Fig.2 is good to show the correlation between CIA and ACC, but it would be better to show the CIA figure here, or in the ablation study section. \n3. explain the symbol $m$ in the Line 170, $p$ in the Line 176.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)