Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection
Authors: Lingchen Meng
Xiyang Dai
Jianwei Yang
Dongdong Chen
Yinpeng Chen
Mengchen Liu
Yi-Ling Chen
Zuxuan Wu
Lu Yuan
Yu-Gang Jiang
Venue: NeurIPS 2023 poster (2023)
Keywords: Long-tail object detection
visual semantics
soft supervision
TL;DR: We presented RichSem, a simple but effective way to leverage classification data by learning rich semantics and coarse locations to boost long-tailed object detection.

Abstract: Long-tailed object detection (LTOD) aims to handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy is to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity---an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity---the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping.
To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional ``soft supervision'' for training detectors. Specifically, we add a semantic branch 
to our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch is only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors. 
Our method achieves state-of-the-art performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method on other long-tailed datasets with additional experiments.

Learning from Rich Semanties and Coarse Locations for Long-tailed Object Detection
Lingehen Meng"* 2.2 Mengehen Liu
Xiyang Dai
Jianwei Yang" Dongdong Chen Yinpeng Chen
Yi-Ling Chen Zuxuan Wu Lu Yuan
Yu-Gang Jiang
"Shanghai Key Lab of Intell. Info. Processing, School of CS, Fudan University Shanghaai Collaborative Innovation Center of Intelligent Visual Computing 3 Microsoft
Abstract
Long-tailed object detection (LTOD) aims handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping- To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional "soft supervision" " for training detectors. Specifically, we add semantic branch our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backhones and detectors. Our method achieves sfate-of-thheart performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method other long-tailed datasets with additional experiments. Code is available at https: //t//ggittt giggth..bb... com/ /ttp///eeoooooiiiii/ooooo RhSem....
Introduction
Object detection for complex scenes has advanced significantly [10, 9, 40, 3, 65) thanks to large-scale datasets [5, 22, 32, 12, 24, 4, 42]. However, current deep models depend on relatively balanced large-scale datasets, where different classes have similar numbers of images and samples, to learn diverse semanties and enough locations. This limits their performance on real-world data, which are often long-tailed, i.e., only a few head classes have plenty of training samples while most classes have very few training samples, making detection more challenging and less effective for rare classes.
A simple and effective way to improve long-tailed object detection (LTOD) is to use extra data to increase the training samples for tail classes. However, collecting bounding box annotations, especially for rare categories, is costly and tedious. Therefore, previous studies resort to datasets with image-level labels to enrich the amount of samples for rare classes by exploring image-level semantics (as shown in Figure (a)). While appealing, directly learning from such data to benefit detection is challenging since they lack bounding box annotations that are essential for object detection. To remedy this, many works [66, 25, 60, 63] focus on estimating bounding boxes for
Corresponding author.
37th Conference on Neural Information Processing Systems (NcurIPS 2023).

---

(a)
(b)
"Bal" in ImageNet
"sugar bowl
boy Rostwa ball Hlog
"piggy bank
Image count
Ietge
LV1S
Rand
lvgo 10:30/ D0
100
frequent
common
rare
sorted id
Figure 1: (a) Category occurrences in the long-tail detection data (LVIS) and the extra classification data (ImageNet-21kk. The green bars show the number of images that each category occurs in LVIS [12]; the blue dots show the number in ImagetNet-21k [5]. Evidently, the classification dataset exhibits a more evenly balanced distribution of occurrences. Moreover, the classification dataset offers broader range of instances. thereby enhancing the diversity of instances for each category. (b) A sample from ImageNet [5]. This sample annotated as "ball" while ignoring another main object "dog". After a random crop during training, the "ball" may be inaccurate on some crops. The class with the highest CLIP confidence score is shown above each crop.
objects in images. They typically consider image-level tags (in the form of one-hot labels) as the ground-truth classes and match with best estimated boxes as pseudo labels for training.
We argue that image-level labels are not well-suited for detection due to their semantics ambiguity and location sensitivity. On one hand, an image-level label can not sufficiently reflect all the semantics within the image. On the other hand, the label mainly focuses on the iconic object; thus, using a different crop might shift the semanties to a different object. Take Figure (b) as an example: the top image is labeled as "ball" in ImageNet [5]. which only depicts part of information in the image. The panel below in Figure (b) shows that the provided image-level label is no longer accurate after data augmentations. Inspired by the recent success of visual-language contrastive models that align a large number of image-text pairs [37, 62, 11, 54, 13], we aim to leverage such models to extract rich semantics that are more informative than the image-level labels in classification datasets. However, naively converting one-shot labels in image classification tasks to a distribution of soft labels is still not optimal and accurate after data augmentation. As shown in Figure I (b), based on different locations (i.e., random crops) of the image, the semantics extracted by CLIP [37] incur more noise than the original image.
To address this, we introduce RichSem, a one-stage training framework that leverages additional image data to boost the detector through learning from rich semantics and coarse locations for long-tailed object detection. In particular, we treat whole-image box as coarse locations (i.e., the coarse bounding box shares the same size as the image) and group multiple images together to build a mosaic. We then use CLIP to extract the semanties according to such coarse bounding boxes. The extracted semantics can serve as "soft labels" to enrich the amount of training data and are more robust to random cropping. We further introduce new branch named as "semantic branch" to object detectors so that it can learn from the derived semantics during training. This allows the detector to fully leverage the rich semanties in classification datasets with minimal modifications Once trained, this branch is discarded during inference, and RichSem can be used readily as a standard detector.
Our contributions are summarized as follows: (1) We point out that using image-level labels from extra classification datasets as supervision is challenging due to semantics ambiguity and location sensitivity, which can be alleviated with distribution of semantics serving as "soft labels". We show that semantics provided by CLIP are not only rich and but also robust to locations, providing better guidance than original image-level labels. (2) We introduce novel semantics learning framework named RichSem, which uses an additional branch to learn from rich semantics and coarse locations for long-tailed object detection without the need to compute pseudo labels. Once trained, the additional branch can be discarded during inference. (3) Our method demonstrates strong results on long-tailed datasets, e.g. LVIS, highlighting that is low-cost way to boost the detector performance for long-tailed object detection by only complementing extra classification data.

---

2 Related Work
Long-tailed object detection (LTOD) has attracted more and more attention. The performance of rare categories is drastically low compared to frequent categories due to the unbalanced distribution and the lack of training samples. Existing works can broadly be divided into two directions: 1) one direction aims to improve the training scheme for balanced learning, including data re-sampling [12), loss re- weightting [46, 45, 51], data augmentations [8] and decoupled training [23, 21]; 2) Another direction leverages extra data to compensate for the data starvation [63, 60]. These methods often trust image-level level- labels to boost the classification capability, We find that is far from optimal to naively treat image-level labels as golden labels and supervision for the classifier. Unlike existing methods, we study a new solution that leverages the rich semantics within the images from classification data with only coarse locations provided by regular augmentations. We leverage CLIP to provide semantics as soft targets on classification data to guide our detector to learn semantics explicitly. In this embarrassingly simple but effective way, we can better leverage classification data for LTOD.
Weakly-supervised object detection aims to train a detector using image-level labels without bounding boxes. Many studies [1, 47, 36] train model using only image-level labels without any bounding box supervision. Another line of work [39, 60] takes the bounding boxes supervision with the whole-image labels together under semi-supervised framework. Unlike prior works, we focus on mining rich semantics within the images instead of bounding box estimation. Thus, we no longer need to estimate precise bounding boxes on classification data.
Language supervision for object detection is a recent topic that aims to leverage linguistic semantics. Since language supervision has rich semantics, each category is related rather than independent in one-hot labels. Thanks to this property. recent works [6. 59, 57, 20, 35, 55] show benefits by pre- training backbones on vision- language tasks. With the rapid progress in contrastive lunguage-iman pre-training [37, 18], many recent approaches [28, 11, 62] apply large-scale pre-trate vre-trainin for object detection. ViLD [11] and RegionCLIP [62] attempt to align the visual-semantic space of a pre-trained CLIP model for open-vocabulary detection. Similar to RegionCLIP and ViLD, our method leverages the
space learned from pre-trained CLIP models. In contrast, our goal is to leverage object semantics to boost object classification, especially for rare categories. Since CLIPs are used to generate "soft labels", our backbone is free of CLIP initialization compared with RegionCLIP [62].
Knowledge distillation and soft label. Knowledge distillation (KD) [17] is a powerful tool to boost the student model with prior knowledge of the teacher model. Recent follow-ups extended and developed many variants, e.g., feature distillation [19, 41]. hard distillation [50]. contrastive distillation [48], etc. To improve the training efficiency of KD, Re-label [58] and FKD [44] use strong teacher model to generate soft labels and store them for efficient image recognition training. Recently, many studies [11, 34, 62, 38] introduce knowledge from CLIP [37] into the KD framework to boost open-vocabulay object detection. Due to the strong semantics capturing capability of CLIP, those methods show encouraging performance open-vocabulary detection. Similar to those works, our approach leverages the semantic knowledge of pre-trained CLIP models. In contrast, our goal is to boost the long-tailed detection, especially for the tailed classes, with the help of rich semantics of classification and detection datasets. Moreover, we introduce a simpler but more effective way to learn visual semantics with an extra semantic branch during training. Unlike [62, 11]. our one-stage training scheme needs no fine-tuning after pre- pre-training. Besides, our approach is more effective since there are no redundant pre-computed boxes [11] and boxes estimation [34, 38).
3 Method
Given a detection dataset denoted as Dod = [(Tod ((bodd (dd) where each image lod is associated with bounding boxes bod and class labels add respectively. Our goal is to leverage an additional classification dataset Dentra = {(iing, ) of which each image is only labeled with image- level label cimg, to improve long-tailed object detection, particularly for rare classes.
In long-tailed object detection, previous approaches typically rely on image-level labels and compute pseudo boxes for corresponding objects. On one hand, the semantic information provided by such labels is limited, (i.e., one one-hot label per image). On the other hand, augmentation strategies like randomized cropping might generate regions that do not contain the annotated class, thus making the provided labels no longer accurate. Motivated by the fact that CLIP has strong capabilities of
3

---

capturing visual semantics conditioned on only coarse locations [62], we build upon CLIP to explore image classification datasets to guide the training of object detectors. Below, we first introduce how to obtain rich semantics from CL.PP in Section 3. and the resulting semantics are then used to guide the training our detector, as will be described in Section 3.2. Then we describe how we unify the training objective and extend RichSem to different types of extra datasets in Section 3.3.
Bod
mixed dataset
original detector for training/ inference
loc
Hive
D
pols
Backbone & Decoder
cls
Hels
ball, sandal ball, ball, dog
ball, ball
f50ft
ghmg
s0ft
H1 ao
Contrast
semantic space
V4
Pooler
Deztra
bal
dog
poul
supervision
dog, puppy. ball, pel, loy, frisbee,... ...
PromptEng P
C
forward
categories Cod
semantic branch for training/ not used in inference
Figure 2: Our RichSem framework. V. and indicate vision encoder and text encoder of the CLIP model. We leverage rich semantics in extra data with image-level labels by pre-trained CLIP. Then semantics behave like "soft labels" for our proposed semantic branch (Haaft followed by Contrast). Blue parts indicate the components for traditional object detection and green parts indicate those for semantics learning.
3.1 Image and Object Semantics
We aim to use image-level labels in classification datasets together with detection datasets to improve long-tin long-tailed object detection. This requires processing these types of data within the same framework. As a result, we treat images from detection and classification datasets equally and we not only derive semantic information at the image-level for classification datasets but also at the object-level for detection datasets, as will be elaborated below.
Image-level semantics. We denote the pre-trained CLIP visual encoder and the language encoder as V; and L., respectively. Consider an image after random cropping, its semantics 8 can be obtained by computing the similarity between its visual features fimo extracted by the visual extractor and the linguistic categories features feats produced by the language encoder. Formally, we compute the visual and linguistic features follows:
fimg V. (Timg): peats = (P(P(C)))
kmg Contrast( fimm,, feats).
Here, P is the prompt engineering function to convert class names into prompts; Contrast is the contrasting paradigm [37] to calculate the similarity between two features, and simg contains semantics information in which each element indicates the likelihood for the corresponding class. Following [37, 62], we use a set of prompts convert the categories in the target vocabulary Col into sentences then use the CLIP text encoder L, to obtain the linguistic features of categories in Cod
Object-level semantics. As mentioned above, to unify the training process with classification and detection data, we also obtain object-level semantics from CLIP as well. To this end, we obtain an object-level representation by pooling from image-level features [9, 15] rather than cropping in the original images, following [62]. More formally, object-level semantics are obtained as:
ft (Vo)
(2)
s Contrast feats)
Please refer our supplementary material for the verification of location

---

where Pooler is RoIAlign [15] that pools region features according to their locations from the entire feature maps, and 6 the location of the object, which could be mound-truth, predicted, or predefined whole-image bounding boxes.
It is worth pointing out with such a formula both detection and classification datasets now be unified, as objects in detection datasets are essentially regions of images, while images from classification datasets are cropped regions from original images. Consequently, the b in Equation (2) indicate tight ground-truth bounding boxes 6od for objects in detection datasets, while b corresponds to the entire image, Le., coarse whole- -mage boxes bimg = (0, O, h, w). for samples from classification datasets, where h, w is the height and width of the augmented image.
Furthermore, unlike previous approaches that use fixed and static semantics provided by image-level labels, we obtain image-level and object-level semantics which dynamically change conditioned on different locations (ie., bounding boxes) in an online fashion. This is particularly useful when location information is not accurate- location could be very loose bounding box for the object of interest or even contains part of the objects. As result, the rich and location-robust semantics can be used to guide the detector.
3.2 Semantics as Soft Labels
Now that we obtain both image-level and object-level semantics with CLIP, the main challenge is how to make our detector learn from them effectively. Traditional detection models [40, 31, 3, 65] usually obtain object features from whole-image features according to locations [9, 40, 31, 49] or arossaattnttion [3, 65]. Then object features are refined with a classification branch classify the region into predefined categories, and a location branch to compute the bounding box, separately. To unify the notations, we denote the object features O, which are fed to different branches to obtain features tailored for each sub-task to generate final outputs with a projection layer. More formally,
floe, fols = (O), (O)
doe, ools (foc)) Wats(fale).
where denotes the sub-task branch for feature refinement, specifically Hor is for localization and Hat is for classification; W, denotes the projection layer to produce final results; fi, and denotes the features and outputs for the corresponding task, i.e., E {loc, cls}.
Now discuss how use the obtained semantics to guide the training of detectors. As ocle logits that are generally used for classification, a straightforward way is, on top of a cross-entropy loss, to use ddls to predict the object semantics s' in similar spirit to knowledge distillation [17, 41]. However, this makes the training of detectors challenging as the cross-entropy loss and distillation loss have confficting purposes: a cross-entropy loss is generally optimized to produce hard prediction results while the distillation loss aims to borrow knowledge from a distribution of semantic scores that serve "soft labels". As result, jointly performing hard predictions and soft distillations conditioned on dl produce unsatisfactory results, as will be shown empirically.
To mitigate the challenge, we introduce additional branch named as "semantic branch", independent of the classification and localization branches in current detectors, to learn the soft semantics obtained by CLIP. The semantic branch also includes feature refinement branch Hooft then feed the refined feature fsoft to Contrast function obtain the semantic prediction =off Such a strategy ensures that semantics from image classification datasets can be explored without interfering with the original training process of detectors. This is essentially using semantics as soft teachers implicitly refine object features for detection. Formally, to train the semantic branch, we use a KI. divergence loss as:
00ft +Contrast(f fooft, foat), froft (Os
N
1
Looft
Σ LKz(o .
N
where of goft is the semantics prediction of the i-th object feature, while of is the corresponding semantic target; N is the number of matched proposals/ queries during training. It is worth noting that we only use the semantic branch for distilling semanties into the detector during training. And once trained, the semantic branch is no longer needed for inference.
5

---

3.3 Unified Objective Functions
We aim to incorporate semantic learning from classification datasets into an end-to- learning scheme instead of fine-tuning after pre-training [62], without redundant pre-computed boxes [1 I] and box estimation [34, 38]. Thus, we unify the objective functions and treat datasets of classification and detection equally in unified way.
In training scheme, classification loss and a soft semantics loss, to boost the capability of object classification. The hard classification loss Lots best works when bounding boxes and class labels are available and sufficient for training. In contrast, the soft semantic learning loss Leoft works well for those categories with few samples, requiring only coarse locations.
unified classification loss Tumicis the combination of hard
L = Mooe Lioc
dets
Lote Asoft Lsoft
(5)
Irunicts
where Atoe, dets and Asoft means the weight of each loss.
Furthermore, the vocabularies of target detection datasets and extra classification datasets often differ. To handle the laxonomy difference between detection and classification datasets, we further unify the supervision for hard classification as follows.
Handling taxonomy differences. To leverage image-level labels in extra data, we need to map the vocabulary of extra data Cextra to the vocabulary of our target detection dataset Cod which is function M Cextra >> Cod One could derive better mapping functions M. yet this orthogonal to manual label mapper leverage the CLIP text
current direction. For convenience, encoder to map the class according to their category semantics automatically.
cod M(cing)
Handling data without any labels. We also consider broader case, in which even image-level labels are not available. Inspired by DeiT [50], we treat the class of highest logits in each semantics target as the label for classification. We also filter hard with threshold filter those images relevant to the vocabulary of the target set.
arg max if conf > th else
hard
where chard is the generated class label for hard classification.
Experiments
We conduct experiments and analysis on the task of long-tailed object detection. We mainly evaluate our method on LVIS [12] val 1.0 for our main experiments and ablations. We also conduct experiments on other datasets of long-tail distribution to further prove the effectiveness. We use DINO [61], advanced DETR-base based detector due to the training efficiency and high performance.
4.1 Datasets and evaluation metrics.
Long-tail detection dataset. We mainly conduct experiments on LVIS [12], which contains 1203 classes with ~ 100K images. The classes are divided into rare, common, and frequent groups based on the number of training images. The category distribution is extremely long- long-tailed since instances in rare categories are usually less than 10. Moreover, we experiment on other datasets, e.g. Visual Genome [22] and OpenImages [24], please refer to our supplementary material.
Extra data. We mainly use ImageNet-21k [5] as additional classification data. ImageNet-21k consists of ~ 14M images for 21K classes, and there are 997 classes that overlap with the vocabulary in LVIS. We follow the label mapping used in Detic [63] and denote the subset of overlapped classes as ImageNet-LVIS,, which contains 115M images with 997 LVIS classes. For the full set, we treat the data as unlabeled images and leverage our learning scheme to learn from it. We also treat the

---

Method
Backbone
Schodule
AP
APse
APrs
APr
AP:
AP
DINO*[61]
[16]
38.4
MosaicOS7 [60]
R30 [16]
1x
25.0
40.8
26.5
20.2
23.9
28.3
Detic-DDETR [65]
R30 [16]
4x
31.7
21.4
30.7
37.5
Detic-DDETRf [63, 65]
R30 [16]
4x
32.5
26.2
31.3
36.6
Detic CenterNet2 [63]
R30+ [16]
4X
35.3
48.7
37.2
28.2
33.8
40.0
Detic CenterNet29 [63]
R50+ [16]
4X
36.8
50.7
38.6
31.4
36.0
40.1
RichSem (Ours)
R50 [16]
1X
32.2
42.3
33.9
24.1
29.9
38.3
RichSem (Ours)
R50 [16]
2x
34,9
45.5
36.6
26.4
32.5
41.3
RichSem (Ouns)
R50
3x
35.1
45.8
36.8
26.0
32.6
41.8
(Ours)
[16]
30.416.3
(Ours)
[16]
37.112.2
29.913.5
RichSom $ (Ours)
R50 [16]
2
40.115.2
51.9
36.2111.8
38.2
44.0
RichSem (Ours)
Swin: [33]
1X
34,9
45.5
36.9
26.0
32.6
41.3
RichSem 3 (Ours)
Swin-T [33]
1x
38.313.4
49.8
40.4
38.1
36.3
42.3
RichSem (Ours)
Swin-T [33]
2x
38.8
49.9
41.0
30.8
36.4
45.0
RichSem & (Our))
Swin-T [33]
2x
41.61
53.3
43.8
37.316.5
39.7
45.5
Ceiie-CenterNe22 [63]
Swin-B [33]
4x
45.4
59.9
47.9
39.9
44.5
48,9
(eiieCeenerNeee2[33]
Swin-B [33]
4x
46.9
62.2
45.8
45.5
49.0
(Ouns)
[33]
2x
38.5
2 (Ours)
Swin [33]
2x
48.21
46.518.0
VITDel [29)
UT.LMMAL [14]
~88
51.27 49.2
VITDet [29]
VIT-H-MAE [14]
~8
53.4/
RichSem (Ours)
Swin-L [33]
1x
47.0
39.9
49.6
41.2
45.9
50.7
RichSem $ (Ours)
Swin-T. [33]
1
49.8
63.7
52.5
48.617.4
49.7
50.5
RichSem (Ours)
Swin-L [33]
2x
49.7
62.9
52.4
42.8
49.2
53.4
RichSem (Ours)
Swin-L 2x
52.012.3
65.7
54.8
50.217.4
51.5
53.3
(Ouns)
Focal-L||56
(Ours)
[ocal-L[[56]
51,512.0
52.315.0
RichSem (Ours)
Focal-L [56]
3x
51.4
64.8
54.2
47.6
49.8
54.9
RichSem 3 (Ours)
Focal-L 156]
3x
53.612.2
67.2
56.7
52.815.2
52.6
55.2
Table 1: Results LVIS val v1.0. indicates that train the model LVIS using their official hyper- parameters best trained on COCO; R50 indicates ResNet50 [16] pre-trained ImageNet- 1k, while R50+ indicates pre-trained on ImageNet-21k: RichSem (Ours) indicates that we train our baseline model under original object detection dataset with traditional loss; RichSem $ (Ours) indicates using additional extra data under training scheme. Besides, Pindicates that the model is trained with extra data, In this table, models the subset of ImageNet-2II,, of which categories overlap with LVIS vocabulary extra dataset. The AP of ViTDet includes the AP reported paper (the former) and the AP reported in their official repo (the later).
full set as unlabeled dataset ignoring the image-level labels, denoted as INet-Unl. Furthermore, we explore additional detection dataset Object365 [42] and an image-text pair dataset CC3M [43].
Evaluation metric. We report box AP on LVIS with the LVIS official evaluator, which also contains AP,, AP.. AP, for rare, common and frequent class, respectively. We mainly focus on the performance gain on overall AP and AP of rare categories.
4.2 Implementation details
Baseline model. We take DINO [61]. a powerful DETR- detector with advanced query denoising [27] as our baseline. Differently, we convert the object classification into vision-language contrasting using the text features of categories.
Mixed dataset training. For experiments with extra data, we combine both detection datasets and extra datasets together to train our detector. We sample images from detection and classification datasets in 1:1 ratio regardless of the original size of each dataset. Images in the same batch are sampled from the same dataset for high training efficiency. For mixed dataset training, we denote the schedule as the number of iterations on the target dataset for fair comparisons.
Training details. We adopt PyTorch for implementation and use 8x V 100 GPUs. We set the initial learning rate as le-4 and multiply 0.1 at the 11-th, 20-th and 30-th epoch for 1x, and 3x, respectively, and set Asoft 0.5 for the soft semantics learning loss. Following [63], we use federated loss [64] and repeat factor sampling [12] for LVIS; we use category aware sampling for OpenImages. We randomly resize input image with its shorter side between 480 and 800, limiting
7

---

the longer size below 1333. Unlike training large models with larger scale images [29, 61], we use the same recipe for all models without any tricks or test time augmentation (TTA), For training with extra data, we use CLIP-RN50 to extract the semantic guidance for most models, while we use CLIP-RN5Ox 16 [37] for Swin-I. [33] and Focal-L. [56] for better performance.
4.3 Main Results
Results on LVIS. Tab. shows the result on LVIS. As shown in the table, detectors trained with our framework outperforms those trained with regular training recipes, especially for rare categories. When using ResNet50 [16] as backbone networks, our baseline model is fully converged under 3x schedule. Our RichSem under 2x still outperforms the fully converged baseline, which demonstrates the effectiveness of our proposed method. Moreover, the experiments with Swin [33] backbones show consis- tent gains. Our model with a Swin-L as backbone achieves 52.0 AP and 50.2 AP,, which outperforms the previous SoTA [29] by a large margin only under 1/4 training schedule. Overall, our RichSem makes the detector better on tailed categories and achieves significant gains the corresponding AP, e.g. rare and common categories. Moreover, with better back- bones, our RichSem achieves even balanced perfor- mance on rare, common and frequent categories. Notably, with a much smaller model size and standard data augmentations for training and testing, our method rivals the ViT [29] that uses huge- sized model and sophisticated training recipes. We also compare compare the models with Faster R-CNN [40] as detector shown in Table We further compare with large-scale vision foundation models [7, 53], please refer to our supplementary material.
Method
AP
AP,.
APc
AP
Faster R-CNN [40]
24.1
14.7
22.2
30,5
EQL. [45]
25.5
17.5
23.9
31,2
BAGS [30]
26.0
17.2
24.9
31.1
Scesaw Loss [51]
26.4
17.5
25.3
[26]
27.5
20.2
26.1
MosaicOS [60]
23.9
15.5
22.4
29.3
CLIS t
29.2
24.4
28.6
31.9
RichSom 1 (Ours)
30.6
27.6
29.7
32,9
Table 2: Results LVIS val v1.0 with Faster R-
CNN as detector. All models use R50 as backbone. 't' indicates using extra classification data.
4.4 Ablation Study
Comparisons with CLIP initialization. We replace the ResNet50 backbone with pre-trained CLIP visual encoder study whether it can lead to balanced learning. As shown in Table 3a, although pre-trained CLIP has a strong ability to capture visual semantics, the detection performance even drops after finetuning on LVIS. The low AP, of CLIP backbone indicates that is still biased towards the long-tailed distribution of downstream datasets. Therefore, using a frozen pre-trained CLIP to extract visual semantics extra "soft supervision" can keep the generalization capability than building upon CLIP as initialization.
Effectiveness of the semantics branch. We study the importance of the proposed semantics branch by sharing the parameters of two heads instead of using two independent heads. As shown in Table 3b, when two heads share the parameters, the performance drops drastically and is even worse than the baseline model. This suggests that the objective of semantics learning differs from object classification. Therefore, semantics should be learned independently with the semantics branch.
Effectiveness of rich semantics of classification data. We conduct experiments using different types of extra data. Initially, we utilized Object365 [42], detection dataset, and utilized its box annotations irrespective of the class labels. However, the performance gain was limited, as shown in Table 3c. This further verifies the assumption that the localization capability is not the primary bottleneck for long-tailed object detection [1]]. Next, we experiment with ImageNet [5] and CC3M [43], two datasets with whole-image labels. We treat these datasets as unlabeled data and generate pseudo labels using Equation (7), denoted as INet-Unl and CC3M-Unl, respectively. As shown in Table 3c, utilizing these two datasets as extra data significantly outperformed the baseline, particularly for the rare categories. Furthermore, we employed ImageNet-LVIS, taxonomy mapped using Equation (6). as extra data, which yields the best performance. Overall, the experiments demonstrate that long- tailed object detection is primarily affected by object classification. Consequently, introducing rich semantics within classification data effectively alleviates this limitation. It worth noting that this showcases our potential for further extension to unlabeled data.
8

---

Method
AP
AP,
Extra branch
AP
AP.
AP,
baseline
32.2
24.1
baseline
32.2
24.1
29.9
38.3
CLIP backbone
30.6|1.6
22.2|1.
X
31.1111
24.670.5
28.3|1.
37.2/1.1
RichSem #
35.012.8
30.416.3
35.012.8
30.476.3
33.113.2
39.010.7
(a) Compared with CLIP initialized bascline.
(b) Ablations on the semantic branch.
Dextro
CLIP models
AP
AP,
Extra data
Loots AP
AP-
Nome
32.2
24.1
Nome
None
32.2
24.1
0365-Box
33.010.8
24.810.7
RNS0
35.072.8
30.476.3
X
33.81
26.912.8
CC3M-Unl
14
28.714.6
RN50x4
36,013.8
33.018.9
INET-LVIS
/
3..07
30.416.3
INer-Unl
34.712.5
28.614.5
RN50X 16
366224400
31.977.8
X
32,970.7
23.7.00.
INET-UBl
INer-LVIS
35.012.8
30.416.3
34,712.5
28.61
(c) Different extra datasets.
(d) Soft semantics provided by different CLIP models.
(e) Soft semantics loss Leoft>
Diackhone
D
AP
AP.
AP.
AP
Meihod
AP
AP,
AP;
AP
IN-IK
32.2
24.1
29.9
Deztre
32.2
24.1
29.9
IN-2IK
LVIS
35.7
25.9
35.0
40.7
+Loofs
33.6
28.674.5
32.4
37.2
IN-IK
LVIS-IN-2IK
35.0
30.
33.1
39.0
+ +Loc
35.0
30.4118
33.1
39.0
IN-21K
LVIS+IN-2IK
37.5
32.4
36.0
41.5
(f) Ablations on backbone pre-training data and down- stream detection data.
(g) Lioft and Lioe on extra data.
Azofe
AP
AP,
APc
AP,
AP,
AP
AP,
Only LVIS
with Image-LVVS
0
32.2
24.1
29.9
38.3
33.8
26.912.8
31,5
39.4
0.2
32.4
23.8,0.3
30.510.6
38.5/0.2
35.0
29.014.9
33.313.4
39.47
32.4
25.010.9
30.510.6
37.7,006
30.416.3
33.113.2
39.070.7
1.0
32.1
27.012.9
29.8.00.1
36.9.14
34,8
29.515.4
33.413.5
38.770.4
(h) Ablations on the weight of soft semantics loss.
Table RichSem ablations. ablations are performed with RN50 as backbone under 1x schedule.
Stronger semantics lead to better detectors. We study the impact of CLIP with different backbones. The results are shown in Table 3d. With the capability increased, the model achieves better perfor- mance on overall AP. Besides, built upon a RN50x 16, detector obtains significant improvements on all metrics.
Semantic learning is effective to leverage extra data. Due the data overlap between backbone pre- training and detector training, we conduct an ablation on them to further demonstrate the effectiveness of method. As shown in the Table 3f, pre-training on large-scale data (ImageNet-211 can provide strong perception capability for the downstream detection task, with overall performance gain. However, the performance does not well alleviate the long-tail effects in detection. In contrast, our method is more effective than pre-training handle long-tailed detection, especially for the tail categories. Notably, our approach still effective with strong pre-trained backbones, further improving performance on long-tailed object detection.
categories is still relatively low, indicating that this approach
Semantics as soft targets leads to balanced learning. The effectiveness of soft labels for low-shot categories can be reaffirmed in Table 3e: when using extra data, detectors trained with the soft semantics learning loss achieve better performance than those trained without the loss, especially for rare categories and unlabeled extra data. Besides, our semantic learning on extra classification boosts detection and classification at the same time thanks to our unified objective functions (Section 3.3). As shown in Table 3g, both rich semanties and coarse locations play significant roles in boosting toil object detection. We further study the impact of soft classification objectives. We first experiment without using extra data. As shown in the Table 3h, with distillation weight to the performance of rare categories increases by ~3AP while the results of frequent categories decrease by 1.4. This suggests that semanties learning benefits those low-shot categories rather than overfitting frequent categories. As the distillation weight decreases, the performance of rare classes also drops but the results of frequent categories improve. After exploring additional data, the performance no longer sensitive to the distillation weight. Thanks to the additional data, semantics learning leads to

---

PM5G
SPEYY
QUNER
DAVERINO SYL
Figure 3: Qualitative results of our RichSem. We show the images containing rare categories from LVIS val 1.0. We show rare categories in red and show others in black. Best viewed sceen.
feature distribution
dragontly (1)
be (f)
basketbal (c)
protzel(e)
(1)
mailbox home) (t)
Booeline
ReitSem
Figure 4: Feature visualization between baseline and RichSem. We randomly sample classes each from the categories rare, common, frequent for visualization. Our RichSem shows well-clustered result across different categories.
consistent performance gains for all categories. We further visualize the object features of our method. Specifically, normalize the features and employ Gaussian Kernel Density Estimation (KDE) in RR,, following [52] to compare the distribution of object features across categories. As shown in the Figure 4, the visualization indeed shows clear distinction between the baseline and our RichSem. Regarding the baseline, the distribution of object features lacks differentiation, often resulting in overlapping patterns among categories, especially between in RichSem, features belonging to each category, even rare categories, are well-clustered. This clear intra-class and inter-class distribution indicate that our approach effectively enhances the region classification capability of diverse categories. Therefore, our method effectively leads balanced leamning among classes with varying frequencies.
and frequent categories. In contrast,
5 Conclusions and Future Work
We presented RichSem, simple but effective way, to leverage extra data by learning rich semantics and coarse locations to boost long-tailed object detection, alleviating the semantics insufficiency and location sensitivity caused by taking image-level labels as supervision. Through extensive experiments, we demonstrate our approach achieves state-of-the-aat performance, without requiring complex training and testing procedures. One possible limitation is that we treat the detection data and classification data equally and use the same unified classification loss, which may be sub-optimal for the categories with sufficient samples.
For future work, we believe our framework can be extended to semi-supervised object detec- tion (SSOD) where golden labels are not annotated on the unlabeled data, since our method is found to be robust when annotations are partially given. Moreover, the soft semantics learning for classification data can be naturally applied for open-vocabulary and multi-dataset detection tasks.
Acknowledgement This project was supported by National Key R&D Program of China (No. 2021ZD01 12805) and National Natural Science Foundation of China (No. 62102092).
10

---

References
[1] H. Bilen and A. Vedaldi. Weakly supervised deep detection networks. In CVPR, 2016.
[2] Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR, 2018.
[3] N. Carion, Massa, Synnaeve, N. Usunier, A. Kirillov, and Zagoruyko. End-to-end object detection with transformers. In ECCV, 2020.
[4] S. Changpinyo. Sharma, N. Ding. and R. Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. CVPR, 2021.
Deng, W. Dong, Socher, L.JJ. Li, Li, and L.. Fei-Fei. Imagenet: large-scale hierarchical image database. CVPR, 2009.
K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations, In CVPR, 2021.
[7] Y. Fang, W. Wang, B. Xie. Q. Sun. L. Wu, X. Wang, T. Huang. X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR, 2023.
[8] G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y. Lin, E. D. Cubuk, Q. V. Le, and B. Zoph. Simple copy-paste is strong data augmentation method for instance segmentation. In CVPR, 2021.
[9] R. Girshick. Fast r-cnn. CVPR, 2015.
[10] R. Girshick, Donahue, T. Darrell, and Malik. Rich feature hierarchies accurate object detection and semantic segmentation. In CVPR, 2014.
[11] X. Gu, T.-Y. Lin, W. Kuo, Cui. Open-vocabulary object detection vision and language knowledge distillation. ICLR, 2021.
[12] A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR, 2019.
[13] G. Han, Ma, S. Huang. L. Chen, R. Chellappa, and S.-F. Chang. Multimodal few-shot object detection with meta-learning based cross moosl prompting. arXiv preprint arXiv:2204. 07841, 2022.
[14] K. Hc. X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders scalable vision learners. In CVPR, 2022.
[15] K. He. G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In ICCV, 2017.
[16] K. He, X. Zhang. S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR, 2016.
[17] G. Hinton, O. Vinyals, Dean, al. Distilling the knowledge in a neural network. arXiv preprint 2015.
[18] C. Jia, Y. Yang, Y. Xia, Y.-T. Chen, Z. Parekh, H. Pham, Q. Le, Y.H.. Sung, 7. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML, 2021.
[19] X. Jiao, Y. Yin, L. Shang, X. Jiang. X. Chen, L. Li, F. Wang, and Q Liu. Tinybert: Distilling bert for natural language understanding. In EMNLP, 2020.
[20] A. Kamath, M. Singh, Y. LeCun, G. Synnacve, I. Misra, and N. Carion. Mdetr-modulatee detection for end-to-end multi-moda understanding. CVPR, 2021.
[21] B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis. Decoupling representation and classifier for tailed recognition. In ICLR, 2019.
[22] R. Krishna. Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y. Kalantidis, L.-J. Li. D. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV, 2017.
[23] J. Kuen, F. Perazzi, Z. Lin, J. Zhang, and Y.-P. Tan. Scaling object detection by transferring classification weights. In CVPR, 2019.
[24] A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings. I. Krasin. J. Pont-Tuset. S. Kamali, S. Popov, M. Malloci. A. Kolesnikov, et al. The open images dataset v4. IJC, 2020.
[25] D.-H. Lee et al. Pabudo-label: The simple and efficient coi-supperiieed learning method for deep neural networks. In Workshop on ICML, 2013.

---

[26] B. Li, Y. Yao, J. Tan, G. Zhang, F. Yu, Lu, and Y. Luo. Equalized focal loss for dense long-tailed object detection. In CVPR, 2022.
[27] F. Li, H. Zhang, S. Liu, J. Guo, M.M. Ni, and L. Zhang, Dn-detr: Accelerate detr training by introducing query denoising. In CVPR, 2022.
[28] L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.NN. Hwang, et al. Grounded language-image pre-training. In CVPR, 2022.
[29] Y. Li. H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In ECCV, 2022.
[30] Y. Li, T. Wang, B. Kang. S. Tang, C. Wang. J. Li, and J. Feng. Overcoming classifier imbalance for long- tail object detection with balanced group softmax. In CVPR, 2020.
[31] T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV, 2017.
[32] T.YY. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.
[33] Z. Liu, Y. Lin, Y. Cao, H. Hu, Y. Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV, 2021.
[34] Z. Ma, G. Luo, J. Gao, L. Li. Y. Chen, S. Wang, C. Zhang, and W. Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In CVPR, 2022.
[35] L. Meng, X. Dai, Y. Chen, P. Zhang, D. Chen, M. Liu, J. Wang, Z. Wu, L. Yuan, and Y.GG. Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In CVPR, 2023.
[36] L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR, 2022.
[37] A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin,
J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML, 2021.
[38] H. Rasheed, M. Maaz, M. U. Khattak, Khan, and F. S. Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. In NeurlPS, 2022.
[39] Redmon and A. Farhadi. Yolo9000: better, faster, stronger. In CVPR, 2017.
[40] S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurlPS, 2015.
[41] V. Sanh, L. Debut, Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint 1r0iv:1110.00100 2019.
[42] S. Shao, Z. Li, Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and Sun. Objects365: A large-scale, high-quality dataset for object detection. In CVPR, 2019.
[43] P. Sharma, Ding. S. Goodman, and Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL, 2018.
[44] Z. Shen and E. Xing. fast knowledge distillation framework for visual recognition. In ECCV, 2022.
[45] J. Tan, X. Lu, G. Zhang, C. Yin. and Q Li. Equalization loss v2: A new gradient balance approach for long-tailes object detection. CVPR, 2021.
[46] J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan. Equalization loss for long- long-tailed object recognition. In CVPR, 2020.
[47] P. Tang, X. Wang. S. Bai, W. Shen, X. Bai, W. Liu, and A. Yuille. Pel: Proposal cluster learning for weakly supervised object detection. TPAMI, 2018.
[48] Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. In ICLR, 2020.
[49] Z. Tian. C. Shen, H. Chen, and T. He. Fcos: Fully convolutional one-stage object detection. In CVPR. 2019.
[50] H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML, 2021.
12

---

[51] J. Wang. W. Zhang, Y. Zang, Y. Cao, J. Pang. T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin. Seesaw loss for long-tailed instance segmentation. In CVPR, 2021.
[52] T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity the hypersphere. In ICML, 2020.
[53] W. Wang, J. Dai, Z. Chen, Z. Huang, Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CPVR, 2023.
[54] 7. Wang, N. Codella, Y.-C. Chen, I.. Zhou, X. Dai, B. Xiao, J. Yang. H. You, K.-W. Chang, S.-f. Chang. et al. Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks. arXiv preprint 2rXiv:2204. 10496, 2022.
[55] Z. Weng, X. Yang, A. Li, Z. Wu, and Y.-G. Jiang. Open-velip: Transforming clip an open-vocabulary video model via interpolated weight optimization. In ICML, 2023.
[56] J. Yang, C. Li, X. Dai, and J. Gao. Focal modulation networks. In NeurlPS, 2022.
[57] K. Ye, M. Zhang, A. Kovashka, W. Li, D. Qin, and J. Berent. Cap2det: Learning amplify weak caption supervision for ohject detection. CVPR, 2019.
[58] S. Yun, J. Oh, B. Heo, D. Han, Choe, and S. Chun. Re-labeling imagenet: from single multi-labes from global to localized labels. In CVPR, 2021.
[59] A. Zarcian, K. D. Rosa, D. H. Hu, and S.-F. Chang, Open-vocabulary object detection using captions. In CVPR, 2021.
[60] C. Zhang. T.-Y. Pan, Li, H. Hu, D. Xuan, S. Changpinyo, B. Gong, and W.-L. Chao. Mosaicos: simple and effective of object-centric images for long-tailed object detection. In ICCV, 2021.
[61] H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y. Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. ICLR, 2022.
[62] Y. Zhong, J. Yang, P. Zhang, C. Li. N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. Regionclip: Region-based language-image pretraining. In CVPR, 2022.
[63] X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and Misra. Detecting thenty-thousand classes using image-level supervision. ECCV, 2022.
[64] X. Zhou, V. Koltun, and Krähenbühl. Probabilistic two-stage detection. arXiv preprint 21X1012103.00461. 2021.
[65] X. Zhu, W. Su, L. Lu, B. Li. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. ICLR, 2021.
[66] B. Zoph, G. Ghiasi, T.Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Lc. Rethinking pre-training and self-training. In NewrlPS, 2020.
13


Reviews:
[{"review_id": "CmYbcYdI4A", "reviewer": "Reviewer_YRik", "summary": "This paper adopts the CLIP model to obtain a 'soft label' supervision to train the detector under the long-tail distribution dataset and derive rich semantics from the CLIP part to enhance the tail-categories representations, which can be removed during the inference. The authors claim that the CLIP model can well capture visual semantics conditioned on only coarse locations, the whole-bbox for extra data in this paper. They then elaborate soft-label supervision to train the detector so as to alleviate semantic ambiguity and location sensitivity issues. Various ablative studies have been conducted to validate the effectiveness of the proposed method.", "questions": "Please regard the weakness parts.", "limitations": "Please regard the weakness parts.", "rating": 5, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1. The novelty of this paper sounds technically reasonable.\n2. The writing of this paper is easy to follow.\n3. The final performances look competitive and the improvements are obvious.", "weaknesses": "1. In Line 172, the subscript of f and o maybe should be superscript.\n2. The proposed semantic branch training is parallel to the detection heads during the training to avoid the training conflicts for the CE loss and KL loss, but the detection heads are only trained on the tailed samples in LVIS without some kind of interactions on the semantic branch and how to ensure that the detection heads can well handle the rare categorical samples during inference?\n3. ImageNet-21K is used as extra data in this paper, but the backbone is pre-trained in ImageNet too. So is there any dataset overlapping between the pertaining and fine-tuning stages?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### q1: L172 subscripts.\nThanks for pointing it out. We will definitely fix it in the next version.\n\n### q2: How can semantic learning on extra classification data boost rare categories detection?\n\nThe detection head is trained not only on the LVIS dataset but also on the extra image classification dataset.\n\nMore specifically, we use a unified objective function $L=L_{loc}+L_{cls}+L_{soft}$ (as detailed in Section 3.3), treating both detection data and extra classification data in a unified way. Consequently, the detection head is trained using both types of datasets. For classification data without bounding box annotations, we employ pre-defined whole-image boxes as pseudo boxes and group multiple images together to create mosaics, which provide location supervision for the detection head.\n\nIn conclusion, our method can help rare classes in two ways:\n\n1) Rich semantics ($L_{soft}$): the semantic supervision flows back to the object features, implicitly enhancing the feature representation and localization capability;\n2) Coarse locations ($L_{loc}$): our training scheme allows the detector to learn from pre-defined pseudo locations on the classification data through $L_{loc}$;\n\nTo further demonstrate the effectiveness of the two perspectives, we conduct an ablation study on $L_{soft}$ and $L_{loc}$ within the training recipe for classification data. The results, shown in the table below, demonstrate that both rich semantics and coarse locations play significant roles in boosting long-tail object detection, offering significant performance gain on overall AP and rare AP. \n\n|Method | AP | AP_r | AP_c | AP_f|\n|---- | ---- |---- |---- |---- |\n| w/o $D_{extra}$ | 32.2 | 24.1 |29.9 |38.3 |\n| + $L_{soft}$ (rich semantics only) | 33.6 | 28.6 (+4.5) | 32.4 | 37.2 |\n| + $L_{loc}$ (coarse location) | 35.0 | 30.4 (+1.8) | 33.1 | 39.0 |\n\n### q3: Data overlapping between backbone pretraining and detection training\nYes, there is data overlap between the backbone pretraining and detection training. There are 246 categories that overlap between ImageNet-1k and LVIS, and 997 categories overlap between ImageNet-21k and LVIS.\n\nDataset | Number of Imgs | Definition|\n|---- | ---- | ----\n|LVIS  | 0.1M | The original LVIS\n|INet-1k  | 1M | The original ImageNet-1k\n|INet-21k  | 14M | The original ImageNet-21k\n|INet-LVIS  | 1M | INet-21k classes overlapped with LVIS\n\n|Method | $D^{backbone}$ | $D^{od}$|AP | AP_r | AP_c | AP_f|\n|---- | ---- |---- |---- |---- |---- |---- |\n| w/o $D_{extra}$ | INet-1k | LVIS | 32.2 | 24.1 |29.9 |38.3 |\n| w/o $D_{extra}$ | INet-21k  | LVIS | 35.7 | 25.9 | 35.0 | 40.7\n| Ours | INet-1k  | LVIS + INet-LVIS | 35.0 | 30.4 | 33.1 | 39.0 |\n| Ours | INet-21k  | LVIS + INet-LVIS | 37.5 | 32.4 | 36.0 | 41.5 |\n\nWe further conduct experiments with R50 backbones pretrained with different amounts of data under the 1$\\times$ schedule. The table shows that pretraining on large-scale data can provide strong perception capability for the downstream detection task, with overall performance gain. However, the performance on rare categories is still relatively low, indicating that this approach does not alleviate the long-tail effects in detection. In addition, the pretraining cost will increase significantly with 10 times more pretraining data.\n\nIn contrast, our method is more effective than pretraining to handle long-tailed detection, especially for the tail categories. Notably, our approach is still effective with strong pretrained backbones, further improving performance on long-tailed object detection.\n\nThanks for the inspiration, and we will add the discussion on data overlapping in the next version."}, {"review_id": "VwPhPHc1Wf", "reviewer": "Reviewer_ndNj", "summary": "To address semantic ambiguity and location sensitivity, this paper introduces a one-stage training framework that leverages additional image data to boost the detector through learning from rich semantics and coarse locations for long-tailed object detection. And their RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors.", "questions": "Please check the paper weaknesses.", "limitations": "Please check the paper weaknesses.", "rating": 6, "confidence": 3, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is clearly written. And main idea of the paper is easy to understand.\n2. This paper introduces a novel semantics learning framework, which uses an additional branch to learn from rich semantics and coarse locations for long-tailed object detection without the need to compute pseudo labels.\n3. Extensive method demonstrates significant results on long-tailed datasets", "weaknesses": "1. In Long-tailed object detection, the scarcity of samples or natural constraints results in a limited number of instances in the tail classes. Can exploring extra data effectively address the issue of scarce tail classes in practical applications?\n2. CLIP is a large-scale model designed for the joint processing of images and text. The lack of representative samples for tail classes may lead to a relatively weak understanding and recognition ability of CLIP for these classes. Can CLIP still provide stable semantic guidance?", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### q1: Can exploring extra data effectively address data scarcity?\n\nYes. Indeed, exploring additional data is a straightforward approach to enhancing the performance of tail categories. However, acquiring bounding box annotations for these rare categories is labor-intensive and costly. \n\nRecognizing this challenge, we introduce a new approach that leverages existing image classification data to alleviate the data scarcity issue faced by tail categories.\n\nUnlike conventional semi-supervised or weakly supervised methods, our approach does not require estimating bounding boxes, which is challenging. Instead, we focus on rich semantics obtained by CLIP models and leverage coarse locations from classification data for improved results.\n\nWe conduct comprehensive experiments involving varying backbones, detectors, schedules, and datasets. Our method shows consistent gains in all experiments, demonstrating that our method can effectively address the challenge caused by the scarcity of detection data.\n\n\n### q2: Can CLIP provide stable semantics on rare categories?\n\nYes, CLIP models showcase balanced recognition capabilities across rare, common, and frequent categories thanks to the web-scale image-text pairs as training data (see supp sec.B, we also include the table here for convenience). \n\nWe use CLIP-RN50 and perform region classification on LVIS utilizing ground truth bounding boxes. Specifically, we obtain object features according to their ground truth bounding boxes and classify them using the contrast with textual features of categories. \nAs shown in the table below, the AP of the top 10 predictions per proposal is around 34%, indicating that CLIP can properly rank labels into the top classes. Furthermore, a key observation is that the results highlight a well-maintained and balanced performance across categories with varying frequencies.\n\n\n| Region classification  |AP | AP_r | AP_c | AP_f|\n|---- | ---- |---- |---- |---- |\n| top1 class per proposal| 16.2 | 16.7 | 16.4 | 15.7\n| top5 class per proposal| 29.6 | 29.9 | 29.3 | 29.8\n| top10 class per proposal | 33.9 | 33.1 | 33.7 | 34.6\n\nWe also find that CLIP models are robust towards location shifts. As shown in Fig.2 in the **rebuttal pdf**, we gradually introduce noise to ground truth boxes, and the top-10 performance experiences only a marginal drop when the noise scale remains relatively small (ranging from 0 to 0.5).\n\nThese observations underscore CLIP's capacity to provide stable and precise semantic information. It allows our method to effectively address data scarcity by leveraging extra classification data."}, {"review_id": "tYD4eZ6dSU", "reviewer": "Reviewer_gFff", "summary": "- This work deals with Long-tail object detection. Authors identify two problems with using additional data, namely Semantic ambiguity and Location sensitivity. \n- Authors identify that semantic ambiguity arises due to supervision with one-hot encoded labels from the image datasets and instead propose to use CLIP scores for supervision.\n- CLIP's ability to provide sufficient semantic information even with course locations is leveraged to tackle location sensitivity.\n- Authors propose RichSem, a simple yet effective method, that adds an additional semantic branch to the detector to learn rich semantics from images. \n- The semantic branch is only required during training and with extensive experiments is shown to achieve state-of-the-art results on LVIS dataset in overall and rare categories.", "questions": "- Authors mention constructing a mosaic with multiple images in L51, where in the whole method is this done?\n- What is $f_v^i$ in Eq. 4? Is that $s^t$ obtained in Eq. 2?\n- From table 2c, rows 4,5 the increase in improvement is because of image level labels. But how are the annotations used in the whole pipeline? The semantic branch only uses CLIP similarities for the KL divergence loss? Is a hard supervised loss also being applied for the image labels? Is there a localization term for the image level labels?\n- What is $f^t$ in the semantic branch in Fig. 2? If $t\\in (\\text{loc},\\text{cls})$, then is the semantic soft loss applied to both $f^{\\text{loc}}$ and $f^{\\text{cls}}$?", "limitations": "I do not foresee any potential negative societal impact of this work. ", "rating": 7, "confidence": 5, "soundness": 4, "presentation": 4, "contribution": 3, "strengths": "- The paper is well written and the presentation makes it easy to follow.\n- The experiment section is exhaustive and supports all the claims made by the authors. Authors test their method with the transformer and R-CNN based family of detectors and the ablation experiments clearly explain the contribution of each component.\n- RichSem is a principled way to leverage additional data for long-tailed object detection.", "weaknesses": "- Authors miss the comparison with [1], which also uses additional data. Please compare appropriately in Table-1.\n- The current work heavily relies on CLIP but it is widely known that CLIP has several limitations [2]. It would be interesting to address the robustness of the current method to the limitations of CLIP.\n\n[1] Bo Li, Yongqiang Yao, Jingru Tan, Xin Lu, Fengwei Yu, Ye Luo, and Jianwei Lu, Improving Long-tailed Object Detection with Image-Level Supervision by Multi-Task Collaborative Learning.\n\n[2] https://stanislavfort.github.io/blog/OpenAI_CLIP_stickers_and_adversarial_examples/", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### q1: Comparison with [1]\nThanks for the good suggestion. We incorporate our method into Faster-RCNN, employing R50 as the backbone for an appropriate comparison following [1].\n\nThe table shows that our method achieves a strong performance and surpasses the previous sota by more than 3 AP on rare categories. Unlike CLIS[1], our approach focuses on digging rich semantics with only pre-defined bounding boxes on extra classification data. Thanks to our semantic learning scheme, our model can effectively leverage the information within classification data for long-tail object detection, achieving a more balanced performance between head and tail categories. \n\nWe greatly appreciate your suggestion and will update the table in our next version.\n\n| Method  |AP | AP_r | AP_c | AP_f|\n|---- | ---- |---- |---- |---- |\n| Faster RCNN| 24.1 | 14.7 | 22.2 | 30.5\n| EQLv2| 25.5 | 16.4 | 23.9 | 31.2 |\n| BAGS| 26.0 | 17.2 | 24.9 | 31.1 \n| Seesaw| 26.4 | 17.5 | 25.3 | 31.5\n| EFL| 27.5 | 20.2 | 26.1 | 32.4 \n| MosaicOS| 23.9 | 15.5 | 22.4 | 29.3 \n| CLIS| 29.2 | 24.4 | 28.6 | 31.9\n| Ours | 30.6 | **27.6** | 29.7 | 32.9\n\n[1] Improving Long-tailed Object Detection with Image-Level Supervision by Multi-Task Collaborative Learning.\n \n### q2: Robustness of CLIP\nThank you for your suggestion. We do agree CLIP is not robust towards adversarial attacks like many other types of neural networks. In this work, we assume images are pristine, and we will study the robustness of CLIP in the future.\n\n### q3: Questions on mosaic augmentation\nThanks. We only apply 2x2 mosaic augmentation on the **extra classification data**. \nSpecifically, we utilize a pre-defined whole-image bounding box for each image and use the mosaic augmentation [1] to randomly concatenate sub-images into a mosaic, thus offering coarse locations on classification data.\n\nUnlike previous works on extra data focusing on bounding box estimation, such as using a pre-trained detector as region generators [2], online predictions [3], and post-processing methods [4],  we emphasize that coarse locations are sufficient and pay more attention on introducing rich semantics from the classification data to the detector.\n\n\n[1] Yolov4: Optimal speed and accuracy of object detection  \n[2] Improving Long-tailed Object Detection with Image-Level Supervision by Multi-Task Collaborative Learning  \n[3] Detecting Twenty-thousand Classes using Image-level Supervision  \n[4] MOSAICOS: A Simple and Effective Use of Object-Centric Images for Long-Tailed Object Detection  \n### q4: Eq.4\nSorry for the confusion. In Equation 4, $f^t_{i}$ refers to the corresponding semantic guidance obtained from CLIP, denoted as $s^t$ earlier. In Eq. 4, we compute the Kullback-Leibler divergence between the soft semantic prediction and the corresponding semantics provided by CLIP models. For the sake of consistency, they should be $o^{soft}$ and $s^{t}$ instead of $f^t_{i}$ and $s^t$ in Eq.4. We will definitely refine the notations in the next version.\n\n### q5: Questions on rows 4-5 of Tab 2c\nBoth line 4 (ImageNet-Unl) and line 5 (Image-LVIS) incorporate the soft KL loss, hard classification loss, and location loss in our training pipeline.\n\nFor the soft KL loss and location loss, both line 4 and line 5 follow the same approach. They utilize pre-defined whole-image bounding boxes as pseudo-locations and extract semantics based on these coarse locations, forming the soft targets of the semantic branch.\n\nRegarding the hard classification loss, line 5 employs image-level labels that are mapped to the LVIS taxonomy as the target. On the other hand, for line 4, the pseudo hard labels for classification are generated by using the class with the highest logits in each semantic target (as described in Eq. 7). Additionally, we incorporate a threshold $th$=0.05 to filter out images that are too irrelevant to the taxonomy.\n\nDue to the distinctions above, line 5 (Image-LVIS) slightly outperforms line 4 (Image-Unl). This improvement can be attributed to the well-matched taxonomy between the extra data and the target detection data, along with a well-designed label mapper.\n\nHowever, it is noteworthy that the difference between Image-LVIS and Image-Unl is relatively small (only 0.3 and 1.8 on AP and AP_rare). This observation shows our potential for unlabeled classification/object-centric data.\n\n### q6: Questions on f^t\n\n$f^t$ in Fig.2 represents the object-level features from CLIP (please refer to Section 3.1 and Equation 2). More specifically, $f^t$ is derived by pooling CLIP features based on the corresponding bounding box. For detection data, this bounding box corresponds to the tight ground-truth bounding box, while for classification data, it involves a coarse whole-image bounding box. The object-level CLIP feature $f^t$ is then employed to derive $s^t$ by calculating the Contrast with linguistic categories features $f^{cat}$, which is soft semantic guidance in our training scheme."}, {"review_id": "lxLwaB0r8r", "reviewer": "Reviewer_vo8U", "summary": "This paper uses the vision-language pre-trained model CLIP for long-tailed object detection.\nThe key idea is to consider not only image-level semantics but also region-level semantics, and fuse them under a soft-label scenario.\nThe overall idea is deployed on different object detectors with different backbones.\nExperiments are conducted on three long-tailed object detection datasets with extra data.\nBesides, the experiments and discussion on ablation studies and foundational models are very extensive.\n", "questions": "The reviewer really appreciate that the work is the first to extend vision-language pre-trained model to the long-tailed object detection task. However, the weakness of this submission is also very obvious.\n\nIn the rebuttal, please address the weakness part point-by-point, on lack insight for long-tailed context, lack state-of the-art comparison, and to-some-extent performance ineffectiveness. \n", "limitations": "The limitation is not properly discussed", "rating": 5, "confidence": 4, "soundness": 2, "presentation": 3, "contribution": 3, "strengths": "+ To the best of the reviewer’s knowledge, it is the first work (excluding CVPR2023) to use vision-language pipelines for long-tailed object detection. The contribution is significant. \n\n+ The ablation studies and other discussions are very extensive, in both the main submission and the\n\n+ This paper is well-written and easy-to-follow.\n", "weaknesses": "- This work, in its current form, lacks theoretical insight, especially on how the proposed method fits into the long-tailed feature distribution. \n\nAs the proposed method is targeted at the context of long-tailed object detection, not the generic object detection, it is very necessary to extensively discuss how the framework and the specific module design can work well or fit into the long-tailed feature distribution.\n\nUnfortunately, as the reviewer goes through the entire manuscript: In Fig.1 motivation case and Fig.2 framework, there is no element for the long-tailed feature space or long-tailed elements. Besides, in the methodology section, from learning the image / object semantics to use soft labels, all these presentations are generic and are suitable for generic recognition, detection, segmentation and etc. More importantly, in the experimental section, there is no feature space visualization on how CLIP benefit the long-tailed feature distribution.  \n\n- The state-of-the-art comparison is insufficient, and to some extent, unfair. Specially,\n\nIt is really strange that in Table 1, the authors only compare some generic vision transformer based detectors such as DERT. Why the methods, using vision-language pre-trained model for generic object detection, such as [11,18,27,34,57], are not involved for comparison on the long-tailed object detection datasets? It is very likely that these methods [11,18,27,34,57] for generic vision-language object detection can work well on long-tailed detection datasets.\n\n- The performance of the proposed method seems to be mainly brought high by the use of Swin-Transformer.\n\nThis remark is not saying that the state-of-the-art performance with Swin-Transformer backbone is not important, but the reviewer would like to raise the attention that, when use the old ResNet-50 backbone (Row 1 of Table 1), the proposed method, even with vision-language pre-trained model, does not lead a significant performance against some SOTA in 2021. \n\nThus, the reviewer feels that in the future, why with Resnet-50 is not that effective for the proposed framework, worth to be further investigated. \n\nOther minor issues to improve:\n\n- Extensive feature visualization on the context of both long-tail and CLIP features, please. \n\n- Please use \\mathcal{} to present the loss function, and use \\mathbf{} to present the tensors and vectors. These notations can distinguish them from scalars.\n", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "### q1: Insight for long-tail context.\nThanks for pointing out this. Exploring extra data is indeed a direct and effective approach to mitigating data scarcity. Our primary motivation arises from the fact that classification data is easy to collect and offers a more balanced distribution compared to detection data.\n\nFor instance, consider the ImageNet-21k and LVIS; there are 997 overlapped categories in ImageNet-21k, covering a significant portion of the LVIS categories with a 10$\\times$ number of images, greatly enhancing instance diversity. Furthermore, the samples in ImageNet-21k exhibit a balanced distribution (please refer to Fig.1 in our **rebuttal pdf**). Therefore, leveraging classification datasets demonstrates great potential in alleviating data scarcity.\n\nHowever, it is still challenging, summarized as (1) semantic ambiguity and (2) location sensitivity. Previous works mainly focus on box estimation to solve location sensitivity but neglect semantic ambiguity and the importance of rich semantics inside classification data.\n\nFortunately, V-L pre-trained models (CLIP) have demonstrated powerful zero-shot recognition capabilities, benefiting from web-scale training data. Our pilot experiments (please refer to Tab.1 and Fig.2 in the **rebuttal pdf**, and Sec. B in supplementary) illustrate that CLIP models exhibit balanced performance across rare, common, and frequent categories; and show robustness towards location shift.\n\nGiven these compelling factors, it becomes natural to utilize CLIP models and delve deeply into leveraging the rich semantics from classification data to address long-tail object detection effectively.\n\n\n### q2: Lack of state-of-the-art comparison\n\nCurrent research on vision-language (V-L) pretraining and foundational models for long-tailed object detection can be categorized into two main directions:\n\n#### 1) **Leveraging the well-aligned Vision-Language knowledge of the pre-trained**\nThese works have primarily focused on open-vocabulary detection tasks.\n\nFor a fair comparison, we thus conduct an experiment on open-vocabulary LVIS, which can be viewed as an extremely long-tailed distribution where tail categories have zero occurrences.\n\n* Open Vocabulary LVIS detection results compared with [11,18,27,34,57]\n\n| Method | backbone |AP | AP_novel \n|---- | ---- |---- |---- |\n| CLIP [34] + gt bbox| ViT-B| 17.7 |18.9 | 18.8 | 16.0 \n| GLIP-zeroshot [27]| Swin-L| 26.9 | 17.1 | 23.3 | 35.4 |\n| ViLD [11] | R50|  27.5 |17.4 | - | - |\n| RegionCLIP [57]| R50| 27.4 | 17.0 | 26.7 | 32.9 |\n| Ours| R50 | 31.5 | **23.0** | 30.9 | 36.0 \n\nAs shown in the table, our method surpasses the mentioned previous sota on their benchmark with nearly 6 AP gains on novel categories, further demonstrating the effectiveness of our method.\n\n\n#### 2) **Scaling up the pretraining data and model size**\n\nResearchers employ large vision foundation models and advanced training techniques to achieve excellent performance, regardless of the computational burden and training recipe consistencies. In contrast, we utilize the advanced Focal-H to compare with these works on LVIS detection.\n\n* Results on LVIS val v1.0\n\n| Method | backbone | $D_{backbone}$ | param|AP | AP_r | AP_c | AP_f|\n|---- | ---- |---- |---- |---- |---- |---- |---- |\n| ViTDet| ViT-H-MAE | IN-1k | 692M | 53.4 | - |- |\n| EVA | EVA-H | merged-30M | 1.1B | 62.2 | 55.1| 62.2| 65.2 |\n| InternImage| DCNv3-H | merged-data | 2.2B |63.2 |- |- | - |\n| Ours| Focal-H | IN-22k | 747M | 61.2 | **61.2** | 60.1 | 62.4\n\nThe table shows our model achieves comparable performance with 0.8B fewer parameters, offering a pretty balanced performance for both overall and rare categories.\n\nNotably, we did not use other advanced training tricks, e.g., large image size (1.5$\\times$ larger) and batch size training (16 vs. 64), and their additional inference post processings (soft NMS and TTA).\n\n### q3: Performance \"ineffectiveness\" on ResNet-50\nSorry for the confusion. In fact, our method is indeed **effective** on R50.\n\nTwo main reasons are contributing to the \"ineffectiveness\" :\n\n1. Backbone pretraining.  \nThe previous sota (Detic) utilized R50-21k [1] (refer to R50$\\star$ in Tab.1 of the main paper), offering strong perception capability for downstream detection. In contrast, we just use the R50-1k, which is significantly weaker than R50-21k.\n2. The size of the CLIP model providing semantics.\nWe utilize CLIP-RN50 as the default semantics provider. However, the previous sota uses CLIP-ViT-B to generate the weight of the classifier. We observe that stronger semantics lead to better performance (L297-L300 in our main paper), indicating that our method can be further improved.\n\nBased on these, we conducted experiments with the stronger backbone and semantics provider. Our best model surpasses the previous sota by a large margin (4.1 AP and 6.1 AP_r), indicating its effectiveness on the R50 backbone. \n\n| Method  | Backbone | CLIP model  |AP | AP_r | AP_c | AP_f|\n| ---- |---- |---- | ---- |---- |---- |---- |\n| Detic  | R50-21k  | CLIP-Vit-B | 36.8 | 31.4 | 36.0 | 40.1\n| Ours  | R50-1k  | CLIP-RN50 | 37.1 | 29.9 | 35.6 | 42.0\n| Ours  | **R50-21k** | CLIP-RN50| 40.1 |36.2 | 38.2 | 44.0\n| Ours  | **R50-21k** | **CLIP-RN50x4** | 40.9 | 37.5 | 39.6 | 43.8\n\n\nThanks for pointing it out and for the great inspiration; we will update this table in the next version.\n\n[1] Imagenet-21k pretraining for the masses. In NeurIPS, 2021. \n\n\n### q4: Limitation part.\n\nA main limitation of our approach is treating detection data and classification data statically with strict equality in the unified objective. An optimal scenario might dynamically entail prioritizing $L_{cls}$ for instances that can be accurately categorized while making $L_{soft}$ precedence in other situations.\n\n### q5: Suggestions on fonts of notations and feature visualization\n\nThanks for your suggestions, and we will refine the fonts to make them distinguished and visualize the features in the next version.\n"}]

Author rebuttal:
First of all, we sincerely appreciate all your valuable comments and suggestions.

We are pleased that all reviewers think our paper is well-written and easy to follow. We are encouraged that reviewers find our proposed RichSem with reasonable novelty (YRik), significant results (ndNj), and extensive ablation and discussion (gFff, vo8U).

We carefully read the comments and attempted to provide comprehensive responses accordingly. Please find the rebuttal below each official review. We hope the responses could answer the questions raised by reviewers and address any concerns about our work.

Thanks again to all reviewers for the time and effort!

Meta-review: Paper received all "accept" ratings from the reviewers post-rebuttal: 2 x Borderline Accept, 1 x Weak Accept and 1 x Accept. There is a consensus that the work is interesting, novel and the paper is well written. The approach also illustrated significant improvements on long-tailed datasets. Questions raised by the reviewers were throughly addressed in the rebuttal, including with provided (significant) additional experimental results. Reviewers were generally satisfied with the rebuttal and acknowledge that their concerns were addressed. AC has read the reviews, rebuttal, and the discussion that followed and concurs with reviewers that the work is interesting and should be accepted. Authors are encouraged to incorporate results from the rebuttal into the main paper (or supplementals) for the camera ready.
Decision: Accept (poster)