Title: Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection
Venue: NeurIPS 2023
Accepted: yes
Keywords: Long-tail object detection

visual semantics

soft supervision
Abstract: Long-tailed object detection (LTOD) aims to handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy is to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity---an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity---the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping.
To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional ``soft supervision'' for training detectors. Specifically, we add a semantic branch 
to our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch is only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors. 
Our method achieves state-of-the-art performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method on other long-tailed datasets with additional experiments.

Learning from Rich Semantics and Coarse Locations for Long-tailed Object Detection

Abstract

Long-tailed object detection (LTOD) aims to handle the extreme data imbalance in real-world datasets, where many tail classes have scarce instances. One popular strategy is to explore extra data with image-level labels, yet it produces limited results due to (1) semantic ambiguity -an image-level label only captures a salient part of the image, ignoring the remaining rich semantics within the image; and (2) location sensitivity -the label highly depends on the locations and crops of the original image, which may change after data transformations like random cropping. To remedy this, we propose RichSem, a simple but effective method, which is robust to learn rich semantics from coarse locations without the need of accurate bounding boxes. RichSem leverages rich semantics from images, which are then served as additional 'soft supervision' for training detectors. Specifically, we add a semantic branch to our detector to learn these soft semantics and enhance feature representations for long-tailed object detection. The semantic branch is only used for training and is removed during inference. RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors. Our method achieves state-of-the-art performance without requiring complex training and testing procedures. Moreover, we show the effectiveness of our method on other long-tailed datasets with additional experiments. Code is available at https://github.com/MengLcool/RichSem .

1 Introduction

Object detection for complex scenes has advanced significantly [10, 9, 40, 3, 65] thanks to large-scale datasets [5, 22, 32, 12, 24, 4, 42]. However, current deep models depend on relatively balanced large-scale datasets, where different classes have similar numbers of images and samples, to learn diverse semantics and enough locations. This limits their performance on real-world data, which are often long-tailed, i.e ., only a few head classes have plenty of training samples while most classes have very few training samples, making detection more challenging and less effective for rare classes.

A simple and effective way to improve long-tailed object detection (LTOD) is to use extra data to increase the training samples for tail classes. However, collecting bounding box annotations, especially for rare categories, is costly and tedious. Therefore, previous studies resort to datasets with image-level labels to enrich the amount of samples for rare classes by exploring image-level semantics (as shown in Figure 1 (a)). While appealing, directly learning from such data to benefit detection is challenging since they lack bounding box annotations that are essential for object detection. To remedy this, many works [66, 25, 60, 63] focus on estimating bounding boxes for

Image count

sorted id

frequent

common

rare

LVIS

'piggy

bank'

'sugar

bowl'

ImageNet-21k

(a)

(b)

frisbee(0.35)

dog(0.58)

ball(0.19)

dog(0.54)

'Ball' in ImageNet

soft semantics

dog

puppy

ball

pet

frisbee

toy

10^4

10^3

10^2

10^1

10^0

objects in images. They typically consider image-level tags (in the form of one-hot labels) as the ground-truth classes and match with best estimated boxes as pseudo labels for training.

We argue that image-level labels are not well-suited for detection due to their semantics ambiguity and location sensitivity . On one hand, an image-level label can not sufficiently reflect all the semantics within the image. On the other hand, the label mainly focuses on the iconic object; thus, using a different crop might shift the semantics to a different object. Take Figure 1 (b) as an example: the top image is labeled as 'ball' in ImageNet [5], which only depicts part of information in the image. The panel below in Figure 1 (b) shows that the provided image-level label is no longer accurate after data augmentations. Inspired by the recent success of visual-language contrastive models that align a large number of image-text pairs [37, 62, 11, 54, 13], we aim to leverage such models to extract rich semantics that are more informative than the image-level labels in classification datasets. However, naively converting one-shot labels in image classification tasks to a distribution of soft labels is still not optimal and accurate after data augmentation. As shown in Figure 1 (b), based on different locations ( i.e ., random crops) of the image, the semantics extracted by CLIP [37] incur more noise than the original image.

To address this, we introduce RichSem, a one-stage training framework that leverages additional image data to boost the detector through learning from rich semantics and coarse locations for long-tailed object detection. In particular, we treat a whole-image box as coarse locations ( i.e ., the coarse bounding box shares the same size as the image) and group multiple images together to build a mosaic. We then use CLIP to extract the semantics according to such coarse bounding boxes. The extracted semantics can serve as 'soft labels' to enrich the amount of training data and are more robust to random cropping. We further introduce a new branch named as 'semantic branch' to object detectors so that it can learn from the derived semantics during training. This allows the detector to fully leverage the rich semantics in classification datasets with minimal modifications. Once trained, this branch is discarded during inference, and RichSem can be used readily as a standard detector.

Our contributions are summarized as follows: (1) We point out that using image-level labels from extra classification datasets as supervision is challenging due to semantics ambiguity and location sensitivity , which can be alleviated with a distribution of semantics serving as 'soft labels'. We show that semantics provided by CLIP are not only rich and but also robust to locations, providing better guidance than original image-level labels. (2) We introduce a novel semantics learning framework named RichSem, which uses an additional branch to learn from rich semantics and coarse locations for long-tailed object detection without the need to compute pseudo labels. Once trained, the additional branch can be discarded during inference. (3) Our method demonstrates strong results on long-tailed datasets, e.g . LVIS, highlighting that it is a low-cost way to boost the detector performance for long-tailed object detection by only complementing extra classification data.

2 Related Work

Long-tailed object detection (LTOD) has attracted more and more attention. The performance of rare categories is drastically low compared to frequent categories due to the unbalanced distribution and the lack of training samples. Existing works can broadly be divided into two directions: 1) one direction aims to improve the training scheme for balanced learning, including data re-sampling [12], loss re-weighting [46, 45, 51], data augmentations [8] and decoupled training [23, 21]; 2) Another direction leverages extra data to compensate for the data starvation [63, 60]. These methods often trust image-level labels to boost the classification capability. We find that it is far from optimal to naively treat image-level labels as golden labels and supervision for the classifier. Unlike existing methods, we study a new solution that leverages the rich semantics within the images from classification data with only coarse locations provided by regular augmentations. We leverage CLIP to provide semantics as soft targets on classification data to guide our detector to learn semantics explicitly. In this embarrassingly simple but effective way, we can better leverage classification data for LTOD.

Weakly-supervised object detection aims to train a detector using image-level labels without bounding boxes. Many studies [1, 47, 36] train a model using only image-level labels without any bounding box supervision. Another line of work [39, 60] takes the bounding boxes supervision with the whole-image labels together under a semi-supervised framework. Unlike prior works, we focus on mining rich semantics within the images instead of bounding box estimation. Thus, we no longer need to estimate precise bounding boxes on classification data.

Language supervision for object detection is a recent topic that aims to leverage linguistic semantics. Since language supervision has rich semantics, each category is related rather than independent in one-hot labels. Thanks to this property, recent works [6, 59, 57, 20, 35, 55] show benefits by pretraining backbones on vision-language tasks. With the rapid progress in contrastive language-image pre-training [37, 18], many recent approaches [28, 11, 62] apply large-scale pre-training for object detection. ViLD [11] and RegionCLIP [62] attempt to align the visual-semantic space of a pre-trained CLIP model for open-vocabulary detection. Similar to RegionCLIP and ViLD, our method leverages the visual-semantic space learned from pre-trained CLIP models. In contrast, our goal is to leverage object semantics to boost object classification, especially for rare categories. Since CLIPs are used to generate 'soft labels', our backbone is free of CLIP initialization compared with RegionCLIP [62].

Knowledge distillation and soft label. Knowledge distillation (KD) [17] is a powerful tool to boost the student model with prior knowledge of the teacher model. Recent follow-ups extended and developed many variants, e.g ., feature distillation [19, 41], hard distillation [50], contrastive distillation [48], etc. To improve the training efficiency of KD, Re-label [58] and FKD [44] use a strong teacher model to generate soft labels and store them for efficient image recognition training. Recently, many studies [11, 34, 62, 38] introduce knowledge from CLIP [37] into the KD framework to boost open-vocabulary object detection. Due to the strong semantics capturing capability of CLIP, those methods show encouraging performance on open-vocabulary detection. Similar to those works, our approach leverages the semantic knowledge of pre-trained CLIP models. In contrast, our goal is to boost the long-tailed detection, especially for the tailed classes, with the help of rich semantics of classification and detection datasets. Moreover, we introduce a simpler but more effective way to learn visual semantics with an extra semantic branch during training. Unlike [62, 11], our one-stage training scheme needs no fine-tuning after pre-training. Besides, our approach is more effective since there are no redundant pre-computed boxes [11] and boxes estimation [34, 38].

3 Method

Given a detection dataset denoted as D od = { ( I od , { ( b od , c od ) } ) } , where each image I od is associated with bounding boxes b od and class labels c od respectively. Our goal is to leverage an additional classification dataset D extra = { ( I img , c img ) } , of which each image is only labeled with an imagelevel label c img , to improve long-tailed object detection, particularly for rare classes.

In long-tailed object detection, previous approaches typically rely on image-level labels and compute pseudo boxes for corresponding objects. On one hand, the semantic information provided by such labels is limited, ( i.e ., one one-hot label per image). On the other hand, augmentation strategies like randomized cropping might generate regions that do not contain the annotated class, thus making the provided labels no longer accurate. Motivated by the fact that CLIP has strong capabilities of

capturing visual semantics conditioned on only coarse locations 1 [62], we build upon CLIP to explore image classification datasets to guide the training of object detectors. Below, we first introduce how to obtain rich semantics from CLIP in Section 3.1, and the resulting semantics are then used to guide the training of our detector, as will be described in Section 3.2. Then we describe how we unify the training objective and extend RichSem to different types of extra datasets in Section 3.3.

o

loc

semantic branch for training/ not used in inference

dog, puppy,

ball, pet, toy,

frisbee, …

categories

/

mixed dataset

original detector for training/ inference

forward

supervision

vector

ball, sandal

dog

ball, ball,

ball, ball

Backbone

& Decoder

semantic space

ball

sandal

dog

c

od

c

img

b

c

D

extra

D

od

V

t

L

t

Contrast

W

cls

H

cls

H

soft

H

loc

W

loc

b

f

cat

PromptEng

f

img

P

s

t

O

f

loc

f

cls

f

soft

f

t

o

cls

o

soft

od

Pooler

bod

C

3.1 Image and Object Semantics

We aim to use image-level labels in classification datasets together with detection datasets to improve long-tailed object detection. This requires processing these two types of data within the same framework. As a result, we treat images from detection and classification datasets equally and we not only derive semantic information at the image-level for classification datasets but also at the object-level for detection datasets, as will be elaborated below.

Image-level semantics. We denote the pre-trained CLIP visual encoder and the language encoder as V t and L t , respectively. Consider an image I after random cropping, its semantics s can be obtained by computing the similarity between its visual features f img extracted by the visual extractor and the linguistic categories features f cats produced by the language encoder. Formally, we compute the visual and linguistic features as follows:

Here, P is the prompt engineering function to convert class names into prompts; Contrast is the contrasting paradigm [37] to calculate the similarity between two features, and s img contains semantics information in which each element indicates the likelihood for the corresponding class. Following [37, 62], we use a set of prompts to convert the categories in the target vocabulary C od into sentences then use the CLIP text encoder L t to obtain the linguistic features of categories in C od .

Object-level semantics. As mentioned above, to unify the training process with classification and detection data, we also obtain object-level semantics from CLIP as well. To this end, we obtain an object-level representation by pooling from image-level features [9, 15] rather than cropping in the original images, following [62]. More formally, object-level semantics are obtained as:

<latexi sh

1_b

64="oWdPT

9H

2

R

c

3

>A

VDLS

N

FJ

U

v

ug

K

j

y

r

0

+

Q

q

I

z

EnX

C

8

5

m

YZ

7

f

O

<latexi sh

1_b

64="TY/8

LJ5MF

N

j9wyu

0

>A

B+X

c

VD

S

2pr fUZd

ovgq

R

H

C

W

3Q

E

z

m

G

n

K

7

I

O

k

P

<latexi sh

1_b

64="Tg p2mVBz

5

3

M

>A

c

DLS

N

F

r

fU

du

v

q

R

H

C

y

J

j

9

PQ

K

7

/

Y

W

I

n

O

<latexi sh

1_b

64="

QL

XVNKERS+

FOG

0

Pr

>A

B9H

c

D

gM

T

7W

q

d

Iv

y

2

j

U

Jp oZm

C

f

u

n

w

z

8

k

5

Y

/

3

<latexi sh

1_b

64="

m

z5r

M

2+L

T

G

ZUNqOEoc

>A

B7n

VD

F

X

W

d

Iv

f

w

9

J

k

R

P8K

C0

j3u/

H

Y

y

p

Q

<latexi sh

1_b

64="OVj0Q7f/Bp zP

5LM

D

Uc2Y

>A

8n

S

N

F

r

Zdu ovgq

T

w

KC

ym

J

H

R

+

G

3

I

9

k

W

X

E

<latexi sh

1_b

64="AS9ocXNwGUdP

Bkyp

u

FM

E

>

H

VDL

g

Oz

r

Y

8

3

W

J

7

K

Q

n

+

Tv

I

j

2

5

/

q

C

m

Z

fR

0

<latexi sh

1_b

64="GnX

2

zuPYV

g5

ZjQvTM9

/JI

>A

B

H

c

DLS

N

EOy

r

fU

o

K

w

W8

m

k

F

R

3

p

d

7

q

0

+

C

<latexi sh

1_b

64="G

Vm

FCOL

ug

H

R

Aj5w

Iz

>

B9

c

D

S

N

2pr

f

Zd

k

K

X

Q

0

7

JM

Evo

P

Y

W

+

n

q

y

U

8

/

T

3

<latexi sh

1_b

64="rUZ2

LD

uygw

CJ7WNI

Xj/Vk

>A

B+H

c

S

M

F

3

P

p0E

qz

v

Y

Q8m d9

G

O

R

n

f

K

T

5

o

<latexi sh

1_b

64="V2UyO5

0Mf n+jQ

gR3

d

So

>A

B7

c

N

8

EJ

r

q/

9

Iv

T

FLz

W

X

Z

m

K

w

k

p

PH

D

u

CY

G

<latexi sh

1_b

64="uRv

MzqZQcP

YnX

j

8oV

>

B7

DLSg

F

T

W+

d

I

y

r

/

K

CGf

9

0

w

3

k

N

O

5

p

J

2

H

U

E

<latexi sh

1_b

64="VyO

d/IS

J5F

0

E

A

KMR

2w

>

B7n

c

DL

gN

r

fUY9

oP

j

X

Q

G

p

mZ

vH

8

q

k

z3

T

C

+

W

u

<latexi sh

1_b

64="

TKr09+QY/

v

dnIz g7XS2

c

>A

B

ZDL

N

F

P

q3W

m8E

uC

J

M

y

p

O

U

j

o

w

H

5

R

G

V

k

f

<latexi sh

1_b

64="NPkLG

rXK3

+J

F50H

w

uUC

AQ

>

B7n

c

VD

SgM

T

W

q

d

Iv

f

9o

2

R

8

j

/

Z

m

O

Y

p

E

z

y

<latexi sh

1_b

64="mU

3J

MP+

wv

E

oY9z

IkW

>A

B7n

c

VDLSgN

Oy

r

f

K

Xj

G

2

u

C

R

8

F

dQ

p

0

5

Z

T

q

/

H

<latexi sh

1_b

64="I

7A

T+L

rf

3F5J

k

MQ

YXvR0

>

B

c

VD

SgN

EOy

U

9

oP

K

2

8

H

Gm

n

Z

p

C

wd jq

W

u

/

z

<latexi sh

1_b

64="5E7n3Yp

2Z

Q9

yzPgB

F

L

+

I

>A

H

c

VD

S

N

O

r

fU

o

K

w

Xj

GM

m

u

CW

8

k

d

0

v

J

T

R

q

/

<latexi sh

1_b

64="Lvnc8NR

9

2AYJ7C

wy

I

UX

mE

>

VD

SgM

F

3

q

d

r

W

G5

P

o

T

k

j

0

u/

z

p

O

Z

K

H

f

Q

<latexi sh

1_b

64="p jcV2frUYB5

LW

JF

q

Mo

>A

7n

D

SgN

EOy

9

P

K

8

I

m

Gz

Z

RXCk

/w

3

HT

v

d

u

0

+

Q

<latexi sh

1_b

64="Bu

LjC

/Z

z

SNDY+X0

U

>A

7n

c

V

8

EJ3

r

9

2P

K

m

oWy

p

I

RF

d

M

H

O

Q

T

f

5

v

k

g

w

<latexi sh

1_b

64="

7duRF

2mgLVN

0S8+

yBo

E

>A

3

c

D

Oz

G

Mr

H

Y

kWP

I5

J

n

Q

Z

p

C

w

v

jq

T

f

U

X

9

K

/

<latexi sh

1_b

64="HT/ENZnBD

R

cpmXd qP

VoQOU

>A

7

JSg

K2

W

Cj

k

8

I

F

L0

3

Gf

9

u

+

M

vr

5

w

z

y

Y

<latexi sh

1_b

64="W

YLG085

mR

N9nD

SuM

P

dT

>A

B7

c

V

J

g

EK2

OX

Cj

k

IZoF

q

3

f

Q

p

+

vr

w

z

U

/

H

y

<latexi sh

1_b

64="

OXYFz

Vu

D

Wg

f

j

d

wI

>A

B

n

c

LS

N

E

G

r

H

8

3

k

P

5

J

m

Q2

7

v

o

q

K

Z

U

R

/9

C

0

T

+

<latexi sh

1_b

64="AS9ocXNwGUdP

Bkyp

u

FM

E

>

H

VDL

g

Oz

r

Y

8

3

W

J

7

K

Q

n

+

Tv

I

j

2

5

/

q

C

m

Z

fR

0

<latexi sh

1_b

64="Rn w0UT

O

KyBH

GgN

jc

>A

VDLS

E

z

Mr

Y

8

3

kWP

J

p

7

Q

F

+

v

I9

Xd

u

2

5

/

q

C

m

o

Z

f

<latexi sh

1_b

64="W

XP0MI9+B

Kf8

LV

Ry

Q

>A

7

c

D

S

N

F

2pr

Zdu ovgq

T

w

C

U

O

/

G

J

3

H

k

mj

n

5

Y

E

z

<latexi sh

1_b

64="EAZ

D

o

VTn75M

U

p

S

I

>

B

c

L

gN

Oz

G

r

H

Y

8

3

dQ

Rj

PC

w+

k

y

m

W

F

q9/

u

v

K2

J

f

X

0

where Pooler is RoIAlign [15] that pools region features according to their locations from the entire feature maps, and b is the location of the object, which could be ground-truth, predicted, or predefined whole-image bounding boxes.

It is worth pointing out with such a formula both detection and classification datasets can now be unified, as objects in detection datasets are essentially regions of images, while images from classification datasets are cropped regions from original images. Consequently, the b in Equation (2) indicate tight ground-truth bounding boxes b od for objects in detection datasets, while b corresponds to the entire image, i.e ., coarse whole-image boxes b img = (0 , 0 , h, w ) , for samples from classification datasets, where h , w is the height and width of the augmented image.

Furthermore, unlike previous approaches that use fixed and static semantics provided by image-level labels, we obtain image-level and object-level semantics which dynamically change conditioned on different locations ( i.e ., bounding boxes) in an online fashion. This is particularly useful when location information is not accurate-the location could be a very loose bounding box for the object of interest or even contains part of the objects. As as result, the rich and location-robust semantics can be used to guide the detector.

3.2 Semantics as Soft Labels

Now that we obtain both image-level and object-level semantics with CLIP, the main challenge is how to make our detector learn from them effectively. Traditional detection models [40, 31, 2, 3, 65] usually obtain object features from whole-image features according to locations [9, 40, 31, 49] or cross-attention [3, 65]. Then object features are refined with a classification branch to classify the region into predefined categories, and a location branch to compute the bounding box, separately. To unify the notations, we denote the object features as O , which are fed to different branches to obtain features tailored for each sub-task to generate final outputs with a projection layer. More formally,

where H denotes the sub-task branch for feature refinement, specifically H loc is for localization and H cls is for classification; W t denotes the projection layer to produce final results; f t , and o t denotes the features and outputs for the corresponding task, i.e ., t ∈ { loc, cls } .

Now we discuss how to use the obtained semantics to guide the training of detectors. As o cls are logits that are generally used for classification, a straightforward way is, on top of a cross-entropy loss, to use o cls to predict the object semantics s t in a similar spirit to knowledge distillation [17, 41]. However, this makes the training of detectors challenging as the cross-entropy loss and distillation loss have conflicting purposes: a cross-entropy loss is generally optimized to produce hard prediction results while the distillation loss aims to borrow knowledge from a distribution of semantic scores that serve as 'soft labels'. As a result, jointly performing hard predictions and soft distillations conditioned on o cls produce unsatisfactory results, as will be shown empirically.

To mitigate the challenge, we introduce an additional branch named as 'semantic branch', independent of the classification and localization branches in current detectors, to learn the soft semantics obtained by CLIP. The semantic branch also includes a feature refinement branch H soft then feed the refined feature f soft to Contrast function to obtain the semantic prediction o soft . Such a strategy ensures that semantics from image classification datasets can be explored without interfering with the original training process of detectors. This is essentially using semantics as soft teachers to implicitly refine object features for detection. Formally, to train the semantic branch, we use a KL divergence loss as:

where o soft i is the semantics prediction of the i -th object feature, while s t i is the corresponding semantic target; N is the number of matched proposals/ queries during training. It is worth noting that we only use the semantic branch for distilling semantics into the detector during training. And once trained, the semantic branch is no longer needed for inference.

3.3 Unified Objective Functions

We aim to incorporate semantic learning from classification datasets into an end-to-end learning scheme instead of fine-tuning after pre-training [62], without redundant pre-computed boxes [11] and box estimation [34, 38]. Thus, we unify the objective functions and treat datasets of classification and detection equally in a unified way.

In our training scheme, we use a unified classification loss L unicls , the combination of a hard classification loss and a soft semantics loss, to boost the capability of object classification. The hard classification loss L cls best works when bounding boxes and class labels are available and sufficient for training. In contrast, the soft semantic learning loss L soft works well for those categories with few samples, requiring only coarse locations.

where λ loc , λ cls and λ soft means the weight of each loss.

Furthermore, the vocabularies of target detection datasets and extra classification datasets often differ. To handle the taxonomy difference between detection and classification datasets, we further unify the supervision for hard classification as follows.

Handling taxonomy differences. To leverage image-level labels in extra data, we need to map the vocabulary of extra data C extra to the vocabulary of our target detection dataset C od , which is a function M : C extra →C od . One could derive better mapping functions M , yet this is orthogonal to our current direction. For convenience, we can use a manual label mapper or leverage the CLIP text encoder to map the class according to their category semantics automatically.

Handling data without any labels. We also consider a broader case, in which even image-level labels are not available. Inspired by DeiT [50], we treat the class of highest logits in each semantics target as the label for classification. We also filter c hard with a threshold to filter out those images relevant to the vocabulary of the target set.

where c hard is the generated class label for hard classification.

4 Experiments

We conduct experiments and analysis on the task of long-tailed object detection. We mainly evaluate our method on LVIS [12] val 1.0 for our main experiments and ablations. We also conduct experiments on other datasets of long-tail distribution to further prove the effectiveness. We use DINO [61], a advanced DETR-based detector due to the training efficiency and high performance.

4.1 Datasets and evaluation metrics.

Long-tail detection dataset. We mainly conduct experiments on LVIS [12], which contains 1203 classes with ∼ 100K images. The classes are divided into rare, common, and frequent groups based on the number of training images. The category distribution is extremely long-tailed since instances in rare categories are usually less than 10. Moreover, we experiment on other datasets, e.g . Visual Genome [22] and OpenImages [24], please refer to our supplementary material.

Extra data. We mainly use ImageNet-21k [5] as additional classification data. ImageNet-21k consists of ∼ 14M images for 21K classes, and there are 997 classes that overlap with the vocabulary in LVIS. We follow the label mapping used in Detic [63] and denote the subset of overlapped classes as ImageNet-LVIS, which contains ∼ 1.5M images with 997 LVIS classes. For the full set, we treat the data as unlabeled images and leverage our learning scheme to learn from it. We also treat the

full set as unlabeled dataset ignoring the image-level labels, denoted as INet-Unl. Furthermore, we explore an additional detection dataset Object365 [42] and an image-text pair dataset CC3M [43].

Evaluation metric. We report box AP on LVIS with the LVIS official evaluator, which also contains AP r , AP c , AP f for rare, common and frequent class, respectively. We mainly focus on the performance gain on overall AP and AP of rare categories.

4.2 Implementation details

Baseline model. We take DINO [61], a powerful DETR-based detector with advanced query denoising [27] as our baseline. Differently, we convert the object classification into vision-language contrasting using the text features of categories.

Mixed dataset training. For experiments with extra data, we combine both detection datasets and extra datasets together to train our detector. We sample images from detection and classification datasets in a 1:1 ratio regardless of the original size of each dataset. Images in the same batch are sampled from the same dataset for high training efficiency. For mixed dataset training, we denote the schedule as the number of iterations on the target dataset for fair comparisons.

Training details. We adopt PyTorch for implementation and use 8 × V100 GPUs. We set the initial learning rate as 1 e -4 and multiply 0.1 at the 11-th, 20-th and 30-th epoch for 1 × , 2 × and 3 × , respectively, and set λ soft = 0 . 5 for the soft semantics learning loss. Following [63], we use a federated loss [64] and repeat factor sampling [12] for LVIS; we use category aware sampling for OpenImages. We randomly resize an input image with its shorter side between 480 and 800, limiting

the longer size below 1333. Unlike training large models with larger scale images [29, 61], we use the same recipe for all models without any tricks or test time augmentation (TTA). For training with extra data, we use CLIP-RN50 to extract the semantic guidance for most models, while we use CLIP-RN50 × 16 [37] for Swin-L [33] and Focal-L [56] for better performance.

4.3 Main Results

Results on LVIS. Tab. 1 shows the result on LVIS. As shown in the table, detectors trained with our framework outperforms those trained with regular training recipes, especially for rare categories.

When using a ResNet50 [16] as backbone networks, our baseline model is fully converged under 3 × schedule. Our RichSem under 2 × still outperforms the fully converged baseline, which demonstrates the effectiveness of our proposed method. Moreover, the experiments with Swin [33] backbones show consistent gains. Our model with a Swin-L as backbone achieves 52.0 AP and 50.2 AP r , which outperforms the previous SoTA [29] by a large margin only under 1/4 training schedule. Overall, our RichSem makes the detector better on tailed categories and achieves significant gains on the corresponding AP, e.g . rare and common categories. Moreover, with better backbones, our RichSem achieves even balanced perfor-

mance on rare, common and frequent categories. Notably, with a much smaller model size and standard data augmentations for training and testing, our method rivals the ViTDet [29] that uses hugesized model and sophisticated training recipes. We also compare compare the models with Faster R-CNN [40] as detector shown in Table 2. We further compare with large-scale vision foundation models [7, 53], please refer to our supplementary material.

4.4 Ablation Study

Comparisons with CLIP initialization. We replace the ResNet50 backbone with pre-trained CLIP visual encoder to study whether it can lead to balanced learning. As shown in Table 3a, although pre-trained CLIP has a strong ability to capture visual semantics, the detection performance even drops after finetuning on LVIS. The low AP r of CLIP backbone indicates that it is still biased towards the long-tailed distribution of downstream datasets. Therefore, using a frozen pre-trained CLIP to extract visual semantics as extra 'soft supervision' can keep the generalization capability than building upon CLIP as initialization.

Effectiveness of the semantics branch. We study the importance of the proposed semantics branch by sharing the parameters of two heads instead of using two independent heads. As shown in Table 3b, when two heads share the parameters, the performance drops drastically and is even worse than the baseline model. This suggests that the objective of semantics learning differs from object classification. Therefore, semantics should be learned independently with the semantics branch.

Effectiveness of rich semantics of classification data. We conduct experiments using different types of extra data. Initially, we utilized Object365 [42], a detection dataset, and utilized its box annotations irrespective of the class labels. However, the performance gain was limited, as shown in Table 3c. This further verifies the assumption that the localization capability is not the primary bottleneck for long-tailed object detection [11]. Next, we experiment with ImageNet [5] and CC3M [43], two datasets with whole-image labels. We treat these datasets as unlabeled data and generate pseudo labels using Equation (7), denoted as INet-Unl and CC3M-Unl, respectively. As shown in Table 3c, utilizing these two datasets as extra data significantly outperformed the baseline, particularly for the rare categories. Furthermore, we employed ImageNet-LVIS, taxonomy mapped using Equation (6), as extra data, which yields the best performance. Overall, the experiments demonstrate that longtailed object detection is primarily affected by object classification. Consequently, introducing rich semantics within classification data effectively alleviates this limitation. It is worth noting that this showcases our potential for further extension to unlabeled data.

(a) Compared with CLIP initialized baseline.

(c)

Different extra datasets.

(b)

Ablations on the semantic branch.

(d)

Soft semantics provided by different CLIP models.

(f) Ablations on backbone pre-training data and downstream detection data. (g) L soft and L loc on extra data.

(h)

Ablations on the weight of soft semantics loss.

Stronger semantics lead to better detectors. We study the impact of CLIP with different backbones. The results are shown in Table 3d. With the capability increased, the model achieves better performance on overall AP. Besides, built upon a RN50 × 16, our detector obtains significant improvements on all metrics.

Semantic learning is effective to leverage extra data. Due to the data overlap between backbone pretraining and detector training, we conduct an ablation on them to further demonstrate the effectiveness of our method. As shown in the Table 3f, pre-training on large-scale data (ImageNet-21k) can provide strong perception capability for the downstream detection task, with overall performance gain. However, the performance on rare categories is still relatively low, indicating that this approach does not well alleviate the long-tail effects in detection. In contrast, our method is more effective than pre-training to handle long-tailed detection, especially for the tail categories. Notably, our approach is still effective with strong pre-trained backbones, further improving performance on long-tailed object detection.

Semantics as soft targets leads to balanced learning. The effectiveness of soft labels for low-shot categories can be reaffirmed in Table 3e: when using extra data, detectors trained with the soft semantics learning loss achieve better performance than those trained without the loss, especially for rare categories and unlabeled extra data. Besides, our semantic learning on extra classification boosts detection and classification at the same time thanks to our unified objective functions ( Section 3.3). As shown in Table 3g, both rich semantics and coarse locations play significant roles in boosting long-tail object detection. We further study the impact of soft classification objectives. We first experiment without using extra data. As shown in the Table 3h, with distillation weight set to 1, the performance of rare categories increases by ∼ 3AP, while the results of frequent categories decrease by 1.4. This suggests that semantics learning benefits those low-shot categories rather than overfitting frequent categories. As the distillation weight decreases, the performance of rare classes also drops but the results of frequent categories improve. After exploring additional data, the performance is no longer sensitive to the distillation weight. Thanks to the additional data, semantics learning leads to

(e)

Soft semantics loss

L

soft

.

armor

space_shuttle

ferret

table-tennis_table

quesadilla_table

quesadilla_table

quesadilla_table

cleat_(for_securing_rope)

dinghy

dinghy

chocolate_mousse

chocolate_mousse

road_map

joystic

combination_lock

ferret

soup_bowl

cocoa (beverage)

H3

table

[coaster

ketknife

scissors

monitor

computerkeyboard

box

DOX

box

tomputerkeyboard

STOP

inet

A1

thumbtack

[thumbtack

chocolate_bar

packet

pastry

tissue_paper

pretzel)

(knife

pencil

eog

boat

baseballcap

spectacles

duckbird

B6

B4

tapestry

bowl

bowl

plate

dumpster

ELLOAD.

pea（food)

bowl

bottle

lersey

plate

shield

shield

bOWI

chopstick

DOWI

dragonfly (r)

bolo_tie (r)

basketball (c)

pretzel (c)

lemon (f)

mailbox_(at_home) (f)

feature distribution

Baseline

RichSem

O

0

-2'2

1

0

-2

1

0

-2-2

I-

-22

-1

0

1

-22

-1

0

-2

-1

0

1

-22

-1

0

2

2

2

2

2

0

D

0

-1

=

-1

-1

-1

0

1

I-

-1

0

-1

consistent performance gains for all categories. We further visualize the object features of our method. Specifically, we normalize the features and employ Gaussian Kernel Density Estimation (KDE) in R 2 , following [52] to compare the distribution of object features across categories. As shown in the Figure 4, the visualization indeed shows a clear distinction between the baseline and our RichSem. Regarding the baseline, the distribution of object features lacks differentiation, often resulting in overlapping patterns among categories, especially between rare and frequent categories. In contrast, in RichSem, features belonging to each category, even rare categories, are well-clustered. This clear intra-class and inter-class distribution indicate that our approach effectively enhances the region classification capability of diverse categories. Therefore, our method effectively leads balanced learning among classes with varying frequencies.

5 Conclusions and Future Work

We presented RichSem, a simple but effective way, to leverage extra data by learning rich semantics and coarse locations to boost long-tailed object detection, alleviating the semantics insufficiency and location sensitivity caused by taking image-level labels as supervision. Through extensive experiments, we demonstrate our approach achieves state-of-the-art performance, without requiring complex training and testing procedures. One possible limitation is that we treat the detection data and classification data equally and use the same unified classification loss, which may be sub-optimal for the categories with sufficient samples.

For future work, we believe our framework can be extended to semi-supervised object detection (SSOD) where golden labels are not annotated on the unlabeled data, since our method is found to be robust when annotations are partially given. Moreover, the soft semantics learning for classification data can be naturally applied for open-vocabulary and multi-dataset detection tasks.

Acknowledgement This project was supported by National Key R&D Program of China (No. 2021ZD0112805) and National Natural Science Foundation of China (No. 62102092).

References

H. Bilen and A. Vedaldi. Weakly supervised deep detection networks. In CVPR , 2016.

Z. Cai and N. Vasconcelos. Cascade r-cnn: Delving into high quality object detection. In CVPR , 2018.

N. Carion, F. Massa, G. Synnaeve, N. Usunier, A. Kirillov, and S. Zagoruyko. End-to-end object detection with transformers. In ECCV , 2020.

S. Changpinyo, P. Sharma, N. Ding, and R. Soricut. Conceptual 12M: Pushing web-scale image-text pre-training to recognize long-tail visual concepts. In CVPR , 2021.

J. Deng, W. Dong, R. Socher, L.-J. Li, K. Li, and L. Fei-Fei. Imagenet: A large-scale hierarchical image database. In CVPR , 2009.

K. Desai and J. Johnson. Virtex: Learning visual representations from textual annotations. In CVPR , 2021.

Y. Fang, W. Wang, B. Xie, Q. Sun, L. Wu, X. Wang, T. Huang, X. Wang, and Y. Cao. Eva: Exploring the limits of masked visual representation learning at scale. In CVPR , 2023.

G. Ghiasi, Y. Cui, A. Srinivas, R. Qian, T.-Y . Lin, E. D. Cubuk, Q. V . Le, and B. Zoph. Simple copy-paste is a strong data augmentation method for instance segmentation. In CVPR , 2021.

R. Girshick. Fast r-cnn. In CVPR , 2015.

R. Girshick, J. Donahue, T. Darrell, and J. Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In CVPR , 2014.

X. Gu, T.-Y. Lin, W. Kuo, and Y. Cui. Open-vocabulary object detection via vision and language knowledge distillation. In ICLR , 2021.

A. Gupta, P. Dollar, and R. Girshick. Lvis: A dataset for large vocabulary instance segmentation. In CVPR , 2019.

G. Han, J. Ma, S. Huang, L. Chen, R. Chellappa, and S.-F. Chang. Multimodal few-shot object detection with meta-learning based cross-modal prompting. arXiv preprint arXiv:2204.07841 , 2022.

K. He, X. Chen, S. Xie, Y. Li, P. Dollár, and R. Girshick. Masked autoencoders are scalable vision learners. In CVPR , 2022.

K. He, G. Gkioxari, P. Dollár, and R. Girshick. Mask r-cnn. In ICCV , 2017.

K. He, X. Zhang, S. Ren, and J. Sun. Deep residual learning for image recognition. In CVPR , 2016.

G. Hinton, O. Vinyals, J. Dean, et al. Distilling the knowledge in a neural network. arXiv preprint arXiv:1503.02531 , 2015.

C. Jia, Y . Yang, Y . Xia, Y .-T. Chen, Z. Parekh, H. Pham, Q. Le, Y .-H. Sung, Z. Li, and T. Duerig. Scaling up visual and vision-language representation learning with noisy text supervision. In ICML , 2021.

X. Jiao, Y. Yin, L. Shang, X. Jiang, X. Chen, L. Li, F. Wang, and Q. Liu. Tinybert: Distilling bert for natural language understanding. In EMNLP , 2020.

A. Kamath, M. Singh, Y. LeCun, G. Synnaeve, I. Misra, and N. Carion. Mdetr-modulated detection for end-to-end multi-modal understanding. In CVPR , 2021.

B. Kang, S. Xie, M. Rohrbach, Z. Yan, A. Gordo, J. Feng, and Y. Kalantidis. Decoupling representation and classifier for long-tailed recognition. In ICLR , 2019.

R. Krishna, Y. Zhu, O. Groth, J. Johnson, K. Hata, J. Kravitz, S. Chen, Y . Kalantidis, L.-J. Li, D. A. Shamma, et al. Visual genome: Connecting language and vision using crowdsourced dense image annotations. IJCV , 2017.

J. Kuen, F. Perazzi, Z. Lin, J. Zhang, and Y.-P. Tan. Scaling object detection by transferring classification weights. In CVPR , 2019.

A. Kuznetsova, H. Rom, N. Alldrin, J. Uijlings, I. Krasin, J. Pont-Tuset, S. Kamali, S. Popov, M. Malloci, A. Kolesnikov, et al. The open images dataset v4. IJCV , 2020.

D.-H. Lee et al. Pseudo-label: The simple and efficient semi-supervised learning method for deep neural networks. In Workshop on ICML , 2013.

B. Li, Y. Yao, J. Tan, G. Zhang, F. Yu, J. Lu, and Y . Luo. Equalized focal loss for dense long-tailed object detection. In CVPR , 2022.

F. Li, H. Zhang, S. Liu, J. Guo, L. M. Ni, and L. Zhang. Dn-detr: Accelerate detr training by introducing query denoising. In CVPR , 2022.

L. H. Li, P. Zhang, H. Zhang, J. Yang, C. Li, Y. Zhong, L. Wang, L. Yuan, L. Zhang, J.-N. Hwang, et al. Grounded language-image pre-training. In CVPR , 2022.

Y. Li, H. Mao, R. Girshick, and K. He. Exploring plain vision transformer backbones for object detection. In ECCV , 2022.

Y. Li, T. Wang, B. Kang, S. Tang, C. Wang, J. Li, and J. Feng. Overcoming classifier imbalance for long-tail object detection with balanced group softmax. In CVPR , 2020.

T.-Y. Lin, P. Goyal, R. Girshick, K. He, and P. Dollár. Focal loss for dense object detection. In ICCV , 2017.

T.-Y. Lin, M. Maire, S. Belongie, J. Hays, P. Perona, D. Ramanan, P. Dollár, and C. L. Zitnick. Microsoft coco: Common objects in context. In ECCV , 2014.

Z. Liu, Y. Lin, Y . Cao, H. Hu, Y . Wei, Z. Zhang, S. Lin, and B. Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In ICCV , 2021.

Z. Ma, G. Luo, J. Gao, L. Li, Y. Chen, S. Wang, C. Zhang, and W. Hu. Open-vocabulary one-stage detection with hierarchical visual-language knowledge distillation. In CVPR , 2022.

L. Meng, X. Dai, Y. Chen, P. Zhang, D. Chen, M. Liu, J. Wang, Z. Wu, L. Yuan, and Y.-G. Jiang. Detection hub: Unifying object detection datasets via query adaptation on language embedding. In CVPR , 2023.

L. Meng, H. Li, B.-C. Chen, S. Lan, Z. Wu, Y.-G. Jiang, and S.-N. Lim. Adavit: Adaptive vision transformers for efficient image recognition. In CVPR , 2022.

A. Radford, J. W. Kim, C. Hallacy, A. Ramesh, G. Goh, S. Agarwal, G. Sastry, A. Askell, P. Mishkin, J. Clark, et al. Learning transferable visual models from natural language supervision. In ICML , 2021.

H. Rasheed, M. Maaz, M. U. Khattak, S. Khan, and F. S. Khan. Bridging the gap between object and image-level representations for open-vocabulary detection. In NeurIPS , 2022.

J. Redmon and A. Farhadi. Yolo9000: better, faster, stronger. In CVPR , 2017.

S. Ren, K. He, R. Girshick, and J. Sun. Faster r-cnn: Towards real-time object detection with region proposal networks. In NeurIPS , 2015.

V. Sanh, L. Debut, J. Chaumond, and T. Wolf. Distilbert, a distilled version of bert: smaller, faster, cheaper and lighter. arXiv preprint arXiv:1910.01108 , 2019.

S. Shao, Z. Li, T. Zhang, C. Peng, G. Yu, X. Zhang, J. Li, and J. Sun. Objects365: A large-scale, high-quality dataset for object detection. In CVPR , 2019.

P. Sharma, N. Ding, S. Goodman, and R. Soricut. Conceptual captions: A cleaned, hypernymed, image alt-text dataset for automatic image captioning. In ACL , 2018.

Z. Shen and E. Xing. A fast knowledge distillation framework for visual recognition. In ECCV , 2022.

J. Tan, X. Lu, G. Zhang, C. Yin, and Q. Li. Equalization loss v2: A new gradient balance approach for long-tailed object detection. In CVPR , 2021.

J. Tan, C. Wang, B. Li, Q. Li, W. Ouyang, C. Yin, and J. Yan. Equalization loss for long-tailed object recognition. In CVPR , 2020.

P. Tang, X. Wang, S. Bai, W. Shen, X. Bai, W. Liu, and A. Yuille. Pcl: Proposal cluster learning for weakly supervised object detection. TPAMI , 2018.

Y. Tian, D. Krishnan, and P. Isola. Contrastive representation distillation. In ICLR , 2020.

Z. Tian, C. Shen, H. Chen, and T. He. Fcos: Fully convolutional one-stage object detection. In CVPR , 2019.

H. Touvron, M. Cord, M. Douze, F. Massa, A. Sablayrolles, and H. Jégou. Training data-efficient image transformers & distillation through attention. In ICML , 2021.

J. Wang, W. Zhang, Y. Zang, Y. Cao, J. Pang, T. Gong, K. Chen, Z. Liu, C. C. Loy, and D. Lin. Seesaw loss for long-tailed instance segmentation. In CVPR , 2021.

T. Wang and P. Isola. Understanding contrastive representation learning through alignment and uniformity on the hypersphere. In ICML , 2020.

W. Wang, J. Dai, Z. Chen, Z. Huang, Z. Li, X. Zhu, X. Hu, T. Lu, L. Lu, H. Li, et al. Internimage: Exploring large-scale vision foundation models with deformable convolutions. In CPVR , 2023.

Z. Wang, N. Codella, Y.-C. Chen, L. Zhou, X. Dai, B. Xiao, J. Yang, H. You, K.-W. Chang, S.-f. Chang, et al. Multimodal adaptive distillation for leveraging unimodal encoders for vision-language tasks. arXiv preprint arXiv:2204.10496 , 2022.

Z. Weng, X. Yang, A. Li, Z. Wu, and Y.-G. Jiang. Open-vclip: Transforming clip to an open-vocabulary video model via interpolated weight optimization. In ICML , 2023.

J. Yang, C. Li, X. Dai, and J. Gao. Focal modulation networks. In NeurIPS , 2022.

K. Ye, M. Zhang, A. Kovashka, W. Li, D. Qin, and J. Berent. Cap2det: Learning to amplify weak caption supervision for object detection. In CVPR , 2019.

S. Yun, S. J. Oh, B. Heo, D. Han, J. Choe, and S. Chun. Re-labeling imagenet: from single to multi-labels, from global to localized labels. In CVPR , 2021.

A. Zareian, K. D. Rosa, D. H. Hu, and S.-F. Chang. Open-vocabulary object detection using captions. In CVPR , 2021.

C. Zhang, T.-Y. Pan, Y. Li, H. Hu, D. Xuan, S. Changpinyo, B. Gong, and W.-L. Chao. Mosaicos: a simple and effective use of object-centric images for long-tailed object detection. In ICCV , 2021.

H. Zhang, F. Li, S. Liu, L. Zhang, H. Su, J. Zhu, L. Ni, and H.-Y. Shum. Dino: Detr with improved denoising anchor boxes for end-to-end object detection. In ICLR , 2022.

Y. Zhong, J. Yang, P. Zhang, C. Li, N. Codella, L. H. Li, L. Zhou, X. Dai, L. Yuan, Y. Li, et al. Regionclip: Region-based language-image pretraining. In CVPR , 2022.

X. Zhou, R. Girdhar, A. Joulin, P. Krähenbühl, and I. Misra. Detecting twenty-thousand classes using image-level supervision. In ECCV , 2022.

X. Zhou, V. Koltun, and P. Krähenbühl. Probabilistic two-stage detection. arXiv preprint arXiv:2103.07461 , 2021.

X. Zhu, W. Su, L. Lu, B. Li, X. Wang, and J. Dai. Deformable detr: Deformable transformers for end-to-end object detection. In ICLR , 2021.

B. Zoph, G. Ghiasi, T.-Y. Lin, Y. Cui, H. Liu, E. D. Cubuk, and Q. Le. Rethinking pre-training and self-training. In NeurIPS , 2020.

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper uses the vision-language pre-trained model CLIP for long-tailed object detection.\nThe key idea is to consider not only image-level semantics but also region-level semantics, and fuse them under a soft-label scenario.\nThe overall idea is deployed on different object detectors with different backbones.\nExperiments are conducted on three long-tailed object detection datasets with extra data.\nBesides, the experiments and discussion on ablation studies and foundational models are very extensive.\n\nSTRENGTHS:\n+ To the best of the reviewer’s knowledge, it is the first work (excluding CVPR2023) to use vision-language pipelines for long-tailed object detection. The contribution is significant. \n\n+ The ablation studies and other discussions are very extensive, in both the main submission and the\n\n+ This paper is well-written and easy-to-follow.\n\nWEAKNESSES:\n- This work, in its current form, lacks theoretical insight, especially on how the proposed method fits into the long-tailed feature distribution. \n\nAs the proposed method is targeted at the context of long-tailed object detection, not the generic object detection, it is very necessary to extensively discuss how the framework and the specific module design can work well or fit into the long-tailed feature distribution.\n\nUnfortunately, as the reviewer goes through the entire manuscript: In Fig.1 motivation case and Fig.2 framework, there is no element for the long-tailed feature space or long-tailed elements. Besides, in the methodology section, from learning the image / object semantics to use soft labels, all these presentations are generic and are suitable for generic recognition, detection, segmentation and etc. More importantly, in the experimental section, there is no feature space visualization on how CLIP benefit the long-tailed feature distribution.  \n\n- The state-of-the-art comparison is insufficient, and to some extent, unfair. Specially,\n\nIt is really strange that in Table 1, the authors only compare some generic vision transformer based detectors such as DERT. Why the methods, using vision-language pre-trained model for generic object detection, such as [11,18,27,34,57], are not involved for comparison on the long-tailed object detection datasets? It is very likely that these methods [11,18,27,34,57] for generic vision-language object detection can work well on long-tailed detection datasets.\n\n- The performance of the proposed method seems to be mainly brought high by the use of Swin-Transformer.\n\nThis remark is not saying that the state-of-the-art performance with Swin-Transformer backbone is not important, but the reviewer would like to raise the attention that, when use the old ResNet-50 backbone (Row 1 of Table 1), the proposed method, even with vision-language pre-trained model, does not lead a significant performance against some SOTA in 2021. \n\nThus, the reviewer feels that in the future, why with Resnet-50 is not that effective for the proposed framework, worth to be further investigated. \n\nOther minor issues to improve:\n\n- Extensive feature visualization on the context of both long-tail and CLIP features, please. \n\n- Please use \\mathcal{} to present the loss function, and use \\mathbf{} to present the tensors and vectors. These notations can distinguish them from scalars.\n\nQUESTIONS:\nThe reviewer really appreciate that the work is the first to extend vision-language pre-trained model to the long-tailed object detection task. However, the weakness of this submission is also very obvious.\n\nIn the rebuttal, please address the weakness part point-by-point, on lack insight for long-tailed context, lack state-of the-art comparison, and to-some-extent performance ineffectiveness.\n\nLIMITATIONS:\nThe limitation is not properly discussed", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-06-16"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\n- This work deals with Long-tail object detection. Authors identify two problems with using additional data, namely Semantic ambiguity and Location sensitivity. \n- Authors identify that semantic ambiguity arises due to supervision with one-hot encoded labels from the image datasets and instead propose to use CLIP scores for supervision.\n- CLIP's ability to provide sufficient semantic information even with course locations is leveraged to tackle location sensitivity.\n- Authors propose RichSem, a simple yet effective method, that adds an additional semantic branch to the detector to learn rich semantics from images. \n- The semantic branch is only required during training and with extensive experiments is shown to achieve state-of-the-art results on LVIS dataset in overall and rare categories.\n\nSTRENGTHS:\n- The paper is well written and the presentation makes it easy to follow.\n- The experiment section is exhaustive and supports all the claims made by the authors. Authors test their method with the transformer and R-CNN based family of detectors and the ablation experiments clearly explain the contribution of each component.\n- RichSem is a principled way to leverage additional data for long-tailed object detection.\n\nWEAKNESSES:\n- Authors miss the comparison with [1], which also uses additional data. Please compare appropriately in Table-1.\n- The current work heavily relies on CLIP but it is widely known that CLIP has several limitations [2]. It would be interesting to address the robustness of the current method to the limitations of CLIP.\n\n[1] Bo Li, Yongqiang Yao, Jingru Tan, Xin Lu, Fengwei Yu, Ye Luo, and Jianwei Lu, Improving Long-tailed Object Detection with Image-Level Supervision by Multi-Task Collaborative Learning.\n\n[2] https://stanislavfort.github.io/blog/OpenAI_CLIP_stickers_and_adversarial_examples/\n\nQUESTIONS:\n- Authors mention constructing a mosaic with multiple images in L51, where in the whole method is this done?\n- What is $f_v^i$ in Eq. 4? Is that $s^t$ obtained in Eq. 2?\n- From table 2c, rows 4,5 the increase in improvement is because of image level labels. But how are the annotations used in the whole pipeline? The semantic branch only uses CLIP similarities for the KL divergence loss? Is a hard supervised loss also being applied for the image labels? Is there a localization term for the image level labels?\n- What is $f^t$ in the semantic branch in Fig. 2? If $t\\in (\\text{loc},\\text{cls})$, then is the semantic soft loss applied to both $f^{\\text{loc}}$ and $f^{\\text{cls}}$?\n\nLIMITATIONS:\nI do not foresee any potential negative societal impact of this work.", "TITLE": "Official Review", "RECOMMENDATION": 7, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-04"}

comments: SUMMARY:
To address semantic ambiguity and location sensitivity, this paper introduces a one-stage training framework that leverages additional image data to boost the detector through learning from rich semantics and coarse locations for long-tailed object detection. And their RichSem achieves consistent improvements on both overall and rare-category of LVIS under different backbones and detectors.

STRENGTHS:
1. The paper is clearly written. And main idea of the paper is easy to understand.
2. This paper introduces a novel semantics learning framework, which uses an additional branch to learn from rich semantics and coarse locations for long-tailed object detection without the need to compute pseudo labels.
3. Extensive method demonstrates significant results on long-tailed datasets

WEAKNESSES:
1. In Long-tailed object detection, the scarcity of samples or natural constraints results in a limited number of instances in the tail classes. Can exploring extra data effectively address the issue of scarce tail classes in practical applications?
2. CLIP is a large-scale model designed for the joint processing of images and text. The lack of representative samples for tail classes may lead to a relatively weak understanding and recognition ability of CLIP for these classes. Can CLIP still provide stable semantic guidance?

QUESTIONS:
Please check the paper weaknesses.

LIMITATIONS:
Please check the paper weaknesses.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06

comments: SUMMARY:
This paper adopts the CLIP model to obtain a 'soft label' supervision to train the detector under the long-tail distribution dataset and derive rich semantics from the CLIP part to enhance the tail-categories representations, which can be removed during the inference. The authors claim that the CLIP model can well capture visual semantics conditioned on only coarse locations, the whole-bbox for extra data in this paper. They then elaborate soft-label supervision to train the detector so as to alleviate semantic ambiguity and location sensitivity issues. Various ablative studies have been conducted to validate the effectiveness of the proposed method.

STRENGTHS:
1. The novelty of this paper sounds technically reasonable.
2. The writing of this paper is easy to follow.
3. The final performances look competitive and the improvements are obvious.

WEAKNESSES:
1. In Line 172, the subscript of f and o maybe should be superscript.
2. The proposed semantic branch training is parallel to the detection heads during the training to avoid the training conflicts for the CE loss and KL loss, but the detection heads are only trained on the tailed samples in LVIS without some kind of interactions on the semantic branch and how to ensure that the detection heads can well handle the rare categorical samples during inference?
3. ImageNet-21K is used as extra data in this paper, but the backbone is pre-trained in ImageNet too. So is there any dataset overlapping between the pertaining and fine-tuning stages?

QUESTIONS:
Please regard the weakness parts.

LIMITATIONS:
Please regard the weakness parts.
TITLE: Official Review
SOUNDNESS_CORRECTNESS: 3
DATE: 2023-07-06