Title: Boosting Semi-Supervised Few-Shot Object Detection with SoftER Teacher
Venue: NeurIPS 2023
Accepted: no
Keywords: label-efficient object detection

semi-supervised few-shot object detection

generalized few-shot object detection
Abstract: Few-shot object detection (FSOD) is an emerging problem aimed at detecting novel concepts from few exemplars. Existing approaches to FSOD assume abundant base labels to adapt to novel objects. This paper studies the task of semi-supervised FSOD by considering a realistic scenario in which both base and novel labels are simultaneously scarce. We explore the utility of unlabeled data and discover its remarkable ability to boost semi-supervised FSOD by way of region proposals. Motivated by this finding, we introduce SoftER Teacher, a robust detector combining pseudo-labeling with representation learning on region proposals, to harness unlabeled data for improved FSOD without relying on abundant labels. Extensive experiments show that SoftER Teacher surpasses the novel performance of a strong supervised detector using only 10% of required base labels, without experiencing catastrophic forgetting observed in prior approaches. Our work also sheds light on a potential relationship between semi-supervised and few-shot detection suggesting that a stronger semi-supervised detector leads to a more effective few-shot detector.

Abstract

Few-shot object detection (FSOD) is an emerging problem aimed at detecting novel concepts from few exemplars. Existing approaches to FSOD assume abundant base labels to adapt to novel objects. This paper studies the task of semi-supervised FSOD by considering a realistic scenario in which both base and novel labels are simultaneously scarce. We explore the utility of unlabeled data and discover its remarkable ability to boost semi-supervised FSOD by way of region proposals. Motivated by this finding, we introduce SoftER Teacher, a robust detector combining pseudo-labeling with representation learning on region proposals, to harness unlabeled data for improved FSOD without relying on abundant labels. Extensive experiments show that SoftER Teacher surpasses the novel performance of a strong supervised detector using only 10% of required base labels, without experiencing catastrophic forgetting observed in prior approaches. Our work also sheds light on a potential relationship between semi-supervised and few-shot detection suggesting that a stronger semi-supervised detector leads to a more effective few-shot detector.

1 Introduction 15

16

Modern object detection systems have enjoyed tremendous progress in recent years, with many successful applications across diverse industries. Their success can be mainly attributed to the availability of large-scale, well-annotated datasets such as the MS-COCO benchmark [28]. However, the demand for more powerful and accurate detection models requires considerable investments in the hand-labeling of massive amounts of data, which are costly to scale. Thus, there is a growing trend in the community to shift toward a more label-efficient paradigm, one that can enable detection models to do more with less hand-labeled data. Such emerging research directions include selfsupervised representation learning from unlabeled data [2, 7, 39], multi-modal pre-training from Web-labeled data [25, 32], semi-supervised detection (SSOD) [30, 48, 52], and few-shot detection (FSOD) [10, 11, 47], all of which have shown great promise in alleviating the dependency on large amounts of instance-level class annotations and bounding boxes.

32

34

36

38

40

42

44

46

Base AP (%)

28

30

32

34

36

Overall Base

+

Novel AP (%)

MS-COCO Benchmark

30-Shot Object Detection

nAP 12.1

nAP 24.5

nAP 21.2

nAP 21.9

nAP 12.4

Ours

nAP 14.0

TFA [ICML'20]

LVC [CVPR'22]

DCFS [NeurIPS'22]

Retentive R-CNN [CVPR'21]

SoftER Teacher (Ours)

Base AP Supervised Upperbound

Base AP Semi-Supervised Upperbound

This paper focuses on the intersection of SSOD and FSOD, which are essentially two sides of the 17 same coin in the context of label-efficient detection. On one side, SSOD investigates the detection 18 problem with a small fraction of images containing ground truth labels. On the other side, FSOD 19 addresses the objective of adapting a base detector to learn novel concepts from few instance-level 20 annotations. Existing approaches to FSOD assume abundant base classes to train the base detector. 21 However, such assumption is not ideal in practical scenarios where labels may be limited for both 22 base and novel classes, giving rise to the research question: can we do more on FSOD with the 23 available unlabeled data without additional hand-labeling? 24

25

26

27

28

29

30

31

We answer this question by introducing the unique task of semi-supervised few-shot detection , in which we explore the utility of unlabeled data for improving detection with label scarcity for both base and novel classes. Inspired by recent advances in SSOD and FSOD, our approach is two-fold: (1) we leverage unlabeled data to improve detection with a small fraction of labeled images; and (2) we generalize the resulting semi-supervised detector into a label-efficient few-shot detector by way of transfer learning. Our chief motivation is to not necessarily depend on an abundance of base classes for robust few-shot detection, which increases the versatility of our approach in realistic applications.

32

33

34

35

36

37

38

39

40

41

42

43

44

45

46

47

48

49

Moreover, our approach to semi-supervised FSOD adapts a base detector to learn novel concepts with reduced performance degradation to base classes , a desirable result missing in most prior approaches. Figure 1 illustrates that while recent work [11, 21, 36] achieve impressive detection on novel categories, they all ignore the importance of preserving base class accuracy . For generalized FSOD [10], the goal is to expand the learned vocabulary of the base detector with novel concepts. As such, base and novel class performances are equally important, since samples at test time may contain instances of both objects. Therefore, the more realistic evaluation metric for FSOD is not only novel AP, but the combined base and novel AP, for which our approach establishes a new state of the art.

We measure the utility of unlabeled data within our integrated semi-supervised few-shot framework, and discover an interesting empirical finding connecting the effectiveness of unlabeled data to semi-supervised FSOD by way of region proposals. Without bells and whistles, by simply adding unlabeled data to a supervised detector, we show a marked improvement on both base and novel class performances while also mitigating catastrophic base forgetting [31].

Summary of Main Contributions. First, we introduce SoftER Teacher, a simple yet effective and versatile detector, to combine the strengths of pseudo-labeling with representation learning on unlabeled images. SoftER Teacher enhances the quality of region proposals to substantially boost semi-supervised FSOD. Our empirical analysis on the relationship between unlabeled data and region proposals extends earlier results on proposal evaluation beyond supervised detection [17, 46].

Second, Figure 2 illustrates a potential relationship suggesting that a strong semi-supervised detector 50 is also a label-efficient few-shot detector, an interesting and non-trivial empirical observation linking 51 the two disparate domains. On the task of semi-supervised FSOD, our SoftER Teacher model exceeds 52 the novel class performance of a strong supervised detector [10] using only 10% of required base 53

labels, while exhibiting less than 9% in base forgetting. When trained on 100% of labeled base 54

few-shot performance using varying amounts of bounding box annotations. 56

classes with supplementary unlabeled data, SoftER Teacher sets a new standard on semi-supervised 55

Third, we establish the Label-Efficient Detection benchmark to quantify the utility of unlabeled data 57 for generalized semi-supervised FSOD. We hope that our benchmark serves as a strong baseline, and 58 a blueprint, to inspire future research toward this new problem setting in the community. 59

60

61

62

63

64

65

66

67

2 Related Work

Semi-Supervised Detection. Recent approaches to SSOD can be summarized into consistencybased and pseudo-labeling categories. The leading consistency method is Humble Teacher [44], which trains a pair of detectors on both labeled and unlabeled data in the student-teacher framework [16, 45]. Humble Teacher learns representations from unlabeled data by enforcing consistency on predicted soft labels from region proposals. Humble Teacher was inspired by CSD [18], which was the first approach to leverage consistency regularization [40] for SSOD, but utilizes strong data augmentation to deliver robust performance.

The state of the art on SSOD, however, belongs to a family of pseudo-labeling methods, which 68 trains a pair of detectors on pseudo labels along with (limited) human labels. One such method is 69 Soft Teacher [52] which vastly improves upon its previous counterparts STAC [42] and Unbiased 70 Teacher [29] by enabling end-to-end pseudo-labeling on unlabeled images. More recently, Consistent 71 Teacher [48] advances the performance envelope by reducing the inconsistency of pseudo targets. In 72 both consistency-based and pseudo-labeling methods, the teacher model is an exponential moving 73 average (EMA) of its student counterpart and is used to predict soft or pseudo labels on unlabeled data. 74 The main difference between the two is how the surrogate labels are used to generate unsupervised 75 targets to be jointly trained with the supervised objective. 76

77

We extend the strong performance of Soft Teacher by incorporating a new module for Entropy

78

79

80

81

82

83

84

85

86

87

88

89

90

91

92

93

94

95

96

97

98

99

100

101

102

103

104

105

106

Regression to learn additional representations from unlabeled images by way of region proposals.

SoftER Teacher

Our model, aptly named

, combines the attractive benefits of pseudo-labeling with supplementary proposal learning to establish a stronger baseline for SSOD.

Few-Shot Detection. Existing methods on FSOD can also be grouped into two categories: meta learning and transfer learning. Early work on meta learning introduced meta models to acquire class-level knowledge for adapting a base detector to novel concepts. Meta learners are jointly trained and fine-tuned with the base detectors to perform tasks like feature re-weighting, such as FSRW [19] and Meta R-CNN [53], or category-specific weight prediction (MetaDet) [49] using few exemplars of support images and ground truth bounding box annotations for the target objects.

Recent work on transfer learning for FSOD found that fine-tuning the last layer of the pre-trained base detector ( i.e. , the box classifier and regressor) on a balanced subset of base and novel classes, while freezing the rest of the detector, can significantly improve detection accuracy. The simple yet effective two-stage fine-tuning approach TFA [47] outperformed all prior meta learning methods on both base and novel detection metrics. Retentive R-CNN [10] extends TFA by introducing the BiasBalanced RPN and Re-Detector modules to achieve strong novel performance without sacrificing base accuracy, a desideratum of FSOD. Other transfer learning methods, such as FSCE [43], DeFRCN [36], FADI [3], and DCFS [11], address the shortcomings of the box classifier to boost FSOD. While these methods obtain impressive performance on novel categories, they suffer from considerable base class forgetting, making them sub-optimal in real-world applications requiring efficient and accurate detection on test samples containing instances of both classes.

We show that both base and novel class performances can be further improved when unlabeled images are incorporated into the two-stage fine-tuning procedure without catastrophic base forgetting, which lead to a new standard for semi-supervised FSOD.

Semi-Supervised Few-Shot Detection. There have been few attempts at leveraging unlabeled data to improve FSOD, but to our knowledge none directly addressed the task of optimizing for semi-supervised few-shot detection, in which setting both base and novel labels are simultaneously scarce. LVC [21] mines novel targets from the same base training dataset via pseudo-labeling to boost novel class detection, but comes at the cost of base performance. UniT [22] obtains impressive results on any-shot detection, but assumes abundant image-level labels for the base and novel targets. And

107

108

109

110

111

112

113

114

115

116

117

118

119

120

121

122

123

124

125

126

127

128

129

130

131

132

133

134

135

136

137

138

139

140

141

142

143

144

145

146

147

148

149

150

151

152

153

154

SSFOD [51] performs semi-supervised FSOD within an episodic meta training and N -way k -shot evaluation framework [20] while also requiring abundant base classes. Our approach is fundamentally different in that we do not strictly depend on abundant base labels, but make effective utilization of external unlabeled data for robust semi-supervised FSOD with reduced base degradation.

3 Approach

We propose to combine the available (limited) labeled examples with supplementary unlabeled images to boost semi-supervised FSOD. We begin with the fully supervised scenario in which we have access to a set of labeled image-target pairs ( x l , y l ) ∈ D sup . The supervised FSOD setting [10, 19, 47] assumes a base dataset C base ∈ D sup with abundant instance-level annotations and a novel dataset C novel ∈ D sup with only a few k ( e.g. , k ∈ { 1 , 5 , 10 } ) labeled instances, or 'shots', per category. The goal of FSOD is to expand the base detector by adapting it to learn new target concepts such that the resulting detector is optimized for accuracy on a test set comprising both classes in C base ∪ C novel .

To maintain parity with existing work, we adopt the simple yet effective two-stage transfer learning approach [10, 47] currently leading the FSOD literature, which comprises an initial stage of base class pre-training followed by a second stage of novel category fine-tuning. We consider the modern Faster R-CNN (FRCN) [37] system as our supervised base detector, which consists of a ResNet [15] backbone and a feature pyramid network (FPN) [27] neck for feature extraction, a region proposal network (RPN), an RoIAlign [14] operation for mapping proposals to region-of-interest (RoI) features, and a fully-connected head for RoI classification and regression. Let FRCN be represented by f θ , a stochastic function parametrized by a set of learnable weights θ . Formally, the base pre-training step is subjected to the standard supervised objective, over a mini-batch of labeled examples b l , given by:

Here, f θ ( x i ) denotes a forward pass on the i -th image to produce box classification and localization predictions from class-agnostic proposals, y i is the i -th ground truth annotation containing box labels and coordinates, and ( L roi cls , L roi reg ) are the cross-entropy and L 1 losses, respectively, for the RoI head. Henceforth for simplicity, we develop our approach only on the RoI head and omit the presentation on the classification and regression losses of the RPN, which remain constant without changes, to predict and localize the 'objectness' of region proposals.

3.1 What Makes for Effective FSOD?

We revisit this question from the perspective of maximizing representation learning while minimizing base forgetting. In two-stage detectors [14, 37], the quality of region proposals is a strong predictor of supervised detection performance [17, 46], since they focus the detector head on candidate RoIs. This is especially true for FSOD approaches based on transfer learning, in which the established procedure is to freeze the RPN during few-shot fine-tuning. Intuitively, if we can incorporate methods and/or data to boost representation learning by way of the RPN, then the detector should have a higher chance of discovering novel categories to improve few-shot performance.

We conduct extensive experiments on the COCO dataset to verify our intuition. We split the dataset into disjoint 60 base and 20 novel categories and pre-train three variants of the FRCN detector on the base classes: (i) FRCN-Base, (ii) FRCN-Base augmented with COCO unlabeled2017 images leveraging the Soft Teacher formulation, and (iii) FRCN-Full using both base and novel classes to represent the upper-bound performance. We also experiment with CRPN-Base, a method specially designed to improve proposal quality and detection performance using a two-stage Cascade RPN [46].

Figure 3a quantifies the 'class-agnosticism' of various RPNs, using the standard metric AR@300 proposals, for varying fractions of base labels. Surprisingly, unlabeled data has the remarkable ability to boost proposal recall on novel-only categories, even in the extremely low 1% label limit. Somewhat unsurprising is the ability of CRPN-Base to propose novel objects competitive with FRCN-Base + Unlabeled when more base labels are available. Consistent with previous findings [10, 21], Figures 3b and 3c show that the vanilla supervised FRCN-Base has a strong tendency to reject novel objects as background, due to the lack of annotations, resulting in the worst recall on novel classes.

As alluded in Section 1, the contribution of unlabeled data to FSOD goes beyond improving base 155 and novel detection performances; unlabeled data can also help mitigate catastrophic base forgetting. 156

157

158

159

160

161

162

163

164

165

166

167

168

169

170

171

172

173

174

175

176

177

178

179

180

181

182

183

184

185

186

187

188

189

190

1

5

10

Percentage of Base Labels

20

30

40

50

60

Novel AR@300 (%)

1

5

10

Percentage of Base Labels

20

30

40

50

60

Overall AR@300 (%)

FRCN-Full

FRCN-Base

+

Unlabeled

CRPN-Base

FRCN-Base

(a)

Impact of unlabeled data on proposal quality

(b)

RPN w/ 1% labels

(c)

RPN w/ 10% labels

We find analogous effectiveness of FRCN-Base + Unlabeled on the combined Overall AR@300 metric, for both base + novel objects, suggesting the RPN, when trained with unlabeled data, has the ability to retain base proposals and help combat base degradation during few-shot fine-tuning.

Discussion. This paper rethinks a different and more versatile way to improve the RPN for FSOD while avoiding catastrophic forgetting. The previous LVC approach proposed to unfreeze the RPN during fine-tuning to obtain large performance gains on novel categories, but comes at the cost of significant base degradation (up to 19%). Similarly, FSCE [43] proposed to unfreeze the RPN while also doubling the number of proposals to encourage novel foreground detection during fine-tuning. However, this method increases the detection overhead and remains unclear whether it helps mitigate base forgetting. We illustrate that simply adding unlabeled data to the base detector leads to a compelling boost in proposal quality, without the need for any ad hoc modifications to the RPN.

We attribute this unique benefit of our approach to the potential base-novel object interactions found in abundant images. When learning with unlabeled data, the base detector can obtain semantically similar cues of novel objects to inform the RPN on foreground detection. Sun et al. [43] showed that visually analogous objects have high cosine similarity scores ( e.g. , sim ( cow , horse ) = 0 . 39 ). With 1% of labels, these base-novel interactions are limited, resulting in a recall of 22 . 7% . Given a sizable unlabeled dataset, the base detector improves its representations to yield a gain of +12 . 3 points.

3.2 Semi-Supervised Base Pre-Training

Motivated by the promising utility of unlabeled data, we now relax the strict assumption on having abundant base classes for FSOD and introduce a new and more general setting of having a small fraction of base labels given abundant unlabeled images. We revisit the task of semi-supervised base pre-training by formulating an unsupervised loss computed on an unlabeled dataset D unsup to be jointly trained with the supervised loss on D sup . We consider the following canonical optimization objective widely adopted as part of the framework for semi-supervised learning [1, 23, 33]:

where λ > 0 is a hyper-parameter controlling the contribution of the unsupervised component. Next, we describe the unsupervised criterion on D unsup to make FRCN into a semi-supervised detector.

Soft Teacher. We adopt Soft Teacher [52] as the baseline SSOD formulation for its simplicity but strong performance. Soft Teacher trains FRCN in a student-teacher fashion on both labeled and unlabeled data. The student is trained on labeled examples in the standard supervised manner per Eq. (1). For unlabeled images, the teacher is treated as a fixed detector to generate thousands of box candidates, most of which are eliminated for redundancy with non-maximum suppression. Additionally, box candidates are thresholded for foreground objects and go through an iterative jittering-refinement procedure to reduce localization variance, resulting in a set of high-quality pseudo boxes to be jointly trained with ground truth annotations.

As is common practice [41, 45], the teacher's parameters ¯ θ are updated from the student's via 191 ¯ θ = EMA ( θ ) at each training step. Integral to the success of Soft Teacher is a student-teacher data 192 augmentation strategy inspired by STAC [42]. The student trains on unlabeled images subjected to 193 complex random perturbations, akin to RandAugment [6], including affine transforms. Separately, 194

195

196

197

198

199

200

201

202

203

204

205

206

207

208

209

210

211

212

213

214

Student Proposals

Teacher Proposals

Student Proposals

Teacher Proposals

M

M

口口

八

the teacher receives weakly augmented images with simple random resizing and flipping. This multi-stream augmentation design allows the teacher to generate reliable unsupervised targets on easy images to guide the student's learning on difficult images for better generalization.

At the time of box classification and regression in the RoI head, we have a set of unlabeled images along with teacher-generated pseudo labels ( x u , ˆ y u ) ∈ D unsup . The unsupervised loss for Soft Teacher on a mini-batch of unlabeled images b u is defined as:

SoftER Teacher. The design of Soft Teacher employs class confidence thresholding and box jittering to select high-quality pseudo-label candidates for unsupervised classification and regression. However, it uses an aggressive threshold of 0.9, resulting in a trade-off between low recall and high precision at 33% and 89%, respectively [52]. We observe that low recall can result in poor detection performance on small and ambiguous objects [26], especially in low-label regimes where the teacher has insufficient confidence about its predicted pseudo labels. We aim to extend Soft Teacher and improve its detection recall by learning additional representations from abundant region proposals.

Given a set of proposals p generated by the student's RPN on a batch of unlabeled images, we apply the student-teacher data augmentation pipeline described above to obtain ( p s , p t ) , denoting transformed student and teacher proposals, which are related to each other by a transformation matrix M . We then forward pass Faster R-CNN twice, as the student f θ and teacher f ¯ θ , to obtain two sets of RoI outputs for predicted box classification logits ( z s , z t ) and localization coordinates ( r s , r t ) . Let g c be the softmax function over the channel dimension c . We define an auxiliary unsupervised criterion for proposal box similarity based on a cross-entropy measure H ( z s , z t ) :

̸

Here, g c outputs a distribution over C classes and w i is the Boolean weight for the predicted positive 215 (foreground) class: w i = 1 if argmax ( z it ) = background , else w i = 0 . 216

Similarly for proposal box regression, we constrain the predicted box coordinates ( r s , r t ) to be 217 close. Since there are complex geometric distortions between the two, we first map teacher proposal 218 coordinates r t to the student space using the transformation M . Then, we align the proposal boxes 219 via the intersection-over-union (IoU) criterion: 220

where we treat the IoU metric as a loss [38] to quantify the discrepancy between student and teacher 221 proposal coordinates. Note that both the cross-entropy and IoU losses, Eqs. (4) and (5), are computed 222 only on predicted foreground classes. 223

Recall that we have two different transformation pipelines operating on each proposal, so we have 224 two augmented views of each proposal. Figure 4 illustrates that by enforcing these randomly 225

226

227

228

229

230

231

232

233

234

235

236

237

238

239

240

241

242

243

244

245

246

247

248

249

250

251

252

253

254

255

256

257

258

259

260

261

262

263

264

265

266

267

268

269

270

271

272

273

274

augmented views, and their box coordinates, to be similar , we enable the student to tap into abundant region proposals to learn diverse feature representations across a spectrum of scale, color, and geometric perturbations. Our formulation draws inspiration from recent research on self-supervised representation learning with multi-augmented views [5, 13]. Note the cross-entropy similarity between the student and teacher predictions, Eq. (4), can be interpreted as a form of entropy regularization [12], which has been proven to work well in various semi-supervised classification scenarios [33, 34]. The overall optimization objective at the RoI head for our SoftER Teacher model is computed as:

where we set α = | b u | | b l | following Soft Teacher and find β = 2 α works well across all experiments.

Discussion. Our approach to proposal learning is unique and different from Humble Teacher [44] in two major ways: (1) we use diverse data transformations, including geometric distortions, to enforce proposal similarity, whereas Humble Teacher considered only simple random flipping and color transforms; and (2) we adopt the IoU metric for box localization consistency over L 2 distance, which has been shown to produce inferior box regression [38]. Further, Humble Teacher acknowledged that matching student-teacher proposals under complex affine transforms is not a trivial task and 'could lead to undesirably complicated details.' We solve this complicated task by tracking the affine matrix M in our entropy regression module, to address a key weakness of Soft Teacher by boosting object recall, thereby enabling SoftER Teacher to demonstrate superior learning with unlabeled data over Humble Teacher and Soft Teacher. Please refer to Appendix B.3 for detailed comparative results.

3.3 Semi-Supervised Few-Shot Fine-Tuning

We propose a simple two-step approach to harness unlabeled data for semi-supervised few-shot fine-tuning. First, we initialize the few-shot detector, f ′ ¯ θ ← f ¯ θ , with parameters copied from the base teacher detector pre-trained with unlabeled data per Eq. (6). And second, we further train the RoI head of f ′ ¯ θ on novel classes using the available few-shot and unlabeled examples while freezing the base backbone, FPN, and RPN components. Then, we fine-tune the few-shot detector on a balanced training set of k shots per class containing both base and novel instances. We only update the RoI box classifier while freezing all other components, including the box regressor, since it is the main source of error [11, 43]. To our knowledge, we are the first to incorporate external unlabeled data with few-shot fine-tuning , which provides a compelling boost to novel performance while enjoying substantial gains in base detection without catastrophic forgetting. We present detailed ablation studies in Appendix A to validate our approach and design choices.

4 Experiments

Datasets. Consistent with the current literature on FSOD, we evaluate our approach on the challenging PASCAL VOC [9] and MS-COCO 2017 [28] detection benchmarks. For VOC, we use the combined VOC07 + 12 trainval splits as the labeled training set and evaluate performance on the VOC2007 test set. For COCO, we utilize the train2017 split as labeled data and test on val2017 . We also leverage COCO-20 , the subset of COCO data having the same 20 class names as VOC, and COCO unlabeled2017 as the sources of supplementary unlabeled data.

Performance Metrics. Following established evaluation protocol, we assess detection performance using AP 50 for the average precision at overlap threshold 0.5 and AP 50:95 for the mean average precision computed over a range of 10 overlap thresholds between 0.5 and 0.95. We also report average recall AR to complement AP for assessing object coverage, which has previously been used to evaluate detection performance of small objects [24, 26].

Implementation Details. For most experiments, we adopt the ResNet-101 backbone pre-trained on ImageNet 1K [8] for a direct comparison with existing work. For some experiments, we also employ ResNet-50 to demonstrate parameter-efficient learning with SoftER Teacher. We implement our models in MMDetection [4] and PyTorch [35]. Complete details are given in Appendix C.

4.1 SoftER Teacher is a Parameter- and Label-Efficient Few-Shot Detector

We conduct our few-shot experiments on the same VOC and COCO samples provided by the TFA benchmark [47]. The VOC dataset is randomly partitioned into 15 base and 5 novel classes, in which

275

276

277

278

279

280

281

282

283

284

285

286

287

288

289

290

291

292

293

294

295

296

297

298

299

300

301

302

303

304

305

306

307

there are k ∈ { 1 , 5 , 10 } labeled boxes per category sampled from the joint VOC07 + 12 trainval splits. And the COCO dataset is divided into 60 base and 20 novel classes having the same VOC category names with k ∈ { 5 , 10 , 30 } shots. We leverage COCO-20 and unlabeled2017 as external unlabeled data to augment base pre-training and novel fine-tuning on VOC and COCO, respectively.

Table 1 compares the effectiveness of SoftER Teacher against transfer-learning methods representing the state of the art on COCO, for the evaluation of both base and novel performances. We report the ideal supervised base AP of 39 . 3 [10, 47] along with our substantially improved semi-supervised base AP of 44 . 4 to measure the extent of base forgetting. Recall that the more realistic evaluation metric for generalized FSOD is not only novel AP, but the combination of base and novel AP. We summarize the following key takeaways: (a) SoftER Teacher with ResNet-101 trained with supplementary unlabeled data is the best model on the combined Overall AP metric for 80 classes, leading the next best Retentive R-CNN by up to +2 . 0 AP; (b) SoftER Teacher with ResNet-50 is on par with Retentive R-CNN while being more parameter-efficient; and (c) SoftER Teacher achieves the state of the art while being more efficient with respect to parameters and labels .

We notice a recurring theme in the FSOD literature that seems to favor novel class performance while ignoring base accuracy, even though the detection of both classes is equally important at test time. Table 2 shows comparative results on VOC, for which there are few existing work evaluating on both base and novel AP. We report the ideal supervisd base AP of 80 . 8 [10, 47] along with our vastly expanded base AP of 85 . 9 using unlabeled data. SoftER Teacher with ResNet-50 incurs negligible base forgetting of less than 1.6% while exceeding MPSR, TFA, and Retentive R-CNN with ResNet-101 by a notable margin on most metrics. Although MPSR achieves impressive one-shot performance on novel categories, it suffers catastrophic base forgetting by as much as 24%. Retentive R-CNN does not exhibit base class degradation, but generally falls behind on novel class performance. Tables 1 and 2 corroborate our observation on the trade-off between novel performance and base forgetting, for which our approach aims to simultaneously optimize. Due to page limit, we refer to Appendix A for detailed ablation studies analyzing the design and benefits of SoftER Teacher.

4.2 How Does Proposal Quality Impact Semi-Supervised Few-Shot Detection?

In this section, we continue our discussion from Sec. 3.1 by analyzing semi-supervised FSOD as a function of proposal quality in Figure 5. We measure proposal quality using the metric AR@ p [46], for p ∈ { 100 , 300 , 1000 } proposals, averaged over thresholds between 0.5 and 0.95. We arrive at the following conclusions: (a) SoftER Teacher produces better proposals than the comparisons across varying fractions of labels; and (b-d) proposal quality is strongly correlated with semi-supervised FSOD, an insightful empirical finding that extends existing results beyond supervised detection [17].

308

309

310

311

312

313

314

315

316

317

318

319

320

321

322

323

324

325

326

327

328

329

1

5

10

Percentage of Base Labels

16

20

24

28

32

AR@100 (%)

(a)

16

18

20

22

24

26

28

30

Proposal Quality AR@100 (%)

2

4

6

8

10

12

Novel AP

50:95

(%)

(b) 30-Shot@1-Percent

Correlation

r

=0.986

22

24

26

28

30

32

Proposal Quality AR@100 (%)

4

6

8

10

12

14

Novel AP

50:95

(%)

(c) 30-Shot@5-Percent

Correlation

r

=0.987

24

26

28

30

32

Proposal Quality AR@100 (%)

6

8

10

12

14

16

Novel AP

50:95

(%)

(d) 30-Shot@10-Percent

Correlation

r

=0.977

SoftER Teacher w/ ResNet-101 (Ours)

Although the strong Soft Teacher baseline is effective at harnessing unlabeled data for semi-supervised FSOD, our approach demonstrates superior learning by addressing a key shortcoming of Soft Teacher . SoftER Teacher boosts object recall via our proposed Entropy Regression module, improving on Soft Teacher by +1 . 2 base AR@100, which yields a gain of +1 . 5 novel AP for the 30-Shot@10-Percent setting. These results further strengthen our empirical observation that a stronger semi-supervised detector leads to a more label-efficient few-shot detector. Future work would explore if our finding can be extended to a more general case with other SSOD formulations including one-stage detectors.

4.3 A New Benchmark for Generalized Semi-Supervised Few-Shot Detection

We present our Label-Efficient Detection benchmark on COCO val2017 in Table 3, evaluated using the supervised Faster R-CNN baseline along with the semi-supervised Soft Teacher and SoftER Teacher models. Our protocol for semi-supervised FSOD is as follows. In the first stage, we pre-train the base detector on the disjoint 60 base categories using { 1 , 5 , 10 } percent of labels per Eq. (6). In the second stage, we transfer the parameters of the base teacher detector to the few-shot detector and fine-tune its RoI box classifier, keeping other components frozen, on a balanced training set of k ∈ { 5 , 10 , 30 } shots per class containing both base and novel examples. In both stages, we supplement base pre-training and novel fine-tuning with images from COCO unlabeled2017 .

We report AP 50:95 performances on both base and novel classes along with the aggregated overall metric. We also report the ideal AP 50:95 and AR 50:95 metrics obtained from the first stage of base pre-training to measure the potential for base forgetting during the few-shot fine-tuning step. We encourage future work to follow suit as we emphasize the importance of optimizing for accuracy on both base and novel classes, a desideratum of generalized few-shot object detection.

5 Conclusion

This paper presented the Label-Efficient Detection framework to quantify the utility of unlabeled data 330 for generalized semi-supervised FSOD. Central to the framework is our SoftER Teacher, a robust 331 detector combining the strengths of pseudo-labeling with representation learning on unlabeled images. 332 We demonstrated two main areas of impact: (1) SoftER Teacher achieves superior learning with 333 unlabeled data to boost semi-supervised FSOD without relying on an abundance of labels; and (2) 334 our framework sheds empirical insight into a potential relationship that a stronger semi-supervised 335 detector leads to a more effective few-shot detector, the basis of which could inspire future research. 336

337

338

339

References

Philip Bachman, Ouais Alsharif, and Doina Precup. Learning with Pseudo-Ensembles. In NeurIPS , 2014. 5

Amir Bar, Xin Wang, Vadim Kantorov, Colorado J. Reed, Roei Herzig, Gal Chechik, Anna Rohrbach, 340 Trevor Darrell, and Amir Globerson. DETReg: Unsupervised Pretraining With Region Priors for Object 341 Detection. In CVPR , 2022. 1 342

Yuhang Cao, Jiaqi Wang, Ying Jin, Tong Wu, Kai Chen, Ziwei Liu, and Dahua Lin. Few-Shot Object 343 Detection via Association and Discrimination. In NeurIPS , 2021. 3 344

Kai Chen, Jiaqi Wang, Jiangmiao Pang, Yuhang Cao, Yu Xiong, Xiaoxiao Li, et al. MMDetection: Open 345 MMLab Detection Toolbox and Benchmark. https://arxiv.org/abs/1906.07155, 2019. 7 346

Xinlei Chen and Kaiming He. Exploring Simple Siamese Representation Learning. In CVPR , 2021. 7 347

Ekin D. Cubuk, Barret Zoph, Jonathon Shlens, and Quoc V. Le. RandAugment: Practical Automated Data 348 Augmentation with a Reduced Search Space. In NeurIPS , 2020. 5 349

Zhigang Dai, Bolun Cai, Yugeng Lin, and Junying Chen. UPDETR: Unsupervised Pre-Training for Object 350 Detection with Transformers. In CVPR , 2021. 1 351

Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. ImageNet: A Large-Scale 352 Hierarchical Image Database. In CVPR , pages 248-255, 2009. 7 353

Mark Everingham, Luc Van Gool, Christopher K.I. Williams, John Winn, and Andrew Zisserman. The 354 PASCAL Visual Object Classes (VOC) Challenge. IJCV , 88(2):303-338, 2010. 7 355

Zhibo Fan, Yuchen Ma, Zeming Li, and Jian Sun. Generalized Few-Shot Object Detection without 356 Forgetting. In CVPR , 2021. 1, 2, 3, 4, 8 357

Bin-Bin Gao, Xiaochen Chen, Zhongyi Huang, Congchong Nie, Jun Liu, Jinxiang Lai, Guannan Jiang, Xi 358 Wang, and Chengjie Wang. Decoupling Classifier for Boosting Few-Shot Object Detection and Instance 359 Segmentation. In NeurIPS , 2022. 1, 2, 3, 7, 8 360

Yves Grandvalet and Yoshua Bengio. Semi-Supervised Learning by Entropy Minimization. In NeurIPS , 361 2004. 7 362

Jean-Bastien Grill, Florian Strub, Florent Altché, Corentin Tallec, Pierre H. Richemond, Elena Buchatskaya, 363 Carl Doersch, Bernardo Avila Pires, et al. Bootstrap Your Own Latent: A New Approach to Self-Supervised 364 Learning. In NeurIPS , 2020. 7 365

Kaiming He, Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask R-CNN. In ICCV , 2017. 4 366

Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep Residual Learning for Image Recognition. 367 In CVPR , 2016. 4 368

Geoffrey Hinton, Oriol Vinyals, and Jeffrey Dean. Distilling the Knowledge in a Neural Network. In 369 NeurIPS Deep Learning and Representation Learning Workshop , 2015. 3 370

Jan Hosang, Rodrigo Benenson, Piotr Dollár, and Bernt Schiele. What Makes for Effective Detection 371 Proposals? IEEE TPAMI , 38(4):814-830, 2016. 2, 4, 8 372

Jisoo Jeong, Seungeui Lee, Jeesoo Kim, and Nojun Kwak. Consistency-Based Semi-Supervised Learning 373 for Object Detection. In NeurIPS , 2019. 3 374

Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-Shot Object 375 Detection via Feature Reweighting. In ICCV , 2019. 3, 4 376

Leonid Karlinsky, Joseph Shtok, Sivan Harary, Eli Schwartz, Amit Aides, Rogerio Feris, Raja Giryes, 377 and Alex M. Bronstein. RepMet: Representative-Based Metric Learning for Classification and Few-Shot 378 Object Detection. In CVPR , 2019. 4 379

Prannay Kaul, Weidi Xie, and Andrew Zisserman. Label, Verify, Correct: A Simple Few Shot Object 380 Detection Method. In CVPR , 2022. 1, 2, 3, 4, 8 381

Siddhesh Khandelwal, Raghav Goyal, and Leonid Sigal. UniT: Unified Knowledge Transfer for Any-shot 382 Object Detection and Segmentation. In CVPR , 2021. 3 383

Samuli Laine and Timo Aila. Temporal Ensembling for Semi-Supervised Learning. In ICLR , 2017. 5 384

Jianan Li, Xiaodan Liang, Yunchao Wei, Tingfa Xu, Jiashi Feng, and Shuicheng Yan. Perceptual Generative 385 Adversarial Networks for Small Object Detection. In CVPR , 2017. 7 386

Yangguang Li, Feng Liang, Lichen Zhao, Yufeng Cui, Wanli Ouyang, Jing Shao, Fengwei Yu, and Junjie 387 Yan. Supervision Exists Everywhere: A Data Efficient Contrastive Language-Image Pre-Training Paradigm. 388 In ICLR , 2022. 1 389

Zeming Li, Chao Peng, Gang Yu, Xiangyu Zhang, Yangdong Deng, and Jian Sun. DetNet: A Backbone 390 Network for Object Detection. In ECCV , 2018. 6, 7 391

Tsung-Yi Lin, Piotr Dollár, Ross Girshick, Kaiming He, Bharath Hariharan, and Serge Belongie. Feature 392 Pyramid Networks for Object Detection. In CVPR , 2017. 4 393

Tsung-Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, 394 and C. Lawrence Zitnick. Microsoft COCO: Common Objects in Context. In ECCV , 2014. 1, 7 395

Yen-Cheng Liu, Chih-Yao Ma, Zijian He, Chia-Wen Kuo, Kan Chen, Peizhao Zhang, Bichen Wu, Zsolt 396 Kira, and Peter Vajda. Unbiased Teacher for Semi-Supervised Object Detection. In ICLR , 2021. 3 397

Yen-Cheng Liu, Chih-Yao Ma, and Zsolt Kira. Unbiased Teacher v2: Semi-Supervised Object Detection 398 for Anchor-Free and Anchor-Based Detectors. In CVPR , 2022. 1 399

David Lopez-Paz and Marc'Aurelio Ranzato. Gradient Episodic Memory for Continual Learning. In 400 NeurIPS , 2017. 2 401

Matthias Minderer, Alexey Gritsenko, Austin Stone, Maxim Neumann, Dirk Weissenborn, Alexey Dosovit402 skiy, Aravindh Mahendran, Anurag Arnab, et al. Simple Open-Vocabulary Object Detection with Vision 403 Transformers. In ECCV , 2022. 1 404

405

406

407

Takeru Miyato, Shin-ichi Maeda, Masanori Koyama, and Shin Ishii. Virtual Adversarial Training: A Regularization Method for Supervised and Semi-Supervised Learning. IEEE TPAMI , 41:1979-1993, 2017. 5, 7

Avital Oliver, Augustus Odena, Colin Raffel, Ekin D. Cubuk, and Ian J. Goodfellow. Realistic Evaluation 408 of Deep Semi-Supervised Learning Algorithms. In NeurIPS , 2018. 7 409

410

411

412

Adam Paszke, Sam Gross, Francisco Massa, Adam Lerer, James Bradbury, et al. PyTorch: An Imperative Style, High-Performance Deep Learning Library. In NeurIPS , pages 8024-8035. Curran Associates, Inc., 2019. 7

Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang. DeFRCN: Decoupled Faster 413 R-CNN for Few-Shot Object Detection. In ICCV , 2021. 2, 3, 8 414

Shaoqing Ren, Kaiming He, Ross Girshick, and Jian Sun. Faster R-CNN: Towards Real-Time Object 415 Detection with Region Proposal Networks. In NeurIPS , 2015. 4 416

Hamid Rezatofighi, Nathan Tsoi, JunYoung Gwak, Amir Sadeghian, Ian Reid, and Silvio Savarese. 417 Generalized Intersection over Union: A Metric and A Loss for Bounding Box Regression. In CVPR , 2019. 418 6, 7 419

Byungseok Roh, Wuhyun Shin, Ildoo Kim, and Sungwoong Kim. Spatially Consistent Representation 420 Learning. In CVPR , 2021. 1 421

Mehdi Sajjadi, Mehran Javanmardi, and Tolga Tasdizen. Regularization with Stochastic Perturbations for 422 Deep Semi-Supervised Learning. In NeurIPS , 2016. 3 423

Kihyuk Sohn, David Berthelot, Chun-Liang Li, Zizhao Zhang, Nicholas Carlini, Ekin D Cubuk, Alex 424 Kurakin, Han Zhang, and Colin Raffel. Fixmatch: Simplifying Semi-Supervised Learning with Consistency 425 and Confidence. In NeurIPS , 2020. 5 426

Kihyuk Sohn, Zizhao Zhang, Chun-Liang Li, Han Zhang, Chen-Yu Lee, and Tomas Pfister. A Simple 427 Semi-Supervised Learning Framework for Object Detection. https://arxiv.org/abs/2005.04757, 2020. 3, 5 428

Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. FSCE: Few-Shot Object Detection via 429 Contrastive Proposal Encoding. In CVPR , 2021. 3, 5, 7 430

Yihe Tang, Weifeng Chen, Yijun Luo, and Yuting Zhang. Humble Teachers Teach Better Students for 431 Semi-Supervised Object Detection. In CVPR , 2021. 3, 7 432

Antti Tarvainen and Harri Valpola. Mean Teachers are Better Role Models: Weight-Averaged Consistency 433 Targets Improve Semi-Supervised Deep Learning Results. In NeurIPS , 2017. 3, 5 434

Thang Vu, Hyunjun Jang, Trung X. Pham, and Chang D. Yoo. Cascade RPN: Delving into High-Quality 435 Region Proposal Network with Adaptive Convolution. In NeurIPS , 2019. 2, 4, 8 436

Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E. Gonzalez, and Fisher Yu. Frustratingly Simple 437 Few-Shot Object Detection. In ICML , 2020. 1, 3, 4, 7, 8 438

Xinjiang Wang, Xingyi Yang, Shilong Zhang, Yijiang Li, Litong Feng, Shijie Fang, Chengqi Lyu, Kai Chen, 439 and Wayne Zhang. Consistent-Teacher: Towards Reducing Inconsistent Pseudo-Targets in Semi-Supervised 440 Object Detection. In CVPR , 2023. 1, 3 441

Yu-Xiong Wang, Deva Ramanan, and Martial Hebert. Meta-Learning to Detect Rare Objects. In ICCV , 442 2019. 3 443

Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-Scale Positive Sample Refinement for 444 Few-Shot Object Detection. In ECCV , 2020. 8 445

Wuti Xiong, Yawen Cui, and Li Liu. Semi-Supervised Few-Shot Object Detection with a Teacher-Student 446 Network. In BMVC , 2021. 4 447

Mengde Xu, Zheng Zhang, Han Hu, Jianfeng Wang, Lijuan Wang, Fangyun Wei, Xiang Bai, and Zicheng 448 Liu. End-to-End Semi-Supervised Object Detection with Soft Teacher. In ICCV , 2021. 1, 3, 5, 6 449

Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta R-CNN : 450 Towards General Solver for Instance-Level Few-Shot Learning. In ICCV , 2019. 3 451

Peer reviews:
{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper studies a new task named semi-supervised few-shot object detection, where both of base and novel classes are supposed to be scarce.\nThe author first finds the vanilla supervised FRCN trained on base classes has a low recall on novel classes, and trained with extra unlabeled novel data can effectively improve the novel recall.\nThen the author follows the SSOD framework Soft Teacher to do semi-supervised base training, where only partial base data is available. However, the original Soft Teacher has a low recall of small and ambiguous objects. Thus the author proposes a new proposal learning method to improve it.\nFinally, the pre-trained model is then semi-supervised fine-tuned on a balance set comprised of both base and novel samples.\nExperiments show Softer Teacher achieves a good GFSOD performance.\n\nSTRENGTHS:\n* The idea is straightforward and has good soundness.\n* The method has promising results in the GFSOD setting.\n\nWEAKNESSES:\n1. The proposed task of semi-supervised few-shot objects is similar to semi-supervised object detection. Since both base and novel classes are scarce, what's the meaning of splitting classes into base and novel? I can't see any practical significance in this task.\n\n2. From my point of view, the setting proposed in the paper is closer to a semi-supervised than a few-shot object detection problem. Particularly, one of the key properties of few-shot learning is that the model does not know the novel classes in training, so it can adapt quickly to new classes with only a few examples in testing either using meta-learning (e.g., meta-RCNN) or small-#step finetuning (e.g., TFA). \nAnother common sense in FSOD is that the novel classes are authentically rare, and we cannot find more images about that class, regardless of label or unlabeled.\nTherefore, the proposed approach works best for semi-supervised object detection rather than few-shot object detection, and it is not fair to compare it with FSOD works.\n\n3. The proposed method SoftER Teacher is an incremental improvement based on existing work SSOD Soft Teacher. The only improvement seems to be that the author appends a new loss to constrain outputs from the teacher and the student should be close to corresponding proposals, but the method is more likely to be only related to semi-supervised learning, it seems nothing about few-shot learning.\n\n4. Section 3.1 is named \"What makes for Effective FSOD\", but 3.1 studies unlabeled data can improve the novel recall of the FSOD model. The title is not very accurate.\n\n5. In line 252, the author argues, \"we are the first to incorporate external unlabeled data with few-shot fine-tuning\", I don't think it is a contribution or anything good.\n\n6. There is no component analysis or ablation experiments to demonstrate the effectiveness of the proposed method. For example, the performance comparison of w/wo  the proposal learning loss.\n\n7. The performance improvement is minor in the FSOD setting (not GFSOD). The novel performance is actually bad. The superior GFSOD performance may attribute to the strong baseline Softer Teacher on base performance.\n\n8. The authors do not show any numbers related to the training resources (memory and time).\n\nQUESTIONS:\n1. What's the difference between the proposed semi-supervised few-shot objection and semi-supervised object detection? What's the practical meaning or practical application scenario that semi-supervised few-shot objection covers but semi-supervised object detection does not?\n\n2. Besides the proposal learning loss, any more improvements upon the baseline framework Soft Teacher? Where is the ablation that demonstrates its effectiveness of it?\n\n3. SoftER teacher adopts FRCN for both teacher and student, how about the performance when replacing the FRCN with other FSOD methods like DeFRCN?\n\nLIMITATIONS:\nN/A", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-06-30"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis paper focuses on Semi-Supervised Few-Shot Object Detection, where both base classes and novel classes have few labeled training set, along with abundant unlabeled data. For model architecture, the softer teacher is proposed to train with unlabeled data and a teacher-student framework. Experiments demonstrate strong performance using only 10% base labels.\n\nSTRENGTHS:\nThe idea of introducing unlabeled data for few-shot object detection is interesting and has great value for real-world application. The idea of  reducing the number of labeled data for base classes in few-shot object detection is also interesting.\n\nWEAKNESSES:\nMy major concerns are as below:\n\n1. What is the sources of unlabeled images? Do the unlabeled images have both base-class and novel-class instances? In this way, it is no surprise that adding abundant images could improve the proposal recall and detection results of few-shot novel classes.\n\n2. The training framework in figure 2 and Section 3.3 are not exactly the same. In Figure 2, the unlabeled images are used for both base-class pre-training and few-shot fine-tuning. But in Section 3.3, it seems that the second stage of few-shot fine-tuning do not use additional unlabeled image. Clarifications are need. If the second stage also use unlabeled images, can we merge the second stage into the first stage because both base classes and novel classes are few-shot? We do not need to have two stages for training in that case, and the problem becomes semi-supervised object detection and each class has very few labeled images. Thus, what is the difference between traditional semi-supervised object detection?\n\n3. This work has improved overall performance of base and novel class. Although the performance of novel class improves compared some baseline model (e.g., Faster RCNN), but has far worse performance compared to the SOTA [1,2]. Does this mean that the additional unlabeled images only work well with base classes, but not for novel class? Using unlabeled image is perhaps a right way for semi-supervised object detection. But does using unlabeled images is the right way for few-shot object detection?\n\n[1] Qiao, Limeng, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang. \"Defrcn: Decoupled faster r-cnn for few-shot object detection.\" In Proceedings of the IEEE/CVF International Conference on Computer Vision, pp. 8681-8690. 2021.\n\n[2] Kaul, Prannay, Weidi Xie, and Andrew Zisserman. \"Label, verify, correct: A simple few shot object detection method.\" In Proceedings of the IEEE/CVF conference on computer vision and pattern recognition, pp. 14237-14247. 2022.\n\n4. The Table 1 and Table 2 are not complete. Table 1 lacks the comparison of Faster R-CNN (Our Impl.) and Soft Teacher (Our Impl.) as in Table 2. Table 2 lacks the comparison of the latest method for few-shot object detection (e.g. LVC, DeFRCN) as in Table 1.\n\n5. The Figure 3 (b) and (c) are very confusing. I can only find one red box. Does it mean that vanilla FRCN-base only have one proposal? This is weird.\n\n6. What is the difference between soft teacher and softer teacher? In L201-L207, the author mentions that soft teacher has an aggressive threshold of 0.9 which is not good. How did the authors address this problem? I do not find the answer in the main text. L208-L225 seems to be a simple extension of soft teacher without big changes.\n\n\n\n\n==========================================================================================\n\nAfter reading author's rebuttal and other reviews, some of the concerns about technical details are clear. But the major concern about the significance of doing few-shot base/novel partition is still there. I would suggest the authors make the problem setting simpler to get broader impact.\n\nQUESTIONS:\nPlease see the weaknesses above\n\nLIMITATIONS:\nPlease see the weaknesses above", "TITLE": "Official Review", "RECOMMENDATION": 4, "REVIEWER_CONFIDENCE": 5, "SOUNDNESS_CORRECTNESS": 2, "DATE": "2023-07-09"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThis article has done a meaningful work, which is a object detection method that combines few-shot with semi-supervised learning. The author introduces a SoftER Teacher for semi-supervised object detection in few-shot scenarios. SoftER Teacher enhances the quality of region proposals to substantially boost semi-supervised FSOD. Compared with LVC, DeFRCN and other methods, the performance has been improved.\n\nSTRENGTHS:\n- This task is very meaningful. As far as I know, the traditional few-shot object detection task is difficult to be directly applied to the industry, and the method combined with semi-supervised target detection will be a good solution. (Although the method in this paper is not the first to consider the combination of few-shot and semi-supervised.)\n- It is good to see that the author provides the source code in the supplementary material, which provides a guarantee for the reproducibility of this article.\n- The authors present rich experimental results in the article and supplementary material.\n\nWEAKNESSES:\n- To my acknowledgment, the current mainstream FSOD method is verified on MS COCO 2014, not MS COCO 2017. \"Consistent with the current literature on FSOD\" may be ambiguous.\n- In Table 1, since the author did not report the results of LVC in 5-Shot, the experimental results of LVC are from the original article? But considering that our method has unlabeled data for additional testing, it seems unfair to compare the results with the LVC experimental setting.\n- In addition to the comparison with the FSOD method, it would be better to add some comparisons with the SSOD method (under the Few-shot setting).\n- In Table 1, I found that the results of the novel class seem to be relatively weak, what is the reason? Because the method in this paper utilizes additional data.\n\nQUESTIONS:\nMy main concern is the experimental comparison, such as whether the comparison method has changed the experimental setting. Another concern is the performance of the novel class. If the author can convincingly solve my problem, I would like to improve my score.\n\n\nI also suggest that the author can also refer to a method on arxiv [1], which seems to have the same task as this article (Of course, this article does not need to compare with its method at this stage.). \n\n[1] Cao, Yuhang, et al. \"MINI: mining implicit novel instances for few-shot object detection.\" *arXiv preprint arXiv:2205.03381* (2022).\n\nLIMITATIONS:\nThe authors describe the limitations of this paper in the supplementary material.", "TITLE": "Official Review", "RECOMMENDATION": 5, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 3, "DATE": "2023-07-10"}

{"IS_META_REVIEW": false, "comments": "SUMMARY:\nThe paper approaches the task of Few-shot Object Detection (FSOD) from a semi-supervised perspective, where in addition to base classes data it uses unlabeled data during the base-pretraining phase, and then fine-tunes on the combination of base and available novel data using the best design choice of freezing appropriate layers (backbone, FPN and RPN) following the past works. The benefit of the approach comes with a higher bar on fully-supervised base class performance, which is attributed to training on additional unlabeled data. This higher performance then translates to a better overall (base + novel) classes performance in the fine-tuning phase, and establishes the effectiveness of semi-supervised learning on the FSOD task. The authors propose a SoftER Teacher approach which, in addition to the Soft Teacher loss, adds a consistency loss between teacher and student using at proposal-level. The authors show that this leads to better proposals (using recall) in low-data regimes.\n\nSTRENGTHS:\n- The paper is well-written with extensive experiments and ablations, and the semi-supervised exposition in the FSOD setup is much appreciated with a potential for realistic low-shot setups\n- The paper is well-positioned with respect to prior related works\n\nWEAKNESSES:\n- Table 2 on VOC07 and lower performance of Novel classes in low-shot setup\n    - Retentive R-CNN compared to SoftER exhibits a trend that its 1-shot performance is much higher that the proposed SoftER, and this trend gets reversed with more shots (such as 10-shot)\n    - The authors acknowledges the phenomenon in lines 295-296, and in line 297 mentions that Retentive R-CNN “generally falls behind on novel class performance”. However, this isn’t true in low-data setup (1-shot).\n    - This trend, however, doesn’t appear in the COCO dataset where novel class performance in 1-shot case is low for both Retentive R-CNN and the proposed SoftER approach\n    - My question is:\n        - Do the authors have any intuition for this behaviour?\n        - The authors leverage COCO-20 and COCO unlabeled2017 as he unlabeled data source for VOC (Table 2) and COCO (Table 1) experiments. Do the authors think that the domain mismatch between VOC and COCO is reflected in low-shot novel class performance in the case of VOC (Table 2)?\n        - In general, the proposed approach does better with relatively higher-shot regimes, which also appears in the claims made in the paper - such as Fig 1 (30-shot). But as a reader, there seems to be little explanation about low-shot regimes, which runs counter-intuitive since the approach uses additional unlabeled data compared to prior approaches, and is expected to perform well especially in low-data regimes\n\n- Presence of novel classes in unlabeled data\n    - Line 277-278 mentions the use of COCO-20 for VOC and COCO unlabeled2017 as the source of unlabeled data for VOC and COCO respectively\n    - This makes the assumption that novel classes in the plots of VOC and COCO experiments are necessarily present in the unlabeled set\n    - In general scenarios, such assumption may not hold true. Do the authors have some intuition of how the proposed approach would work if the percentage of novel classes in the chosen unlabeled set is low / absent?\n\n\n### Minor concerns\n- Figure 3a for more percentage of base labels\n    - Not a head-to-head comparison between FRCN-Base and FRCN-Base + Unlabeled, since the latter assumes more data\n    - Does the difference narrow with more percentage of base labels?\n\n- Conflicting claims\n    - Fig 1: exhibiting less than 7% in base degradation\n    - line 54: exhibiting less than 9% in base forgetting\n\n\n### Justification of the rating\nMy main concern is highlighted above. In general, explanations towards the above mentioned concern would help the readers\n\nQUESTIONS:\nPut together in the weaknesses section\n\nLIMITATIONS:\nYes", "TITLE": "Official Review", "RECOMMENDATION": 6, "REVIEWER_CONFIDENCE": 4, "SOUNDNESS_CORRECTNESS": 4, "DATE": "2023-07-26"}