Multi-Perspective Data Augmentation for Few-shot Object Detection
Authors: Anh Khoa Nguyen Vu
Truong Quoc Truong
Vinh-Tiep Nguyen
Thanh Duc Ngo
Thanh-Toan Do
Tam V. Nguyen
Venue: ICLR 2025 Poster (2025)
Keywords: few-shot object detection
controllable diffusion
data augmentation
Primary area: applications to computer vision, audio, language, and other modalities

Abstract: Recent few-shot object detection (FSOD) methods have focused on  augmenting synthetic samples for novel classes, show promising results  to the rise of diffusion models. However, the diversity of such datasets is often limited in representativeness because they lack awareness of typical and hard samples, especially in the context of foreground and background relationships. To tackle this issue, we propose a Multi-Perspective Data Augmentation (MPAD) framework. In terms of foreground-foreground relationships, we propose in-context learning for object synthesis (ICOS) with bounding box adjustments to enhance the detail and spatial information of synthetic samples. Inspired by the large margin principle, support samples play a vital role in defining class boundaries. Therefore, we design a Harmonic Prompt Aggregation Scheduler (HPAS) to mix prompt embeddings at each time step of the generation process in diffusion models, producing hard novel samples. For foreground-background relationships, we introduce a Background Proposal method (BAP) to sample typical and hard backgrounds. Extensive experiments on multiple FSOD benchmarks demonstrate the effectiveness of our approach. Our framework significantly outperforms traditional methods, achieving an average increase of $17.5\%$ in nAP50 over the baseline on PASCAL VOC.

Published as a conference paper at ICLR 2025
MULTI-PERSPECTIVE DATA AUGMENTATION FOR FEW-SHOT OBJECT DETECTION
Anh-Khoa Nguyen Vu Quoe-Truong Truong!.2 Vinh-Tiep Nguyen 1,2,*
Thanh Due Ngo!.2
Thanh-Toan Do Tam V. Nguyen'
Uniiveriity Information Technology, Ho Chi Minh City, Vietnam
Vieetna National University, Ho Chi Minh City, Vietnam DDepartment of Data Science and AI, Monash University, Australia UUniversity of Dayton, Dayton, OH 45469, United States
ABSTRACT
Recent few-shot object detection (FSOD) methods have focused on augmenting synthetic samples for novel classes, show promising results to the rise of diffusion models. However, the diversity of such datasets is often limited in representa- tiveness because they lack awareness of typical and hard samples, especially the context of foreground and background relationships. To tackle this issue, we propose a Multi-Persspettvve Data Augmentation (MPAD) framework. In terms of
relationships, we propose in-context learning for object synthesis (ICOS) with bounding box adjustments to enhance the detail and spatial information of synthetic samples. Inspired by the large margin principle, support samples play a vital role in defining class boundaries. Therefore, we design Har- monic Prompt Aggregation Scheduler (HPAS) mix prompt emheddings at each time step of the generation process in diffusion models, producing hard novel samples. For gregrounddbackgrron Proposal method (BAP) to sample typical and hard backgrounds. Extensive ex- periments on multiple FSOD benchmarks demonstrate the effectiveness of our approach. Our framework significantly outperforms traditional methods, achiev- ing an average increase of 17.5% in nAP50 over the baseline on PASCAL VOC. Code is available at atgthhuh..o//aaaaha//PAAl
relationships, we introduce Background
INTRODUCTION
Humans can recognize new objects after seeing them just a few times, a remarkable ability that is simulated and studied in few-shot object detection (FSOD). In FSOD setup, there two distinct datasets: the base dataset and the novel dataset. The base dataset is extensive and comprises numerous classes with abundant training instances. This dataset helps the model learn a wide variety of object features and characteristics, forming general knowledge for detection tasks. In contrast, the novel dataset is limited, with only a few samples per novel class, posing a significant challenge for object detection. This constraint makes FSOD a critical research area Yan et al.) 2019 Kang 2019 Fan et al. 2020 Wang et al. 2020 et al.) 2021 Li & Li [022bb] /hhang et al., 202
al.
Han et al. 2022b Bulat et al. 2023). with potential applications in fields such as robotics, autonomous driving, and medical imaging, where models need to handle critical but rare scenarios.
Earlier FSOD approaches (Wang al. 2020 Qiao al. 202 Yan et al. 2019 Zhang et al.) 2021 firstly train a model on the base dataset to establish a generalized detector. This detector is then fine-tuned on the novel dataset recognize and detect new objects. Still, this approach could lead to overfitting due to the limited amount of data available. Other methods (Zhu et al.) 2021 Li
al. 2023a) leverage the general knowledge of the large language models (LLMMs) to alleviate this issue. A simply yet effective approach for FSOD is data augmentation. Recent works (Zhang & Wang 2021] Vu al. 2023b) utilize the prior knowledge to create hallucinations in feature space to fine-tune classifiers. However, these synthetic samples often lack essential information for object detection, such as low level details, spatial information. Meanwhile, other methods (Li & 2021a
"Corresponding Authors: tiepnv wuit.cduiv

---

Published as a conference paper at ICLR 2025
Demirel et al. 2023) rely solely on traditional geometric transformations (e.g.. flipping, cropping, rotating) to create variations of given samples from novel classes, which limits the diversity of synthesized datasets.
Recently, diffusion models have achieved remarkable strides in producing high-quality and diverse
datasets (Nichol et al. 2021 Rombach et al. 2022 Ramesh et al. 2022 Saharia al. 2022)
Furthermore, large-scale text-to-image diftusion models have shown significant flexibility and scal- ability in image editing tasks by incorporating lightweight adapter modules for additional conditions
(e.g.. bounding box, semantic map, depth map. human pose) (Zhang et al., 2023) Li et al. 2023b
Zhuang al. 2023). Consequently, several FSOD methods (Lin et al. [023]] Fang et al., 2024 Wang et al.J (024) leverage controllable diffusion but often reply on simple prompts to generate synthetic objects, without exploring attributes such as colors, shapes, details, sizes, types of objects. As result, most synthesized novel samples are typical objects.
To address the above problem, we propose In- Context learning for Object Synthesis (ICOS). ICOS leverages general knowledge from LLMs to deeply explore the attributes of novel classes and diversify prompt inputs. Additionally, the diversity of class is derived from both typi- cal and hard samples, as illustrated in Figure 1 Inspired by the large margin principle (e.g. SVM (Cortes 1995)), support vectors play a cru- cial role in learning generalized model. These samples, considered hard samples, often exhibit characteristics not only of the main class but also of neighboring ones. In other words, in this pa- per, we define typical samples as those that con- tain features of a single class, whereas hard sam- ples exhibit features of two classes. Leveraging this aspect, we aim to blend the characteristies of two classes during the data generation pro- cess. Unlike image classification, where only the foreground-foregroundd
Classes
acroplane bird
train
horse
bus
cow
types hard
typical sample
Figure 1: T-SNE visualization of novel syn- thetic samples and base real samples in Novel Set 1 of PASCAL VOC. We only generate syn- thetic samples for three novel classes ("bird", "bus", "cow"") and use real samples for three base classes ("aeroplane", "train", "horse"). Typical and hard samples in novel classes created by using ICOS and HPAS, respectively. Base real samples are considered typical samples.
foreground relations are considered and the main objects are roughly centered, object detection must take into account the foreground- background relations. To our knowledge, this is the first work to use ChatGPT to diversify prompts and embed the foreground-backgree relations when synthesizing diverse datasets in few-shot object detection.
In terms of the foreground-foregrouund relation, we propose Harmonic Prompt Aggregation Sched- uler (HPAS) to mix prompt embeddings at each time step of the generation process in the diffusion model. This approach guides the diffusion model to synthesize objects with high-level features (c.g., object parts) of the main class and low-level features (e.g., shape, color, size) of a selected base class. By mixing the low-level features of the base class, we leverage the prior knowledge acquired during the base training stage. Regarding bvegrrund-backgronn Background Proposal method (BAP) to sample typical and hard backgrounds from the base dataset. For typical backgrounds, we select the most cluttered backgrounds based on an entropy metric. For hard backgrounds, we select those with the highest similarity to foreground objects in the embed- ding space. During the base training stage, the model learns to classify novel objects as backgrounds when trained on base classes. This phenomenon creates ambiguities in learning and detecting novel classes. Therefore, in the novel training stage, we guide the model to distinguish novel classes from similar base backgrounds, utilizing the knowledge gained from base training.
relation, we introduce a
In summary, our contributions can be summarized as follows:
We propose a Multi-Persppectvy Data Augmentation (MPAD) framework for synthesizing data which better prevents the overfitting problem for FSOD.
2

---

Published as a conference paper at ICLR 2025
We introduce ICOS, HPAS, and BAP methods to enhance synthesis by considering brreground-bccgrrounn relation. Specifically, ICOS diversifies prompts using fine- ane-grained attributes from the general knowledge of LLMs. HPAS supports the controllable diffusion model to create hard samples containing characteristics of two foregrounds, while BAP proposes typical and hard backgrounds in relation to the foreground.
We conduct comprehensive experiments on FSOD benchmarks to demonstrate the effec- tiveness of our method. The results show that our method outperforms the baseline model by a large margin and achieves state-of-the-art performance on few-shot object detection.
MPAD METHOD
2.1 FORMULATION
In few-shot object detection, the base data is characterized by large number of base classes Chase with an abundance of samples. In contrast, the novel data comprises a few novel classes Crowel each with K samples (K E (1, 2,,5,, 5, 10} in the PASCAL VOC setting). It is important note that the base classes and novel classes are disjoint sets (i.e., Chasa O Cnovel
0). As outlined in previous works (Yan et al.) 2019 Wang et al. 2020; Qiao et al.) (021). we define two data sets D2 = {(L and number of samples in set s, respectively. An annotation A (c, b) represents pair consisting of a class name с E C2 and the bounding box of the j-th object in the i-th image.
where E {base, novelf. Lar Ag and N, denote the images, annotations
Typically, FSOD methods involve two stages: base training stage and novel fine-tuming stage. In the base training stage, detectors are trained on Daese to acquire extensive knowledge, learn concept features, and build the feature extractor In the novel fine-tuning stage, the base models fine- tuned on a balanced set Dst with K samples for each base and novel class to detect both base and novel objects the image.
2.2 FOREGROUND- FOREGROUND-BACKGROLLN BACKGROUND RELATION- DATA AUGMENTATION
In object detection, image comprises two main components: the background and the foreground. The foreground highlights the primary objects, while the background provides contextual informa- tion that aids in object inference within the images. To augment data with class representativeness, we synthesize both typical and hard samples. For typical foreground samples, ICOS uses input samples to generate novel objects with characteristics pointed out by general knowledge of L.MMs. To create hard samples, HPAS mixes prompt embeddings at each time step of the data generation process in the diffusion model. Different from the classification task, the background plays an im- portant role in object detection. Therefore, BAP proposes hard background samples in relation to foreground features. As shown in Figure 2 our overall framework contains three main components: ICOS, HPAS, and BAP, as detailed in the tollowing subsections.
2.3 IN-CONTEXT LEARNING FOR OBJECT SYNTHESIS
Controllable diffusion. We utilize PowerPaint (Zhuang et al. 2023) model for the object inpainting task, ensuring that the generated object seamlessly conforms to the specified mask shape. We process the object's bounding box by applying masking and padding, and then use it as the mask input for controllable diffusion. The controllable diffusion 0(-) takes prompt embedding Ço, bounding box b. and an image / as inputs. The reverse diffusion process is sequence of denoising steps with time step& & T,T 1,....1.
2t-1 = p(2--10 b)),
where 27 is the reference image and 2g is the synthesized image j. For each novel class c E Cnovets generate N synthesis samples. We add novel objects random base images Ihuses defined as:
f- 0 0(Thase Sey b),
3

---

Published as a conference paper at ICLR 2025
0: "oow""
Hand buclaground
C
"eococatt P"blacict, "bcownt, whody"s ("ccurdy"> Trusculas", "Lean", *atosky"), "sasenic ("abore"",, "bead"s ("eound"y
2 cos(s. fhace)
(oose) .P.
"oual", Peal) (Plong"), (telomen") 1y *aliseyy (Ned "Casge"), fine-grained cianseas
2ijes F(bbased I1 11F
Typical background
BAP
ileau Σ (oanotop(
polurter;
Backgrodd
Yesr images
Cuo softmmast (bmsr)
Typical hackground
balssis ef cow las shur oval eyes, seri- creat cas, long tall, legesiee"
Typical for fgrounddd
Foreground generation
(0
covet
Typical forepround
pard jeatter
Hard forgground
c: "A lus duet luir, cvaleyro, acti- cruct can, long tall large size"
Harmooiic Promupt Asgregation Schoddleer (HIPAS)
Zo-1 p(Z--1 (2,, Yey, b))
Ycs (a, a be + (1 (6) (asse)
Oraec la horss"
Ge m. Rio-s) + (1 m)
(w
1-2 (1 - w))
p-3
Figure 2: The overall framework. To exploit the ability of controllable diffusion model for FSOD, proposed novel data augmentation method that incorporates various aspects to generate diverse data. Our method includes ICOS, BAP, HPAS. ICOS aims to deeply explore the attributes of novel classes and diversify the prompt for controllable diffusion models. BAP selects hard and typical backgrounds while HPAS generates hard (mixed) instances
where i ~ U1, Niase), So E(prompt.) is the prompt embedding and È is the CLIP text encoder. The bounding box is randomly chosen from the annotations of Base unless otherwise specified. The class label of the selected bounding box b is replaced by
Simple Prompting. Following previous works (Lin et al., 2023 Fang et al. 2024 Wang et al, 2024]. simple prompt is created by concatenating the prefix "a" and the class name, resulting in the simple prompt input prompt, = "a photo of a [CLASSNAME]"
In-Context learning for Object Synthesis (ICOS). The simple prompt mentioned above only out- lines a general concept of the object without detailed information, which can result in similar objects within a class and limit diversity. To address this, propose using in-context learning (Reynolds & McDonell 2021] to collect and incorporate specific characteristics and class information, enhancing the diversity of the prompts for the diffusion model.
In-Context learning for Attribute Analysis. Based on a recent work (Zhu et al.) 2024). we explore the attributes of a specific class using LLMs. Specifically, we construct an input and output template to extract appearance information of a class using ChatGPT. Figure 3 (a) demonstrates a in-context learning approach for analyzing parts and attribute values of a class, where the target class name is input for the next inference. We then parse the attributes into dictionary. with keys and values repre- senting the general appearances and detailed attributes of the class. We randomly select key-value attributes list [attr] (key,, val ppe))....... to additional provide information and diversify the prompt. Specifically, we construct the new prompt from [attr] by the template as prompt. "a (CLASS NAME] has [keyı] [valuei], [key2] [valuez], .... [key...] [valuem,]]]
In- In-Context learming for Fine-Grained Categories. Fine-grained categories are crucial for assessing the diversity within class. Several methods (Vu et al. 2023a Wu et al. 2024) exploit this aspect to improve model generalization. SMS (Vu et al.] (023a) introduces a technique that utilizes fine- grained categories in few-shot instance segmentation by generating hallucinated superclasses from base and novel classes. Inspired by SMS, we leverage LLMs by querying ChatGPT to list the fine- grained categories of class c using the prompt illustrated in Figure (b).
4

---

Published as a conference paper at ICLR 2025
In-Context learning for attribute analysis
In- Aa-Contextt learning for fine-grained objects
Describe the features and attributes of cach class after we enter the input class. With cach class, return the response, and attribute following this instruction:
List the fine-grained categories within an object
class, With each class, return attribute following this instruction:
Input sample:
Input sample: cow
Describe the appearance attributes of cow categorics.
Output sample:
Ouput sample:
atrr:
response: Cows typically have colors such as black, white, brown, or combination of these. Their fur is usually short and coarse...
"cow": {
"Fine-grained classes" ["Holstein Friesian", "Jersey", "Guernsey", "Brown Swiss", "Ayrshire", "Herefard", "Angus", "Charolais", "Limousin", "Sinmental" "Brahnan", "Gelbvieh", "Shorthorn", "Texas Longhorn", "Highland", "Red Poll", "Devon", "Galloway", "Belgian Blue", "Watusi"] ),.,)
atrr:
"cow" {
"colors": ["black", "white", "brown"], "fur": ["short", "coarse"], "body": ["large", "rounded", "stocky"], "ears": ("medium-sized "floppy"1... h..)
Input: [CLASS NAME]
Input: [CLASS NAMR]
(a)
(b)
Figure 3: In-context learning technique for exploring (a) attributes and (b) fine- fine-grained object cat- egories of novel class given sample. The input [CLASSNAME] is replaced by class name c E Cnovel.
The result of this query is parsed and added to [attr] to generate diverse of prompts. The final prompt, is randomly sampled from the attribute list [attr] and then used for synthesizing novel class samples. See Figure 6 and Figure 7 in Appendix for detailed responses of ICOS.
2.4 HARMONIC PROMPT AGGREGATION SCHEDULER
In addition to leveraging hard novel samples for the few-shot object detection model, we in- troduce a mechanism called Harmonic Prompt Aggregation Scheduler (HPAS). The main idea is to mix a base class with similar novel class to enhance the diversity of the synthetic dataset. This is achieved in the prompt embed- ding space, step by step, throughout the genera- tion process of the diffusion model. The prompt embedding aggregation scheduler is defined:
02
10
Figure 4: Visualization of the weighted values of the Harmonic Prompt Aggregation Scheduler across the timesteps of controllable diffusion.
Ye,t (1)) * Se + ar Cbose:
where Ot = m X ((t-1) + (1 m) x ae is the weighted value at t-th time step and w is the starting value. By gradually increasing the weight of novel class features and reducing that of the base class, we create a synthetic object that incorporates novel detailed characteristics within the low-level features of the base class. Inspired by He et al. (2020), the momentum m is used to retain the main features of the prompt embedding. The weighted values are shown in Figure 4 We substi-
(
tute Te.t from Eq. 3 into Eq. (.. In this way, we can create hard samples, as shown Figure The reverse diffusion process of the diffusion model becomes:
Zt-1 1 = -(( 10(z,, Te,ts b))
(4)
5

---

Published as a conference paper at ICLR 2025
Bird Airplane
Bus
Tiger Pig
Pig Tiger
Figure 5: Visualization of the mixed instances of the Harmonie Prompt Aggregation Scheduler during the generation data process in the controllable diffusion model.
2.5 BACKGROUND PROPOPOSAL
The background plays an important role in object detection tasks, where the model must distinguish only between foreground objects. but also between foreground and background. In the base training stage, the model is trained to classify novel classes as background due to the condition Cbase Covee! = D. Therefore, in the novel training stage, we need to guide the model efficiently distinguish novel classes from new backgrounds by utilizing backgrounds with similar visual fea- tures. To address this issue, we introduce the background proposal (BAP), which includes both the hard background proposals and the typical background proposals.
Hard background proposal. Inspired by [e al. (2019) where objects concealing in the back- grounds, these camouflaged objects have foreground that visually resembles the background and it creates difficulties for the model when detecting them. Therefore, we introduce a visual similarity background technique to create hard samples.
select backgrounds from base images Josee that share similar features with the novel class by employing cosine similarity. Instead of using textual embeddings, which may not capture essential visual information, we use a pretrained visual encoder F(-) (e.g., ViT (Dosovitskiy et al. 2021 ) In FSOD, the number of novel samples is insufficient represent the general class distribution. Therefore, we use the stable diffusion model (Rombach et al. 2022) SD()) to synthesize a set of samples for class c, denoted by fc. This synthetic set is used for selecting hard backgrounds. The cosine similarity metric is defined as follows:
1 F(I -F(!))
cos(c, [sase)
(5)
n [F [I]F(((()
where f. - { f -SD(r=ppppt)) are synthesized images of class c. We select the top base backgrounds with the highest similarity scores to the novel class c, denoted by Ihard
Typical clutter background. For the typical background, we sample from Losee ones that have clut- ter features representing scenes with crowded and complex environments (as defined in Rosenholuz et al. (2007)). These samples with noise features force the model to improve its localization ability.
In this paper, we use the entropy score to quantify the clutter level of an image. Specifically, we normalize the feature embedding of the image using the softmax function. Then, we apply the entropy formula follows:
Hiase -
Σ Abuse log(abase),
(6)

---

Published as a conference paper at ICLR 2025
Method
2
Novel Set
3
5
10
1
2
Notel
3
5
10
1
2
3
10
Mean
TEA w/ le Wone et 2020
22.9
34.5
60.4
46.7
52.0
16.9
26.4
30.5
34.5
39.7
15.7
27.2
34.7
40.8
44.6
33.8
THA w/ can Wang ar 20120
25.3
36.4
42.1
47.9
52.8
18.3
27.5
30.9
34.1
39.5
17.9
21.2
54.3
40.8
45.6
34.7
PSDe
2020
24.2
35.3
422
49.1
57.4
21.6
24.6
31.9
37.0
45.7
21.2
30.0
57.2
43.8
49.6
36.7
MPSR
42.5
55.2
244
29.3
39.2
47.8
35.6
41.8
42.3
43.4
Surgeelle F3/48
52.9
59.7
23.7
30.6
38.4
43.0
22,6
33.4
39.5
SRR-FSITaZzuRR [d12
47.8
50.5
51.3
55.2
56.8
32.5
15.3
39.1
40.8
43.8
40.1
41.5
44.3
46.9
46.4
44.8
DCNNe Hilremmn 20045
33.9
37,4
43.7
51,1
59.6
23.2
24.8
30.6
36.7
45.5
32.3
34,9
39.7
42.6
50.7
39.2
Meta DETR panarr +1
2021
19.0
53.2
57.4
62.0
27.9
32.3
18.4
43.2
51.8
34.9
11.8
47.1
54.1
58.2
45.8
45.8
Mota F RRUNN [Hil P1r23
43.0
54.5
606
66.1
65.4
27.7
35.5
46.1
47.8
51.4
40.6
46.4
53.4
59.9
58.6
50.5
KISOD (2hang
2012
54.4
60.9
68.8
37.8
43.1
18.1
504
34,8
DeFRCN
53.6
66.5
29.5
35.0
40.4
MHD Wur p37z2
63.4
66.3
67.7
69.4
68.1
42.1
46.5
53A
55.3
53.8
56.1
58.5
59.0
62.2
61.7
59.0
FCT
(n Funeu)
57.1
$7.9
63.2
67.1
37.6
34.5
43.7
49.2
51.2
39.5
54.7
52.3
57.0
58.7
50.9
500
PS-DETRIIRI 0 2023
45.0
48.5
$1.5
52.7
56.1
57.1
41.3
41.4
46.5
49.0
43.8
47.1
50.6
52.1
56.9
48.1
D&R [Lietun (1
41.0
51.7
55.7
618
65.4
30.7
39.0
42.5
16.5
51.7
37.9
47.1
51.7
56.8
59.5
49.3
VHA
47.4
54,4
58.5
64.5
06.5
33.7
38.2
43.5
48.5
52.4
43.8
48.9
53.3
58.1
00.0
51.
FSKN
20233a
33,9
42.3
51.9
55.1
27.3
35.2
47.5
26.7
37.0
41.
37.3
39.4
48.5
58.6
61.5
18.9
NIFF
Grnguus 2t 20236
46.0
57.2
62.0
65.5
67.2
30.1
39.6
45.0
49.4
52.8
41.1
52.5
56.4
59.7
52.4
Using
deep learning nagunentttion
technique
TIP Li
2021a
27.7
36.5
43.3
50.2
59.6
22.7
30.1
33.8
40.9
$6.9
21.7
30.5
38.1
44.3
50.9
38.3
Hallur (Zhint Wing 2021
47.0
46,9
66.5
54.7
54.7
26.3
31.8
37.4
37.4
41.2
10.4
42.1
43.3
51.4
49.6
43.2
LVC
58.8
52.8
29.2
50.6
48.4
52:7
52.3
Norm X24 A013
62.1
64.9
67.5
69.2
67.5
39.9
4688
34.4
54.2
53.6
58.2
60.3
51.0
64.0
59.3
SFOT ef pizsi0
47.9
60.4
62.7
67.3
69.1
32.4
41.2
45.7
50.2
54.0
43.5
56.9
60.6
62.5
53.9
Lin et (1ir p026)
67.5
69.8
71.1
71,5
$2.0
54.3
57.5
57.4
55.9
58.6
59.6
63.9
61.6
SNIDA Wongg 2004
59.3
60.8
64.3
65.4
65.6
352
40.8
50.2
54.5
50.0
51.6
52.4
55.9
58.5
62.6
MPAD
69.1
69.5
69.5
89.9
68.9
58.4
59.7
61.8
61.8
03.5
70.1
69.8
69.9
70.4
71.4
66.9
Table 1: Generalized few-shot object detection performance (nAP50) on the PASCAL VOC dataset. The best and second performances are marked in boldface and underlined, respectively. iindicates using post-detection process.
where tlsse softmax (ofttmxx(F[Jjmmrre (F(Jbase) and H hase are the feature distribution and information entropy of the background image Th base: respectively. We select the top base backgrounds with the highest entropy score, denoted by jelutter.
2.6 DATA GENERATION AND MODEL TRAINING PROCESS
In summary, for each novel class c, we use the proposed backgrounds 1e = I U jclutter to synthesize novel images Inovet- We generate two types of foregrounds: the typical foreground and the hard foreground. For typical foregrounds, we use in-context learning with diverse attributes and fine-grained categories to create prompt... Then, images with the novel class e are then synthesized through the reverse diffusion process conditioned on prompt embedding Çe. as illustrated in Eq. ( For hard foregrounds, we use Eq. (4) to generate mixed instances between class c and selected base class. The annotation Anovee for synthesized image ihove! is copied from the original annotation data of the selected base image, with the class name within the bounding box replaced by c. The novel synthetic dataset Dnove! = {(frovel: A 1 N U UD is used to fine-tune detectors during the novel training stage. See Figure g in Appendix B for additional visualizations of our synthetic dataset.
EXPERIMENTS
3.1 DATASETS AND SETTINGS
Dataset settings and evaluation. Following previous works (Yan et al. 2019 Kang et al. 2019;
Qiao al. 2021 1. we assess our MPAD method in the FSOD setting of PASCAL VOC (Everingham
et al. 2010 2015) and MS COCO (Lin et al. 2014). For PASCAL VOC, 20 classes are separated
into three sets each set, five classes are designated as novel classes Cnovet, and the remaining fifteen classes are used as the base classes Chase. There are K samples for each novel class (K E (1,2,3, 5, 10)). Regarding MS COCO, the dataset serves as a challenging benchmark for FSOD. 80 classes are split into 60 base classes and 20 novel classes (identical to the 20 PASCAL VOC classes). We select a value of K from the set (f1, 2, 5)) for each novel and base class to fine-tune detectors. To evaluate the model performance, we follow TFA (Wang et al. 2020), DeFRCN (Qiao jet al.] (02T) and use the Generalized Few-Shot Object Detection (G-FSOD) which contains both base and novel classes to train and test models in the novel fine-tuning stage. We report AP50 of
7

---

Published as a conference paper at ICLR 2025
Method
RAP
nAP50
I-sbot
RAP75
nAP
RAP50
2-shet
nAP75
RAP
RAP50
3-shoi
RAPT5
pAP
nAP50
5-shot
nAP15
TEA
2020
1.3
9.9
6.9
6.3
TFA cos Wanz et
2020
1.9
1.7
3.9
7.8
3.6
5.1
9.9
7.0
13.3
6.5
MPSR Wa è mi. 20221
2.3
4.1
2.3
3.5
6.3
3.4
5.2
6.7
PSDedView
Marlet
2020
3.2
1.4
13.3
8.1
4.4
DeFRCN c1 2024
4.8
4.4
16.3
7.8
10.7
20.0
10.3
13.5
24.7
13.0
Meta-D211 DETR (inin
2021
7.5
12.5
7.7
13.5
21.7
14
15.4
15.8
FCT
20220
AirDer c
2012
6.1
11.4
6.0
8.7
16.2
8.4
10.0
19.4
9.1
10.8
20.8
10.3
Meta IG [Ta
20228
5.1
10.7
4.3
7.6
16.3
6.2
9.8
20.2
8.2
10.8
22.1
9.2
FSRN [Guirreut a1
20
8.7
16.1
8.2
PS-DETT (Rome er
2023
7.0
13.6
7.5
8,9
17.5
9.0
10.0
18.8
10.0
10.9
20.7
10.8
Using deep
leurning augrenntation technique
Haliuc Zhang & Wane 2021
4.4
4.5
5.6
9.9
5.9
1.2
13.3
7.4
SEOT [Vll er
6.7
13.2
6.0
10.5
20.3
12.5
23.6
11.8
14.9
27.8
14.2
Norm- VAB X mi 122
9.5
8.8
13.7
13.7
14.3
14.4
15.9
15.3
SNIDA Warg 3i 2023
9.3
12.9
14.8
16.1
MPAD
18.3
18.8
18.5
31.6
18.9
18.8
31.8
19.1
18.9
32.4
19.3
Table 2: Generalized few-shot object detection performance on 2, 5-shot of MS COCO dataset. The best and second performances are marked in boldface and underlined, respectively.
novel classes (nAP50) PASCAL VOC dataset and nAP, nAP50, nAP75 metrics for experiments on the COCO dataset.
Implementation details. Our model adopts DeFRCN (Qiao et al. 2021 In both the base train- ing and the fine-tuning stage, we use the same hyper-pssameters as DeFRCN (Qiao et al.] 2021 During the fine-tuning stage, we utilize both real novel data and synthetic data train models on
a single NVIDIA GeForce RTX 2080 Ti GPU, We employ Powerpaint (Zhuang et al. 2023 for the conditional diffusion model 0(-), and the CLIP text encoder (Radford et al. 2021) for 8(-) In our experiments, we aim to generate an equal number of synthetic instances for each novel class. The image feature extractor FF.) is a pre-trained ViT model (Dosovitskiy al., (021) on Ima- geNet (Deng et al. 2009). We set w = 0.7, m = 0.8 and Naug = 300. The number of inference steps is fixed T 80. Several methods show that training with multi-scale objects is crucial in FSOD. Therefore, we implement a fundamental method to increase the diversity under this aspect. In particular, scale the selected bounding box in the data generation process with weight. We randomly select weight value in [1.25, 1.5, 1.75, 2} in our settings.
3.2
RESULTS AND DISCUSSION
3.2.1
MAIN RESULTS
We conduct G-FSOD experiments on PASCAL VOC (Everingham et al. 2010 2015) and MS COCO (Lin et al.) 2014) and report the results in Table and Table 2 respectively. These num- bers indicate that our method MPAD generally outperforms the baseline and other state-of-the-art methods FSOD benchmarks by large margin.
T shows the results from the three novel sets of PASCAL VOC, comparing our approach with baselines and state-of-the-art methods. Our MPAD method consis- tently outperforms the baselines across all splits and shots. Notably, our method, based on De- FRON (Qiao et al. 202T) achieves the highest performance of 66.9%. exceeding the baseline by an average margin of 17.5%. In extremely low-shot scenarios, our method delivers significantly larger performance gains, with an average increase of +31.0% nAP50 in the 1-shot setting. Considerably,
Results on PASCAL VOC.
our MPAD surpasses previous works (Wang et al., 2024 Lin et al. 2023 Kaul et al. 2022 Li et al.)
2023a Zhu et al.) 2021 Xu al., 2023) that use pretrained CLIP, ViT, diffusion models, language models, or post-processing in detection. Mcanwhile, methods (Wang et al. 2024 Lin et al.) 2023) are statc-of-the-art data augmentation methods in FSOD. Overall, our approach demonstrates supe- rior performance compared most existing methods across various splits and shots, highlighting the robustness and generalization capabilities of our method.
Results on MS COCO. We present the experimental results for MS COCO in Table By using method, baseline DeFRCN improves by about 11.5% on average, particularly in extremely few-shot settings and 2-shot). Specifically, our method enhances nAP, nAP50, and nAP75 by over 13%,
8

---

Published as a conference paper at ICLR 2025
1-shot
2-shot
3-shot
5-shot
10-shot
Mean
Cutout
54.7
57.2
62.3
64.0
62.5
60.1
GridMask
54.3
58.0
62.0
63.7
63.2
60.2
Augment
51.3
54.4
59.6
62.1
61.0
57.7
CutMix
55.5
57.6
61.4
63.9
63.5
60.4
MPAD
69.1
69.5
69.6
69.9
68.9
69.4
Table Few-shot object detection performance (nAP50) of other augmentation methods on Novel Set of PASCAL VOC dataset. The best and second performances are marked in boldface and underlined, respectively.
21%, and 14%, respectively, in the 1-shot setting. These results highlight the promising approach to improving FSOD performance by employing controllable diffusion model.
Comparison with different augmentation methods. Following Wang et al. (2024), also show few-shot object detection results on PASCAL VOC Novel Set of other augmentation methods Cutout Vries 2017), GridMask (Chen al. 2020), AutoAugment (Zoph al. 2020), and Cut- Mix Yun et al. 2019) in 3 The nAP50 results show that our method consistently outperforms these augmentation methods by large margin (+9%). This evidence demonstrates the effectiveness of our approach in the context of few-shot object detection. Detailed ablations on the number of generated images and training schemes are provided in Appendix and Appendix respectively.
3.2.2 IS THE DIVERSITY A CLASS NECESSARY?
We investigate the importance of class di- versity in Table The table indicates that applying different augmentation tech- niques, which create typical and hard fore- grounds, improves the performance of de- tectors. Specifically, controllable diffu- sion using ICOS in the third row diver- sifies prompts, enhancing the diversity of the synthetic dataset and increasing detector performance by approximately 6% nAP50 compared not using ICOS (i.e., directly using PowerPaint with simple prompting. shown in the first row). Additionally,
ICOS
HPAS
Atrihutes
ena-ggrined.
62.1
38.7
65.8
39.0
41.2
68.5
42.6
42.8
69.1
45.1
Table 4: songrround-oregroouue ablation studies about ICOS and HPAS. nAP, nAP50, nAP75 metrics on Novel Set of PASCAL VOC are reported to eval- uate the importance of each modules.
by using HPAS, our method generates hard samples for FSOD, which boosts performance to 42.8%/69. 1 1%/45.1% in nAP/AAP50/APP5.. These results demonstrate that both typical and hard foregrounds are crucial for data augmentation, especially in FSOD. Detailed ablation studies on the foreground forerooundoorerroudd approach are provided Table Figure and Figure 10 in Appendix
3.2.3 THE IMPACT BACKGROUND SELECTION
In addition to studying the foreground, we also conduct ablation experiments back- ground selection, a crucial but often over-
Random
Typical
Hard
nAP
nAP50
nAP75
looked component. Table 5 demonstrates the effectiveness of background selection. With random selection, nAP, nAP50, and nAP75 metrics achieve only 34.6%. 62.1%. and 34.4%. respectively, which are lower than other background proposal strategies. By using both our typical and hard back- ground proposal technique, the detector be improved to 37 37.4%/64.114%
34.6
62.1
34.4
37.0
64.8
38.3
35.2
61.9
36.0
36.8
64.4
37.5
V
v
37.4
64.1
39.6
37.0
64.5
37.7
63.6
38.2
Table 5: BAP. Metrics on Novel Set of PASCAL VOC are reported to evaluate the importance of each selection.
ablation study about
in These results high-

---

Published as a conference paper at ICLR 2025
light the importance of gnreground-backrree relations and the effectiveness of our BAP method. Therefore, we hope these types of relations can explored more few-shot object detection.
3.3 LIMITATION
There are several issues with diffusion models. The hallucinations still occur in the generated im- ages. These circumstances can lead to parts or the entire generated object being unrelated to the prompt or resulting low-quality synthetic images, as shown in the last two rows of Figure g There several potential ways reduce the number of hallucinations in generated data. We can apply filter as a post-process for data generation, which can filter out objects that significantly apply LoRA in PEFT (Mangrulkar
deviate from the general characteristies. Additionally, et al. 2022) to fine-tune the diffusion model on the few-shot data, which could generate synthetic samples with greater similarity to the current dataset and reduce ballucinations in the synthetic data. Another issue relates to the starting value w. This value is fixed, which may not be suitable for all novel classes.
CONCLUSION
In this paper, we introduced novel nulti-perspective data augmentation framework that enhances few-shot object detection by addressing the challenges of sample diversity and representativeness. Our approach effectively leverages in-context learning for object synthesis and incorporates har- monic prompt aggregation scheduler to create challenging novel samples, while also improving the representation of foreground-backgree Extensive experiments several FSOD benchmarks, including PASCAL VOC, demonstrate the significant advantages of our framework over state-of-the-ee methods.
relationships through our Background Proposal method.
ACKNOWLEDGMENTS
This research is funded by Vietnam National University Ho Chi Minh City (VNU-HCM) under grant D82024-26-06.
REFERENCES
Adrian Bulat, Ricardo Guerrero, Brais Martinez, and Georgios Tzimiropoulos. Fs-detr: Few-shot detection transformer with prompting and without re-training. In ICCV, pp. 11793-11802, 2023.
Pengguang Chen, Shu Liu, Hengshuang Zhao, and Jiaya Jia. Gridmask data augmentation. arXiv preprint arXiv:2001.0448S: 2020.
Corinna Cortes. Support-vector networks. Machine Learning, 1995.
Berkan Demirel, Orhun Bugra Baran, and Ramazan Gokberk Cinbis. Meta-tuning loss functions and data augmentation for few-shot object detection. In CVPR, pp. 7339-7349, 2023.
Jia Deng, Wei Dong, Richard Socher, Li-Jia Li, Kai Li, and Li Fei-Fei. Imagenet: A large-scale hierarchical image database. CVPR, pp. 248-255. leee, 2009.
Terrance DeVries. Improved regularization of convolutional neural networks with cutout. arXiv preprint arXiv: 04552, 2017.
Alexey Dosovitskiy, Lucas Beyer, Alexander Kolesnikov, Dirk Weissenborn, Xiaohua Zhai, Thomas Unterthiner, Mostafa Dehghani, Matthias Minderer, Georg Heigold, Sylvain Gelly, et al. An image is worth 16x 16 words: Transformers for image recognition at scale. ICLR, 2021.
Mark Everingham, Luc Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The paseal visual object classes (voc) challenge. IJCV, 88(2), 2010.
Mark Everingham, SM Ali Eslami, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes challenge: A retrospective. IJCV, 111(1), 2015.
10

---

Published as a conference paper at ICLR 2025
Qi Fan, Wei Zhuo, Chi- Chi-Keung Tang, and Yu- Wing Tai. Few-shot object detection with attention-rye and multirrelation detector. In CVPR, 2020.
Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmen- tation for object detection via controllable diffusion models. In WACV, pp. 1257-1266, 2024.
Karim Guirguis, Mohamed Abdelsamad, George Eskandar, Ahmed Hendawy, Matthias Kayser, Bin Yang, and Juergen Beyerer. Towards discriminative and transferable one-stage few-shot object detectors. WACV, pp. 3760-3769, 2023a.
Karim Guirguis, Johannes Meier, George Eskandar, Matthias Kayser, Bin Yang, and Jürgen Beyerer. Niff: Alleviating forgetting in generalized few-shot object detection via neural instance feature forging. In CVPR, pp. 24193-24202, 2023b.
Guangxing Han, Shiyuan Huang, Jiawei Ma, Yicheng He, and Shih-Fu Chang. Meta faster r-cmm: Towards accurate few-shot object detection with attentive feature alignment. In AAAI, volume 36, pp. 780-789, 2022a.
Guangxing Han, Jiawei Ma, Shiyuan Huang. Long Chen, and Shih-Fu Chang. Few-shot object detection with fully cross-transformer. In CVPR, pp. 5321-5330, 2022b.
Jiaming Han, Yuqiang Ren, Jian Ding, Ke Yan, and Gui-Song Xia. Few-shot object detection via variational feature aggregation. In AAAI, 2023.
Kaiming He, Haoqi Fan, Yuxin Wu, Saining Xie, and Ross Girshick. Momentum contrast for unsupervised visual representation learning. In CVPR, pp. 9729-9738, 2020.
Hanzhe Hu, Shuai Bai, Aoxue Li, Jinshi Cui, and Liwei Wang. Dense relation distillation with context-aware aggregation for shot object detection. In CVPR, 2021.
Bingyi Kang, Zhuang Liu, Xin Wang, Fisher Yu, Jiashi Feng, and Trevor Darrell. Few-shot object detection via feature reweighting. In ICCV, 2019.
Prannay Kaul, Weidi Xie, and Andrew Zisserman. Label, verify, correct: A simple few shot object detection method. In CVPR, pp. 14237-14247, 2022.
Trung-Nghia Le, Tam V Nguyen, Zhongliang Nie, Minh-Triet Tran, and Akihiro Sugimoto. Anabranch network for camouflaged object segmentation. Computer vision and image under- standing, 184:45-56, 2019.
Aoxue Li and Zhenguo Li. Transformation invariant few-shot object detection. In CVPR, pp. 3094- 3102, 2021a.
Aoxue Li and Zhenguo Li. Transformation invariant few-shot object detection. CVPR, 2021b.
Bohao Li, Boyu Yang, Chang Liu, Feng Liu, Rongrong Ji, and Qixiang Ye. Beyond max-man max-margin: Class margin equilibrium for few-shot object detection. In CVPR, 2021.
Bowen Li, Chen Wang, Pranay Reddy, Seungehan Kim, and Sebastian Scherer. Airdet: Few-shot detection without tun-tuning for autonomous exploration. In ECCV, pp. 427 Springer, 2022.
Jiangmeng Li, Yanan Zhang, Wenwen Qiang, Lingyu Si, Chengbo Jiao, Xiaohui Hu, Changwen Zheng, and Fuchun Sun. Disentangle and remerge: Interventional knowledge distillation for few- shot object detection from a conditional causal perspective. In AAAI, volume 37, pp. 1323-1333, 2023a.
Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded text-to-image generation. In CVPR, pp. 22511- 22521, 2023b.
Shaobo Lin. Kun Wang, Xingyu Zeng, and Rui Zhao. Explore the power of synthetic data on few- shot object detection. In CVPRW, pp. 638-647, 2023.
Tsung- Yi Lin, Michael Maire, Serge Belongie, James Hays, Pietro Perona, Deva Ramanan, Piotr Dollár, and C. Lawrence Zitnick. Microsoft coco: Common objects in context. In ECCV, 2014.

---

Published as a conference paper at ICLR 2025
Jiawei Ma, Yulei Niu, Jincheng Xu, Shiyuan Huang, Guangxing Han, and Shih-Fu Chang. Digeo: Discriminative geometry- aware learning for generalized few-shot object detection. CVPR, pp. 3208-3218, 202..
Sourab Mangrulkar, Sylvain Gugger, Lysandre Debut, Younes Belkada, Sayak Paul, and Benjamin Bossan, Peft: State-of-the-ar parameter-effiient fine-tuning methods. https://githu/ com//aaggingface//eet 2022.
Alex Nichol, Prafulla Dhariwal, Aditya Ramesh, Pranav Shyam, Pamela Mishkin, Bob McGrew, Ilya Sutskever, and Mark Chen. Glide: Towards photorealistic image generation and editing with text-guided diffusion models. arXiv preprint 1rXiv:2112.10777 2021.
Limeng Qiao, Yuxuan Zhao, Zhiyuan Li, Xi Qiu, Jianan Wu, and Chi Zhang. Defren: Decoupled faster r-cnn for few-shot object detection. ICCV, pp. 8681-8690, 2021.
Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, et al. Learning transferable visual models from natural language supervision. In International conference on machine learning, pp- 8748-8763. PMLR, 2021.
Aditya Ramesh, Prafulla Dhariwal, Alex Nichol, Casey Chu, and Mark Chen. Hierarchical text- conditional image generation with clip latents. arXiv preprint arXiv:2204.06125, 1(2):3, 2022.
Laria Reynolds and Kyle McDonell. Prompt programming for large language models: Beyond the few-shot paradigm. In Extended abstracts of the 2021 CHI conference on human factors in computing systems, pp. 1-7, 2021.
Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Bjöm Ommer. High- resolution image synthesis with latent diffusion models. In CVPR, pp. 10684-10695, 2022.
Ruth Rosenholtz, Yuanzhen Li, and Lisa Nakano. Measuring visual clutter. Journal of vision, 7(2):
17-17, 2007.
Chitwan Saharia, William Chan, Saurabh Saxena, Lala Li, Jay Whang, Emily L Denton, Kamyar Ghasemipour, Raphael Gontijo Lopes, Burcu Karagol Ayan, Tim Salimans, et al. Photorealistic text-to-image diffusion models with deep language understanding. NeurIPS, 35:36479-36494 2022.
Bo Sun, Banghuai Li, Shengcai Cai, Ye Yuan, and Chi Zhang. Fsce: Few-shot object detection via contrastive proposal encoding. In CVPR, pp. 7352-7362. 2021.
Anh-Khoa Nguyen Vu, Thanh-Toan Do, Nhat-Duy Nguyen, Vinh-Tiep Nguyen, Thanh Duc Ngo, and Tam Nguyen. Instance-level few-shot learning with class hierarchy mining. TIP, 2023a.
Khoa Nguyen Vu, Thanh-Toae Moan-Toan Do, Vinh-Tiep Nguyen, Tam Le, Minh-Triet Tran, and Tam V Nguyen. Few-shot object detection via synthetic features with optimal transport. arXiv preprint arXiv:2308. 15005, 2023b.
Xin Wang, Thomas E. Huang, Trevor Darrell, Joseph E Gonzalez, and Fisher Yu. Frustratingly simple shw- object detection. In ICMI,, 2020.
Yanjie Wang, Xu Zou, Luxin Yan, Sheng Zhong, and Jiahuan Zhou. Snida: Unlocking few-shot object detection with non-linear semantic decoupling augmentation. In CVPR, pp. 12544-12553, 2024.
Jiaxi Wu, Songtao Liu, Di Huang, and Yunhong Wang. Multi-scale positive sample refinement for few-shot object detection. In ECCV, 2020.
Shuang Wu, Wenjie Pei, Dianwen Mei, Fanglin Chen, Jiandong Tian, and Guangming Lu. Multi- faceted distillation of base-novel commonality for few-shot object detection. In ECCV, pp. 578- 594. Springer, 2022.
Tianxu Wu, Shuo Ye, Shuhuang Chen, Qinmu Peng, and Xinge You. Detail reinforcement diffusion model: Augmentation fine- men-graieed visual categorization in few-shot conditions. IEEE Transac- fions Emerging Topics in Computational Intelligence, 2024.
12

---

Published as a conference paper at ICLR 2025
Yang Xiao and Renaud Marlet. Few-shot object detection and viewpoint estimation for objects in the wild. In ECCV, pp. 192-210. Springer, 2020.
Jingyi Xu, Hieu Le, and Dimitris Samaras. Generating features with increased crop-related diversity for few-shot object detection. In CVPR, pp. 19713-19722, 2023.
Xiaopeng Yan, Ziliang Chen, Anni Xu, Xiaoxi Wang, Xiaodan Liang, and Liang Lin. Meta r-cnm: Towards general solver for instance-level low-shot learning. In ICCV, 2019.
Sangdoo Yun, Dongyoon Han, Scong Joon Oh, Sanghyuk Chun, Junsuk Choe. and Youngjoon Yoo. Cutmix: Regularization strategy to train strong classifiers with localizable features. CVPR, pp. 6023 6032, 2019.
Gongjie Zhang, Zhipeng Luo, Kaiwen Cui, and Shijian Lu. Meta-detr: Image-level few-shot object detection with inter-class correlation exploitation. arXiv preprint arXiv:2103.11731, 2021.
Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control to text-to-image diffusion models. ICCV, pp. 3836-3847, 2023.
Shan Zhang, Lei Wang, Naila Murray, and Piotr Koniusz. Kernelized few-shot object detection with efficient integral aggregation. In CVPR, pp. 19207-19216, 2022.
Weilin Zhang and Yu-Xiong Wang. Hallucination improves few-shot object detection. In CVPR, pp. 13008-13017, 2021.
Chenchen Zhu, Fangyi Chen, Uzair Ahmed, Zhiqiang Shen, and Marios Savvides. Semantic relation reasoning for shot-stable few-shot object detection. In CVPR, 2021.
Lanyun Zhu, Tianrun Chen, Deyi Ji, Jieping Ye, and Jun Liu. Llafs: When large language models meet few-shot segmentation. In CVPR, pp. 3065-3075, 2024.
Junhao Zhuang, Yanhong Zeng, Wenran Liu, Chun Yuan, and Kai Chen. A task is worth one word: Learning with task prompts for high-quality versatile image inpainting. arXiv preprint arXiv:2312.03594, 2023.
Barret Zoph, Ekin Cubuk, Golnaz Ghiasi, Tsung- Yi Lin, Jonathon Shlens, and Quoe Le. Learn ing data augmentation strategies for object detection. In ECCV, pp. 566-583. Springer, 2020.
13

---

Published as a conference paper at ICLR 2025
We provide more detailed information about our work in this Appendix. The structure includes ICOS outputs (Appendix A), visualizations of our synthesis dataset (Appendix [B), detailed studies for the foreground-foregrouudd approach (Appendix [ ablations about the number of generated images (Appendix D) and comparisons with different fine-tuning schemes (Appendix
A THE RESPONSES IN ICOS
We provide ChatGPT responses ICOS for exploring class attributes and fine-grained classes in Novel Set of PASCAL VOC, shown in Figure Figure respectively.
In-Context learning for fine-grained objects
List the fine-gruined categories within object class, With each class, return attribute following this instruction:
"Avventure Bike", "Scooter", "Moped", "Café
Racer", "Chopper", "Endura Bike", "Babber", "Scranbler", "Track Bike", "Electric Motorcycle", "Mini Bike", "Supernoto", "Trials Bike"]
Input sample: cow
}}
"caw":
Fine-grained ("Holstein Friesian", "Jersey", "Guernsey", "Brown Swiss", "Ayrshire", "Hereford", "Angus", "charolais", "L "Linousin", "Simental", "Brahnan", "Gelbvieh", "Shorthorn", "Texas Longhorn", "Highland", "Red Poll", "Devon", "Galloway", "Belgian Blue", "Watusi"] h...3
Input: bus
attr: {
"bus": {
"Fine-grained classes" (""Shoool Bus", "Double-Decker Bus", "City Transit Bus", "Articulated Bus", "Minibus", "Coach Bus", "Shuttle Bus", "Tour Bus "Electric Bus", "Hybrid Bus", "Trolleybus", "Airport Bus", "party Bus", foo-floor Bus", "Intercity Bus", "Express Bus", "Sightseeing Bus", "Vintage Bus", "Luxury Bus", "Charter Bus*] }}
Input: bird
attr:
"bird": {
"Fine-grained classes" NNoorhhemm Cardinal", "Anerican Robin", "Blue Jay", "Mallard Duck", "Bald Eagle*, "Peregrine Falcon", "House Sparrow", "Barn swallow", "Ruby-throatee" Hummingbird", "Great Horned Owl", "European Starling', 'Red-tailes Hawk", "Anerican Goldfinch", "Eastern Bluebird', "Snowy Oml", "Rock Pigeon", "Wood Duck", "Yellow Narbler", "Northern Mockingbird", "Cedar Maxwing"]
Iaput: sofa
attr: {
"sofa":
"Fine-grained "Lasses" Sofa", "Chesterfield Sofa", "Sleeper Sofa", "Loveseat", "Reclining Sofa", "Kid-Century Modern Sofa", "Lawson "Bridgewater Sofa", "Camelback sofa", "Chaise Lounge",
}}
Input: mottorbike
"Tuxedo Sofa",
Sofa", "Divan Safa", "Settee", "English Roll Are sofa", "Modular sofa", "Cabriole Sofa", "Curved Sofa", "Pallet sofa", "Daybed Sofa"]
"notorbike":
"Fine-grained classes": ("Sportbikkk"" "Cruisen", "Tauring Bike", "Standard Bike", "pual-Sport Bike", "Dirt Bike", "Naked Bike",
}}
Figure 6: In-context learning results for exploring fine-grained classes Novel on the PAS- CAL VOC dataset.
14

---

Published as a conference paper at ICLR 2025
In- Context learning for attribute analysis
atre:
Dowrihe the features and altribuics of cach class after we enter the inpul clos. With each query, reepome,, and attributes of cach class above following this insirutiin::
'sotorbikk':
"colors"; Pelsek', "red", "blue", 'eite" 'sncen"),
"body": ("strwellnnd" "Sluek" "roundud",,
"erane": ("esposed", "covered with tateings"),
['spoked", "alloy"],
Input sample:
"tines": ["srooth". "knosey"],
appereance sitritutes entegories
"hanilabars"s ("varied stap= "different height"), "seat"; ["Kingle", "split", "flat", "contourel") ) "echoust": Maingge", "dail", "chreee", "snto") )
Ouput sample:
response: Cons typically have colors such black, white, browe, conbination of these, their for is ussully shart and
"Ligets" Pround", "rectanquiar", "sharp-edged")
")
coarse. Cous have large, rounded bostes with stocky build. Their eans are rectur-stzed and florpy, and their catls are long with tuft of halr at the end, Coass have a distinctive face with browd, stort snout and large, promiment eyes that are meal ur slaond-shaped, often with cals or docile expression.
Input: bus
response:
Buses colors like yellow, eften displayint branding, advert! sements. shepw with large windos along visibility. The front of bis have flat slightly angled wirdshield, and the beellights are caudly lerge and rectanguiar or circalar. buses have
vehicles typteally painted nulti-colored information,
scew cows have herrs, which can be shark, long, or absent, depanding on the bread. They oftes have distinct narkings, such as patches or spots, and prenounced udder in fanales. atre:
have besy slightly counded body
vides for
"cow": [
"colons": ["black", "white", "brown", "contiration"), ",
or sere axles with large whoels, often fitted with huocags or exposed rins. Doors are typically located on the sides and can be single or cousle, opening imford sliding. The noof can be flat or slightly curved, sonetiics with adsttional restures like air consitiining units, roor racks, or roottop advertisements. The rear of the bus usally features tail lights, sonettees large and rectangulaer, along with exhaust out lets. Insioe, the seating arrongement is ofton rows of seats with handroils, and the driver's arco is sectionnd off with control ponels and a steering whuul.
"fun": L'short", "csarse"],
"body": ["large", "roundsd", "stocky"),
"ears": "wodise-sized" "Elospy"!.
"txti": ["long with tuft heir"],
"Eacs": ["bncod", , "short sncut");
"eyes"; ["val", "oleond-shoped"" ), "expression": L'cate", "coctle"]
"hores": ["short", "long", "absest")
warking": [ ("patetes", "spats"), "uddee": ("peanounced freales"]
atre:
"bue":
Input: bird
"calars"; ["y=llme", "White", "hlue", "red", "grees",
'sulti-solorwd").
response:
Birds disolay
colors, including thades wnite, vibront
"body": ["buay". , "Llightly roundel"],
brown, gra,, yellow. their fwathers be sleak, fluffy, bluw, green
"windows": 1"large", "sid"",
"sindshield": [fflatt, "sligicly angled"),
that in size, shape, and longth, oftan adapted for thair flying style, such as broad sings for soaring short wings for repld flapping, their beats
pattorned with stripes
Birds have
"neadiighes": ["receanguiae" "circala"],
"wheels": [""aage",, "with huncaps', ', "exposed rim")) "),
"doors": ["stingee,, "double", "irward opening", "siiding",, "root": ["rlatt, "slightly curved', "Mith additional features"),
pointed, curved, or stout, depending on their feeding habits. Btres have slender or stout legs with scales, and thei- feet myy have clas, tolons, or webbing. Their eyes are usustly round and can range fros smill to lange, providing a been sense of viston. The tatl shape can be long, short, rourded, or farked, contrisuting to their belance and flight control.
"rear": ["tall Hights" "exhaust outlets"),
"intorio"": ["ross seats", "handroils", "driven's area control panels"]
atrr: !
lapat: soft
"bire": {
"colors": [fbrown", "gray", "Block", "Wiite". tred",
respanse: Sofas cose in wide range of styles, often uphol stered natectals surs fabric, lesther, velut, or synthetic blands. tomso colors inc neutral tones like twige, gray, black, white, and brown, as will as vibrant hum blue, green, res, pattornes designs. Sofes varying shapes, iscluding straight, Lishaped, or curved designs, arerests that can square, rounded, absent. plush,
"blus", "aneon". "yellow"],
"feathers"; ["sleek", "fluffy, "striped", "spotted"),
"uings": ["oroad"; "short", "iong"], "beat": ["panntes "surved", "stusit), "lags": ("llnder" "stout", "scalad"),
have
"= ("elaas" "talons", "watthe"") ).
"large", "rrund"", "toil": ("ong", "short", "roundod", "forkod"]
"syes":
usually ly nodols feataring renovable or attached cushtons. Backrests can los, tufted,
seating
slements
sefa the style, which exposed hidden, contributing traditional include acditional festures like recliners, chiise bullt
or netal stitching, lags
Input: motnobike
Notorbikes wons varicus colors, including black, red, blue, white, and green, often with castes decals patterns. Tay festure streanlined body with fuel tank, usaally
response:
stre:
"sofa":
exposed covered with foirings, deponding on the style, suth sports, allay, knobby off-raad. handlehars vary in contributing the riding fron
"satertals"s ["fabric', "leather", "velvet", "synthetic
blaade"],
sheels,
"colors": ["beige", "gray", "black", "Wite", "brown', "blar", "grwen", "rwd", "pestumed"],
"shap=": ["straight", "L-shapud", "curvee]], "ararests": ["sqiare", "rounded") "absent"), "cushioos": ["soft", "plish", "reaovable", "attacsed"], "barcnees";; ["high", "Iaw", "ruftes", "with buttass", "with stitching"), ),
split, designed confort penforeance. Hotorbikes dusl, orten natte. The lights Snctuse headlights, lights, and indiestors, round, rectangolar, sharp-edges, entancing the astarhike's is nverall
"legs": L"woodon", "actal", "exposod", "hsdden"),
"stylu": ["traditional"" "Iadern"
"additional features": (frucliners", "chaise lounge". "built-in storage"] 1
))
Figure 7: In-context learning results for exploring class attributes in Novel Set on the PASCAL VOC dataset.
B SYNTHESIS IMAGES VISUALIZATION
We provide visualization for synthesis images in Figure 8
They are created separately for each method. We visualize synthetic samples for five novel classes in Novel Set 1 on PASCAL VOC.
15

---

Published as a conference paper at ICLR 2025
Methods
CoW
moturbike
Syoth
image
image
Syoth
Hard background
Typical background
lunningked
Triamph
Typical foreground
bird
garbage truck
tutlestus
huder
foreground
Figure 8: visualization for synthesis images. Each row represent type of augmentation.
C DETAILED STUDIES FOR THE FOREGROUND-FOREGRRONN APPROACH
N.o. fine-grained classes
nAP
nAP50
DAP75
40.0
66.8
42.3
2
39.6
66.3
40.7
39.9
65.4
43.3
4
41.2
68.5
42.6
40.5
67.1
42.9
38.8
64.8
39.3
Table 6: Ablations about the number of fine-graing -rained classes in MPAD. nAP, nAP50, nAP75 metrics on Novel Set of PASCAL VOC are reported.
As shown in 6 and Figure we provide ablation studies in our foreground generation method. We use nAP, nAP50 and nAP75 for Table and the average of these three metrics for Figure 9
Firstly, evaluate the impact of the number of fine-grained classes in Table An upward trend is experienced when the number of fine-grained classes increases up to 4. These experiments reveal that models cannot capture datasets with very high diversity. The reason for this that models are optimized with default training hyper-parameter (e.g., training iterations, learning rate, batch size, etc.) for FSOD.
result, detectors are unable to converge on such datasets.
In Figure 9 we test the applied, overall performance improves. Specifically, when m increases from 0.7 to 0.9, the model performance improves by about 1 to 2%. However, when increases to 0.99 with low the base class features are retained over time steps, which will generate objects too similar to the base class and reduce the model performance. Similarly, when is too low, the generated objects may retain many of the base's 's features. More visualization of generated samples with different w and m shown in Figure 10
of the HPAS method. It shows that when momentum is
16

---

Published as a conference paper at ICLR 2025
48.10
48.40
48.90
49.00
0.6
46.90
48.40
47.80
48.30
0.7
49.10
51.00
49.80
50.00
0.8
46.80
51.60
48.50
48.80
0.9
49.10
49.60
49.80
49.90
0.99
47.60
48.10
48.70
49.40
0.6
0.7
0.8
0.9
W
Figure 9: Ablations about momentum and starting value in Eq. (3. The average value of nAP, nAP50, nAP75 metrics on Novel Set of PASCAL VOC are reported.
D ABLATIONS ABOUT THE NUMBER OF GENERATED IMAGES
We also provide an ablation study on the number of generated images in Table 7 These results show upward trend in performance the number of images increases, achieving the best performance at 300 images.
N.o. images
50
100
200
300
400
nAP75
42.0
41.0
43.7
42.8
43.8
nAP50
67.6
66.9
68.8
69.1
68.0
Table 7: Ablations about the number of generated images per class.
E COMPARISONS WITH DIFFERENT FINE-TUNING SCHEMES
To analyze the impact of different training schemes, conducted experiments comparing our aug- mentation framework (MPAD) with alternative approaches, such as using simple prompting with ChatGPT and the diffusion model for generating training samples and fine-tuning strategies, as shown in Table Specifically, we present training scheme (1), where base models are only trained on the synthetic dataset, and training scheme (2), where models pre-trained with generated data before being tunettned with the original few-shot data. All models are evaluated under the 1-shot setting of Novel Set of PASCAL VOC.
#images
100
200
300
400
600
800
1000
49.5
56.7
62.3
62.4
63.8
63.1
64.5
65.1
65.4
64.5
2
62.3
60.5
60.4
61.7
61.0
61.6
62.1
62.1
62.3
62.5
Table Performance comparison between directly training on generated data (I) and fine-tuning in real data (2) using different numbers of generated samples.
17

---

Published as a conference paper at ICLR 2025
The results indicate that performance saturates 600 samples, and additional samples do not im- prove results. Compared to the best performance obtained in this experiment, MPAD still outper- forms simple prompting by 3.7, even when using only 300 samples, as shown in Table 7
Our training scheme is also superior to alternative pre-training and fine-tuning approaches. Specif- ically, MPAD effectively leverages both real high-quality few-shot samples and diverse generated samples within a single fine-tuning phase, avoiding potential overfitting issues. When pre-training with large number of training data and fine-tuning with too few samples, the model risks "forget- ting" knowledge from the pre-training phase. Moreover, MPAD is computationally efficient, requir- ing a simpler training process compared alternative approaches that involve separate pre-training and fine-tuning steps.
These ablations demonstrate the effectiveness MPAD in leveraging generated data while main- taining robustness and efficiency in training.
18

---

Published as a conference paper at ICLR 2025
Momentum
0.1
0.5
0.9
Class
bird
0.1
motobike
cow
bird
0.7
motobike
COW
bird
0.9
motobike
cow
Figure 10: Visualization of generated samples with different momentum (m) and w.
19


Reviews:
[{"review_id": "4DDidDdM3q", "reviewer": "Reviewer_RKTD", "summary": "This paper presents a Multi-Perspective Data Augmentation (MPAD) framework to improve few-shot object detection (FSOD) by generating diverse and representative synthetic samples. The framework includes three components: Chain-of-Thought Prompting for Object Synthesis (CPOS) uses large language models to enhance prompt diversity with fine-grained attributes; Harmonic Prompt Aggregation Scheduler (HPAS) mixes base and novel class features in the diffusion process, producing hard-to-classify samples; and Background Proposal (BAP) selects complex or visually similar backgrounds to improve object-background differentiation. Extensive experiments on PASCAL VOC and COCO few-shot object detection benchmarks show the effectiveness of the proposed method.", "questions": "1: In the Introduction part (especially for Figure 1), what's the definition of typical and hard objects on the novel set? How to distinguish them?\n\n2: Which are the novel synthetic samples and the base real samples in Novel Set 1 in Figure 1? Why do some colors (classes) only have solid circle samples without any \"x\" samples? (I am deeply Confused by Figure 1.) \n\n3: In Table 2, is it a typo that sometimes the metric is \"nAP\" while sometimes not?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 2, "contribution": 1, "strengths": "1: The MPAD framework leverages Chain-of-Thought Prompting to generate prompts with fine-grained attributes, enabling diverse and representative data synthesis for few-shot object detection. \n\n2: The proposed method is easy to follow and not limited to some specific object detection architectures, making the technique be applied in different scenarios without further modifications. \n\n3: The main results on PASCAL VOC and COCO few-shot datasets illustrate that the proposed method can improve non-trivial performance on many few-shot object detection benchmarks.", "weaknesses": "1: The description that \"this work is the first ...\" in Lines 81-83 is a little bit over-claimed. From my perspective, there is already a large amount of work exploring using large-scale pretrained diffusion models for object detection by generation (like general object detection and corner case generation for autonomous driving). Simply extending the setting to few-shot object detection isn't that significant to me.\n\n2: Section 2.3 (CoT prompting for object synthesis) actually belongs to prompt engineering. The Chain-of-Thought emphasizes solving a problem step by step. For object synthesis discussed in this paper, the task simply needs to list all possible attributes of a category ( or use a pre-defined template). This can be regarded as in-context learning or prompt engineering rather than Chain-of-Thought. Meanwhile, the inpainting model is also modified from other previous works.\n\n3: The overall technique contribution is limited. Continuing from point 2, the method is more like a combination of several existing verified techniques for object detection data generation without a clear logical chain. The HPAS is a prompt embedding level mixup data augmentation which is also not novel (the insight of embedding mixup is common in conditional data generation using diffusion model). It's also hard to convince me that the BAP part brings much novelty.   \n\n4: Comparisons with deep learning methods (using large-scale pretrained models like diffusion model and CLIP, etc). In Table 3, the authors only compare their proposed method with some non-deep learning-based methods. However, given the widely used conditional diffusion model for object detection object generation, the authors also should add more baselines such as simply using the **Powerpoint** inpainting model for data generation, to further verify the effectiveness of the proposed module that is claimed to enhance the diversity.\n\nOverall, I tend to vote for rejection at this time because of the technical contribution and the proposed method itself, along with comprehensive comparisons. I am waiting and will hear from other reviewers.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "giBqjdyoiv", "reviewer": "Reviewer_iPh4", "summary": "This paper attempts to enhance the diversity of data generation by analyzing typical and hard samples. Specifically, a data generation architecture is proposed that comprehensively considers the typicality and difficulty of foreground and background selections, establishing the relationship between foreground and background. Experimental evidence demonstrates significant performance improvements compared to recent methods.", "questions": "1. What are the definitions of typical and hard samples? Detailed assessment criteria need to be further elaborated.\n2. In Section 2.2, lines 159-161 mention that HPAS can generate hard samples. Strategically, it may indeed involve a wider range of categories. However, for the model, the level of difficulty in recognition is not solely dependent on the number of categories in the image. Why are samples generated based on mixed prompt embedding considered hard samples in HPAS?\n3. In the last paragraph on the fifth page, it mentions the concept of camouflage targets. Camouflaged objects typically refer to objects where the foreground and background have high similarity. However, in the subsequent text, the selection of data with high similarity between the background in the base stage and the classes in the novel stage as challenging data does not explicitly demonstrate the connection between selecting difficult backgrounds and camouflaged objects. Therefore, what is the relationship between the selection of challenging backgrounds and camouflaged objects?\n4. In Equation 6, it is mentioned that typical clutter background is obtained by calculating entropy. Such selection heavily relies on the performance of the feature extractor F. If the predictive results are good, few background images may be selected. However, if the predictive results are poor, do the selected background images hold any useful value?\n5. The architecture of this paper incorporates various methods with pre-trained models, such as CLIP. Is the fairness of the comparison between the current augmentation strategy and previous methods lacking?", "limitations": "", "rating": 6, "confidence": 5, "soundness": 2, "presentation": 2, "contribution": 2, "strengths": "+ From the experimental data, the current data augmentation methods show a certain degree of performance improvement.\n+ Integrating various mainstream generative models and zero-shot learning models, including diffusion, CLIP, etc.", "weaknesses": "- The present approach heavily depends on utilizing pre-trained models to select typical and challenging samples, potentially causing interference when assessing the efficacy of augmentation strategies.\n- Additional clarification is needed regarding the fairness of the experiments.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "AJYUlUyfmA", "reviewer": "Reviewer_jFnB", "summary": "This paper proposes a Multi-Perspective Data Augmentation (MPAD) framework aimed at enhancing few-shot object detection (FSOD) by generating diverse and challenging synthetic samples. The MPAD framework utilizes techniques such as Chain-of-Thought Prompting for Object Synthesis (CPOS), Harmonic Prompt Aggregation Scheduler (HPAS), and Background Proposal (BAP) to create representative and hard samples. The authors present results demonstrating significant performance improvements on PASCAL VOC and MS COCO benchmarks, showing an average increase of 17.5% in nAP50 over baseline methods.", "questions": "Please refer to the Weaknesses box.", "limitations": "", "rating": 6, "confidence": 5, "soundness": 3, "presentation": 3, "contribution": 3, "strengths": "1.\tThe CPOS and HPAS introduce novel ways to leverage both typical and hard samples, leading to a more representative synthetic dataset.\n2.\tThe use of BAP to generate diverse backgrounds helps enhance detection accuracy by allowing the model to distinguish between foreground and background more effectively.\n3.\tThe proposed framework achieves notable gains over state-of-the-art baselines on multiple FSOD benchmarks, particularly in challenging low-shot settings.", "weaknesses": "1. The framework combines multiple advanced techniques, including diffusion models, harmonic prompt scheduling, and complex background sampling, which may make it challenging for practitioners to implement effectively in real-world scenarios. At the same time, it increases the complexity of the model. Please analyze the model complexity and real-time inference of the generated model.\n2. The paper demonstrates performance gains, but it would benefit from a more granular analysis comparing the effectiveness of each augmentation component (CPOS, HPAS) against similar elements in other FSOD methods. Has the pre-trained model of the generated model already seen various categories to generate more realistic new class samples when regenerated again? Can other generative models also generate samples to improve the performance of small samples, and does the proposed method have such a significant performance improvement compared to this type of method\nData Efficiency in Synthetic Generation:\n3. While the method performs well, additional experiments assessing data efficiency could be valuable. Evaluating how much synthetic data is optimal or exploring the impact of different amounts of augmented data on performance could provide insights into the scalability and efficiency of the framework.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}, {"review_id": "V1T2hRQjFo", "reviewer": "Reviewer_B8R6", "summary": "This paper proposes a Multi-Perspective Data Augmentation (MPAD) framework for few-shot object detection (FSOD), aiming to address the limitations of existing data augmentation methods. By considering both foreground-foreground and foreground-background relationships in MPAD, the authors provide an interesting solution to the problem of data augmentation. The experimental results on PASCAL VOC and MS COCO datasets demonstrate the effectiveness of the proposed methods, outperforming many state-of-the-art methods.", "questions": "See \"Weaknesses\"", "limitations": "", "rating": 6, "confidence": 4, "soundness": 3, "presentation": 3, "contribution": 2, "strengths": "1. The paper is generally well-written and the idea is presented clearly. \n\n2. The introduction provides a good background and motivation for the research\n\n3. The proposed methods are described in detail and mostly are easy to follow. \n\n4. The experimental setup and results are also presented in a clear and organized manner.\n\n5. The authors also conduct a series of ablation studies to better understand the proposed method.", "weaknesses": "1. My main concern lies in the use of Diffusion Model (DM) and ChatGPT. From the perspective of the FSOD task, data is always the most crucial element. Consequently, it is essential to employ ChatGPT and DM to generate data as a means to address this concern, as the authors did in this study. However, the following questions arise: if ChatGPT and DM are being used, why not simply generate a substantial number of samples of the corresponding categories and conduct direct training on the generated data? Alternatively, another option could be to pre-train the detector on the generated data and subsequently perform fine-tuning on few-shot categories. Currently, the authors haven't furnished a compelling rationale to either explain, oppose, or support these viewpoints, which are of utmost significance for this paper. \n\n    To answer these questions, I suggest the authors: \n    \n    - (1)  conduct a comprehensive analysis comparing the performance of direct training on generated data with the proposed method; \n\n    - (2) explore the potential benefits and drawbacks of pretraining on generated data and then fine-tuning on few-shot categories. This could include experiments to determine the optimal amount of pretraining and the impact on final performance; \n\n    - (3) provide a detailed explanation of why the proposed approach of using ChatGPT and DM in a specific way is more advantageous than (1) and (2). \n\n2. Some details could be further elaborated. For example, in the description of the Harmonic Prompt Aggregation Scheduler (HPAS), the role of the momentum parameter and its impact on the generated samples could be explained more clearly. For example, the authors can provide a figure to visualize generated samples across a range of momentum values (e.g. 0.1, 0.5, 0.9) to illustrate how this parameter impacts the mixing of base and novel class features.\n\n3. Additionally, the discussion on the limitations of the diffusion model in the appendix could be more in-depth, perhaps exploring potential solutions or future research directions. For example, (1) analyzing whether the hallucination will affect the final result of FSOD, and if so, what possible means can be used to solve it; or (2) is there any possible way to train (in a PEFT manner) an FSOD-aware DM to further improve the performance. In addition, I suggest putting the \"limitation\" part to the main text; With these changes, this paper becomes more inspiring to the reader.", "quality": null, "clarity": null, "significance": null, "originality": null, "strengths_and_weaknesses": "", "was_revised": false, "final_justification": "", "rebuttal": "", "claims_and_evidence": "", "methods_and_evaluation_criteria": "", "theoretical_claims": "", "experimental_designs_or_analyses": "", "relation_to_broader_scientific_literature": "", "essential_references_not_discussed": "", "other_strengths_and_weaknesses": "", "other_comments_or_suggestions": "", "questions_for_authors": "", "summary_of_paper": "", "summary_of_recommendation": "", "technical_quality": "", "clarity_of_presentation": "", "potential_impact": "", "robotics_focus": "", "quality_of_limitations": "", "questions_for_rebuttal": "", "reasons_to_accept": "", "reasons_to_reject": "", "questions_to_authors": "", "extra_scores": {}}]

Author rebuttal:

Decision: Accept (Poster)