Sample-Efficient Multi-Round Generative Data Augmentation for Long-Tail Instance Segmentation
Authors: Byunghyun Kim
Minyoung Bae
Jae-Gil Lee
Venue: NeurIPS 2025 poster ()
Keywords: instance segmentation
data synthesis
diffusion
copy-paste augmentation
Primary area: applications
TL;DR: A sample-efficient diffusion-based data augmentation method for instance segmentation, achieved by multi-round copy-paste augmentation with model feedback.

Abstract: Data synthesis has become increasingly crucial for long-tail instance segmentation tasks to mitigate class imbalance and high annotation costs. Previous methods have primarily prioritized the selection of data from a pre-generated image object pool, which frequently leads to the inefficient utilization of generated data. To address this inefficiency, we propose a **collaborative** approach that incorporates feedback from an instance segmentation model to guide the augmentation process. Specifically, the diffusion model uses feedback to generate objects that exhibit high uncertainty. The number and size of synthesized objects for each class are dynamically adjusted based on the model state to improve learning in underrepresented classes. This augmentation process is further strengthened by running **multiple rounds**, allowing feedback to be refined throughout training. In summary, **multi-round collaborative augmentation (MRCA)** enhances sample efficiency by providing optimal synthetic data at the right moment. Our framework requires **only 6\%** of the data generation needed by state-of-the-art methods while outperforming them.

Sample-Efficient Multi-Round Generative Data Augmentation for Long-Tail Instance Segmentation
Byunghyun Kim, Minyoung Bae, Jae-Gil Lee"
KAIST frooknpowm, mybae, jaegil) ac.
Abstract
Data synthesis has become increasingly crucial for long-tail instance segmentation tasks to mitigate class imbalance and high annotation costs. Previous methods have primarily prioritized the selection of data from a pre-generated image object pool, which frequently leads to the inefficient utilization of generated data. To address this inefficiency, we propose a collaborative approach that incorporates feedback from instance segmentation model to guide the augmentation process. Specifically, the diffusion model uses feedback to generate objects that exhibit high uncertainty. The number and size of synthesized objects for each class dynamically adjusted based on the model state to improve learning in underrepresented classes. This augmentation process further strengthened by running multiple rounds, allowing feedback to be refined throughout training. In summary. multi-round collaborative augmentation (MRCA) enhances sample efficiency by providing optimal synthetic data at the right moment. Our framework requires only 6% of the data generation needed by state-of-the-an methods while outperforming them.
Introduction
The advent of diffusion models has made generating high-quality images significantly easier with
lower cost 99] 23 30 31). These advancements naturally opened new possibilities for generating
training data in data-hungry vision tasks like object detection [16 and instance segmentation [4]
21 27
Diffusion models can synthesize images conditioned on textual descriptions, enabling the creation of diverse and detailed datasets with labels ( [2). Especially for long-tail instance segmentation, where class imbalance and high annotation costs pose significant challenges, diffusion-
based image synthesis has become crucial solution.
Copy-paste-based methods, originally intended for augmenting training data using real image ob- jects [8] [5), now have been extended to utilize high-quality synthetic image objects generated by diffusion models. That is, these recent methods first generate large pool of image objects and then augment the training data by pasting them to real images [1] 37 41). The advantage arises from generating objects in a training-free manner, comparably lower cost than layout- methods that train the diffusion model with the training data as shown in Figure Ia 5] 40). However, some studies have shown that continuously training with synthetic data can eventually degrade model performance [12 13 Such finding has naturally led to research on active and curriculum learning strategies that can incrementally select most informative synthetic objects from a pre- reeegeeeeated object pool as shown in Figure Tb 26 41.. Nevertheless, preemptively constructing the object pool before training may suboptimal, as can result in significant number of unused objects and may not include object characteristics that could address the model's deficiencies encountered during training.
* "Corresoonding author
'hhiee we designate instance segmentation as our target task, object detection is equally relevant. 39th Conference on Neural Information Processing Systems (NcurIPS 2025).

---

Generator
Stamentation Model
Generator gen
train
Seunettation
Instacce
Model
Syn Data
Real Data
Syn Objects
W
(a) Layout-hased generative augmentation (existing).
(b) Training-free free generative augmentation (existing).
Feedback: Rare class? Increase difficulty, number and scale!
Generator
gen
paste
train
Instance Segmentation
Model
round r-1
round r
round r+1
Syn Objects
Real
(c) MRCA: multi-round collaborative augmentation (ours).
Figure 1: Novelty of our generative augmentation strategy. Existing methods (a) train generator separately or (b) discard part of pre-generated data and incur extra cost. Ours (c) incorporates feedback from the training model into the generation process, optimizing cost and enhancing quality.
To solve this problem, we propose multi-round collaborative augmentation between a generator and an instance segmentation model to guide the generator to build an augmented training set which can achieve high accuracy with a smaller number synthetic objects- -ict, toward being
Collaborative Augmentation To effectively improve the power of the instance segmentation model using synthetic images, we transfer its feedback to the augmentation process. As illustrated in Figure [c] the feedback is exploited in two distinct places: (i) in the generation of synthetic objects and (ii) in the pasting of these objects to real images. For the generation step, the classifier gradients employed to produce more challenging objects for the instance segmentation model's 's classifier, akin to prior research on image classification II 19]. Also, the class-wise -wise validation accuracy and classifier weights used to ascertain adequately mandated quantity of synthetic objects for each class, thereby enhancing the classifier's s capacity to learn from underrepresented or difficult classes. For the pasting step, the class-wise validation accuracy is employed assign greater space to underperforming classes, as larger objects are more effective enhancing the performance of instance segmentation [28) In short, this two-fold feedback approach facilitates the object generator in supplying the most beneficial data to the instance segmentation model.
Multi-Round This novel collaborative augmentation is boosted through multiple rounds. It is well-known that the most useful training instances vary depending on the training stage, as witnessed by curriculum learning B. Thanks to the collaborative augmentation, the feedback from the instance segmentation model can be effectively transferred to the object generator. Then, it is more effective to transfer such feedback periodically throughout the training stage. As result, the generation and pasting of synthetic objects occur incrementally, resulting in a reduced overall quantity of synthetic objects. Regarding each incremental generation, better customized images can be supplied for the training of the instance segmentation model at the appropriate time.
In summary, our proposed Multi-Round Collaborative Augmentation (MRCA) achieves higher or comparable performance with only 6% object generation compared to state-of-the-ar methods [ the LVIS 1.0 [18] dataset,
37 41 in generative augmentation for long-tail instance segmentation
particularly outperforming rare classes. Extensive ablation experiments show the effectiveness of the two-fold feedback components of MRCA. The source code is publicly available at https: -itt:m/miiiiiiiaa//MRR.... dmlab/MRCA
//githb.ccom//eee
2 Related Work
Data Synthesis for Instance Segmentation Data synthesis for instance segmentation can be broadly categorized into two main approaches. The former involves cut-and-paste (or copy-paste based) methods, where objects from a source image are extracted with precise boundaries and pasted onto destination images to create new synthesized images. This approach originates from cut-and- paste augmentation [8] and copy-paste augmentation [5]. which were initially designed for pasting real objects to enhance data diversity.

---

Building upon these foundational methods for the former approach, X-Paste B7 extends the basic cut-and-pastt method by incorporating objects from multiple sources, including both real and diffusion-bann basodd synthetic objects, to further diversify training examples. More recently, BSGAL [4]] builds upon X-Paste by integrating instance-wise data selection strategies based on gradient and loss information, ensuring that pasted objects contribute effectively to model training. Additionally, DiverGen [I]] focuses on diversity-centered instance generation, optimizing synthetic data to enhance representation learning by promoting diversity among generated objects.
The latter approach involves layout- based methods, where bounding boxes or segmentation masks are given a priori for generating images for object detection. GeoDiffusion [5] and ODGEN [40] generate bounding box coordinates derived from the dataset and condition the diffusion model to produce objects accordingly. InstaGen [4] instead synthesizes images and creates bounding box annotation by training an instance-level grounding head. However, these methods cannot be extended to instance segmentation tasks because of the high complexity of segmentation masking. Even for layout-based methods that provide segmentation masks as condition [25]] the diverse geometric knowledge of the diffusion model cannot be fully utilized owing to the fixed segmentation mask.
Diffusion with Guidance The diffusion model generates the data following data distribution p(x) by learning the denoising process. Recent diffusion models are based on classifier-uuidance [6] or classifier-free guidance [22] that learns the data distribution conditioned on class labels p( y). That is. V log pe,y (x) y) = Vz log Pe (x) + VV,
x log ply(z), where pe (zz is the generative model's data distribution of an image æ parametrized by 0, p(y|z) is the classifier's probability of a class y given and y is guidance weight controlling classifier influence.
ControlNet 36 proposed more structured approach by injecting additional spatial conditions into the diffusion process without retraining the base model, allowing fine-grained control over outputs. More recently, Universal Guidance [2] emerged as general framework that can direct diffusion models using arbitrary signals through plug-and-play mechanisms, broadening the flexibility and applicability of guided generation. Altogether, these methods have significantly expanded the controllability and reliability of diffusion- generative models.
3 Methodology: MRCA
3.1 Problem Formulation
An instance segmentation model is trained using training set Dreat {(1), H where I; is an image, y; is set of annotations for the image Tú, and T is the total number of images. Here, y, [ır.) [ii, (bbs.j, ma.j ((.)) [bx]] where bbi,j, ma,j and Ctj denote the bounding box, segmentation mask, and class of the j-th object.
We aim to augment the original training set Dreal through data synthesis to increase data diversity, rectify class imbalance, and enhance model generalization. To achieve this goal, a set of image objects Oyyn = for oo (I,, 1,23 | is created using generative model & with given label y: from the set of classes as well as a dichotomous segmentation model S, where B represents the given budget of data synthesis. Two functions" are internally used for this augmentation process A(Dreal) = Doug, as shown in Figure
Gen B): It generates B image objects to compose Osym, where the number of objects per class is automatically determined by the learning progress of the instance segmentation model.
Map-Paste(o): For each object E 0yyn, it first selects the real image and annotations, (z, y) E Dreal, that o will be pasted. Consequently, the mapping between objects and images, P, is constructed. For each pair of an object and an image (o. (z. y)) E P, then pastes into x and accordingly updates the corresponding set of annotations That is, it returns an augmentation (x',y) from yz) E Dreal and E Fym
As result, an augmented training set Dang = y( ET is used instead of Dreal Note that the number of images is fixed to N whereas the number of annotations in these images is increased by B, following the common convention 37 41). It is also possible to expand the set of images that used for augmentation, and we leave it as a topic of future work.
Auxiliary arguments (e.g.. generative model) of the functions are omitted for simplicity.
3

---

Round
Round +
Round r
Gen(B)
Dreat
Doug
Generator
g
"Bulldozer"
Model Guidance
Train
"Seahorse"
Round
Instance
Segmentation
Model
"Coconut"
Map
Paste
Segmentation Model
M
y
(x.y)
M
Map-Paste(O)
Feedback to the Next Round
Class-wise Metrics
Figure 2: Inside the multi-round collaborative augmentation with Gen(-) and Map-Paste(( The model guidance and class-wise metrics are delivered from the model of the previous round to the copy-paste pipeline the current round to optimize the Gen(-) and Map-Paste(-)) processes.
Let's consider two instance segmentation models trained on Dreal and Dung respectively. The performance difference between these models the test set Diest indicates the quality(() of the data augmentation. Therefore, the primary challenge is to find an augmentation process Ag that maximizes the quality of data augmentation within a specified budget B with respect to Dreat, Specifically, let Mreal and Ma denote the instance segmentation models trained on Dreat and Doug, respectively, and L denote the loss function. Then, the objective is formalized as finding Ag such that
arg max ((Ane()))))) = arg max (oe((--EE-x-DB-- Du L(Moug (x))) ( (B L( (x),
An
An
(I)
3.2 Multi-Round Collaborative Augmentation
To address the challenge, propose the frame- work of multi-round collaborative augmenta- tion (MRCA), which exploits the feedback from an instance segmentation model through mul- riple rounds. Figure shows its overall proce- dure. MRCA consists of warm-up stage and N rounds, where each round comprises one sev- eral epochs of the instance segmentation model training. We initially the warm-up stage by training the instance segmentation model with real data and generating objects with no feed- back. After the warm-up stage or round is completed, the feedback from the instance seg- mentation model is applied to both Gen(-) and Map- Paste(-) of the next round. Therefore, the instance segmentation model and the object generator collaboratively augment the training data. See Appendix A.1 for the pseudocode of this augmentation pipeline.
Warm-up
Round
Round N
Final Result
Testance Segmentation
Instance Model
Instance
Segmentation
Trained
Model
Mode!
Model
M
M
Mos
Real Data:
Syn Data
Syn Data
Syn
Training
Object Generator
Object Generator
Object Generator
Feedback
Gen(-)
Map-Paster
Figure Multi-round collaborative augmentation.
For every round, the training of the instance segmentation model and the generation of image objects are done simultaneously on multiple GPUs. However, the training is dependent on the generation since the generated objects are required for the augmentation of the training set. To overcome the delay caused by the dependency, the generated objects are passed to the next round for training. In summary, at the round r, we generate objects with the feedback from the previous round 1 and provide the generated objects to the next round 1.
Budget Constraint We set the generation budget B equal the maximum number of generations possible in the training time of each round. By doing so, both the generation and training steps can overlap different GPUs and thus are processed without delays.
Collaborative Augmentation Given the budget constraint, our main idea is to leverage feedback from the instance segmentation model M during the augmentation process. The intuition is that the

---

model M holds valuable information that can help generate the most useful objects in Gen(-) and guide optimal placement during Map-Paste(-), thereby using the budget more effectively. Employing model feedback for generative data augmentation has been shown effective in classification tasks by directly aligning augmentation with the model's 's objective as previous studies [144] We formalize this widely-aceee accepted idea as Assumption 3.1 with specific conditions.
Assumption 3.1 (Collaborative Augmentation Improves Quality). Let A denote data augmentation applied to a dataset D. and let fa denote the feedback from a model Mo trained on the dataset D. Then, under the conditions of (C1) sufficient model capacity, (C2) controlled distribution shift, and (C3) proper feedback direction, the augmentation guided by So has higher quality than the unguided augmentation. That is,
(C1) A (C2) (C2)A((3) - Q(A(D: Q(A(D))
(2)
Conditions for the Assumption We first define the information metric ID ID(Ma) measure of how well the model Mo captures the distribution of the dataset DD. It can be measured by the average accuracy, i.e., (Ma) = To E(EYEED +ी Eyey 1 3i E Ma(x) s.t. IoU(v. ú) 2 0.5).
(C1) Sufficient model capacity: Model capacity can be represented by the empirical risk Ramp, which defined as the average loss L over the dataset D. Formally, Remp [2] E(soy)e (MM) To ensure that the model retains available learning capacity, the
empirical risk should be decreasing over time, i.e., d 0.
de
(C2) Controlled distribution shift: The knowledge gained through augmentation should surpass the adverse effect of distribution shift caused by the augmentation. Formally, given the data distribution
dPxz(poll dt
(t) PD and the model's learned distribution PM2 over time t,
0.
(C3) Proper feedback direction: Given the information from the model Ms. on the dataset D, the feedback fe should properly guide the augmentation process toward quality maximization. For example, the feedback should be applied in the direction that increases diversity and uncertainty of the dataset (proper direction), not decreasing them (improper direction). That is, (Mp) 2 [D) A (C3) + (D; fo)) 2 ((A())).
Under the sufficient model capacity and controlled distribution shift conditions, the average accuracy of the model Ma on the dataset D increases through training according to the definitions. That is, (C1) A (C2) + ID (M₂) 2 (V)) Combining it with the proper feedback direction condition (C3), we obtain the overall guarantee that (AA(D; fo)) 2 Q(A(D)) holds under these conditions.
Similarly, the model is trained by the data with higher quality of augmentation, the quality of the feedback from the model will increase accordingly. We incrementally formalize this concept as Lemma 3.2]
Lemma 3.2 (Collaborative Augmentation with Further Trained Model Improves Quality). Let A
be a data augmentation applied to a dataset D, and let for be the model feedback obtained from a model trained for rounds. Then, under Assumption 3.1 1.Jlool with the three conditions (C1, C2, and C3), the augmentation guided by further trained model Jo, provides higher quality than the one guided
by JØ. That
Q(A(D; 2 ((A(D: for 2r>0 O.
Proof. By Assumption 3.1 under sufficient model capacity (C1) and controlled distribution shift
(C2), Ip (Mo,) 2 (D( Ma Also, more information guides augmentation of higher quality under proper feedback direction (C3), so (C1) A (C2) A (C3) - ((A(D; fo-)) 2 Q(A(D: fo..)).
Multi-Round Augmentation We further claim that applying feedback in multiple rounds is supe- rior to one-time feedback under certain conditions. The high-level idea is that iterative augmentation incorporates feedback from recently trained model instance, ensuring that the augmented data aligns with the model's evolving needs. We formalize this statement Theorem 3.3
Theorem 3.3 Mulli--ound Collaborative Augmentation Improves Quality). Let A a data aug- mentation applied real dataset Do = Dreal, and let fo. denote the model feedback obtained
5

---

from a model MR. trained on the dataset available at the round r. Consider a multi-round augmenta- tion, where augmentation performed over N rounds. The augmented dataset at the round with cumulative budget NV
B is defined as
1
for forr=1,2,... 1.2,...,. .N, Then, multi- multi-round collaborative augmentation has higher quality compared to a single-round collab- orative augmentation. That is,
Dr A (Du: fo.
10
B.
(4)
N
Q(DN,B) Q(PLB), for N 1.
(5)
Proof. The detailed proof is provided in Appendix
To ensure that the theoretical claims hold in practice, we specifically design MRCA to keep the assumption better. First, we run the training process until the model validation accuracy saturates to verify that the current round has more information than the previous round, [D(,,,)) Ip(Mo..1) While this statement is usually true since the model learns from newly generated objects, learning on only portion of the dataset sometimes leads to biases and information loss. Therefore, train the model for multiple epochs between rounds to incorporate the entire dataset. Second, the feedback fo. itself should be properly applied to the augmentation process toward the direction that maximizes the quality of augmentation. Since the mechanism of how feedback improves augmentation quality solely depends on its design, explain the details of how each feedback guides the augmentation process toward quality maximization in the next section.
3.3 Feedback on Object Generation
Class- Budget Optimization Class-wise budget optimization dynamically adjusts the number of synthetic objects for each class based on the model's state. Specifically, the number of objects to create for class E (1, .maa at the round B, is computed
B
1
r 2 Br(((
T> Br a 2r-2c) 2C
(6)
Cmaz
Sp_2.c
where Cr,e is the classifier weight, Ar,e is the validation accuracy, and Sr.c the number of objects in the augmented training set. The scale factor scales the budget for each round so that the sum of (c) is equal to B. Eq. (6) targets generation on classes with low accuracy, low coverage, and scarce data. It is straightforward how budget is weighed on classes with low validation accuracy and low number of training instances for sample efficiency. The classifier weight Cr,e is defined as the weight vector corresponding to a class in the final classification weight matrix at the round r. The classifier weight difference term |Cr-2cc Cr-3.ell reflects how much the class has learned between the rounds, indicating room for improvement. Therefore, we provide more budget to classes with larger weight differences maximize the learning efficiency.
The two-round gap between the left-hand term and the right-hand term in Eq. (6) results from two factors. One round gap occurs because the generation uses metrics from the model trained on the previous round. Another round gap is required to earn time for generating objects. This intentional gap allows efficient pipelining of generation with feedback, resulting in minimal halts.
Feedback-Guided Object Generation To imbue each object with richer class information, we incorporate feedback from the instance segmentation model Ma into the diffusion process. Specifi- cally, the diffusion sampling is steered by the gradients of Ma with respect to our chosen criterion C. By using entropy the criterion, we synthesize objects that become progressively more difficult for the model to learn over successive rounds. Mathematically. the diffusion process is conditioned as
Va log.e. ( = Vz (o)) + V logr(ylæ) + v.CC,, Ma),
(7)
where y is the scale of classifier guidance and w is the scale of feedback guidance. Compared to previous work [I] on a classifier model, computing feedback guidance from an instance segmentation model requires an additional technique. A general instance segmentation model produces multiple objects' bounding boxes and segmentation masks from a given image which cannot be directly used for guidance. Fortunately, we aim to generate a single object per image in our method. Therefore, we set the whole image as the box proposal and compute the remaining layers of the instance segmentation model. The detailed algorithm is provided in Appendix A.2

---

3.4 Feedback on Map-Paste of Objects
The first stage is Map, where create set of mappings P between o E Osyn and (xr,) Dreal The second stage is Paste, where each mapped object in (o, ()) E P is scaled by 8, which is adaptively determined by class accuracy, and is pasted onto the mapped image create the final augmented image (a', y'). We aim to maximize the learning of the instance segmentation task through optimizing each stage.
Recall the two stages of the Map-Paste process defined in Section 3.1
Unique Mapping Previous studies 37 41 adopt purely random assignment of synthetic objects to real images, P = {(o, (x.y)) V(r.y) E 'Dreal, o ~ Uniform( (O*r)) While random mapping serves a strong baseline, it becomes suboptimal under tight object generation budget. Furthermore, random mapping hinders our class-wing budget optimization since some generated objects may never be selected.
To address these limitations, we introduce quota-balanced unique mapping. In this scheme, once an object o is drawn from the synthetic object pool OF it is removed so that each synthetic object contributes exactly once toward its class quota. When O is exhausted, it is refilled with the initially generated object pool. This scheme ensures that, across the entire augmentation process, each class is represented according to its budget ratio prescribed in Section 3.3
Accuracy-Based Object Resizing When pasting an object, existing approaches 37 draw object size from the distribution of each class in the real dataset, i.e., N (He, o²), where Me and of are the mean and variance of objects of a class in Dreal While such approach helps match the overall scale between synthetic and real objects, it fails to boost learning for rare or low-accuracy classes, since their typically smaller object sizes remain hard to distinguish and learn from.
To remedy this imbalance, we introduce a resizing scheme based on the validation accuracy of each class. Specifically, objects from low-accuracy classes are scaled larger to enhance the visibility of their features, thereby facilitating better feature extraction. In contrast, objects from high-accuracy classes are generated at smaller scales reduce their influence during training and prevent the model from overfitting to already well-represented classes. This simple yet effective strategy balances the contribution of each class by providing richer visual cues for underrepresented classes, enhancing the model's ability to recognize rare objects. The detailed algorithm is provided in Appendix A.3
Experiments
4.1 Experiment Settings
Datasets and Implementation Details Instance segmentation and object detection experiments mainly conducted the LVIS v1.0 dataset (18 in the relevant literature, with supporting experiments on the Pascal VOC dataset [0] and the Open Images V5 dataset [24]. CenterNet2 [39], with ResNet-50 (20 backbone, is used as our main instance segmentation model, implemented
in Detectron2 [33] The L (27 backbone is also used for comparison with baseline methods.
Stable Diffusion Medium 9 is used as our generation model, and BiRefNet [88 is used for
segmentation on the generated objects. After generation or segmentation, CLIP
29 is used to filter
out the low low-quality objects whose score is smaller than 0.25.
Our model is trained for 10 rounds, with each round consisting of 9,000 iterations and a batch size of 16. Our multi-round collaborative augmentation serves the sole modification the baseline training pipeline. We evaluate performance using average precision (AP) for both bounding box detection and instance segmentation while also analyzing results across different class frequencies (rave, common, and frequens classes) as defined in LVIS. We use 8 NVIDIA GeForce RTX 3090 GPUs, where 4 GPUs are used for training the model, GPUs for generating objects, and I GPU for segmentation. Only for the Swin-L experiment, we use NVIDIA A40 GPUs instead to fit the model in the GPU memory. To run both training and generation without waiting for another process, the number of object generations per round is empirically set to 6 x 1203 (number of classes in LVIS). Besides, y and w in Eq. (7) are set 5.0 and 0.03, respectively. Further details about the experiments are provided in Appendix
Compared Methods We compare our MRCA with strong generative copy-paste methods, in- cluding X-Paste (37], BSGAL [41, and DiverGen [1]. The numerical results of the baselines are borrowed from the BSGAL and DiverGen papers owing to the same experiment settings.
7

---

4.2 Main Experiment Results
Table and Table 2 show the mean average precision (mAP) the instance segmentation and object detection tasks on the LVIS v1.0 dataset for the ResNet-50 20] and Swin-L [27] backbones. The superscripts "box" and "mask" denote the mAP for object detection and instance segmentation, respectively. The subscripts "r", "c", and "f" signify rare, common, and frequent classes, respectively. Overall, MRCA outperforms the state-of-the-are generative augmentation methods while generating only 6% as many synthetic objects. These results indeed demonstrate the sample efficiency of MRCA, because its multi-round collaborative augmentation delivers the most essential training data the appropriate time. Furthermore, the performance improvement for rare classes more pronounced than for other classes, because the rare classes receive superior treatment due to increased generation budgets ($3.3 and expanded space ($3.4). See Appendix D for augmented image examples.
Table 1: Comparison with the ofate-o-tte ofte--o-the-are art methods using the ResNet-50 backbone.
Method
Gen Objects
A
pomesk
APp
A
A.Phox
A APmosk
AP
APp
No Aug.
31.50
28.20
22.60
20.20
29.30
26.70
37.80
33.40
X-Paste
1200k
34.20
30.39
24.33
22.21
33.23
29.57
39.63
34.89
X-Paste CLIP
1200k
34.35
30.70
25.99
24.38
32.83
29.41
39.71
34.92
BSGAL
1200k
35.40
31.56
25.43
34.14
30.56
40.07
35.37
MRCA
72k
35.56
31.81
28.14
25.93
34.33
30,86
40.18
35.44
Table 2: Comparison with the state-of of the-art art methods using the Swin-L backbone.
Method
# Gen Objects
A
A poass
A
A Apmask
A.hoox
A Apmesk
A A
A
No Aug.
47.43
42,30
41.00
36.75
47.53
43.10
50.14
43.83
X-Paste
1200k
49.57
43.85
44.87
39.66
49.74
44.64
51.46
44.82
X-Paste CLIP
1200k
49.80
44.51
45.28
40.62
49.33
44.96
$2.30
45.72
BSGAL
1200k
50.47
44.85
47.55
42.37
50.43
45.47
$1.79
45.26
DiverGen
1200k
51.24
45.48
50.07
45.85
51.33
45.83
51.64
44.96
MRCA
72k
51.80
45.91
51.58
46.84
51.86
46.31
51.84
45.05
4.3
Ablation Study and Further Analysis
Effect of the Feedback on Generation and Map-Paste The ablation study is conducted on the feedback components: class-wise wise budget optimization, model gradient-based generation, and object resizing. We apply each component individually and collectively in a multi-round manner to confirm its efficacy. Table 3 shows that each individual component contributes to perfor- mance gain without interfering with other components.
Table 3: Ablation the components of MRCA.
Class Rudget Misiel Crudien Reiixng A p
A Aprws
A
APpus
34,86
31.35
27.47
24.91
33.04
31 31.14
27.13
24.25
34.91
31.13
27.52
24.89
33.15
31.42
26.54
24.35
38.56
31.81
28.14
25.93
Effect of the Class-Wiss Budget Optimization Components We also conduct fine-grat ane-ggained ablation study on class-wise budget optimization while keeping other main methods (model gra- dient and resizing) enabled, as shown in Table
Table 4: Ablation on the components of class-wise budget optimization.
Number Obsects (8)
Validation
Classiter
Accuracy
Apbax
A
Apbsc
APmaa
(A)
When using only the number of objects, we observe notable improvements for rare classes.
2805
5 The validation accuracy term further supports common and frequent classes with low accuracy. Nonetheless, some low-accuracy classes continue to produce similar objects across rounds. To mitigate this issue, we incorporate the classifier weight, which reflects whether gradient feedback being updated over rounds. As evidenced by the improvements in the result, both validation accuracy and classifier weight contribute effectively as intended.
35.51
27.93
35.56
28.14
25.93
Effect of the Generation Model In Table
Table 5: Effect of the generation model.
we compare the performance when using difter- ent generative models, Stable Diffusion 1.5 30 and Stable Diffusion 3 Medium (9. The hyper- parameters of the diffusion models are set equal, with classifier guidance scale as 5.0, generation step as 30, and image resolution as 512x512. The recent version of Stable Diffusion generates higher-quality objects, leading to better results. When using the same version, Stable Diffusion 1.5, the superiority of MRCA over BSGAL is still maintained in terms of the mAP for all and rare classes.
Method
Apbox
Apmssk
A
A
BSGAL (SD 1.5)
34.82
31.21
26.76
24.84
MRCA (SD 1.5)
35.12
31.38
27.42
25.21
MRCA (SD
35.56
31.81
28.14
25.93
8

---

# of Rounds Effect of the Number of Rounds We investigate the effectiveness of applying feedback in multi- round manner in T The number of training rounds adjusted from 30, given fixed number of total training iterations. As shown in Theorem 3.3 when the number of rounds increases from to 10, the quality of instance segmentation and object detection improves, indicating that the quality of the augmentation improves accordingly. Over 10 rounds, however, the performance drops because the model does not learn sufficiently between the short training rounds; that is, a single round is unable to cover whole epoch over in such conditions.
Table Effect of the number of rounds with a fixed number of total training iterations.
# Gen Objects
A.b
A.m.maa
APhox
APr
APho
APe.akk
APP
APpuaak
1
72k
34.79
31.17
27.64
24.63
33.11
30.09
39.81
35.25
2
72k
34,96
31.09
27.33
24.35
33.28
29.90
40.19
35.39
5
72k
35,09
31.38
26.16
24.04
33.76
30.44
40.39
35.54
10
72k
35.56
31.81
28.14
25.93
34.33
30,86
40.18
35.44
20
72k
34.96
31.46
27.81
24.85
33.48
30.55
39.76
35.38
30
72k
34.81
31.13
26.86
24.49
33.45
30.14
39.82
35.15
Effect of the Curriculum Learning The pro- posed MRCA generates objects and trains with the curriculum divided by rounds. To check the effect of the curriculum learning strategy, we separately train a model from scratch given the objects generated over all rounds of MRCA. Table shows the difference depending on the existence of the curriculum learning. Although training without the curriculum has more object quantity at the beginning, the performance becomes lower most cases. This result suggests that while the objects generated by MRCA can be used for the input to any training pipeline, the optimal strategy is to use the curriculum learning strategy provided by our method.
Table 7: Effect of the curriculum learning.
Method
Apbbx
A Apanna
Apblax
A
MRCA (w.o. curriculum)
35.23
31.27
28.28
25.46
MRCA (w. curriculum)
35.56
31.11
28.14
25,93
Analysis on the Generation Budget In pre- vious experiments, we empirically chose gen-
Table 8: Analysis on the generation budget.
Gen Objects
Training Time Aphis
Aplis
A
eration budget B of 72k to efficiently pipeline our generation and training processes, ensuring no bubbles. This additional test examines the effect of larger budgets on the mAP as well as the training time excluding the warm-up stage. Table shows that the training time is proportional to the number of object generations, because multi-round training needs to wait for the object generation to be used for augmentation. Only small performance improvement is observed with an increased generation budget, decreasing the sample efficiency. This
361.
30h
34.82
31.21
26.76
24.84
72k
34h
35.5 56
31.81
28.14
25.93
144
35.58
25,99
limitation of low scalability is further discussed in Appendix B
Analysis on the Feedback Guidance Scale The feedback guidance scale, w in Eq. 77. has a significant impact on the image quality. Table
Table 9: Analysis on the feedback guidance scale.
Apbox
Apmask
APbox
A.mass
[ supports that feedback guidance can largely
0.01
34.87
30.95
26.90
23.83
affect the generated image quality. The default
0.03
35,56
31.81
28.14
25.93
value w = 0.03 usually offers the best perfor-
0.05
34.78
31.15
27.34
25.21
mance. The performance drops significantly for
0.10
34.25
30.42
23.98
22.93
a large w because excessively large model guidance overwhelms the trained diffusion denoising gradients. Its effective range can be quickly identified with qualitative analysis, as a high w produces unnatural artifacts, while a low w results images nearly identical to unguided outputs.
Analysis on the Classifier Guidance Scale I00le[00preeen the effect of the classifier guid- ance scale, in Eq. (7). A higher amplifies the influence of the conditioning signal (e.g., text prompt or class label) on the generation process, which strengthens fidelity and prompt alignment but pushes samples toward a narrower region of the data distribution, thus reducing diversity among outputs. Conversely, when 7y is too low, the diffusion model behaves closer to its unconditional form. Therefore, the generated samples exhibit higher variability but lower fidelity and semantic accuracy, leading to visibly degraded quality. The
Table 10: Analysis the classifier guidance scale.
7
Aphox
Apmask
Apbox
APmask
3.0
35.21
31.35
26.59
24.33
5.0
35.56
31.81
28.14
25.93
7.0
35.32
31.58
26.89
24.68
default value of 5.0, which has been adopted in prior studies (37 41, is also found to be effective in our experiments by stabilizing the tradeoffs.
9

---

Experiments on Additional Datasets We conduct only object detection on Pascal VOC 2012 [0] because the annotations only sup- port semantic segmentation. For Open Images V5 (24), due to the large size of the dataset, we use pareto sampling method from previ- ous work [32] to create its long-tailed version.
Table 11: Object detection VOC 2012 and in-
stance segmentation on Open Images V5 long-tail
Pascal VOC 2012
Open Images V5 (long tail) Apbox
Method
AP30
AP75
AP
Ap
No Aug.
68.45
49.35
44.89
45.97 47.10
36.53 37.30
MRCA
72.16
55.08
50.85
Again, MRCA boosts the performance through our novel data augmentation pipeline in these datasets.
4.4 Qualitative Analysis
Round-Wiss Object Visualization and Entropy Figure visualizes a portion of generated objects and their average entropy for each round. As shown in Figure 4a the generated objects maintain their class characteristics over all rounds. We assess the entropy of objects with respect to two instances of the model: (1) the warmed-up model instance, which is trained solely on real training data, and (2) the feedback model instance, which is the model that guides the generation of the synthetic images in the corresponding round. Given that the guidance scale is constant, the entropy (uncertainty) respect the feedback model Mg, remains similar across rounds, shown in Figure 4b) However, from the perspective the initial model Mo., the feedback received from more advanced models in subsequent rounds results in the generation of increasingly uncertain challenging objects, as depicted Figure 4c This tendency is also evident when examining the average entropy across all classes in Figure 4d Overall, MRCA forms effective easy-to-hard curriculum.
Dove
Dragonfly
Dustpan
Round
Round
Round
Round
Round 4
Round Round Round 7
Round
Round
(a) Examples of generated objects for each round.
Dove
Dovi
Entray via Warm up
Dragerrly
Ductson
Board
hound
(b) Mean entropy of objects respect the feedback model (Ms.).
(c) Mean entropy of objects respect the warm-up model (Mao).
(d) Class average entropy respect to the warm-up and feedback models.
Figure 4: Examples of generated objects for each round (a), with their class-wise entropy respect to the feedback model (b), the warm-up model (c), and their comparison over average of all classes (d).
Conclusion
The proposed multi-round collaborative augmentation (MRCA) framework enables generating objects that are continuously more informative to the instance segmentation model with a considerably small amount of data generations (e.g., 6%). The instance segmentation model consistently guides the diffusion generation process to create objects that elicit greater uncertainty from the model, hence generating more informative samples, while employing class-wise budget optimization determine the appropriate number of generations for each class. Then, accuracy-based scaling effectively places the generated objects in real images to focus learning of rare classes. Experiment results demonstrate that MRCA outperforms the state-of-the-art generative copy-paste augmentation methods for instance segmentation. In conclusion, we assert that the MRCA framework presents highly promising avenue for enhancing the performance of instance segmentation.
10

---

Acknowledgements
This work was supported by Institute of Information & Communications Technology Planning Evaluation (IITP) grant funded by the Korea government (MSIT) (No. RS-2020-1120866. DB4DL: High-Usability and Performance In-Memory Distributed DBMS for Deep Learning, 50% and No. RS- 2025-25410841 Beyond the Turing Test: Human-Level Game-Playing Agents with Generalization and Adaptation, 50%).
References
[1] Reyhane Askari- Hemmit, Mohammad Pezeshki, Florian Bordes, Michal Drozdzal, and Adriana Romero- Soriano. Feedback guudded data synthesis for imbalanced classification. arXiv preprint 2000::3310.00/58 2023.
[2] Arpit Bansal, Mong-Min Chu, Avi Schwarzschild, Soumyadip Sengupta, Micah Goldblum, Jonas Geiping, and Tom Goldstein. Universal guidance for diffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 843 433-852,22 2023.
Proceedings of the International Conference Machine Learning (ICML), pages 41 488 2009. [3] Yoshua Bengio, Jérôme Louradour, Ronan Collobert, and Jason Weston. Curriculum learning. In
[4] Zhaowei Cai and Nuno Vasconcelos. Cascade r-cnn: High quality object detection and instance segmenta- tion. IEEE Transactions on Pattern Analysis and Machine Intelligence, 43(5):1483-1499 2019.
[5] Kai Chen, Enze Xie, Zhe Chen, Yibo Wang, Lanqing Hong, Zhenguo Li, and Dit Yan Yeung. Geodiffusion: Text-prompted geometric control for object detection data generation. arXiv preprint ar 2466:2300..46677 2023.
[6] Prafulla Dhariwal and Alexander Nichol. Diffusion models beat gans on image synthesis. Proceedings of the Conference Neural Information Processing Systems (NeurIPS), 34:8780-8794, 2021.
[7] Lisa Dunlap, Alyssa Umino, Han Zhang. Jiezhi Yang. Joseph E Gonzalez, and Trevor Darrell, Diversify your vision datasets with automatic diflusiin-bassd augmentation. Proceedings the Conference on Neural Information Processing Systems (NeurIPS), 01779024-79344 2023.
[8] Debidatta Dwibedi, Ishan Misra, and Martial Hebert. Cut, paste and learn: Surprisingly easy synthesis instance detection. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1301-1310, 2017.
[9] Patrick Esser, Sumith Kulal, Andreas Blattmann, Rahim Entezari, Jonas Müller, Harry Saini, Yam Levi, Dominik Lorenz, Axel Sauer, Frederic Boesel, et al. Scaling rectified flow transformers for high-resolution image synthesis. In Proceedings of the International Conference on Machine Learning (ICML), 2024.
[10] Mark Everingham, Lue Van Gool, Christopher KI Williams, John Winn, and Andrew Zisserman. The pascal visual object classes (voc) challenge. International Journal of Computer Vision, 88:303-338. 2010.
[11] Chengxiang Fan, Muzhi Zhu, Hao Chen, Yang Liu, Weijia Wu, Huaqi Zhang, and Chunhua Shen. Divergen: Improving instance segmentation by learning wider data distribution with more diverse generative data. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 3986-3995, 2024.
[12] Lijie Fan, Kaifeng Chen, Dilip Krishnan, Dina Katabi, Phillip Isola, and Yonglong Tian, Scaling laws of synthetic images for model training... for now. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 7382-732 7392, 2024.
[13] Haoyang Fang, Boran Han, Shuai Zhang, Su Zhou, Cuixiong Hu, and Wen-Ming Ye. Data augmentation for object detection via controllable diffusion models. In Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision, pages 1257-1266, 2024.
[14] Chengjian Feng, Yujie Zhong, Zequn Jie, Weidi Xie, and Lin Ma. Instagen: Enhancing object detection by triniigg on synthetic dataset. In Proceedings of the IEEE/CVF Conference on Computer Vision and Puttern Recognition (CVPR), pages 14121-14130, 2024.
[15] Golnaz Ghiasi, Yin Cui, Aravind Srinivas, Rui Qian, Tsung-Yl Yi Lin, Ekin Cubuk. Quoc V Le, Barret Zoph. Simple copy-paste a strong data augmentation method for instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2918 2928, 2021.
11

---

[16] Ross Girshick. Fast r-can. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 1440-1448, 2015.
[17] Ross Girshick, Jeff Donahue. Trevor Darrell, and Jitendra Malik. Rich feature hierarchies for accurate object detection and semantic segmentation. In Proceedings of the IEEF/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 580-587, 2014.
[18] Agrim Gupta, Piotr Dollar, and Ross Girshick. Lvis: A dataset for large vocabulary instance segmentation. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 5356-5364, 2019.
[19] David S Hayden, Mao Ye, Timur Garipov, Gregory P Meyer, Carl Vondrick, Zhao Chen, Yuning Chai, Eric Wolff, and Siddhartha S Srinivasa. Generative data mining with longtail-guided diffusion. arXiv preprint wrXiv:2502.0196 2025.
[20] Kaiming He, Xiangyu Zhang, Shaoqing Ren, and Jian Sun. Deep residual learning for image recognition. In Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 770-778, 2016.
[21] Kaiming He. Georgia Gkioxari, Piotr Dollár, and Ross Girshick. Mask r-cmn. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 2961-2969, 2017.
[22] Jonathan Ho and Tim Salimans. Classitier- freee diffusion guidance. arXiv preprint arXiv:2207.12540 2022.
[23] Jonathan Ho, Ajay Jain, and Pieter Abbeel. Denoising diffusion probabilistic models. Proceedings of the Conference on Neurul Information Processing Systems (NeurIPS), 33:6840-6851, 2020.
[24] Ilya Krylov, Sergei Nosov, and Vladislav Sovrasov. Open images v5 text annotation and yet another mask text spotter. In Proceedings of the Asian Conference on Machine Learning, pages 379-389, 2021.
[25] Yuheng Li, Haotian Liu, Qingyang Wu, Fangzhou Mu, Jianwei Yang, Jianfeng Gao, Chunyuan Li, and Yong Jae Lee. Gligen: Open-set grounded texl-to-image generation. In Proceedings the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR), pages 2251 12211-,, 2023.
[26] Yijun Liang, Shweta Bhardwaj, and Tianyi Zhou. Diffusion curriculum: Synthetic-to-real generative curriculum learning via image-guided diffusion. arXiv preprint arXiv:2410. 13674, 2024.
[27] Liu, Yutong Lin, Yue Cao, Han Hu, Yixuan Wei, Zheng Zhang, Stephen Lin, and Baining Guo. Swin transformer: Hierarchical vision transformer using shifted windows. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages
2021.
[28] Jiangmiao Pang, Kai Chen, Jianping Shi, Huajun Feng, Wanli Ouyang, and Dahua Lin. Libra r-cnm: Towards balanced learning for object detection. In Proceedings of the IEEE/CVF Conference on Computer Vision and Patiern Recognition, pages 821-830 2019.
[29] Alec Radford, Jong Wook Kim, Chris Hallacy, Aditya Ramesh, Gabriel Goh, Sandhini Agarwal, Girish Sastry, Amanda Askell, Pamela Mishkin, Jack Clark, al. Learning transferable visual models from natural language supervision. Proceedings of the International Conference on Machine Learning (ICML), pages 8763, 2021.
[30] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, and Björn Ommer. High-resolution image synthesis with latent difffusion models. In Proceedings of the IEEE/CVF Conference Computer Vision and Pattern Recognition (CVPR), pages 10684-10695, 2022.
[31] Jiaming Song, Chenlin Meng, and Stefano Ermon. Denoising diffusion implicit models. arXiv preprint arXiv:2010.02502, 2020.
[32] Tong Wu, Qingqiu Huang, Ziwei Liu, Yu Wang, and Dahua Lin. Distribution-bllaceed loss for multi-label classification long-tailed tailed datasets. Proceedings of the European Conference on Computer Vision (ECCV), pages 162 108,1 2020.
[33] Yuxin Wu, Alexander Kirillov, Francisco Massa, Wan- Men-Yen Lo, and Ross Girshick. Detectron2. https fethhb..om/nncboookeeesttttttttd 2019.
[34] Teresa Yeo, Andrei Atanov, Harold Benoit, Aleksandr Alekseev Ruchira Ray, Pooya Esmaeil Akhoondi, and Amir Zamir. Controlled training data generation with diffusion models. arXiv preprint arXiv:2403.15309, 2024.
12

---

[35] Sangdoo Yun. Dongyoon Han, Seong Joon Oh. Sanghyuk Chun, Junsuk Choe, and Youngjoon Yoo. Cutmix: Regularization strategy train strong classifiers with localizable features. In Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 6023-6032, 2019.
[36] Lvmin Zhang, Anyi Rao, and Maneesh Agrawala. Adding conditional control text-to-image diffusion models. Proceedings of the IEEE International Conference on Computer Vision (ICCV), pages 3836- 3847, 2023.
[37] Hanqing Zhao, Dianmo Sheng. Jianmin Bao, Dongdong Chen, Dong Chen, Fang Wen, Lu Yuan, Ce Liu, Wenbo Zhou, Qi Chu, et al. X-paste Revisiting scalable copy popy-paste for instance segmentation using clip and stablediffusion. In Proceedings of the International Conference on Machine Learning (ICML), pages 42098 42109. PMLR, 2023.
[38] Peng Zheng, Dehong Gao, Deng-Ping Fan, Li Liu, Jorma Laaksonen, Wanli Ouyang, and Nicu Sebe. Bilateral reference for high-resolution dichotomous image segmentation, arXiv preprint arXiv:2401.03407, 2024.
[39] Xingyi Zhou, Vladlen Koltun, and Philipp Krähenbühh. Probabilistic two-stage detection. arXiv preprint 103.4:2103.07461, 2021.
[40] Jingyuan Zhu, Shiyu Li, Yuxuan Andy Liu, Jian Yuan, Ping Huang, Jiulong Shan, and Huimin Ma. Odgen: Domain-speciin object detection data generation with diffusion models. Proceedings of the Conference on Neural Information Processing Systems (NeurIPS), 37:63599-63633, 2025.
[41] Muzhi Zhu, Chengxiang Fan, Hao Chen, Yang Liu, Weian Mao, Xiaogang Xu, and Chunhua Shen. Generative active learning for long-tailed instance segmentation. In Proceedings of the International Conference on Machine Learning (ICML). PMLR, 2024.
13

---

A Algorithm Pseudocode
A.1 Multi-Round Collaborative Augmentation Pipeline
The overall procedure of MRCA is explained in Algorithm D composed of two phases: the warm-up phase and the multi-round phase. At the warm-up phase, the instance segmentation model Ma is trained with real training data in parallel with the generation of objects by G. When both processes terminate, the multi-round phase starts. Similar to the warm-up phase, the multi-round phase also undergoes parallel training and generation. The difference lies in applying the feedback from the previous round's model to both object generation and training data augmentation. For the final round, the object generation does not proceed since there are more training rounds left to use them. Eventually, MRCA returns the trained instance segmentation model and the set of generated objects.
Algorithm Multi-round collaborative augmentation pipeline
1: Given instance segmentation model Ma. parametrized by e at a round that creates feedback
fer: object generator G, real training data D, Dreal, total round N
2: Do in parallel
Warm-up phase
3: Train Moner on Do - MMo
4: Generate objects by G with no feedback + 01
5:
end
6: for to N do
D. Multi-round phase
7: Do in parallel
8:
Augment Do with U and feedback fo.., D,, Train Mo.., D, Mo. if # then
9:
10:
Generate objects by 9 with feedback fo.-, from M Opy end
11:
12:
end
13:
end for
Return trained instance segmentation model Møx generated objects U
A.2 Diffusion with Feedback from the Instance Segmentation Model
The feedback-guider diffusion- object generation is explained by two hierarchical algorithms. At a higher level, Algorithm [] where the feedback is applied in the diffusion process. general diffusion process is composed of multiple denoising steps, where the scheduled denoising is applied sequentially. We apply the feedback guidance selectively after the scheduled denoising to keep the uncertainty in range that the generated images remain in the context of the input class. In our experiments, the feedback guidance is applied every steps out 30 steps, for a total of 6 times image. The hyperparameter in Line 6 is set to 0.03.
Algorithm 2 Denoising the diffusion model with a criterion function
1: Given stable diffusion model latent state zr, denoising U-Net ep with scheduler (L)), conditioning Tp. stable diffusion vae decoder Udees instance segmentation model Mş. scale of feedback guidance w, and criterion function C for each step & from I' to 1 do
2:
3:
é eø (2t, T8)
Predict noise
4:
z|-1 = 2t (t)e
Denoise according to the scheduler
5:
if t step for object detector feedback then
6:
2 2/_1 w 2/_ 1. Ms, Udec) & Apply instance segmentation model's d feedback
7:
else
8:
26-1 2/_11
9:
end
10:
end for
Return denoised latent state 20
14

---

Aexplibmmeexplains in more detail how the instance segmentation model and the criterion function compute the gradients for the diffusion model. The key idea is setting the whole image as the object proposal for extracting the features, since we generate a single object per image. Lines 2-5 explain the criterion function computation line by line.
Algorithm Applying the criterion function to the instance segmentation model
1: Given intermediate stable diffusion model latent state z 1. stable diffusion vae decoder Vdecs instance segmentation model Ms consisting of backbone (feature extractor) by and classifier Co. and criterion function
2: _t-1 (4
Create an image from latent variable b. Extract features from detector's backbone With whole image proposal, classify corresponding features Calculate gradients respect to criterion function
3: feal = by (T+-1)
4: logils co (Jeal(prop))
5: grads = Cilogils, 2
6: Return gradients on latent state grads
A.3 Object Resizing Based Class-Wise Evaluation
Algorithm provides a more detailed explanation of the resizing method. Given the average precision A. of specific class, assign scale factor to class, which will cause its objects be resized in accordance with the scale factor. For three values of A., 0%, A (mean), and 100%, For other
Ag 0% scale Smaz: Ag Å scale 1, Ag 100%
scale 8mar
values of A,, we linearly interpolate in log scale.
Algorithm 4 Computing scale factor based average precision (AP)
1: function COMPUTE_ SSPPLLE FACTOR(Ap. A)
2: Input: A. (Average precision of class c), A (Mean average precision of all classes), Smoz (Maximum scale)
3: Output: Computed scale factor
Log scale values for interpolation Scale A. 0%
4: log_ se__scle_miil <- In(s
5: log scals_maaass In
Scale at A. 100%
>
Piccewise linear interpolation in log scale
6:
if A then
7:
log_scale + Log_scale- min
A-Ac
8: else
9:
log_scale log scale max
A--A
100-A
10: end if
return scale exp(log
12:
end function
B Proof of Theorem
We begin by expressing the quality of the final dataset in the multi-round setting
Q(Dx.B) = Q (A ( (Do; Sa. ,.
(8)
Consider the 1B objects created at a round r. Since the feedback from the recent model for., is more informative than for.z. ID(Ma...) 2 (M by Lemma 3.2
Q (AA fo f 2 Q (A (Do; for 20
(9)
Similarly, continuing for $ B objects at rounds r - 1,r-2 -2 2....
Q (AAzo(DoiSe. 1. -. 2 (AAge(Duile -
(10)
15

---

o @(Asne(Doilan Sa,, ,f") 2 Q (Ax (Doi (Doi
(11)
By multiplying all inequalities from Eq. 9 to Eq.
@ (MA Sor 1. ,;")) Q (Dxn
(12)
Q
2 (DDiB))
(13)
C More Details for Experiments
Object Edge Smoothing When pasting objects, we do not employ any methods for smoothing object edges, such as Gaussian blurring or alpha/Poisson blending. Based on previous work [33] and own evaluation, the edge handling methods have significant effect the overall performance.
Reason for Using the ResNet Backbone Since the overall performance trend remains consistent across different backbone models (ResNet and Swin-L), we choose lighter model to enable more extensive ablation studies within given time. Notably, the performance improvement reported in Table [ and Table 2 is even larger when using the Swin-I. backbone, supporting that our ResNet- based ablation results do not overstate the effectiveness of MRCA. The ResNet backbone is more memory efffcceefficieet and can be trained on 24GB RTX 3090, whereas the Swin-L backbone requires higher-memory setup, such as 48GB A6000 or larger. We believe that both models are important, as they represent a trade-off between model size and performance, which is an important consideration for practical deployment, particularly on-device scenarios.
Overlap of Augmented Objects The augmented objects can be overlapped in our experiment setting. When an object is overlapped, its object mask is adjusted eliminate the obscured portion. This scheme is a common practice in the relevant studies 7 37 41).
Unrealistic Augmented Objects Our approach follows the line of research established by X- Paste 377. BSGAL [4]], and DiverGen (II,, which leverage synthetically generated, often unrealistic, map-pasted images improve performance. While the augmented images may not share the same distribution as the original images in terms of object size, they still contribute positively to instance segmentation performance by incorporating knowledge from the generation model. The MRCA framework further enhances this effect by explicitly optimizing both object generation and pasting strategies. It is worth noting that the use of visually unrealistic images for data augmentation has been widely observed to be effective in improving downstream performance. For instance, also in image classification, CutMix (35 often produces implausible examples such as images featuring a cat's head on a dog's body.
D Visualization of Augmented Images
We visualize some of the augmented images in Figure During training, we paste 1-3 generated objects onto an image from the real LVIS training set. We keep the number of pasted objects small to mínimize the deviation in the average number of objects per image compared to real images. The sizes of the objects are managed with our accuracy-bar -based resizing scheme (334).. For example, in the first image, the class "cockroach" has low accuracy, and therefore an object of that class is pasted with larger size compared to its actual size from real images.
E Limitations
The current implementation MRCA shows low scalability, meaning that generating more objects with the current framework does not lead to a large increase in performance. This downside is due to our method's focus on uncertainty, which effective in generating small-scale datasets since there is low chance of overlap between objects. However, generating large-scale datasets without a Civersity-focused method may generate similar objects, resulting in low scalability. Increasing the diversity of the generated objects effectively addresses the scalability issue, reducing overlap between objects allows for the acquisition of more information. DiverGen [[I. prior study that focuses the diversity of the generated dataset, supports earlier argument with scalable results.
16

---

bilc
697
033
2013
Rodhy
Figure Examples of augmented images.
Furthermore, the current framework only works with single generator to efficiently feedback. Previous studies, such as BSGAL [1] and DiverGen [1], demonstrate that using multiple generation models for generating objects can enhance data quality by increasing diversity. Therefore, employing multiple generation models in our multi-round augmentation framework remains a future task. Such an approach can increase diversity in another dimension, well solve the scalability issue.
We also note that scaling of rare classes can lead undesirable effects domains where object size carries critical contextual meaning. For instance, in medical imaging other domains with consistent scales, resizing objects may distort important spatial cues and disrupt the natural size distribution. To mitigate such risks, resizing strategies should ideally be avoided applied only within narrow bounds when using MRCA these settings. However, in real-world image datasets such as LVIS [8], PASCAL [10), and OpenImages (24) where object scales vary greatly even within the same category, we did not observe notable negative impacts on detection performance. By carefully constraining the maximum scaling coefficient, we ensured that objects did not become unnaturally large or small while maintaining overall realism. Moreover, as the representation of rare classes improves through successive training rounds, their resulting scale distribution tends to align more closely with natural proportions, further reducing the potential adverse effects of scaling.
17

---

NeurIPS Paper Checklist
1. Claims
Question: Do the main claims made in the abstract and introduction accurately reflect the paper's contributions and scope?
Answer: [Yes]
Justification: The main claims are clearly provided in the abstract and introduction of the paper with its scope.
Guidelines:
The answer NA means that the abstract and introduction do not include the claims made the paper.
The abstract and/or introduction should clearly state the claims made, including the contributions made in the paper and important assumptions and limitations. A No or
NA answer to this question will not be perceived well by the reviewers.
The claims made should match theoretical and experimental results, and reflect how much the results can be expected to generalize to other settings.
is fine to include aspirational goals as motivation as long as is clear that these goals not attained by the paper.
2. Limitations
Question: Does the paper discuss the limitations the work performed by the authors?
Answer: [Yes]
Justification:
The limitations are provided at separate appendix section.
Guidelines:
The answer NA means that the paper has no limitation while the answer No means that the paper has limitations, but those are not discussed in the paper.
The authors are encouraged to create a separate "Limitations" section their paper.
The paper should point out any strong assumptions and how robust the results are to violations of these assumptions (e.g., independence assumptions, noiseless settings, model well-specification, asymptotic approximations only holding locally). The authors should reflect how these assumptions might be violated in practice and what the implications would be.
The authors should reflect on the scope of the claims made, e.g., if the approach was only tested on few datasets or with few runs. In general, empirical results often depend on implicit assumptions, which should be articulated.
The authors should reflect on the factors that influence the performance of the approach. For example, facial recognition algorithm may perform poorly when image resolution is low images are taken in low lighting. Or speech-to-text system might not be used reliably to provide closed captions for online lectures because it fails to handle technical jargon.
The authors should discuss the computational efficiency of the proposed algorithms and how they scale with dataset size.
If applicable, the authors should discuss possible limitations of their approach to address problems of privacy and fairness.
While the authors might fear that complete honesty about limitations might used by reviewers as grounds for rejection, worse outcome might be that reviewers discover limitations that aren't 't acknowledged judgment and recognize that individual actions in favor of transparency play impor- tant role in developing norms that preserve the integrity of the community. Reviewers will be specifically instructed to not penalize honesty concerning limitations.
the paper. The authors should use their best
3. Theory assumptions and proofs
Question: For each theoretical result, does the paper provide the full set of assumptions and complete (and correct) proof?
Answer: [Yes]
18

---

Justification:
The assumptions and theorems are completely provided.
Guidelines:
The answer NA means that the paper does not include theoretical results.
All the theorems, formulas, and proofs in the paper should be numbered and cross- referenced.
All assumptions should be clearly stated referenced in the statement of any theorems.
The proofs can either appear in the main paper or the supplemental material, but they appear in the supplemental material, the authors are encouraged to provide short proof sketch provide intuition.
Inversely, any informal proof provided in the core of the paper should be complemented by formal proofs provided in appendix or supplemental material.
Theorems and Lemmas that the proof relies upon should be properly referenced.
4. Experimental result reproducibility
Question: Does the paper fully disclose all the information needed to reproduce the main ex- perimental results of the paper to the extent that affects the main claims and/or conclusions of the paper (regardless whether the code and data are provided or not)?
Answer: [Yes]
Justification: All the details are provided in the experiment section and the attached appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
If the paper includes experiments, a No answer to this question will not be perceived well by the reviewers: Making the paper reproducible is important, regardless of whether the code and data are provided or not.
If the contribution is dataset and/or model, the authors should describe the steps taken to make their results reproducible or verifiable.
Depending on the contribution, reproducibility can be accomplished in various ways. For example, if the contribution is a novel architecture, describing the architecture fully might suffice, or if the contribution is a specific model and empirical evaluation, it may be necessary to either make possible for others to replicate the model with the same dataset, or provide access to the model. In general. releasing code and data is often one good way accomplish this, but reproducibility can also be provided via detailed instructions for how to replicate the results, access to a hosted model (e.g.. in the case of large language model), releasing of a model checkpoint, or other means that are appropriate to the research performed.
While NeurIPS does not require releasing code, the conference does require all submis- sions to provide some reasonable avenue for reproducibility, which may depend on the nature of the contribution. For example
(a) If the contribution is primarily a new algorithm, the paper should make clear how to reproduce that algorithm. (b) the contribution is primarily a new model architecture, the paper should describe the architccture clearly and fully.
(c) If the contribution is new model (e.g., a large language model), then there should either way to access this model for reproducing the results or way to reproduce the model (e.g., with an open-source dataset instructions for how to construct the dataset).
(d) We recognize that reproducibility may be tricky in some cases, in which case authors are welcome to describe the particular way they provide for reproducibility. the case of closed-source models, may be that access to the model is limited in some way (e.g., to registered users), but it should be possible for other researchers have some path to reproducing or verifying the results.
5. Open access to data and code
Question: Does the paper provide open access to the data and code, with sufficient instruc- tions to faithfully reproduce the main experimental results, as described in supplemental material?
19

---

Answer: [Yes]
Justification:
The code for data generation and training and testing are provided.
Guidelines:
The answer NA means that paper does not include experiments requiring code.
Please see the NeurIPS code and data submission guidelines (htips://nips.cc/ubblic guddesSCodeSubmissionn
for more details.
While we encourage the release of code and data, we understand that this might not be possible, so "No" an acceptable answer. Papers cannot be rejected simply for not including code, unless this is central to the contribution (e.g., for new open-source benchmark).
The instructions should contain the exact command and environment needed to run reproduce the results. See the NeurlPS code and data submission guidelines (htips) for more details.
The authors should provide instructions on data access and preparation, including how access the raw data, preprocessed data, intermediate data, and generated data, The authors should provide scripts to reproduce all experimental results for the new proposed method and baselines. If only subset of experiments are reproducible, they should state which ones are omitted from the script and why.
At submission time, to preserve anonymity, the authors should release anonymized versions (if applicable).
Providing as much information as possible in supplemental material (appended the paper) is recommended, but including URLs to data and code is permitted.
6. Experimental setting/details
Question: Does the paper specify all the training and test details (e.g., data splits, hyper- parameters, how they were chosen, type of optimizer, etc.) necessary to understand the results?
Answer: [Yes]
Justification: The training and test details are provided in the experiment section and the appendix.
Guidelines:
The answer NA means that the paper does not include experiments.
The experimental setting should be presented in the core of the paper to level of detail that is necessary to appreciate the results and make sense of them.
The full details can be provided either with the code, in appendix, or as supplemental material.
7. Experiment statistical significance
Does the paper report error bars suitably and correctly defined or other appropriate information about the statistical significance of the experiments?
Question:
Answer: [No]
Justification: This paper does not provide error bars, but we conduct extensive experiments over many ablation settings.
Guidelines:
The answer NA means that the paper does not include experiments.
The authors should answer "Yes" if the results are accompanied by emror bars, confi- dence intervals, or statistical significance tests, at least for the experiments that support the main claims of the paper.
The factors of variability that the error bars capturing should be clearly stated (for example, train/test split, initialization, random drawing of some parameter, or overall with given experimental conditions).
The method for calculating the error bars should be explained (closed form formula, call to a library function, bootstrap, etc.)
The assumptions made should be given (e.g., Normally distributed errors).
20

---

It should be clear whether the error bar is the standard deviation or the standard error of the mean.
is OK to report 1-sigma error bars, but one should state it. The authors should preferably report a 2-sigma error bar than state that they have 96% CI, if the hypothesis of Normality of errors is not verified.
For asymmetric distributions, the authors should be careful not to show in tables or figures symmetric error bars that would yield results that are out range (e.g. negative enror rates).
If error bars reported in tables plots, The authors should explain in the text how they were calculated and reference the corresponding figures or tables in the text.
8. Experiments compute resources
For each experiment, does the paper provide sufficient information on the com- puter resources (type of compute workers, memory, time of execution) needed reproduce the experiments?
Question:
Answer: [Yes]
Justification: The compute resources and their training time are provided in the experiment section.
Guidelines:
The answer NA means that the paper does not include experiments.
The paper should indicate the type of compute workers CPU or GPU, internal cluster, or cloud provider, including relevant memory and storage.
The paper should provide the amount of compute required for each of the individual experimental runs as well as estimate the total compute.
The paper should disclose whether the full research project required more compute than the experiments reported in the paper (c.g., preliminary or failed experiments that didn't 't make into the paper).
9. Code of ethics
Question: Does the research conducted in the paper conform, in every respect, with the NeurlPS Code of Ethics https://eeuris..ccomool//ttttttu
Answer: [Yes]
Justification:
This paper is conducted under the guidance Code Of Ethics.
Guidelines:
The answer NA means that the authors have not reviewed the NeurIPS Code of Ethics.
If the authors answer No, they should explain the special circumstances that require deviation from the Code of Ethics.
The authors should make sure to preserve anonymity (e.g., there a special consid- eration due to laws or regulations in their jurisdiction).
10. Broader impacts
Question: Does the paper discuss both potential positive societal impacts and negative societal impacts of the work performed?
Answer:
[NA]
Justification:
The societal impact of this paper is intractable.
Guidelines:
The answer NA means that there no societal impact of the work performed.
If the authors answer NA or No, they should explain why their work has no societal impact why the paper does not address societal impact.
Examples of negative societal impacts include potential malicious or unintended uses (e.g., disinformation, generating fake profiles, surveillance), fairness considerations (e.g., deployment of technologies that could make decisions that unfairly impact specific groups), privacy considerations, and security considerations.
21

---

The conference expects that many papers will be foundational research and not tied particular applications, let alone deployments. However, if there is direct path to any negative applications, the authors should point it out. For example, it is legitimate to point out that an improvement in the quality of generative models could be used to generate deepfakes for disinformation. On the other hand, it is not needed to point out that a generic algorithm for optimizing neural networks could enable people to train models that generate Deepfakes faster.
The authors should consider possible harms that could arise when the technology is being used as intended and functioning correctly, harms that could arise when the technology is being used as intended but gives incorrect results, and harms following from (intentional or unintentional) misuse of the technology.
If there negative societal impacts, the authors could also discuss possible mitigation strategies (e.g., gated release of models, providing defenses in addition to attacks, mechanisms for monitoring misuse, mechanisms monitor how system learns from feedback over time, improving the efficiency and accessibility of ML).
11. Safeguards
Does the paper describe safeguards that have been put in place for responsible release of data models that have high risk for misuse (e.g., pretrained language models, image generators, or scraped datasets)?
Question:
Answer: [NA]
Justification: This paper do not release data models that pose such risks.
Guidelines:
The answer NA means that the paper poses such risks.
Released models that have a high risk for misuse or dual-use should be released with necessary safeguards to allow for controlled use of the model, for example by requiring that users adhere to usage guidelines or restrictions to access the model or implementing safety filters.
Datasets that have been scraped from the Internet could pose safety risks. The authors should describe how they avoided releasing unsafe images.
We recognize that providing effective safeguards is challenging, and many papers do not require this, but we encourage authors to take this into account and make best faith effort.
12. Licenses for existing assets
Are the creators or original owners of assets (e.g., code, data, models), used in the paper, properly credited and are the license and terms of use explicitly mentioned and properly respected?
Question:
Answer: [Yes]
Justification: This paper uses LVIS v1.0, Pascal VOC 2012, OpenImages datasets, Stable Diffusion 1.5 and 3 generation models, Detectron2 and X-paste code, and properly cite them.
Guidelines:
The answer NA means that the paper does not use existing assets.
The authors should cite the original paper that produced the code package dataset.
The authors should state which version the asset used and, possible, include URI.
The name of the license (e.g., CC-BY 4.0) should be included for each asset.
For scraped data from a particular source (c.g.. website), the copyright and terms of service of that source should be provided.
assetts are released, the license, copyright information, and terms of use package should be provided. For popular datasets, paperswithcode, cttprooadd.commaaaatttttssss has curated licenses for some datasets. Their licensing guide can help determine the license of dataset.
22

---

For existing datasets that are re-packaged, both the original license and the license of the derived asset (if has changed) should be provided.
If this information is not available online, the authors are encouraged to reach out to the asset's s creators.
13. New assets
Question: Are new assets introduced in the paper well documented and is the documentation provided alongside the assets?
Answer: [NA]
Justification:
This paper do not introduce any new assets.
Guidelines:
The answer NA means that the paper does not release new assets.
Researchers should communicate the details of the dataset/code/moddl as part of their submissions via structured templates. This includes details about training, license, limitations, etc.
The paper should discuss whether and how consent was obtained from people whose asset used.
At submission time, remember to anonymize your assets (if applicable). You can either create anonymized URL or include anonymized zip file.
14. Crowdsourcing and research with human subjects
Question: For crowdsourcing experiments and research with human subjects, does the paper include the full text of instructions given to participants and screenshots, if applicable, as well as details about compensation (if any)?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subject,
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Including this information in the supplemental material is fine, but if the main contribu- tion of the paper involves human subjects, then as much detail as possible should be included in the main paper.
According the NeurIPS Code of Bthies, workers involved in data collection, curation, or other labor should be paid at least the minimum wage in the country of the data collector.
15. Institutional review board (IRB) approvals or equivalent for research with human subjects
Question: Does the paper describe potential risks incurred by study participants, whether such risks were disclosed to the subjects, and whether Institutional Review Board (IRB) approvals (or equivalent approval/review based the requirements of your country or institution) were obtained?
Answer: [NA]
Justification: This paper does not involve crowdsourcing nor research with human subject.
Guidelines:
The answer NA means that the paper does not involve crowdsourcing nor research with human subjects.
Depending on the country in which research is conducted, IRB approval (or equivalent) may be required for any human subjects research. If you obtained IRB approval, you should clearly state this in the paper.
We recognize that the procedures for this may vary significantly between institutions and locations, and we expect authors to adhere to the NeurlPS Code of Ethics and the guidelines for their institution.
For initial submissions, not include any information that would break anonymity (if applicable), such the institution conducting the review.
23

---

16. Declaration of LLM usage
Question: Does the paper describe the usage of LLMs if it is an important, original, or non-standard component of the core methods in this research? Note that if the LLM is used only for writing, editing, or formatting purposes and does not impact the core methodology. scientific rigorousness, originality of the research, declaration is not required.
Answer: [NA]
Justification: This paper does not involve LLM as our core method development.
Guidelines:
The answer NA means that the core method development in this research does not involve LLMs as any important, original, non-standard components.
Please refer to our LLM policy https://eetii.cccccccmceeeeeee/s for what should or should not be described.
24


Reviews:
[{"review_id": "yzHFQc7u4r", "reviewer": "Reviewer_KqRM", "summary": "This paper proposes a novel framework called Multi-Round Collaborative Augmentation (MRCA) to tackle the challenges of class imbalance and annotation costs in long-tail instance segmentation. MRCA operates by forming a feedback loop between a diffusion-based generative model and an instance segmentation network, selectively generating additional synthetic objects for classes that are either underrepresented or challenging for the model. By iteratively incorporating these synthetic samples across multiple rounds, MRCA progressively enhances the training dataset. In each round, both the number and size of generated objects are adaptively adjusted according to the current performance of the segmentation model, aiming to maximize sample efficiency. On benchmarks such as LVIS, MRCA achieves higher performance than existing state-of-the-art methods using only 6% as much synthetic data, and ablation studies demonstrate the contribution of each core component.", "questions": "1) In cases where the instance segmentation model trains much faster than object generation, have the authors observed any decline in the effectiveness of feedback-driven augmentation when the round gap is increased? \n2) For rare classes that are inherently small in real images, have the authors experienced any negative effects on detection performance from scaling these objects up during training? I’m also curious if the authors have considered or tried any alternative resizing strategies for these cases.\n3) While MRCA demonstrates strong performance with only 6% of the synthetic data compared to previous methods, I am curious whether further improvements are possible if the generation budget is increased to match that of methods like BSGAL or X-Paste+CLIP. (~1200k of #Gen Objects) Have the authors explored the performance of MRCA under higher generation budgets?", "limitations": "The accuracy-based resizing strategy involves artificially enlarging rare or low-accuracy objects rather than preserving their natural size distribution. This may disturb the true data size distribution, which could be particularly problematic in domains like medical imaging, where the physical size of objects often carries important diagnostic meaning. In such cases, this approach might introduce bias or lead to unintended side effects in model training and deployment.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 2, "significance": 1, "originality": 2, "strengths_and_weaknesses": "Strengths: \n- Among prior works addressing long-tailed instance segmentation with generative models, directly leveraging feedback from the instance segmentation model to guide the generative process is a fresh and interesting idea. The proposed framework is novel and, to my knowledge, has not been seen in previous literature.\n- The paper conducts thorough ablation studies and analyses for each key component (such as class-wise budget allocation, model gradient feedback, and accuracy-based resizing), which provides valuable insights into the contribution of each part.\n- The authors also provide reproducible code, which is a strong point in terms of transparency and research usability.\n\nWeaknesses: \n- Depending on the relative speeds of instance segmentation training and object generation, the required round gap in Equation 6 may need to be increased. For example, if the segmentation model is lightweight and trains much faster than object generation, a larger gap (e.g., 3 or more rounds) might be needed to prevent the training process from stalling. In this situation, the feedback used to guide object generation would come from an older model rather than the most recent checkpoint. This could reduce the method’s ability to adapt quickly to the model’s current weaknesses, and might weaken the core advantage of feedback-driven augmentation, especially in fast training scenarios. \n- Limitation in accuracy-based resizing strategy is that some rare classes are inherently small in real life (e.g., coin, ant..) Enlarging these objects during training may harm the model’s ability to recognize them in realistic contexts where their small size is important.\n- While the paper shows impressive sample efficiency given a fixed generative budget (e.g., 6%), I wonder whether constraints on synthetic data generation are as significant in practiceas reducing human annotation costs. The importance of being sample-efficient with generative models may not be as high as with real human-labeled data, so I am somewhat uncertain about the broader impact or significance of the proposed efficiency gains in this context.", "was_revised": true, "final_justification": "The authors have thoroughly addressed my concerns in the rebuttal, including issues related to feedback timing, object resizing, and sample efficiency. Given the clarity of the responses and the novelty of the proposed feedback-guided generation framework, I have updated my rating to borderline accept.", "rebuttal": "We sincerely appreciate the reviewer's insightful comments, which have significantly contributed to improving the quality of our paper. Below, we address each of the concerns in detail.\n\n`Q1(W1). In the case where the instance segmentation model trains much faster than object generation, is there any decline in the effectiveness of feedback-driven augmentation when the round gap is increased?`\n\nWe first clarify that, when instance segmentation training is much faster than object generation, MRCA waits for the generation of objects from the **previous** round rather than increasing the round gap. **This stall is minimized** by adjusting two experimental settings. (1) **The number of GPUs assigned for each task** (instance segmentation training and object generation) is chosen to keep the speed similar between the two tasks. (2) **The number of object generations (72k)** is further set to better match the speed of training. \n\nWe assigned 4 GPUs and 3 GPUs for instance segmentation training and object generation, respectively. For a lightweight segmentation model, for example, we can assign 2 GPUs and 5 GPUs instead. Also, generating 36k objects instead of 72k is a reasonable option according to Table 8.\n\nIn summary, by adjusting the two parameters of MRCA, we exploit the **up-to-date** feedback from the previous round while minimizing the stall between instance segmentation training and object generation. \n\n`Q2(W2). Can scaling rare classes lead to negative effects? Are there any alternative approaches tested for resizing?`\n\nThis is a very good point. Especially in domains like medical imaging, as you mentioned, where the scale of the images is usually consistent, the size of the objects conveys important contextual information, and altering it may distort the natural size distribution. When using MRCA on such data domains, we agree that using resizing strategies should be prohibited or be applied only on a small scale. This domain-specific issue will be included in Appendix D. Limitations.\n\nHowever, in the context of ***real-world images* on which we test (LVIS, PASCAL, OpenImages)**, the size of the objects varies significantly depending on the resolution or scale of the image, even within the same class. Please consider a person depicted in a close-up photograph and in a landscape photograph. Also, for rare classes, because the number of objects is small (0--9), their size distribution does not carry statistically significant meaning.\n\nAs a result, we haven't observed any critical negative effects on detection performance. By setting the maximum scaling coefficient ($s_{max}=4.0$ in Algorithm 4) properly, we prevent objects from occupying the entire image or high-performing objects from becoming too small. Furthermore, it keeps the objects from deviating largely from their natural sizes. More importantly, as the performance of rare objects is improved in the upcoming rounds, their scale will approach their natural sizes.\n\n`Q3. Have the authors explored the performance of MRCA under higher generation budgets (i.e., 72k → 1200k)? (W3) Also, are constraints on synthetic data generation as significant in reducing human annotation costs?`\n\n`Higher Generation Budget:` Great point, as we also wanted to verify in such a case where the number of generated objects is similar to that of previous works. In **Table 8**, we tested 144k generated objects but observed only small improvements compared to 72k. Because of such observations of early saturation and **long training/generation time**, we have decided not to increase the generation budget over 144k.\n\nWe believe that since a lot of redundant generations from previous works [1,2,3] (i.e., generation of similar objects and/or generation on already high-performing classes) are removed by our pipeline, MRCA can outperform previous works with only a small number of object generations. Using multiple generation models could be a way to further improve our pipeline, which we leave as future work.\n\n`Sample Efficiency with Synthetic Data:` The significance of sample efficiency is indeed larger in real human-labeled data compared to synthetic data, as you pointed out. However, we contend that **running GPUs for synthetic data generation is also costly**. For example, running eight RTX 3090 GPUs for 3 weeks to generate and use 1200k objects incurs a cost of **766.08 US\\$** (\\$0.19/hr in vast.ai), whereas our MRCA framework requires only **51.68 US\\$**. For the broader impact, using fewer GPUs reduces electricity consumption and helps protect the environment.\n\n---\nOverall, we will incorporate these responses and additional results into the final version of the paper. We hope that most of your concerns have been addressed and would be glad to engage in further discussion if needed.\n\n---\n[1] \"X-paste: Revisiting scalable copy-paste for instance segmentation using clip and stablediffusion\", In ICML, 2023\n\n[2] \"Generative active learning for long-tailed instance segmentation\", In ICML, 2024\n\n[3] \"Diversify your vision datasets with automatic diffusion-based augmentation\", In CVPR, 2024"}, {"review_id": "IxQO8kSX15", "reviewer": "Reviewer_CkKJ", "summary": "This manuscript presents a collaborative, multi-round approach (MCRA) using feedback from an instance segmentation model, to guide data augmentation after synthesis. This method is claimed to require only a small fraction (about 6%) of the amount of synthetic generated data compared to other current methods, while improving average precision for bounding box and instance segmentation.", "questions": "1. In Section 3.1, it might be clarified as to whether the same generated image (from the frozen generator $G$) may be associated with different segmentation $S$ in different rounds, due to the effect of the evolving segmentation model.\n\n2. In Section 3.1, it might be clarified as to whether there is any processing/blurring (gradient?) of object edges, for the Map-Paste() function.\n\n3. In Section 3.2, the proof of Lemma 3.2 (i.e. \"increased information\" of D necessarily guides augmentation of higher quality) appears somewhat informal. The concept of \"information\" applied here, and how increased dataset information implies improved augmentation, could be explained or derived in greater detail.\n\n4. In Section 3.2, \"as round pass\" might be \"as rounds pass\".\n\n5. In Section 3.3, it is stated that entropy is used as the criterion to steer diffusion sampling. It might be clarified as to how entropy is measured for data instances (also with respect to the warmup and feedback models, as explored in Section 4.4).\n\n6. From Section 4.2 (Tables 1 and 2), the Swin-L backbone significantly outperforms the ResNet-50 backbone. Given this, was there a reason to use the ResNet-50 backbone as the baseline main instance segmentation model, since that could exaggerate the potential gains from MCRA in practical use?\n\n7. For the augmentation examples in Appendix C, it might be clarified as to whether augmented objects can be overlapped.", "limitations": "yes", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "Strengths\n - (Quality) Comprehensive ablation experiments isolating effects of feedback, generation model, curriculum learning, guidance scale and generation budget\n - (Originality) Integrates well-known ideas on model feedback and guided learning to the segmentation task\n\n\nWeaknesses\n - (Quality) Ablation on classifier guidance appears missing\n - (Significance) While the requirement for fewer synthetic examples is claimed as a major benefit of MCRA, no ablation analysis on the effect of reduced synthetic examples (i.e. 72k instead of 1200k) is performed for the competing methods; this is important as Table 8 suggests potential early saturation of the impact of additional synthetic examples\n - (Significance) Performance improvements appear task-specific, and findings (e.g. on optimal hyperparameters) may not be fully generalizable", "was_revised": true, "final_justification": "The authors have addressed all points raised in the original review, with the additional ablation studies appreciated. The concerns on generalizability nonetheless remain. The justification for ResNet versus Swin-L as backbone moreover suggests that the impact of the proposed MCRA method may be diminished with stronger base models. Nonetheless, it remains a useful empirical technique for consideration.", "rebuttal": "We sincerely appreciate the positive feedback on our work. We hope that our rebuttal has addressed all major concerns and clarified any remaining questions.\n\n`W1. Ablation on classifier guidance.`\n\n**We have conducted an ablation study on the classifier guidance ($\\gamma$) in Table R1** by adding the results with $\\gamma$=3.0 and $\\gamma$=7.0. The value of 7.0 shows decreased diversity among generated objects, and the value of 3.0 shows decreased quality of generated objects, both resulting in lower performance. The classifier guidance's default value of 5.0 was also used in previous works [1, 2], and it was found to be effective also in our work. \n\nTable R1: Ablation on the classifier guidance using LVIS on ResNet-50.\n| **Classifier Guidance** ($\\gamma$) |   $\\mathbf{AP^{box}}$  | $\\mathbf{AP^{mask}}$  |  $\\mathbf{AP^{box}_r}$ | $\\mathbf{AP^{mask}_r}$ |\n|------------|:----------------:|:---------------:|:----------:|:----------:|\n| 3.0     |   35.21        |   31.35     |   26.59    | 24.33\n| **5.0 (default)** |     **35.56** |    **31.81**    |    **28.14**   | **25.93**|\n| 7.0     |     35.32        |   31.58     |   26.89    | 24.68 |\n\n`W2. Ablation on reduced synthetic examples for competing methods.`\n\nIn Tables R2 and R3, we borrow the experimental results based on a reduced quantity of synthetic examples from the original papers (BSGAL [2] and DiverGen [3]). **Previous works require more synthetic examples to saturate** since their generation method is class-balanced, resulting in redundant synthetic objects (i.e., generation of similar objects and/or generation on already high-performing classes). Thus, we confirm that **the competing methods lack sample efficiency** as opposed to our MRCA framework.\n\nTable R2: Result on a reduced quantity of synthetic examples for BSGAL [2] using LVIS on ResNet-50.\n| **Method (# Gen)** |   $\\mathbf{AP^{box}}$  | $\\mathbf{AP^{mask}}$  |  $\\mathbf{AP^{box}_r}$ | $\\mathbf{AP^{mask}_r}$ |\n|------------|:----------------:|:---------------:|:----------:|:----------:|\n| BSGAL(12k)     |   34.90        |   -     |   -    | -\n| BSGAL(480k)     |   35.12        |   -     |   -    | -\n| BSGAL(1200k)     |   35.40        |   31.56     |   27.95    | 25.43\n| **MRCA (72k)**     |     **35.56** |    **31.81**    |    **28.14**   | **25.93**|\n\nTable R3: Result on a reduced quantity of synthetic examples for DiverGen [3] using LVIS on Swin-L.\n| **Method (# Gen)** |   $\\mathbf{AP^{box}}$  | $\\mathbf{AP^{mask}}$  |  $\\mathbf{AP^{box}_r}$ | $\\mathbf{AP^{mask}_r}$ |\n|------------|:----------------:|:---------------:|:----------:|:----------:|\n| DiverGen(300k)      |   49.65 | 44.01 | 45.68 | 41.11\n| DiverGen(600k)     |   50.67  | 44.99  | 48.52  | 43.63\n| DiverGen(1200k)     |   51.24        |   45.48     |   50.07    | 45.85\n| **MRCA (72k)**     |    **51.80**|   **45.91**|   **51.58**| **46.84**\n\n`W3. Peformance improvement seems task specific and findings (on hyperparameters) maybe not generalizable.`\n\nWhile we focus on the instance segmentation task in this paper, we believe that the idea of collaborative augmentation between the generator and the target task model can be further generalized to other tasks with synthetic data generation. For example, the image classification task can benefit from the notion of the collaborative augmentation.\n\nWe admit that some hyperparameters may not be generalizable, but the classifier guidance (refer to W1) and the guidance scale (Table 7) seem to be generalizable under the same generation model. Also, we would like to clarify that the necessity of hyperparameter tuning is not a specific issue unique to our work, but rather a common aspect of most machine learning methods.\n\n`Q1. Clarification on whether the generated images are processed with different(evolved) segmentation model over rounds.`\n\nWe first clarify that there are two segmentation models used. (1) $M_{\\phi_r}$: the target instance segmentation model with parameters $\\phi_r$ with respect to a round $r$, which is trained for the downstream task; (2) $\\mathcal{S}$: a frozen dichotomous segmentation model (BiRefNet [4]) that extracts an object from a generated image. While the model $M_{\\phi_r}$ evolves over rounds, the dichotomous segmentation model $\\mathcal{S}$ is fixed across all rounds. We will revise the notation in Section 3.1 to explicitly denote the round dependency of $M_{\\phi_r}$.\n\n`Q2. Clarification on object edges for the Map-Paste() function.`\n\nWe did not use any methods on object edges (e.g., Gaussian blurring and alpha/Poisson blending) when pasting objects. Based on previous work [5] and our own evaluation, the edge handling methods did not have a meaningful effect on the overall performance. \n\n`Q3. More detail on the concept of information and how increased dataset information implies improved augmentation is required.`\n\nWe appreciate the feedback on clarifying the concept of _information_ in Lemma 3.2 and its role in guiding better augmentation.\n\n`Clarification on the Information Metric:` The information metric $\\mathcal{I_D}(M_{\\phi})$ is a flexible measure that quantifies how effectively the model $M_{\\phi}$ captures the characteristics of the dataset $\\mathcal{D}$. It can be instantiated using simple metrics such as average accuracy or defined more formally, for example, as the reciprocal of the KL divergence between the data distribution $p_{\\mathcal{D}}$ and the model's distribution $p_{M_{\\phi}}$: $\\mathcal{I_D}(M_{\\phi})= {1\\over{D_{KL}(p_{\\mathcal{D}} || p_{M_{\\phi}})}}$.\n\n`Implication on Better Augmentation:` A higher $\\mathcal{I_D}(M_{\\phi})$ means that the model better represents the data distribution, enabling more meaningful feedback $f_\\phi$ to improve augmentation quality. For example, this feedback can balance class distributions by generating more objects for rare classes or focus on difficult examples to enhance learning.\n\n`Q4. \"as round pass\" → \"as rounds pass\".`\n\nWe will fix this typo.\n\n`Q5. Clarification on how entropy is measured for data instances.`\n\nSince an instance segmentation model predicts a class label for each segment, we compute the entropy based on the classification logit given an input object image. Here, the whole image is set as the object proposal since we generate only a single object per image. Specifically, for the $i$-th object at round $r+1$, denoted as $o_{i,r+1}$, the entropy is calculated as $\\text{entropy}(o_{i,r+1}; M_{\\phi}) = -\\sum_{j} p_j \\log(p_j)$, where $p = c_{\\phi}(\\mathcal{F}(o_{i,r+1}))$ is the softmax-normalized class probability vector obtained from the classifier head $c_{\\phi}$, given features $\\mathcal{F}(o_{i,r+1})$ extracted by the backbone of the model $M_{\\phi}$. Using this formulation, the entropy results in Section 4.4 are obtained as follows.\n* Figure 4.4(b) presents $\\text{entropy}(o_{i,r+1}; M_{\\phi_{r}})$, where the feedback model from the previous round ($M_{\\phi_r}$) is used. \n* Figure 4.4(c), on the other hand, shows $\\text{entropy}(o_{i,r+1}; M_{\\phi_{0}})$, where the warm-up model ($M_{\\phi_0}$) is applied.\n\n`Q6. Why use a weaker ResNet backbone when Swin-L significantly outperforms it?`\n\nSince the overall performance trend remained consistent across different backbone models, we chose a lighter model **to enable more extensive ablation studies** within a given time. Notably, the performance improvement reported in Tables 1 and 2 is even larger when using the Swin-L backbone, supporting that our ResNet-based ablation results do not overstate the effectiveness of MRCA.\n\nThe ResNet backbone is more memory-efficient and can be trained on a 24GB RTX 3090, whereas the Swin-L backbone requires a higher-memory setup, such as a 48GB A6000 or larger. We believe that both models are important, as they represent a trade-off between model size and performance, which is an important consideration for practical deployment, particularly in on-device scenarios.\n\n`Q7. Can augmented objects be overlapped?`\n\nYes, the objects can be overlapped. When an object is overlapped, its object mask is adjusted to eliminate the obscured portion. This is a common practice in the relevant studies [1, 2, 3].\n\n---\nOverall, we will incorporate these responses and additional results into the final version of the paper. We hope that most of your concerns have been addressed and would be glad to engage in further discussion if needed.\n\n---\n[1] \"X-paste: Revisiting scalable copy-paste for instance segmentation using clip and stablediffusion\", In ICML, 2023\n\n[2] \"Generative active learning for long-tailed instance segmentation\", In ICML, 2024\n\n[3] \"Diversify your vision datasets with automatic diffusion-based augmentation\", In CVPR, 2024\n\n[4] \"Bilateral reference for high-resolution dichotomous image segmentation\", In CAAI AIR, 2024\n\n[5] \"Simple Copy-Paste is a Strong Data Augmentation Method for Instance Segmentation\", In CVPR, 2021"}, {"review_id": "Efm9uCpYTn", "reviewer": "Reviewer_SKJ8", "summary": "This paper introduces Multi-Round Collaborative Augmentation (MRCA), a novel generative data augmentation framework designed to enhance sample efficiency for long-tail instance segmentation tasks. MRCA addresses the inefficient utilization of pre-generated data by incorporating feedback from an instance segmentation model to guide the augmentation process. It dynamically adjusts the number and size of synthetic objects for each class based on the model's state, improving learning for underrepresented classes. This multi-round process allows feedback to be refined throughout training , enabling MRCA to outperform state-of-the-art methods while requiring only 6% of their data generation.", "questions": "**Effectiveness of Guidance from Noisy Intermediate States:** The paper's method applies guidance from the instance segmentation model (Mϕ) to noisy intermediate diffusion states. A key question is whether this guidance remains effective early in the diffusion process, when latent states are still heavily dominated by noise. Features extracted from such noisy representations might be unreliable, potentially leading to suboptimal or counterproductive signals for guiding object generation.", "limitations": "yes", "rating": 4, "confidence": 2, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 3, "clarity": 3, "significance": 3, "originality": 3, "strengths_and_weaknesses": "strengths\n\n1. **Methodological Soundness and Intuitive Design:** The proposed Multi-Round Collaborative Augmentation (MRCA) framework is conceptually strong and highly intuitive. By incorporating feedback from the instance segmentation model to guide the generative process, MRCA ensures that synthetic data is optimally tailored to the model's evolving needs, which makes inherent sense for improving learning efficiency. The dynamic adjustment of object generation and resizing based on model uncertainty and class accuracy directly targets areas of weakness, providing a logically sound approach to data augmentation.\n2. **Empirical Validation and Significant Performance Gains:** The paper provides compelling experimental evidence demonstrating MRCA's effectiveness. Notably, MRCA outperforms state-of-the-art generative augmentation methods on the LVIS v1.0 dataset while requiring only 6% of the data generation. This substantial reduction in generated objects coupled with superior performance, particularly in challenging rare classes, clearly validates the method's sample efficiency and practical utility.\n3. **Exemplary Presentation and Clarity:** The paper is exceptionally well-structured and clearly written, enhancing its readability and impact. The figures, such as Figure 1 illustrating the novelty of MRCA and Figure 4 visualizing round-wise object generation and entropy, are highly informative and visually clear, effectively conveying complex ideas. The use of tables to present detailed experimental results and ablation studies further contributes to the paper's comprehensiveness and allows for easy understanding of the contributions of individual components.\n\nweakness\n\n1. **Limited Impact of Core Feedback Mechanisms Compared to Simpler Design:** Table 3  in the ablation study suggests that the more complex feedback mechanisms, such as \"Class Budget\" and \"Model Gradient,\" contribute less to overall performance improvement than the seemingly simpler \"Resizing\" component. This observation raises questions about the relative efficacy and necessity of intricate feedback loops, as the straightforward resizing based on accuracy  appears to be a more dominant factor in the gains. Further analysis or alternative designs for the feedback-guided generation could be explored to maximize the impact of these sophisticated components.\n2. **Insufficient Granularity in Ablation of Equation (6) Components:** The paper's current ablation study does not fully explore the individual contributions of the terms within Equation (6), which governs the Class-Wise Budget Optimization. Specifically, it would be beneficial to ablate the impact of including or excluding components like the classifier weight difference term and the validation accuracy . A more detailed breakdown would provide clearer insights into which specific feedback signals are most influential for dynamic budget allocation and why, thereby strengthening the empirical foundation of this crucial part of the methodology.", "was_revised": false, "final_justification": "", "rebuttal": "We sincerely appreciate the positive feedback on our work. We hope that our rebuttal has addressed all major concerns and clarified any remaining questions.\n\n`W1. Is resizing's effect much larger than complex class budget and model gradient?`\n\nWe apologize for any confusion caused by our presentation. The contributions of the three components, \"Class Budget,\" \"Model Gradient,\" and \"Resizing,\" are comparable in magnitude. Since the result using only the \"Resizing\" component was omitted from Table 3 in the original submission, we have now included it in Table R1. As shown in Table R1, **the \"Resizing\" component alone does *not* yield a dominant performance gain**, which supports our claim that **a sophisticated feedback-guided generation mechanism is essential**. The best performance is achieved through the synergistic integration of all three components. \n\nTable R1: Ablation on the main components of MRCA using LVIS on ResNet-50. **The 3rd row is added to Table 3 of the original submission.**\n| Class Budget | Model Gradient | Resizing |   $\\mathbf{AP^{box}}$  | $\\mathbf{AP^{mask}}$  |  $\\mathbf{AP^{box}_r}$ | $\\mathbf{AP^{mask}_r}$ |\n|:------------:|:------------:|:------------:|:----------------:|:---------------:|:----------:|:----------:|\n| $\\checkmark$    | |  |    34.86 | 31.33 |27.47 | 24.91\n|    |$\\checkmark$ |  |    35.04 | 31.14 | 27.13 | 24.28\n|      |  | $\\checkmark$ |   34.91 |  31.10 |  27.52 |  24.89\n|  $\\checkmark$     | $\\checkmark$  |  |   35.15 |  31.42 |  26.54 |  24.35\n|  $\\checkmark$  |  $\\checkmark$ | $\\checkmark$ |  **35.56** | **31.81** | **28.14** | **25.93**\n\n\n`W2. Insufficient granularity in ablation of class budget optimization (Equation (6)).`\n\nWe conducted a **fine-grained** ablation study on class-wise budget optimization while keeping other components enabled, as shown in Table R2. Equation (6) incorporates three types of information: the number of objects ($S$), validation accuracy ($A$), and classifier weight ($C$). When using only the number of objects, we observed notable improvements for **rare** classes (see the difference in $\\mathbf{AP^{box}_r}$ and $\\mathbf{AP^{mask}_r}$ between the first and second rows). Then, to better support **common** and **frequent** classes with low accuracy, we introduced the validation accuracy. Nonetheless, some low-accuracy classes continued to produce similar objects across rounds. To mitigate this issue, we incorporated the classifier weight, which reflects whether gradient feedback is being updated over rounds. As evidenced by the improvements in $\\mathbf{AP^{box}}$ and $\\mathbf{AP^{mask}}$ in the third and fourth rows, both validation accuracy and classifier weight contribute effectively as intended.\n\nTable R2: **Fine-grained** ablation on the components of class budget optimization using LVIS on ResNet-50.\n| Number of objects($S$) | Validation accuracy($A$) | Classifier weight($C$) |   $\\mathbf{AP^{box}}$  | $\\mathbf{AP^{mask}}$  |  $\\mathbf{AP^{box}_r}$ | $\\mathbf{AP^{mask}_r}$ |\n|:------------:|:------------:|:------------:|:----------------:|:---------------:|:----------:|:----------:|\n|   | |      | 35.11  | 31.35  | 27.36 |  25.02 \n| $\\checkmark$  | |      | 35.33   | 31.60  | 28.05 | 25.87\n| $\\checkmark$  |$\\checkmark$  |   | 35.51    | 31.79  | 27.93 | 25.67 \n| $\\checkmark$ |$\\checkmark$  |   $\\checkmark$    |  **35.56** | **31.81** | **28.14** | **25.93**\n\n`Q1. Effectiveness of guidance from noisy intermediate states.`\n\nThank you for the insightful comment. We indeed explored **three** different strategies for applying feedback guidance from the instance segmentation model to the diffusion model.\n\n1. In the selected method, feedback guidance is **applied every 5 steps out of a total of 30 diffusion steps** (i.e., 6 times in total). This method was empirically found to offer the best trade-off between performance and image quality.\n1. The next method is exactly what you mentioned. **Feedback guidance is applied only during the last 6 steps**. While this method reduces the influence of noisy intermediate features, we found that applying guidance in consecutive steps disrupted the denoising process, leading to degraded image quality.\n1. Another method involves performing an approximate **single-step denoising from an intermediate noisy state to the final state**, followed by extracting features as described in [1]. However, this method significantly increased generation time and yielded negligible performance improvements.\n\n---\nOverall, we will incorporate these responses and additional results into the final version of the paper. We hope that most of your concerns have been addressed and would be glad to engage in further discussion if needed.\n\n---\n[1] \"Test-time Alignment of Diffusion Models without Reward Over-optimization\", In ICLR, 2025."}, {"review_id": "5xmvZpYoOw", "reviewer": "Reviewer_3V2k", "summary": "This paper proposes a multi-round collaborative augmentation (MRCA) framework for long-tail instance segmentation. By integrating feedback from an instance segmentation model into both object generation and placement, MRCA adaptively synthesizes challenging and informative training samples. It uses diffusion models guided by model uncertainty and class-wise accuracy, and dynamically adjusts generation budgets. Experiments on LVIS show that MRCA achieves superior performance with only 6% of the synthetic data used by prior methods.", "questions": "How do you ensure that the augmented data and the original data share the same distribution? From my perspective, the map-pasted pictures seem not quite realistic.", "limitations": "Yes, the authors have adequately addressed the limitations and potential negative societal impact of their work.", "rating": 4, "confidence": 4, "soundness": null, "presentation": null, "contribution": null, "strengths": "", "weaknesses": "", "quality": 2, "clarity": 3, "significance": 3, "originality": 2, "strengths_and_weaknesses": "## Strengths \n\n1. The paper is well-organised with clear structure.\n2. The paper integrates two-fold model feedback (class-wise budget optimization + gradient-guided generation) in a multi-round loop, which is a principled extension of curriculum learning to generative augmentation.\n3. The paper effectively disentangles contributions of each feedback component (budget optimization, gradient-based generation, resizing).\n\n## Weaknesses\n\n1. Theoretical analysis is superficial. Assumption 3.1 merely states that feedback from a trained model improves augmentation quality, but does not formalize under what conditions this holds (e.g. model capacity limits, data distribution shifts). Furthermore, The paper references information gain $\\mathcal{I}\\_{\\mathcal{D}}(M\\_{\\phi})$ but does not define how it is computed or measured in practice.\n2. Feedback guidance hyperparameter sensitivity. Performance is sensitive to the feedback guidance scale (ω). While the paper provides analysis, tuning such parameters can be non-trivial in practice.", "was_revised": true, "final_justification": "Most of my concerns have been addressed. I especially appreciate it that the authors explain the model sensitivity to hyperparameters and the use of \"unrealistic\" images. While I still believe that the theoretical analysis is rather superficial and only marginally insightful, the proposed method is in itself a contribution to the field. Overall, I decide to keep my rating at 4.", "rebuttal": "We sincerely appreciate the positive feedback on our work. We hope that our rebuttal has addressed all major concerns and clarified any remaining questions.\n\n`W1. Require more detailed theoretical analysis on information gain and the conditions where Assumption 3.1 holds.`\n\nWe fully understand your perspective. Yes, Assumption 3.1 states that feedback from a trained model improves augmentation quality. However, a similar assumption has been **widely adopted in recent data augmentation literature**, where model feedback consistently improves augmentation quality over unguided approaches. This assumption is empirically validated across numerous established methods[1, 2, 3, 5]. Also, the two conditions raised by the reviewer, (1) the model having sufficient capacity to learn from synthetic data and (2) controlled distribution shift between real and augmented data, are naturally met as training converges. \n\nRegarding the information metric $\\mathcal{I_D}(M_{\\phi})$, it can be any metric that quantifies how well the model $M_{\\phi}$ captures the dataset $\\mathcal{D}$. In practice, it can be measured via standard metrics like accuracy or as the reciprocal of the KL divergence between the true and model's distributions: $\\mathcal{I_D}(M_{\\phi})= {1\\over{D_{KL}(p_{\\mathcal{D}} || p_{M_{\\phi}})}}$. A higher $\\mathcal{I_D}(M_{\\phi})$ means closer alignment to the true data distribution, enabling more effective feedback $f_\\phi$ to improve augmentation quality, such as addressing class imbalance or focusing on uncertain examples.\n\nPer your comment, **we will revise our Assumption 3.1** by explicitly stating the required conditions. \n\n> **Assumption 3.1 (revised)**\n> Let $\\mathcal{A}$ denote a data augmentation applied to a dataset $\\mathcal{D}$, and let $f_\\phi$ denote the feedback from a model $M_\\phi$ trained on the dataset $\\mathcal{D}$. We define the information metric $\\mathcal{I_D}(M_{\\phi})$ as a measure of how well the model $M_{\\phi}$ captures the distribution of the dataset $\\mathcal{D}$. In practice, it can be measured by the average accuracy, i.e., $\\mathcal{I_D}(M_{\\phi})= \\frac{1}{|\\mathcal{D}|} \\sum_{(x, \\mathbf{y}) \\in \\mathcal{D}} \\frac{1}{|\\mathbf{y}|} \\sum_{y \\in \\mathbf{y}} \\mathbf{1}\\left[ \\exists \\hat{y} \\in \\hat{\\mathbf{y}}(x) \\text{ s.t. } \\text{IoU}(y, \\hat{y}) \\geq 0.5 \\right]$.\n>\n> **($C1$) Model Capacity**: Model capacity can be represented by the empirical risk $\\mathcal{R_{emp}}$, which is defined as the average loss $\\ell$ over the dataset $\\mathcal{D}$. Formally, $\\mathcal{R_{emp}}(M_{\\phi}) = \\frac{1}{|\\mathcal{D}|} \\sum_{(x_i, \\mathbf{y_{\\mathcal{i}}}) \\in \\mathcal{D}} \\ell(M_\\phi(x_i),\\mathbf{y_{\\mathcal{i}}})$. To ensure the model retains available learning capacity, the empirical risk should be decreasing over time, i.e., $\\frac{d\\mathcal{R_{emp}}(M_\\phi^{(t)})}{dt} < 0$.\n>\n> Then, $C1 \\rightarrow \\mathcal{I_D}(M_\\phi) \\geq \\mathcal{I_D}(\\emptyset)$.\n>\n> **($C2$) Controlled Distribution Shift**:  The knowledge gained through augmentation should surpass the adverse effects of distribution shift. Formally, given the model's learned distribution $p_{M_{\\phi}}^{(t)}$ over time $t$,  $\\frac{d{D_{KL}(p_{\\mathcal{D}} || p_{M_{\\phi}}^{(t)})}}{dt} < 0$.\n>\n> **($C3$) Feedback Quality**: The feedback $f_\\phi$ effectively guides the augmentation process toward quality maximization.\n>\n> Then, $[\\mathcal{I_D}(M_\\phi) \\geq \\mathcal{I_D}(\\emptyset)] \\land (C2) \\land (C3) \\rightarrow Q(\\mathcal{A}(\\mathcal{D}; f_\\phi)) \\geq Q(\\mathcal{A}(\\mathcal{D}))$.\n\nCondition ($C1$) and ($C2$) are typically satisfied as the accuracy saturates during the training. For condition ($C3$), the design of the feedback mechanism $f_\\phi$ and how it guides the augmentation process toward quality maximization are detailed in Sections 3.3 and 3.4.\n\n`W2. Highly sensitive feedback guidance scale can be difficult to tune.`\n\nAlthough the feedback guidance scale $\\omega$ may appear sensitive, its effective search space can be quickly narrowed through simple qualitative analysis. When $\\omega > 0.10$, the diffusion model fails to produce natural images, often resulting in artifacts such as white blurs and black holes. These artifacts diminish as $\\omega$ decreases and become infrequent around $\\omega \\approx 0.05$. Conversely, for $\\omega < 0.01$, the generated images with feedback are nearly indistinguishable from those without feedback, indicating that meaningful improvements only begin to emerge when $\\omega \\geq 0.01$. Therefore, the practical and effective range for $\\omega$ lies between 0.01 and 0.05, which is **tractable in practice**.\n\n`Q1. The map-pasted images do not seem realistic. Do the augmented images and the original image share the same data distribution?`\n\nOur approach follows the line of research established by X-Paste[4], BSGAL[5], and DiverGen[6], which leverage synthetically generated, **often unrealistic**, map-pasted images to improve performance. While the augmented images may not share the same distribution as the original images in terms of object size, they still contribute positively to instance segmentation performance by incorporating knowledge from the generation model. The MRCA framework further enhances this effect by explicitly optimizing both object generation and pasting strategies. It is worth noting that **the use of visually unrealistic images for data augmentation has been widely observed to be effective in improving downstream performance**. For instance, also in image classification, CutMix[7] often produces implausible examples such as images featuring a cat's head on a dog's body.\n\n---\nOverall, we will incorporate these responses into the final version of the paper. We hope that most of your concerns have been addressed and would be glad to engage in further discussion if needed.\n\n---\n\n[1] \"AdaAug: Learning Class- and Instance-adaptive Data Augmentation Policies\", In ICLR, 2022\n\n[2] \"Feedback-guided Data Synthesis for Imbalanced Classification\", In TMLR, 2024\n\n[3] \"Controlled Training Data Generation with Diffusion Models\", In TMLR, 2025\n\n[4] \"X-paste: Revisiting scalable copy-paste for instance segmentation using clip and stablediffusion\", In ICML, 2023\n\n[5] \"Generative active learning for long-tailed instance segmentation\", In ICML, 2024\n\n[6] \"Diversify your vision datasets with automatic diffusion-based augmentation\", In CVPR, 2024\n\n[7] \"CutMix: Regularization Strategy to Train Strong Classifiers with Localizable Features\", In ICCV, 2019"}]

Author rebuttal:

Meta-review: ***(a) Scientific Claims and Findings***

The paper proposes a novel framework called Multi-Round Collaborative Augmentation (MRCA) for long-tail instance segmentation. The core idea is to create a feedback loop between a diffusion-based generative model and an instance segmentation network. The generative model selectively produces synthetic objects for classes that are either underrepresented or difficult for the current model to segment. This process is performed over multiple rounds, with the number and size of synthesized objects for each class dynamically adjusted based on the model's state. The authors claim that MRCA enhances sample efficiency and outperforms state-of-the-art methods while using only 6% of the data generation required by competing methods.

***(b) Strengths***
+ Novelty: The framework is considered a fresh and interesting idea, leveraging feedback from an instance segmentation model to guide the generative process. This is a new approach to generative data augmentation.
+ Empirical Performance: The paper provides compelling experimental evidence that MRCA outperforms state-of-the-art methods while using significantly less synthetic data. The authors conducted extensive evaluations, which reviewers found to be thorough.
+ Methodological Soundness: The framework is conceptually strong and intuitive. The logical structure and clear presentation of the method were also praised by reviewers.

***(c) Weaknesses***
+ Theoretical Superficiality: The theoretical analysis was initially criticized as being "superficial" and only "marginally insightful". The paper's core assumption that feedback improves augmentation quality lacked formal conditions.
+ Generalizability Concerns: A reviewer noted that the performance improvements might be task-specific and that the findings on optimal hyperparameters may not be fully generalizable. They also suggested the method's impact might be diminished when using stronger backbone models, though the authors provided a counterargument.
+ Practical Limitations: One reviewer raised concerns about the resizing strategy, noting that it might be problematic for classes that are inherently small and could disturb the true data size distribution, particularly in domains like medical imaging.


***(d) Reasons for Decision to Accept***

I recommend Acceptance as poster for this paper. The core idea is highly innovative and addresses a critical problem in the field: the inefficiency of data generation for long-tail instance segmentation. The authors' approach of using a feedback loop to guide a diffusion model is a significant step forward. 

***(e) Summary of Rebuttal and Discussion***

While the initial submission had some theoretical and clarity issues, the authors provided a detailed and convincing rebuttal. They successfully addressed all major concerns, providing additional ablation studies, clarifying their experimental setup, and adding a more rigorous analysis of their theoretical foundations. The authors received positive initial reviews but were asked to address several key points. They provided a new ablation study on classifier guidance, clarified their experimental setup, and added a more formal theoretical analysis to address the "superficial" critique. They also provided convincing arguments and additional data to address concerns about sample efficiency, object resizing, and the generalizability of their findings. The rebuttal successfully addressed all major concerns raised by the reviewers, leading to a unanimous favorable rating among all four. This clear consensus among the reviewers, combined with the paper's strong empirical results and practical efficiency, justifies its acceptance as a valuable contribution to the NeurIPS community.
Decision: Accept (poster)